(part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline - PowerPoint PPT Presentation

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . weight 1 * applies 1 ( 🗏 , A TTACK ) exp ( ) ) weight 2 * applies 2 ( 🗏 , A TTACK ) weight 3 * applies 3 ( 🗏 , A TTACK ) …

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . weight 1 * applies 1 ( 🗏 , A TTACK ) exp ( ) ) weight 2 * applies 2 ( 🗏 , A TTACK ) weight 3 * applies 3 ( 🗏 , A TTACK ) … K different for K different weights … features

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . weight 1 * applies 1 ( 🗏 , A TTACK ) exp ( ) ) weight 2 * applies 2 ( 🗏 , A TTACK ) weight 3 * applies 3 ( 🗏 , A TTACK ) … K different for K different multiplied and weights … features … then summed

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . exp ( ) Dot_product of weight_vec feature_vec( 🗏 , A TTACK ) K different for K different multiplied and weights … features … then summed

Maxent Modeling Three people have been fatally shot, and five people, including ) ∝ p( | a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . 𝜄 𝑈 𝑔( 🗏 , A TTACK ) exp ( ) K different for K different multiplied and weights … features … then summed

Maxent Modeling Three people have been fatally shot, and five people, including p( | ) = a mayor, were seriously wounded as a result of a A TTACK Shining Path attack today against a community in Junin department, central Peruvian mountain region . Q: How do we define Z? 𝜄 𝑈 𝑔( 🗏 , A TTACK ) 1 exp ( ) ) Z

Normalization for Classification Z = Σ 𝜄 𝑈 𝑔( 🗏 , Y ) exp ( ) label y 𝑞 𝑧 𝑦) ∝ exp(𝜄 𝑈 𝑔 𝑦, 𝑧 ) classify doc x with label y in one go

Normalization for Classification (long form) Z = Σ weight 1 * applies 1 ( 🗏 , y) exp ( ) weight 2 * applies 2 ( 🗏 , y) weight 3 * applies 3 ( 🗏 , y) … label y 𝑞 𝑧 𝑦) ∝ exp(𝜄 𝑈 𝑔 𝑦, 𝑧 ) classify doc x with label y in one go

Core Aspects to Maxent Classifier p(y|x) • features 𝑔 𝑦, 𝑧 between x and y that are meaningful; • weights 𝜄 (one per feature) to say how important each feature is; and • a way to form probabilities from 𝑔 and 𝜄 exp(𝜄 𝑈 𝑔(𝑦, 𝑧)) 𝑞 𝑧 𝑦) = σ 𝑧′ exp(𝜄 𝑈 𝑔(𝑦, 𝑧′))

Outline Maximum Entropy models 1. Defining Appropriate Features Defining the model 2. Understanding features Defining the objective in conditional models Learning: Optimizing the objective Math: gradient derivation Neural (language) models

Defining Appropriate Features in a Maxent Model Feature functions help extract useful features (characteristics) of the data They turn data into numbers Features that are not 0 are said to have fired

Defining Appropriate Features in a Maxent Model Feature functions help extract useful features (characteristics) of the data They turn data into numbers Features that are not 0 are said to have fired Generally templated Often binary-valued (0 or 1), but can be real-valued

Templated Features Define a feature f clue ( 🗏 , label) for each clue you want to consider The feature f clue fires if the clue applies to/can be found in the ( 🗏 , label) pair Clue is often a target phrase (an n-gram) and a label

Templated Features Define a feature f clue ( 🗏 , label) for each clue you want to consider The feature f clue fires if the clue applies to/can be found in the ( 🗏 , label) pair Clue is often a target phrase (an n-gram) and a label Q: For a classifier p(label | 🗏 ) are clues that depend only on 🗏 useful?

Maxent Modeling: Templated Binary Feature Functions ) ∝ p( | Three people have been fatally shot, and five A TTACK people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region . weight 1 * applies 1 ( 🗏 , A TTACK ) exp ( ) ) weight 2 * applies 2 ( 🗏 , A TTACK ) weight 3 * applies 3 ( 🗏 , A TTACK ) … applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise binary

Example of a Templated Binary Feature Functions applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise applies hurt,ATTACK 🗏 , ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise

Example of a Templated Binary Feature Functions applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise applies hurt,ATTACK 🗏 , ATTACK = Q: What does this ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK function check? 0, otherwise

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? applies hurt,ATTACK 🗏 , ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? A1: 𝑊𝑀 applies hurt,ATTACK 🗏 , ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? A1: 𝑊𝑀 2. How many features are defined if applies hurt,ATTACK 🗏 , ATTACK = bigram targets are used? ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? A1: 𝑊𝑀 2. How many features are defined if applies hurt,ATTACK 🗏 , ATTACK = bigram targets are used? ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise A2: 𝑊 2 𝑀

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? A1: 𝑊𝑀 2. How many features are defined if applies hurt,ATTACK 🗏 , ATTACK = bigram targets are used? ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise A2: 𝑊 2 𝑀 3. How many features are defined if unigram and bigram targets are used?

Example of a Templated Binary Feature Functions Q: If there are V vocab types and L label applies target,type 🗏 , ATTACK = types: ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 1. How many features are defined if 0, otherwise unigram targets are used? A1: 𝑊𝑀 2. How many features are defined if applies hurt,ATTACK 🗏 , ATTACK = bigram targets are used? ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0, otherwise A2: 𝑊 2 𝑀 3. How many features are defined if unigram and bigram targets are used? A2: (V + 𝑊 2 )𝑀

More on Feature Functions applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise binary ??? Non-templated real-valued ??? Templated real- valued

More on Feature Functions applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise binary applies 🗏 , ATTACK = log 𝑞 🗏 ATTACK) Non-templated real-valued ??? Templated real- valued

More on Feature Functions applies target,type 🗏 , ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0, otherwise binary applies 🗏 , ATTACK = log 𝑞 🗏 ATTACK) Non-templated real-valued applies target,type 🗏 , ATTACK = log 𝑞 🗏 ATTACK) + log 𝑞 type ATTACK) + log 𝑞(ATTACK |type) Templated real- valued

Understanding Conditioning 𝑞 𝑧 𝑦) ∝ count(𝑦) Q: Is this a good model?

Understanding Conditioning 𝑞 𝑧 𝑦) ∝ exp(𝜄 ⋅ 𝑔 𝑦 ) Q: Is this a good model?

https://www.csee.umbc.edu/courses/undergraduate/473/f19/loglin-tutorial/ Lesson 11

Earlier, I said Maxent Models as Featureful n-gram Language Models of text x p(Colorless green ideas sleep furiously | Label) = p(Colorless | Label, <BOS>) * … * p(<EOS> | Label , furiously)20 Model each n-gram term with a maxent model 𝑞 𝑦 𝑗 𝑧, 𝑦 𝑗−𝑂+1:𝑗−𝑗 ) = maxent(𝑧, 𝑦 𝑗−𝑂+1:𝑗−𝑗 , 𝑦 𝑗 ) Q: What would this look like?

Language Model with Maxent n-grams label 𝑁 𝑞 𝑜 🗏 𝑧) = ෑ maxent(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 ) 𝑗=1 n-gram

Language Model with Maxent n-grams label 𝑁 𝑞 𝑜 🗏 𝑧) = ෑ maxent(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 ) 𝑗=1 n-gram 𝑁 exp(𝜄 𝑈 𝑔(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 )) = ෑ σ 𝑦′ exp(𝜄 𝑈 𝑔(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦′)) 𝑗=1

Language Model with Maxent n-grams 𝑁 𝑞 𝑜 🗏 𝑧) = ෑ maxent(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 ) 𝑗=1 n-gram 𝑁 exp(𝜄 𝑈 𝑔(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 )) = ෑ 𝑎(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 ) 𝑗=1

Language Model with Maxent n-grams 𝑁 𝑞 𝑜 🗏 𝑧) = ෑ maxent(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 ) 𝑗=1 n-gram 𝑁 exp(𝜄 𝑈 𝑔(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 , 𝑦 𝑗 )) = ෑ 𝑎(𝑧, 𝑦 𝑗−𝑜+1:𝑗−𝑗 ) 𝑗=1 Q: Why is this Z a function of the context?

Outline Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

p θ (u | v ) probabilistic model 𝐺(𝜄; 𝑣, 𝑤) objective

Primary Objective: Likelihood • Goal: maximize the score your model gives to the training data it observes • This is called the likelihood of your data • In classification, this is p(label | 🗏 ) • For language modeling, this is p( 🗏 | label)

Objective = Full Likelihood? (in LM) exp(𝜄 𝑈 𝑔 𝑦 𝑗 , ℎ ) ෑ 𝑞 𝜄 𝑦 𝑗 ℎ 𝑗 ∝ ෑ 𝑗 𝑗 Differentiating this These values can have very small magnitude ➔ underflow product could be a pain (assume ℎ 𝑗 has whatever context and n-gram history necessary)

Logarithms (0, 1] ➔ (- ∞, 0] Products ➔ Sums log(ab) = log(a) + log(b) log(a/b) = log(a) – log(b) Inverse of exp log(exp(x)) = x

Log-Likelihood (n-gram LM) Wide range of (negative) numbers Sums are more stable log ෑ 𝑞 𝜄 𝑦 𝑗 ℎ 𝑗 = ෍ log 𝑞 𝜄 (𝑦 𝑗 |ℎ 𝑗 ) 𝑗 𝑗 Products ➔ Sums log(ab) = log(a) + log(b) Differentiating this log(a/b) = log(a) – log(b) becomes nicer (even though Z depends on θ )

Log-Likelihood (n-gram LM) Wide range of (negative) numbers Sums are more stable log ෑ 𝑞 𝜄 𝑦 𝑗 ℎ 𝑗 = ෍ log 𝑞 𝜄 (𝑦 𝑗 |ℎ 𝑗 ) 𝑗 𝑗 Inverse of exp 𝜄 𝑈 𝑔 𝑦 𝑗 , ℎ 𝑗 − log 𝑎(ℎ 𝑗 ) = ෍ log(exp(x)) = x 𝑗 Differentiating this becomes nicer (even though Z depends on θ )

Log-Likelihood (n-gram LM) Wide range of (negative) numbers Sums are more stable log ෑ 𝑞 𝜄 𝑦 𝑗 ℎ 𝑗 = ෍ log 𝑞 𝜄 (𝑦 𝑗 |ℎ 𝑗 ) 𝑗 𝑗 𝜄 𝑈 𝑔 𝑦 𝑗 , ℎ 𝑗 − log 𝑎(ℎ 𝑗 ) = ෍ 𝑗 = 𝐺 𝜄

Outline Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

How will we optimize F( θ )? Calculus

F( θ ) θ

F( θ ) θ θ *

F( θ ) F’(θ ) derivative of F wrt θ θ θ *

What if you can’t find the roots? Follow the derivative F( θ ) F’(θ ) derivative of F wrt θ θ θ *

What if you can’t find the roots? Follow the derivative Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) θ θ 0 θ *

What if you can’t find the roots? Follow the derivative Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) 2. Get derivative g t = F’(θ t ) g 0 θ θ 0 θ *

What if you can’t find the roots? Follow the derivative Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) 2. Get derivative g t = F’(θ t ) 3. Get scaling factor ρ t 4. Set θ t+1 = θ t + ρ t *g t g 0 5. Set t += 1 θ θ 0 θ 1 θ *

What if you can’t find the roots? Follow the derivative z 1 Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) 2. Get derivative g t = F’(θ t ) 3. Get scaling factor ρ t 4. Set θ t+1 = θ t + ρ t *g t g 0 g 1 5. Set t += 1 θ θ 0 θ 1 θ 2 θ *

What if you can’t find the roots? Follow the derivative z 3 z 2 z 1 Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) 2. Get derivative g t = F’(θ t ) 3. Get scaling factor ρ t 4. Set θ t+1 = θ t + ρ t *g t g 0 g 1 5. Set t += 1 g 2 θ θ 0 θ 1 θ 2 θ 3 θ *

What if you can’t find the roots? Follow the derivative z 3 z 2 z 1 Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged : of F wrt θ 1. Get value z t = F( θ t ) 2. Get derivative g t = F’(θ t ) 3. Get scaling factor ρ t g 0 4. Set θ t+1 = θ t + ρ t *g t g 1 g 2 θ 5. Set t += 1 θ 0 θ 1 θ 2 θ 3 θ *

Remember: Common Derivative Rules

Gradient = Multi-variable derivative K-dimensional input K-dimensional output

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

What if you can’t find the roots? Follow the gradient z 3 z 2 z 1 Set t = 0 F( θ ) F’(θ ) z 0 Pick a starting value θ t derivative Until converged: of F wrt θ 1. Get value z t = F( θ t ) 2. Get gradient g t = F’(θ t ) 3. Get scaling factor ρ t 4. Set θ t+1 = θ t + ρ t *g t g 0 g 1 5. Set t += 1 g 2 θ θ 0 θ 1 θ 2 θ 3 θ * K-dimensional vectors

Outline Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

Reminder: Expectation of a Random Variable number of pieces of candy 1 2 3 4 5 6 1/6 * 1 + 1/6 * 2 + 1/6 * 3 + 𝔽 𝑌 = ෍ 𝑦 𝑞 𝑦 = 3.5 1/6 * 4 + 𝑦 1/6 * 5 + 1/6 * 6

Reminder: Expectation of a Random Variable number of pieces of candy 1 2 3 4 5 6 1/2 * 1 + 1/10 * 2 + 1/10 * 3 + 𝔽 𝑌 = ෍ 𝑦 𝑞 𝑦 = 2.5 1/10 * 4 + 𝑦 1/10 * 5 + 1/10 * 6

Reminder: Expectation of a Random Variable number of pieces of candy 1 2 3 4 5 6 1/2 * 1 + 1/10 * 2 + 𝔽 𝑌 = ෍ 𝑦 𝑞 𝑦 1/10 * 3 + = 2.5 1/10 * 4 + 𝑦 1/10 * 5 + 1/10 * 6

Expectations Depend on a Probability Distribution number of pieces of candy 1 2 3 4 5 6 1/2 * 1 + 1/10 * 2 + 𝔽 𝑌 = ෍ 𝑦 𝑞 𝑦 1/10 * 3 + = 2.5 1/10 * 4 + 𝑦 1/10 * 5 + 1/10 * 6