(part 1) CMSC 473/673 UMBC Some slides adapted from 3SLP Outline - - PowerPoint PPT Presentation

Maxent and Neural Language Models (part 1)

CMSC 473/673 UMBC

Some slides adapted from 3SLP

Outline

Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

Terminology

Log-Linear Models (Multinomial) logistic regression Softmax regression Maximum Entropy models (MaxEnt) Generalized Linear Models Discriminative Naïve Bayes Very shallow (sigmoidal) neural nets

as statistical regression a form of viewed as based in information theory to be cool today :) common NLP term

Maxent Models are Flexible

Maxent models can be used:

• to create featureful language models, or
• to design discriminatively trained classifiers

Maxent Models as Featureful n-gram Language Models

generatively trained: learn to model (class-specific) language

𝑞 𝑦𝑗 𝑧, 𝑦𝑗−𝑂+1:𝑗−𝑗) = maxent(𝑧, 𝑦𝑗−𝑂+1:𝑗−𝑗, 𝑦𝑗)

p(Colorless green ideas sleep furiously | Label) = p(Colorless | Label, <BOS>) * … * p(<EOS> | Label , furiously)20

Model each n-gram term with a maxent model

Maxent Models for Classification: Discriminatively …

𝑞 𝑍 𝑌) = 𝐧𝐛𝐲𝐟𝐨𝐮(𝑌; 𝑍)

Discriminatively trained classifier

Directly model the posterior

Maxent Models for Classification: Discriminatively or Generatively Trained

𝑞 𝑍 𝑌) ∝ 𝒒 𝑌 𝑍) ∗ 𝑞(𝑍)

𝑞 𝑍 𝑌) = maxent(𝑌; 𝑍)

Discriminatively trained classifier Generatively trained classifier with maxent-based language model

Directly model the posterior Model the posterior with Bayes rule

Maximum Entropy (Log-linear) Models For Discriminatively Trained Classifiers

discriminatively trained: classify in one go

𝑞 𝑧 𝑦) = maxent 𝑦, 𝑧

(we’ll start with this one)

Discriminative ML Classification in 30 Seconds

• Common goal: probabilistic classifier p(y | x)
• Often done by defining features between x

and y that are meaningful

Discriminative ML Classification in 30 Seconds

• Common goal: probabilistic classifier p(y | x)
• Often done by defining features between x

and y that are meaningful

– Denoted by a general vector of K features 𝑔 𝑦, 𝑧 = (𝑔

1 𝑦, 𝑧 , … , 𝑔 𝐿(𝑦, 𝑧))

Discriminative ML Classification in 30 Seconds

• Common goal: probabilistic classifier p(y | x)
• Often done by defining features between x

and y that are meaningful

– Denoted by a general vector of K features 𝑔 𝑦, 𝑧 = (𝑔

1 𝑦, 𝑧 , … , 𝑔 𝐿(𝑦, 𝑧))

• Features can be thought of as “soft” rules

– E.g., POSITIVE sentiments tweets may be more likely to have the word “happy”

Discriminative ML Classification in 30 Seconds

• Common goal: probabilistic classifier p(y | x)
• Often done by defining features between x

and y that are meaningful

– Denoted by a general vector of K features 𝑔 𝑦, 𝑧 = (𝑔

1 𝑦, 𝑧 , … , 𝑔 𝐿(𝑦, 𝑧))

• Features can be thought of as “soft” rules

– E.g., POSITIVE sentiments tweets may be more likely to have the word “happy” Q: What are the features in a Naïve Bayes classifier?

Core Aspects to Maxent Classifier p(y|x)

We need to define

• features 𝑔 𝑦, 𝑧 between x and y that are

meaningful;

• weights 𝜄 (one per feature) to say how

important each feature is; and

• a way to form probabilities from 𝑔 and 𝜄

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region. Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result

• f a Shining Path attack today

against a community in Junin department, central Peruvian mountain region .

p( | )

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

• # killed:
• Type:
• Perp:

shot

ATTACK

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Discriminative Document Classification

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

These extractions are all features that have fired (likely have some significance)

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

We need to score the different extracted clues.

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

Score and Combine Our Clues

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

…

COMBINE posterior probability of ATTACK

…

scorek(🗏, ATTACK)

Scoring Our Clues

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

score( , ) =

ATTACK

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

…

(ignore the feature indexing for now)

Scoring Our Clues

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

score( , ) =

ATTACK

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

…

Learn these scores… but how? What do we

• ptimize?

https://www.csee.umbc.edu/courses/undergraduate/473/f19/loglin-tutorial/

Lesson 1

SNAP(score( , ))

ATTACK

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

What function…

• perates on any real number?

is never less than 0?

What function…

• perates on any real number?

is never less than 0? f(x) = exp(x)

exp(score( , ))

ATTACK

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

exp( ))

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

…

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

exp( ))

score1(🗏, ATTACK) score2(🗏, ATTACK) score3(🗏, ATTACK)

…

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

Learn the scores (but we’ll declare what combinations should be looked at)

exp( ))

weight1 * applies1(🗏, ATTACK) weight2 * applies2(🗏, ATTACK) weight3 * applies3(🗏, ATTACK)

…

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

exp( ))

weight1 * applies1(🗏, ATTACK) weight2 * applies2(🗏, ATTACK) weight3 * applies3(🗏, ATTACK)

…

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

K different weights… for K different features

exp( ))

weight1 * applies1(🗏, ATTACK) weight2 * applies2(🗏, ATTACK) weight3 * applies3(🗏, ATTACK)

…

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

K different weights… for K different features… multiplied and then summed

exp(

)

Dot_product of weight_vec feature_vec(🗏, ATTACK)

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

K different weights… for K different features… multiplied and then summed

exp(

)

𝜄𝑈𝑔(🗏, ATTACK)

Maxent Modeling

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | )∝

ATTACK

K different weights… for K different features… multiplied and then summed

𝜄𝑈𝑔(🗏, ATTACK)

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region .

p( | ) =

ATTACK

exp( ))

Maxent Modeling

1 Z

Q: How do we define Z?

exp( )

Σ

label y

Z =

Normalization for Classification

𝑞 𝑧 𝑦) ∝ exp(𝜄𝑈𝑔 𝑦, 𝑧 )

classify doc x with label y in one go

𝜄𝑈𝑔(🗏, Y)

exp( )

…

Σ

label y

Z =

Normalization for Classification (long form)

𝑞 𝑧 𝑦) ∝ exp(𝜄𝑈𝑔 𝑦, 𝑧 )

classify doc x with label y in one go

weight1 * applies1(🗏, y) weight2 * applies2(🗏, y) weight3 * applies3(🗏, y)

Core Aspects to Maxent Classifier p(y|x)

• features 𝑔 𝑦, 𝑧 between x and y that are

meaningful;

• weights 𝜄 (one per feature) to say how

important each feature is; and

• a way to form probabilities from 𝑔 and 𝜄

𝑞 𝑧 𝑦) = exp(𝜄𝑈𝑔(𝑦, 𝑧)) σ𝑧′ exp(𝜄𝑈𝑔(𝑦, 𝑧′))

Outline

Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

• 1. Defining Appropriate

Features

• 2. Understanding features

in conditional models

Defining Appropriate Features in a Maxent Model

Feature functions help extract useful features (characteristics) of the data They turn data into numbers Features that are not 0 are said to have fired

Defining Appropriate Features in a Maxent Model

Feature functions help extract useful features (characteristics) of the data They turn data into numbers Features that are not 0 are said to have fired Generally templated Often binary-valued (0 or 1), but can be real-valued

Templated Features

Define a feature fclue(🗏, label) for each clue you want to consider The feature fclue fires if the clue applies to/can be found in the (🗏, label) pair Clue is often a target phrase (an n-gram) and a label

Templated Features

Define a feature fclue(🗏, label) for each clue you want to consider The feature fclue fires if the clue applies to/can be found in the (🗏, label) pair Clue is often a target phrase (an n-gram) and a label

Q: For a classifier p(label | 🗏) are clues that depend only on 🗏 useful?

exp( ))

weight1 * applies1(🗏, ATTACK) weight2 * applies2(🗏, ATTACK) weight3 * applies3(🗏, ATTACK)

…

Maxent Modeling: Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

p( | )∝

ATTACK

binary

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: What does this function check?

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used?

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used?

A1: 𝑊𝑀

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used? 2. How many features are defined if bigram targets are used?

A1: 𝑊𝑀

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used? 2. How many features are defined if bigram targets are used?

A1: 𝑊𝑀 A2: 𝑊2𝑀

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used? 2. How many features are defined if bigram targets are used? 3. How many features are defined if unigram and bigram targets are used?

A1: 𝑊𝑀 A2: 𝑊2𝑀

Example of a Templated Binary Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applieshurt,ATTACK 🗏, ATTACK = ቊ1, hurt 𝑗𝑜 🗏 and ATTACK == ATTACK 0,

• therwise

Q: If there are V vocab types and L label types: 1. How many features are defined if unigram targets are used? 2. How many features are defined if bigram targets are used? 3. How many features are defined if unigram and bigram targets are used?

A1: 𝑊𝑀 A2: 𝑊2𝑀 A2: (V + 𝑊2)𝑀

More on Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

Templated real- valued Non-templated real-valued binary

??? ???

More on Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

applies 🗏, ATTACK = log 𝑞 🗏 ATTACK)

Non-templated real-valued binary Templated real- valued

???

More on Feature Functions

appliestarget,type 🗏, ATTACK = ቊ1, target 𝑛𝑏𝑢𝑑ℎ𝑓𝑡 🗏 and type == ATTACK 0,

• therwise

appliestarget,type 🗏, ATTACK = log 𝑞 🗏 ATTACK) + log 𝑞 type ATTACK) + log 𝑞(ATTACK |type)

Templated real- valued

applies 🗏, ATTACK = log 𝑞 🗏 ATTACK)

Non-templated real-valued binary

Understanding Conditioning 𝑞 𝑧 𝑦) ∝ count(𝑦)

Q: Is this a good model?

Understanding Conditioning 𝑞 𝑧 𝑦) ∝ exp(𝜄 ⋅ 𝑔 𝑦 )

Q: Is this a good model?

https://www.csee.umbc.edu/courses/undergraduate/473/f19/loglin-tutorial/

Lesson 11

Earlier, I said Maxent Models as Featureful n-gram Language Models of text x

𝑞 𝑦𝑗 𝑧, 𝑦𝑗−𝑂+1:𝑗−𝑗) = maxent(𝑧, 𝑦𝑗−𝑂+1:𝑗−𝑗, 𝑦𝑗)

p(Colorless green ideas sleep furiously | Label) = p(Colorless | Label, <BOS>) * … * p(<EOS> | Label , furiously)20

Model each n-gram term with a maxent model

Q: What would this look like?

Language Model with Maxent n-grams

𝑞𝑜 🗏 𝑧) = ෑ

𝑗=1 𝑁

maxent(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗)

n-gram label

Language Model with Maxent n-grams

𝑞𝑜 🗏 𝑧) = ෑ

𝑗=1 𝑁

maxent(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗)

= ෑ

𝑗=1 𝑁

exp(𝜄𝑈𝑔(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗)) σ𝑦′ exp(𝜄𝑈𝑔(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦′))

n-gram label

Language Model with Maxent n-grams

𝑞𝑜 🗏 𝑧) = ෑ

𝑗=1 𝑁

maxent(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗)

n-gram

= ෑ

𝑗=1 𝑁 exp(𝜄𝑈𝑔(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗))

𝑎(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗)

Language Model with Maxent n-grams

𝑞𝑜 🗏 𝑧) = ෑ

𝑗=1 𝑁

maxent(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗)

n-gram

= ෑ

𝑗=1 𝑁 exp(𝜄𝑈𝑔(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗, 𝑦𝑗))

𝑎(𝑧, 𝑦𝑗−𝑜+1:𝑗−𝑗)

Q: Why is this Z a function of the context?

Outline

Maximum Entropy models Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

pθ(u | v )

probabilistic model

• bjective

𝐺(𝜄; 𝑣, 𝑤)

Primary Objective: Likelihood

• Goal: maximize the score your model gives to

the training data it observes

• This is called the likelihood of your data
• In classification, this is p(label | 🗏)
• For language modeling, this is p(🗏 | label)

Objective = Full Likelihood? (in LM)

These values can have very small magnitude ➔ underflow Differentiating this product could be a pain

ෑ

𝑗

𝑞𝜄 𝑦𝑗 ℎ𝑗 ∝ ෑ

𝑗

exp(𝜄𝑈𝑔 𝑦𝑗, ℎ )

(assume ℎ𝑗 has whatever context and n-gram history necessary)

Logarithms

(0, 1] ➔ (-∞, 0] Products ➔ Sums log(ab) = log(a) + log(b) log(a/b) = log(a) – log(b) Inverse of exp log(exp(x)) = x

Log-Likelihood (n-gram LM)

Differentiating this becomes nicer (even though Z depends on θ) Wide range of (negative) numbers Sums are more stable

Products ➔ Sums log(ab) = log(a) + log(b) log(a/b) = log(a) – log(b)

log ෑ

𝑗

𝑞𝜄 𝑦𝑗 ℎ𝑗 = ෍

𝑗

log 𝑞𝜄(𝑦𝑗|ℎ𝑗)

Log-Likelihood (n-gram LM)

Wide range of (negative) numbers Sums are more stable Differentiating this becomes nicer (even though Z depends on θ)

log ෑ

𝑗

𝑞𝜄 𝑦𝑗 ℎ𝑗 = ෍

𝑗

log 𝑞𝜄(𝑦𝑗|ℎ𝑗) = ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

Inverse of exp log(exp(x)) = x

Log-Likelihood (n-gram LM)

Wide range of (negative) numbers Sums are more stable

log ෑ

𝑗

𝑞𝜄 𝑦𝑗 ℎ𝑗 = ෍

𝑗

log 𝑞𝜄(𝑦𝑗|ℎ𝑗) = ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗) = 𝐺 𝜄

Outline

Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

How will we optimize F(θ)?

Calculus

F(θ) θ

F(θ) θ

θ*

F(θ) θ F’(θ)

derivative of F wrt θ

θ*

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)

θ0 z0

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get derivative g t = F’(θ t)

θ0 z0 g0

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

θ0 z0 θ1 g0

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

θ0 z0 θ1 z1 θ2 g0 g1

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

θ0 z0 θ1 z1 θ2 z2 z3 θ3 g0 g1 g2

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the derivative

Set t = 0

Pick a starting value θt

Until converged:

• 1. Get value zt = F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

θ0 z0 θ1 z1 θ2 z2 z3 θ3 g0 g1 g2

Remember: Common Derivative Rules

Gradient = Multi-variable derivative

K-dimensional input K-dimensional output

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

Gradient Ascent

F(θ) θ F’(θ)

derivative

• f F wrt θ

θ*

What if you can’t find the roots? Follow the gradient

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get gradient g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

θ0 z0 θ1 z1 θ2 z2 z3 θ3 g0 g1 g2 K-dimensional vectors

Outline

Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

Reminder: Expectation of a Random Variable

1 2 3 4 5 6 number of pieces of candy 1/6 * 1 + 1/6 * 2 + 1/6 * 3 + 1/6 * 4 + 1/6 * 5 + 1/6 * 6 = 3.5

𝔽 𝑌 = ෍

𝑦

𝑦 𝑞 𝑦

Reminder: Expectation of a Random Variable

1 2 3 4 5 6 number of pieces of candy 1/2 * 1 + 1/10 * 2 + 1/10 * 3 + 1/10 * 4 + 1/10 * 5 + 1/10 * 6 = 2.5

𝔽 𝑌 = ෍

𝑦

𝑦 𝑞 𝑦

Reminder: Expectation of a Random Variable

1 2 3 4 5 6 number of pieces of candy 1/2 * 1 + 1/10 * 2 + 1/10 * 3 + 1/10 * 4 + 1/10 * 5 + 1/10 * 6 = 2.5

𝔽 𝑌 = ෍

𝑦

𝑦 𝑞 𝑦

Expectations Depend on a Probability Distribution

1 2 3 4 5 6 number of pieces of candy 1/2 * 1 + 1/10 * 2 + 1/10 * 3 + 1/10 * 4 + 1/10 * 5 + 1/10 * 6 = 2.5

𝔽 𝑌 = ෍

𝑦

𝑦 𝑞 𝑦

Log-Likelihood (n-gram LM)

Wide range of (negative) numbers Sums are more stable Differentiating this becomes nicer (even though Z depends

• n θ)

log ෑ

𝑗

𝑞𝜄 𝑦𝑗 ℎ𝑗 = ෍

𝑗

log 𝑞𝜄(𝑦𝑗|ℎ𝑗) = ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗) = 𝐺 𝜄

Log-Likelihood Gradient

Each component k is the difference between:

Log-Likelihood Gradient

Each component k is the difference between: the total value of feature fk in the training data ෍

𝑗

𝑔

𝑙(𝑦𝑗, ℎ𝑗)

Log-Likelihood Gradient

Each component k is the difference between: the total value of feature fk in the training data

and

the total value the current model pθ thinks it computes for feature fk ෍

𝑗

𝔽𝑦′~ 𝑞 𝑔

𝑙(𝑦′, ℎ𝑗)

෍

𝑗

𝑔

𝑙(𝑦𝑗, ℎ𝑗)

https://www.csee.umbc.edu/courses/undergraduate/473/f19/loglin-tutorial/

Lesson 6

𝛼𝜄𝐺 𝜄 = 𝛼𝜄 ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

Log-Likelihood Gradient Derivation for LM p 𝑦𝑗 | ℎ𝑗

𝑧𝑗

𝛼𝜄𝐺 𝜄 = 𝛼𝜄 ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

= ෍

𝑗

𝑔 𝑦𝑗, ℎ𝑗 −

Log-Likelihood Gradient Derivation for LM p 𝑦𝑗 | ℎ𝑗

𝑧𝑗

𝑎 ℎ𝑗 = ෍

𝑦′

exp(𝜄 ⋅ 𝑔 𝑦′, ℎ𝑗 )

𝛼𝜄𝐺 𝜄 = 𝛼𝜄 ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

= ෍

𝑗

𝑔 𝑦𝑗, ℎ𝑗 − ෍

𝑗

෍

𝑦′

exp 𝜄𝑈𝑔 𝑦′, ℎ𝑗 𝑎 ℎ𝑗 𝑔(𝑦′, ℎ𝑗)

Log-Likelihood Gradient Derivation for LM p 𝑦𝑗 | ℎ𝑗

𝜖 𝜖𝜄 log 𝑕(ℎ 𝜄 ) = 𝜖𝑕 𝜖ℎ(𝜄) 𝜖ℎ 𝜖𝜄 use the (calculus) chain rule

scalar p(x’ | hi) vector of functions

𝛼𝜄𝐺 𝜄 = 𝛼𝜄 ෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

= ෍

𝑗

𝑔 𝑦𝑗, ℎ𝑗 − ෍

𝑗

෍

𝑦′

exp 𝜄𝑈𝑔 𝑦′, ℎ𝑗 𝑎 ℎ𝑗 𝑔(𝑦′, ℎ𝑗)

Log-Likelihood Gradient Derivation for LM p 𝑦𝑗 | ℎ𝑗

Do we want these to fully match? What does it mean if they do? What if we have missing values in our data?

Gradient Optimization for LM p 𝑦𝑗 | ℎ𝑗

Set t = 0 Pick a starting value θt Until converged:

• 1. Get value z t = F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

𝜖𝐺 𝜖𝜄𝑙 = ෍

𝑗

𝑔

𝑙 𝑦𝑗, ℎ𝑗 − ෍ 𝑗

෍

𝑦′

𝑔

𝑙 𝑦′, ℎ𝑗 𝑞 𝑦′ ℎ𝑗)

෍

𝑗

𝜄𝑈𝑔 𝑦𝑗, ℎ𝑗 − log 𝑎(ℎ𝑗)

Gradient Optimization for Classifier p 𝑧 |🗏

Set t = 0 Pick a starting value θt Until converged:

• 1. Get func. value F(θ t)
• 2. Get derivative g t = F’(θ t)
• 3. Get scaling factor ρ t
• 4. Set θ t+1 = θ t + ρ t *g t
• 5. Set t += 1

𝜖𝐺 𝜖𝜄𝑙 = 𝑔

𝑙 🗏, 𝑧 − ෍ 𝑧′

𝑔

𝑙 🗏, 𝑧′ 𝑞 𝑧′ 🗏)

𝜄𝑈𝑔 🗏, 𝑧 − log 𝑎(🗏)

Preventing Extreme Values

Naïve Bayes

Extreme values are 0 probabilities

𝑞 𝑣 𝑤 ∝ count(𝑤, 𝑣) 𝑞 𝑣 𝑤 ∝ count 𝑤, 𝑣 + 𝜇

Preventing Extreme Values

Naïve Bayes Log-linear models

Extreme values are 0 probabilities Extreme values are large θ values

𝑞 𝑣 𝑤 ∝ count(𝑤, 𝑣) 𝑞 𝑣 𝑤 ∝ count 𝑤, 𝑣 + 𝜇 𝐺 𝜄 = ෍

𝑗

log 𝑞𝜄(𝑣𝑗|𝑤𝑗) 𝐺 𝜄 = ෍

𝑗

log 𝑞𝜄(𝑣𝑗|𝑤𝑗) − 𝑆 𝜄

Preventing Extreme Values

Naïve Bayes Log-linear models

Extreme values are 0 probabilities Extreme values are large θ values

regularization

𝑞 𝑣 𝑤 ∝ count(𝑤, 𝑣) 𝑞 𝑣 𝑤 ∝ count 𝑤, 𝑣 + 𝜇 𝐺 𝜄 = ෍

𝑗

log 𝑞𝜄(𝑣𝑗|𝑤𝑗) 𝐺 𝜄 = ෍

𝑗

log 𝑞𝜄(𝑣𝑗|𝑤𝑗) − 𝑆 𝜄

(Squared) L2 Regularization

https://www.csee.umbc.edu/courses/undergraduate/473/f19/loglin-tutorial/

Lesson 8

Outline

Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models

Revisiting the SNAP Function

softmax

𝑞 𝑣 𝑤) ∝ exp(𝜄 ⋅ 𝑔 𝑤, 𝑣 )

Revisiting the SNAP Function

softmax

𝑞 𝑣 𝑤) ∝ exp(𝜄 ⋅ 𝑔 𝑤, 𝑣 ) softmax 𝒜 𝑗 = exp(𝑨𝑗) σ𝑘 exp(𝑨

𝑘)

N-gram Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1

N-gram Language Models

predict the next word given some context…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) ∝ 𝑑𝑝𝑣𝑜𝑢(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗)

wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

N-gram Language Models

predict the next word given some context…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) ∝ 𝑑𝑝𝑣𝑜𝑢(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗)

wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

Maxent Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄 ⋅ 𝑔(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗))

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗))

can we learn the feature function(s)?

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝒙𝒋 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

can we learn the feature function(s) for just the context? can we learn word-specific weights (by type)?

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝑥𝑗 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

create/use “distributed representations”… ei-3 ei-2 ei-1 ew

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝑥𝑗 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

create/use “distributed representations”… ei-3 ei-2 ei-1 combine these representations… C = f

matrix-vector product

ew

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝑥𝑗 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

create/use “distributed representations”… ei-3 ei-2 ei-1 combine these representations… C = f

matrix-vector product

ew θwi

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝑥𝑗 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

create/use “distributed representations”… ei-3 ei-2 ei-1 combine these representations… C = f

matrix-vector product

ew θwi

“A Neural Probabilistic Language Model,” Bengio et al. (2003)

Baselines

LM Name N- gram Params. Test Ppl. Interpolation 3

• 336

Kneser-Ney backoff 3

• 323

Kneser-Ney backoff 5

• 321

Class-based backoff 3 500 classes 312 Class-based backoff 5 500 classes 312

“A Neural Probabilistic Language Model,” Bengio et al. (2003)

Baselines

LM Name N- gram Params. Test Ppl. Interpolation 3

• 336

Kneser-Ney backoff 3

• 323

Kneser-Ney backoff 5

• 321

Class-based backoff 3 500 classes 312 Class-based backoff 5 500 classes 312

NPLM

N-gram Word Vector Dim. Hidden Dim. Mix with non- neural LM Ppl. 5 60 50 No 268 5 60 50 Yes 257 5 30 100 No 276 5 30 100 Yes 252

“A Neural Probabilistic Language Model,” Bengio et al. (2003)

Baselines

LM Name N- gram Params. Test Ppl. Interpolation 3

• 336

Kneser-Ney backoff 3

• 323

Kneser-Ney backoff 5

• 321

Class-based backoff 3 500 classes 312 Class-based backoff 5 500 classes 312

NPLM

N-gram Word Vector Dim. Hidden Dim. Mix with non- neural LM Ppl. 5 60 50 No 268 5 60 50 Yes 257 5 30 100 No 276 5 30 100 Yes 252 “we were not able to see signs of over- fitting (on the validation set), possibly because we ran only 5 epochs (over 3 weeks using 40 CPUs)” (Sect. 4.2)

Outline

Maximum Entropy classifiers Defining the model Defining the objective Learning: Optimizing the objective Math: gradient derivation Neural (language) models