Expectation Maximization CMSC 473/673 UMBC Recap from last time - - PowerPoint PPT Presentation

Introduction to Latent Sequences & Expectation Maximization

CMSC 473/673 UMBC

Recap from last time (and the first unit)…

N-gram Language Models

predict the next word given some context…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) ∝ 𝑑𝑝𝑣𝑜𝑢(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗)

wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

Maxent Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄 ⋅ 𝑔(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1, 𝑥𝑗))

Neural Language Models

predict the next word given some context… wi-3 wi-2 wi wi-1 compute beliefs about what is likely…

𝑞 𝑥𝑗 𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1) = softmax(𝜄𝑥𝑗 ⋅ 𝒈(𝑥𝑗−3, 𝑥𝑗−2, 𝑥𝑗−1))

create/use “distributed representations”… ei-3 ei-2 ei-1 combine these representations… C = f

matrix-vector product

ew θwi

(Some) Properties of Embeddings

Capture “like” (similar) words Capture relationships

Mikolov et al. (2013)

vector(‘king’) – vector(‘man’) + vector(‘woman’) ≈ vector(‘queen’) vector(‘Paris’) - vector(‘France’) + vector(‘Italy’) ≈ vector(‘Rome’)

Learn more in:

• Your project
• Paper (673)
• Other classes (478/678)

Four kinds of vector models

Sparse vector representations

1. Mutual-information weighted word co-

• ccurrence matrices

Dense vector representations:

2. Singular value decomposition/Latent Semantic Analysis 3. Neural-network-inspired models (skip-grams, CBOW) 4. Brown clusters

Shared Intuition

Model the meaning of a word by “embedding” in a vector space The meaning of a word is a vector of numbers Contrast: word meaning is represented in many computational linguistic applications by a vocabulary index (“word number 545”) or the string itself

Intrinsic Evaluation: Cosine Similarity

Divide the dot product by the length of the two vectors This is the cosine of the angle between them Are the vectors parallel?

• 1: vectors point in
• pposite directions

+1: vectors point in same directions 0: vectors are orthogonal

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule

Basics of language modeling Goal: model (be able to predict) and give a score to language (whole sequences of characters or words) Simple count-based model Smoothing (and why we need it): Laplace (add-λ), interpolation, backoff Evaluation: perplexity

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule Basics of language modeling Goal: model (be able to predict) and give a score to language (whole sequences of characters or words) Simple count-based model Smoothing (and why we need it): Laplace (add-λ), interpolation, backoff Evaluation: perplexity

Tasks and Classification (use Bayes rule!) Posterior decoding vs. noisy channel model Evaluations: accuracy, precision, recall, and Fβ (F1) scores Naïve Bayes (given the label, generate/explain each feature independently) and connection to language modeling

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule Basics of language modeling Goal: model (be able to predict) and give a score to language (whole sequences of characters or words) Simple count-based model Smoothing (and why we need it): Laplace (add-λ), interpolation, backoff Evaluation: perplexity Tasks and Classification (use Bayes rule!) Posterior decoding vs. noisy channel model Evaluations: accuracy, precision, recall, and Fβ (F1) scores Naïve Bayes (given the label, generate/explain each feature independently) and connection to language modeling

Maximum Entropy Models Meanings of feature functions and weights Use for language modeling or conditional classification (“posterior in

• ne go”)

How to learn the weights: gradient descent

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule Basics of language modeling Goal: model (be able to predict) and give a score to language (whole sequences of characters or words) Simple count-based model Smoothing (and why we need it): Laplace (add-λ), interpolation, backoff Evaluation: perplexity Tasks and Classification (use Bayes rule!) Posterior decoding vs. noisy channel model Evaluations: accuracy, precision, recall, and Fβ (F1) scores Naïve Bayes (given the label, generate/explain each feature independently) and connection to language modeling Maximum Entropy Models Meanings of feature functions and weights Use for language modeling or conditional classification (“posterior in one go”) How to learn the weights: gradient descent

Distributed Representations & Neural Language Models What embeddings are and what their motivation is A common way to evaluate: cosine similarity

Course Recap So Far

Basics of Probability Requirements to be a distribution (“proportional to”, ∝) Definitions of conditional probability, joint probability, and independence Bayes rule, (probability) chain rule Basics of language modeling Goal: model (be able to predict) and give a score to language (whole sequences of characters or words) Simple count-based model Smoothing (and why we need it): Laplace (add-λ), interpolation, backoff Evaluation: perplexity Tasks and Classification (use Bayes rule!) Posterior decoding vs. noisy channel model Evaluations: accuracy, precision, recall, and Fβ (F1) scores Naïve Bayes (given the label, generate/explain each feature independently) and connection to language modeling Maximum Entropy Models Meanings of feature functions and weights Use for language modeling or conditional classification (“posterior in one go”) How to learn the weights: gradient descent Distributed Representations & Neural Language Models What embeddings are and what their motivation is A common way to evaluate: cosine similarity

LATENT SEQUENCES AND LATENT VARIABLE MODELS

Is Language Modeling “Latent?”

p(Colorless green ideas sleep furiously) = p(Colorless) * p(green | Colorless) * p(ideas | Colorless green) * p(sleep | green ideas) * p(furiously | ideas sleep)

Is Language Modeling “Latent?” Most* of What We’ve Discussed: Not Really

p(Colorless green ideas sleep furiously) = p(Colorless) * p(green | Colorless) * p(ideas | Colorless green) * p(sleep | green ideas) * p(furiously | ideas sleep)

these values are unknown but the generation process (explanation) is transparent

*Neural language modeling as an exception

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Is Document Classification “Latent?”

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Is Document Classification “Latent?” As We’ve Discussed

argmax𝑍 ෑ

𝑗

𝑞(𝑌𝑗|𝑍) ∗ 𝑞(𝑍) argmax𝑍 exp 𝜄 ⋅ 𝑔 𝑦, 𝑧 𝑎(𝑦) ∗ 𝑞(𝑍) argmax𝑍 exp(𝜄 ⋅ 𝑔 𝑦, 𝑧 )

ATTACK

Three people have been fatally shot, and five people, including a mayor, were seriously wounded as a result of a Shining Path attack today against a community in Junin department, central Peruvian mountain region.

Is Document Classification “Latent?” As We’ve Discussed: Not Really

argmax𝑍 ෑ

𝑗

𝑞(𝑌𝑗|𝑍) ∗ 𝑞(𝑍) argmax𝑍 exp 𝜄 ⋅ 𝑔 𝑦, 𝑧 𝑎(𝑦) ∗ 𝑞(𝑍) argmax𝑍 exp(𝜄 ⋅ 𝑔 𝑦, 𝑧 )

these values are unknown but the generation process (explanation) is transparent

Ambiguity → Part of Speech Tagging

British Left Waffles on Falkland Islands British Left Waffles on Falkland Islands British Left Waffles on Falkland Islands

Adjective Noun Verb Noun Verb Noun

• rthography

morphology

Adapted from Jason Eisner, Noah Smith

lexemes syntax semantics pragmatics discourse

• bserved text

Adapted from Jason Eisner, Noah Smith

Latent Modeling

explain what you see/annotate with things “of importance” you don’t

• rthography

morphology lexemes syntax semantics pragmatics discourse

• bserved text

Latent Sequence Models: Part of Speech p(British Left Waffles on Falkland Islands)

Latent Sequence Models: Part of Speech p(British Left Waffles on Falkland Islands)

Adjective Noun Verb Noun Verb Noun Prep Noun Noun Prep Noun Noun (i): (ii):

Latent Sequence Models: Part of Speech p(British Left Waffles on Falkland Islands)

1. Explain this sentence as a sequence of (likely?) latent (unseen) tags (labels) Adjective Noun Verb Noun Verb Noun Prep Noun Noun Prep Noun Noun (i): (ii):

Latent Sequence Models: Part of Speech p(British Left Waffles on Falkland Islands)

1. Explain this sentence as a sequence of (likely?) latent (unseen) tags (labels) 2. Produce a tag sequence for this sentence Adjective Noun Verb Noun Verb Noun Prep Noun Noun Prep Noun Noun (i): (ii):

Noisy Channel Model

Decode Rerank

𝑞 𝑌 𝑍) ∝ 𝑞 𝑍 𝑌) ∗ 𝑞(𝑌)

possible (clean)

• utput
• bserved

(noisy) text translation/ decode model (clean) language model

Latent Sequence Model: Machine Translation

Decode Rerank

𝑞 𝑌 𝑍) ∝ 𝑞 𝑍 𝑌) ∗ 𝑞(𝑌)

possible (clean)

• utput
• bserved

(noisy) text

translation/ decode model

(clean) language model

Latent Sequence Model: Machine Translation

Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

Latent Sequence Model: Machine Translation

The cat is on the chair. Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

Latent Sequence Model: Machine Translation

The cat is on the chair. Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

How do you know what words translate as? Learn the translations!

Latent Sequence Model: Machine Translation

The cat is on the chair. Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

How do you know what words translate as? Learn the translations! How? Learn a “reverse” latent alignment model p(French words, alignments | English words)

Latent Sequence Model: Machine Translation

The cat is on the chair. Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

How do you know what words translate as? Learn the translations! How? Learn a “reverse” latent alignment model p(French words, alignments | English words) Alignment? Words can have different meaning/senses

Latent Sequence Model: Machine Translation

The cat is on the chair. Le chat est sur la chaise.

Eddie Izzard, “Dress to Kill” (MPAA: R) https://www.youtube.com/watch?v=x1sQkEfAdfY

How do you know what words translate as? Learn the translations! How? Learn a “reverse” latent alignment model p(French words, alignments | English words)

𝑞 English French) ∝ 𝑞 French English) ∗ 𝑞(English)

Why Reverse? Alignment? Words can have different meaning/senses

How to Learn With Latent Variables (Sequences)

Expectation Maximization

Example: Unigram Language Modeling

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗

Example: Unigram Language Modeling

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗

maximize (log-)likelihood to learn the probability parameters

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

add complexity to better explain what we see

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

add complexity to better explain what we see

examples of latent classes z:

• part of speech tag
• topic (“sports” vs. “politics”)

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

add complexity to better explain what we see

goal: maximize (log-)likelihood

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values we just see the words w

add complexity to better explain what we see

goal: maximize (log-)likelihood

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values we just see the words w

add complexity to better explain what we see

goal: maximize (log-)likelihood if we did observe z, estimating the probability parameters would be easy… but we don’t! :(

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = 𝑞 𝑥1 𝑞 𝑥2 ⋯ 𝑞 𝑥𝑂 = ෑ

𝑗

𝑞 𝑥𝑗 𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values we just see the words w

add complexity to better explain what we see

goal: maximize (log-)likelihood if we knew the probability parameters then we could estimate z and evaluate likelihood… but we don’t! :( if we did observe z, estimating the probability parameters would be easy… but we don’t! :(

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values goal: maximize marginalized (log-)likelihood

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values goal: maximize marginalized (log-)likelihood w

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values goal: maximize marginalized (log-)likelihood w

z=1 & w z=2 & w z=3 & w z=4 & w

Marginal(ized) Probability

w

𝑞 𝑥 = 𝑞 𝑨 = 1, 𝑥 + 𝑞 𝑨 = 2, 𝑥 + 𝑞 𝑨 = 3, 𝑥 + 𝑞(𝑨 = 4, 𝑥)

z=1 & w z=2 & w z=3 & w z=4 & w

(assuming z can take on only four possible values…)

Marginal(ized) Probability

w

𝑞 𝑥 = 𝑞 𝑨 = 1 , 𝑥 + 𝑞 𝑨 = 2, 𝑥 + 𝑞 𝑨 = 3, 𝑥 + 𝑞 𝑨 = 4, 𝑥 = ෍

𝑨=1 4

𝑞(𝑨 = 𝑗, 𝑥)

z=1 & w z=2 & w z=3 & w z=4 & w

(assuming z can take on only four possible values…)

Marginal(ized) Probability

w

𝑞 𝑥 = ෍

𝑨

𝑞(𝑨, 𝑥)

z=1 & w z=2 & w z=3 & w z=4 & w

Marginal(ized) Probability

w

𝑞 𝑥 = ෍

𝑨

𝑞(𝑨, 𝑥) = ෍

𝑨

𝑞 𝑨 𝑞 𝑥 𝑨)

z=1 & w z=2 & w z=3 & w z=4 & w

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂 = ෑ

𝑗

𝑞 𝑥𝑗|𝑨𝑗 𝑞 𝑨𝑗

we don’t actually observe these z values goal: maximize marginalized (log-)likelihood w

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = ෍

𝑨1

𝑞(𝑨1, 𝑥1) ෍

𝑨2

𝑞(𝑨2, 𝑥2) ⋯ ෍

𝑨𝑂

𝑞(𝑨𝑂, 𝑥𝑂)

z=1 & w z=2 & w z=3 & w z=4 & w

Example: Unigram Language Modeling with Hidden Class

𝑞 𝑨1, 𝑥1, 𝑨2, 𝑥2, … , 𝑨𝑂, 𝑥𝑂 = 𝑞 𝑨1 𝑞 𝑥1|𝑨1 ⋯ 𝑞 𝑨𝑂 𝑞 𝑥𝑂|𝑨𝑂

goal: maximize marginalized (log-)likelihood w

𝑞 𝑥1, 𝑥2, … , 𝑥𝑂 = ෍

𝑨1

𝑞(𝑨1, 𝑥1) ෍

𝑨2

𝑞(𝑨2, 𝑥2) ⋯ ෍

𝑨𝑂

𝑞(𝑨𝑂, 𝑥𝑂)

if we did observe z, estimating the probability parameters would be easy… but we don’t! :( if we knew the probability parameters then we could estimate z and evaluate likelihood… but we don’t! :(

z=1 & w z=2 & w z=3 & w z=4 & w

http://blog.innotas.com/wp-content/uploads/2015/08/chicken-or-egg-cropped1.jpg

http://blog.innotas.com/wp-content/uploads/2015/08/chicken-or-egg-cropped1.jpg

if we did observe z, estimating the probability parameters would be easy… but we don’t! :( if we knew the probability parameters then we could estimate z and evaluate likelihood… but we don’t! :(

http://blog.innotas.com/wp-content/uploads/2015/08/chicken-or-egg-cropped1.jpg

if we did observe z, estimating the probability parameters would be easy… but we don’t! :( if we knew the probability parameters then we could estimate z and evaluate likelihood… but we don’t! :(

Expectation Maximization (EM)

• 0. Assume some value for your parameters

Two step, iterative algorithm

• 1. E-step: count under uncertainty (compute expectations)
• 2. M-step: maximize log-likelihood, assuming these

uncertain counts

Expectation Maximization (EM): E-step

• 0. Assume some value for your parameters

Two step, iterative algorithm

• 1. E-step: count under uncertainty, assuming these

parameters

• 2. M-step: maximize log-likelihood, assuming these

uncertain counts

count(𝑨𝑗, 𝑥𝑗) 𝑞(𝑨𝑗)

Expectation Maximization (EM): E-step

• 0. Assume some value for your parameters

Two step, iterative algorithm

• 1. E-step: count under uncertainty, assuming these

parameters

• 2. M-step: maximize log-likelihood, assuming these

uncertain counts

count(𝑨𝑗, 𝑥𝑗) 𝑞(𝑨𝑗)

We’ve already seen this type of counting, when computing the gradient in maxent models.

Expectation Maximization (EM): M-step

• 0. Assume some value for your parameters

Two step, iterative algorithm

• 1. E-step: count under uncertainty, assuming these

parameters

• 2. M-step: maximize log-likelihood, assuming these

uncertain counts

𝑞 𝑢+1 (𝑨) 𝑞(𝑢)(𝑨)

estimated counts

EM Math

max 𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

EM Math

max

𝜄

𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

maximize the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

EM Math

max

𝜄

𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

maximize the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

EM Math

max

𝜄

𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

posterior distribution

maximize the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

current parameters

EM Math

max

𝜄

𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

current parameters new parameters new parameters posterior distribution

maximize the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

EM Math

max

𝜄

𝔽𝑨 ~ 𝑞𝜄(𝑢)(⋅|𝑥) log 𝑞𝜄(𝑨, 𝑥)

E-step: count under uncertainty M-step: maximize log-likelihood

current parameters new parameters new parameters posterior distribution

maximize the average log-likelihood of our complete data (z, w), averaged across all z and according to how likely our current model thinks z is

Three Coins/Unigram With Class Example

Imagine three coins Flip 1st coin (penny) If heads: flip 2nd coin (dollar coin) If tails: flip 3rd coin (dime)

Three Coins/Unigram With Class Example

Imagine three coins Flip 1st coin (penny) If heads: flip 2nd coin (dollar coin) If tails: flip 3rd coin (dime)

• nly observe these

(record heads vs. tails

• utcome)

don’t observe this

Three Coins/Unigram With Class Example

Imagine three coins Flip 1st coin (penny) If heads: flip 2nd coin (dollar coin) If tails: flip 3rd coin (dime)

• bserved:

a, b, e, etc. We run the code, vs. The run failed unobserved: vowel or constonant? part of speech?

Three Coins/Unigram With Class Example

Imagine three coins Flip 1st coin (penny) If heads: flip 2nd coin (dollar coin) If tails: flip 3rd coin (dime)

𝑞 heads = 𝜇 𝑞 tails = 1 − 𝜇 𝑞 heads = 𝛿 𝑞 heads = 𝜔 𝑞 tails = 1 − 𝛿 𝑞 tails = 1 − 𝜔

Three Coins/Unigram With Class Example

Imagine three coins

𝑞 heads = 𝜇 𝑞 tails = 1 − 𝜇 𝑞 heads = 𝛿 𝑞 heads = 𝜔 𝑞 tails = 1 − 𝛿 𝑞 tails = 1 − 𝜔 Three parameters to estimate: λ, γ, and ψ

Three Coins/Unigram With Class Example

If all flips were observed

H H T H T H H T H T T T

𝑞 heads = 𝜇 𝑞 tails = 1 − 𝜇 𝑞 heads = 𝛿 𝑞 heads = 𝜔 𝑞 tails = 1 − 𝛿 𝑞 tails = 1 − 𝜔

Three Coins/Unigram With Class Example

If all flips were observed

H H T H T H H T H T T T

𝑞 heads = 4 6 𝑞 tails = 2 6 𝑞 heads = 1 4 𝑞 heads = 1 2 𝑞 tails = 3 4 𝑞 tails = 1 2 𝑞 heads = 𝜇 𝑞 tails = 1 − 𝜇 𝑞 heads = 𝛿 𝑞 heads = 𝜔 𝑞 tails = 1 − 𝛿 𝑞 tails = 1 − 𝜔

Three Coins/Unigram With Class Example

But not all flips are observed → set parameter values

H H T H T H H T H T T T

𝑞 heads = 𝜇 = .6 𝑞 tails = .4 𝑞 heads = .8 𝑞 heads = .6 𝑞 tails = .2 𝑞 tails = .4

Three Coins/Unigram With Class Example

But not all flips are observed → set parameter values

H H T H T H H T H T T T

𝑞 heads = 𝜇 = .6 𝑞 tails = .4 𝑞 heads = .8 𝑞 heads = .6 𝑞 tails = .2 𝑞 tails = .4 𝑞 heads | observed item H = 𝑞(heads & H) 𝑞(H)

Use these values to compute posteriors

𝑞 heads | observed item T = 𝑞(heads & T) 𝑞(T)

Three Coins/Unigram With Class Example

But not all flips are observed → set parameter values

H H T H T H H T H T T T

𝑞 heads = 𝜇 = .6 𝑞 tails = .4 𝑞 heads = .8 𝑞 heads = .6 𝑞 tails = .2 𝑞 tails = .4

𝑞 heads | observed item H = 𝑞 H heads)𝑞(heads) 𝑞(H)

Use these values to compute posteriors

marginal likelihood rewrite joint using Bayes rule

Three Coins/Unigram With Class Example

But not all flips are observed → set parameter values H H T H T H H T H T T T

𝑞 heads = 𝜇 = .6 𝑞 tails = .4 𝑞 heads = .8 𝑞 heads = .6 𝑞 tails = .2 𝑞 tails = .4

𝑞 heads | observed item H = 𝑞 H heads)𝑞(heads) 𝑞(H)

Use these values to compute posteriors

𝑞 H | heads = .8 𝑞 T | heads = .2

Three Coins/Unigram With Class Example

But not all flips are observed → set parameter values H H T H T H H T H T T T

𝑞 heads = 𝜇 = .6 𝑞 tails = .4 𝑞 heads = .8 𝑞 heads = .6 𝑞 tails = .2 𝑞 tails = .4

Use these values to compute posteriors

𝑞 H = 𝑞 H | heads ∗ 𝑞 heads + 𝑞 H | tails * 𝑞(tails) = .8 ∗ .6 + .6 ∗ .4

𝑞 heads | observed item H = 𝑞 H heads)𝑞(heads) 𝑞(H) 𝑞 H | heads = .8 𝑞 T | heads = .2

Three Coins/Unigram With Class Example

H H T H T H H T H T T T

𝑞 heads | obs. H = 𝑞 H heads)𝑞(heads) 𝑞(H) = .8 ∗ .6 .8 ∗ .6 + .6 ∗ .4 ≈ 0.667

Use posteriors to update parameters

𝑞 heads | obs. T = 𝑞 T heads)𝑞(heads) 𝑞(T) = .2 ∗ .6 .2 ∗ .6 + .6 ∗ .4 ≈ 0.334 (in general, p(heads | obs. H) and p(heads| obs. T) do NOT sum to 1)

Three Coins/Unigram With Class Example

H H T H T H H T H T T T Use posteriors to update parameters

𝑞 heads = # heads from penny # total flips of penny fully observed setting

• ur setting: partially-observed

𝑞 heads = # 𝑓𝑦𝑞𝑓𝑑𝑢𝑓𝑒 heads from penny # total flips of penny 𝑞 heads | obs. H = 𝑞 H heads)𝑞(heads) 𝑞(H) = .8 ∗ .6 .8 ∗ .6 + .6 ∗ .4 ≈ 0.667 𝑞 heads | obs. T = 𝑞 T heads)𝑞(heads) 𝑞(T) = .2 ∗ .6 .2 ∗ .6 + .6 ∗ .4 ≈ 0.334 (in general, p(heads | obs. H) and p(heads| obs. T) do NOT sum to 1)

Three Coins/Unigram With Class Example

H H T H T H H T H T T T Use posteriors to update parameters

• ur setting: partially-observed

𝑞(𝑢+1) heads = # 𝑓𝑦𝑞𝑓𝑑𝑢𝑓𝑒 heads from penny # total flips of penny = 𝔽𝑞(𝑢)[# 𝑓𝑦𝑞𝑓𝑑𝑢𝑓𝑒 heads from penny] # total flips of penny 𝑞 heads | obs. H = 𝑞 H heads)𝑞(heads) 𝑞(H) = .8 ∗ .6 .8 ∗ .6 + .6 ∗ .4 ≈ 0.667 𝑞 heads | obs. T = 𝑞 T heads)𝑞(heads) 𝑞(T) = .2 ∗ .6 .2 ∗ .6 + .6 ∗ .4 ≈ 0.334

Three Coins/Unigram With Class Example

H H T H T H H T H T T T Use posteriors to update parameters

• ur setting:

partially-

• bserved

𝑞(𝑢+1) heads = # 𝑓𝑦𝑞𝑓𝑑𝑢𝑓𝑒 heads from penny # total flips of penny = 𝔽𝑞(𝑢)[# 𝑓𝑦𝑞𝑓𝑑𝑢𝑓𝑒 heads from penny] # total flips of penny = 2 ∗ 𝑞 heads | obs. H + 4 ∗ 𝑞 heads | obs. 𝑈 6 ≈ 0.444 𝑞 heads | obs. H = 𝑞 H heads)𝑞(heads) 𝑞(H) = .8 ∗ .6 .8 ∗ .6 + .6 ∗ .4 ≈ 0.667 𝑞 heads | obs. T = 𝑞 T heads)𝑞(heads) 𝑞(T) = .2 ∗ .6 .2 ∗ .6 + .6 ∗ .4 ≈ 0.334

Expectation Maximization (EM)

• 0. Assume some value for your parameters

Two step, iterative algorithm:

• 1. E-step: count under uncertainty (compute expectations)
• 2. M-step: maximize log-likelihood, assuming these

uncertain counts

Related to EM

Latent clustering K-means:

https://www.csee.umbc.edu/courses/undergraduate/473/f19/kmeans/

Gaussian mixture modeling