Lecture 16 Jan-Willem van de Meent TOPIC MODELS (part 3) Borrowing - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 16

Jan-Willem van de Meent

TOPIC MODELS

(part 3)

Borrowing from:
 David Blei
 (Columbia)

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

βk: Topics (shared) θd: Topic Proportions

Review: Topic Modeling

zd: Assignments (document-specific) xd: Words

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

Calculating the Likelihood for each Word

Word Probabilities

gene 0.04 dna 0.02 genetic 0.01 .,,

β1 β2

life 0.02 evolve 0.01

• rganism 0.01

.,,

…

Topic Probabilities

θd

1 2 K …

How many genes does an

• rganism need to survive?

Vocabulary

1: the … 513: genes 712: organism 918: survive … V: …

Document

x d,6 = 721, zd,6 = 1

log probability of all words in document d

YY log p(x d | β,θ d) =

N d

X

n=1 V

X

v=1

I[x d,n = v]log Ç K X

k=1

β k,vθ d,k å

X d,v =

N d

X

n=1

I[x d,n = v]

bag of words

= X d log(θ dβ)

inner product between bag of word vector, and log weighted average over topics

Interpretation as Dimensionality Reduction

  '   112 · 0.91 ... 112 · 0.01 ... 234 · 0.02 ... 234 · 0.86     0.0081 ... 0.0002 ... 0.0001 ... 0.0072  

stocks chairman the wins game stocks game “sports” “finance”

  2 4 8 ... 1 ... 1 7 ... 2 3  

(D x V) (D x K) (K x V) Word Counts

E     X1,1 ... X1,V ... X D,1 ... X D,V    

Topic Counts Topic Word Probabilities

    =   N 1θ 1,1 ... N 1θ 1,K ... N Dθ D,1 ... N Dθ D,K     β1,1 ... β1,V ... β K,1 ... β K,V  

log(p(X d | β,θ d)) = X d log(θ dβ)>

X d,v =

N d

X

n=1

I[x d,n = v]

Bag of Word Vector Log marginal likelihood

Relationship to Latent Semantic Analysis

LSA: Factorize matrix of word counts (using PCA) LSA: Assume Gaussian distribution

⇥

u

⇥ ⇥

(

stocks: 2 · · · · · · · · · 0 chairman: 4 · · · · · · · · · 1 the: 8 · · · · · · · · · 7 · · · . . . · · · · · · · · · . . . wins: 0 · · · · · · · · · 2 game: 1 · · · · · · · · · 3)

u(

0.4 ·· -0.001 0.8 ·· 0.03 0.01 ·· 0.04 . . . ·· . . . 0.002 ·· 2.3 0.003 ·· 1.9 )( z1 . . . zn)

X u

) (

Ud Zk

(V x D) (V x K) (K x D)

Topic Models: Assume mixture of Discrete distributions

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

βk: Topics (shared) θd: Topic Proportions

LDA: Use Dirichlet Priors

zd: Assignments (document-specific) xd: Words

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

θ d ∼ Dirichlet(α) β k ∼ Dirichlet(ηk)

Review: Dirichlet Distribution

Review: Dirichlet Distribution

LDA: αk = 0.001 – Enforces Sparsity of Topic Weights θd

LDA: Summary so far

• Idea: Model documents as mixtures over topics
• Model parameters: 


θd Topic probabilities for each document 
 (K-dimensional vector for each document) 
 βk Word probabilities for each topic
 (V-dimensional vector for each topic)

• Interpretation Dimensionality Reduction: 


Similar to LSA, but assumes Discrete mixture 
 instead of Gaussian distribution on word counts

• Dirichlet Priors: Enforce sparsity, associate a small


number of topics which each document

Estimating Model Parameters

• 1. Expectation Maximization
• 2. Variational Inference 


(will discuss at a high level)

• 3. Gibbs Sampling

Question: How can we estimate βk and θd?

E-step: Update assignments Generative Model M-step: Update parameters

*(Probabilistic Latent Semantic Indexing, a.k.a. Probabilistic Latent Semantic Analysis)

Calculate probability that word n in document d belongs to topic k Use assignment probabilities φd to update topics αssignment probabilities θd and topic word 
 probabilities βk

φd,n,k = p(zd,n=k | x d,n,β,θ d)

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

PLSI/PLSA*: EM for Topic Models

• rganism 0.01

βk θd zd xd

φd,n,k = p(zd,n=k | x d,n = v,β,θ d)

(Apply Bayes’ Rule)

= p(x d,n=v, zd,n=k | β,θ d) p(x d,n=v | β,θ d)

(Substitute results from previous slides)

= θ d,kβ k,v PK

l=1 θ d,lβ l,v

φd,n,k = θ d,k ÄPV

v=1 β k,vI[x d,n = v]

ä PK

l=1 θ d,l

ÄPV

v=1 β l,vI[x d,n = v]

ä

General Form, with One-hot Indexing Trick

PLSI/PLSA: E-step

E-step: Update assignments Generative Model

• rganism 0.01

βk θd zd xd

φd,n,k = θ d,k ÄPV

v=1 β k,vI[x d,n = v]

ä PK

l=1 θ d,l

ÄPV

v=1 β l,vI[x d,n = v]

ä

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

PLSI/PLSA: EM for Topic Models

M-step: Update parameters Use assignment probabilities φd to update topics αssignment probabilities θd and topic word 
 probabilities βk

PLSI/PLSA: M-Step

Intuition: If we know topic zdn for each word,
 then we can estimate θd and βk

Find θd and βk that maximize probability of words xd given zd

Use Sufficient Statistics:

argmax

θ,β D

X

d=1

log p(x d | zd,θ d,β)

Number of times topic k

• ccurs in document d

N θ

d,k = Nd

X

n=1

I[zd,n = k]

N β

k,v = D

X

d=1 Nd

X

n=1

I[zd,n = k]I[x d,n = v]

Number of times word v

• ccurs in topic k

ML Estimates:

θ d,k = N θ

d,k

Nd

β k,v = N β

k,v

PD

d=1 N θ d,k

Fraction of topic k in document d Fraction of word v in topic k

PLSI/PLSA: M-Step

Sufficient Statistics

N θ

d,k = Nd

X

n=1

I[zd,n = k]

N β

k,v = D

X

d=1 Nd

X

n=1

I[zd,n = k]I[x d,n = v]

ML Estimates

θ d,k = N θ

d,k

Nd

β k,v = N β

k,v

PD

d=1 N θ d,k

Fraction of topic k in document d Fraction of word v in topic k

(hard assignment to topics) EM Sufficient Statistics (soft assignment to topics)

N β

k,v = D

X

d=1 Nd

X

n=1

φd,n,kI[x d,n = v]

N θ

d,k = Nd

X

n=1

φd,n,k

Probability that word n in document d belongs to topic k

φd,n,k = p(zd,n=k | x d,n,β,θ d)

E-step: Update assignments Generative Model

• rganism 0.01

βk θd zd xd

φd,n,k = θ d,k ÄPV

v=1 β k,vI[x d,n = v]

ä PK

l=1 θ d,l

ÄPV

v=1 β l,vI[x d,n = v]

ä

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

PLSI/PLSA: EM for Topic Models

M-step: Update parameters

θ d,k = N θ

d,k

Nd

β k,v = N β

k,v

PD

d=1 N θ d,k

N β

k,v = D

X

d=1 Nd

X

n=1

φd,n,kI[x d,n = v]

N θ

d,k = Nd

X

n=1

φd,n,k

Estimating Model Parameters

• 1. Expectation Maximization
• 2. Variational Inference 


(will discuss at a high level)

• 3. Gibbs Sampling


(won’t discuss) Question: How can we estimate βk and θd?

Variational Inference

Idea: Maximize Evidence Lower Bound (ELBO)

Maximizing the ELBO is equivalent to minimizing the KL divergence

Review: Conjugacy

Prior on Assignments Prior on Topic Weights Question: What distribution is the posterior?

Variational Expectation Maximization

(analogous to MAP estimation – need to know this)

Variational E-step: Update φ

φd,n,k = exp Ç Eq ñ logθ d,k +

V

X

v=1

I[x d,n = v]logβ k,v ôå

Idea: Approximate p(z, θ, β | x) with q(z, θ, β)

q(z,θ,β) = q(z;φ) q(θ;γ) q(β;λ)

φ,γ,λ = argmin

φ,γ,λ

KL(q(z,θ,β) || p(z,θ,β | x))

(high-level understanding is sufficient) (won’t derive this – but can be computed in closed form)

Variational M-step: Update γ and λ

λk,v = ηk,v + N β

k,v

γd,k = αk + N θ

d,k

EM vs. Variational EM

E-step: M-step:

λk,v = ηk,v + N β

k,v

γd,k = αk + N θ

d,k

φd,n,k = exp Ç Eq ñ logθ d,k +

V

X

v=1

I[x d,n = v]logβ k,v ôå

M-step: E-step:

φd,n,k ∝ θ d,k Ç V X

v=1

β k,vI[x d,n = v] å

θ d,k = N θ

d,k

Nd

β k,v = N β

k,v

PD

d=1 N θ d,k

φ,γ,λ = argmin

φ,γ,λ

KL(q(z,θ,β) || p(z,θ,β | x))

Variational EM EM

θ,β = argmax

θ,β

log p(x | θ,β)

EM vs. Variational EM

Commonalities: Both compute sufficient statistics

Probability that word n in document d belongs to topic k Number of words in document d that belong to topic k

φd,n,k

+ N θ

d,k

+ N β

k,v

Number of times word v appears in topic k (across all documents in corpus)

Differences: Point estimates vs Distributions

Fraction of words in document d for topic k

θ d,k

β k,v

EM: Computes most likely values for parameters Variational EM: Estimate Posterior over Parameters

q(β k;λk)

q(θ d;γd)

Approximation of topic distribution for document d Approximation of word distribution for topic k Fraction of words in topic k for vocabulary entry v

Performance Metric: Perplexity

10 20 30 40 50 60 70 80 90 100 1400 1600 1800 2000 2200 2400 2600 2800 3000 3200 3400

Number of Topics Perplexity

Smoothed Unigram Smoothed Mixt. Unigrams LDA Fold in pLSI 7000

Nematode abstracts Associated Press

20 40 60 80 100 120 140 160 180 200 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000

Number of Topics Perplexity

Smoothed Unigram Smoothed Mixt. Unigrams LDA Fold in pLSI

Marginal likelihood (evidence) of held out documents

Perplexity = exp ñ 1 D0

D0

X

d=1

1 Nd log(x 0

d | α,η)

ô {x 0

1,..., x 0 D0}

Extensions of LDA

• EM inference (PLSA/PLSI) yields similar results 


to Variational inference or MAP inference (LDA) 


• n most data
• Reason for popularity of LDA: 


can be embedded in more complicated models

Extensions: Supervised LDA

θd Zd,n Wd,n N D K

βk

α Yd η, σ2

1 Draw topic proportions θ | α ∼ Dir(α). 2 For each word

• Draw topic assignment zn | θ ∼ Mult(θ).
• Draw word wn | zn, β1:K ∼ Mult(βzn).

3 Draw response variable y | z1:N, η, σ2 ∼ N

• η>¯

z, σ2 , where ¯ z = (1/N) PN

n=1 zn.

Extensions: Supervised LDA

both motion simple perfect fascinating power complex however cinematography screenplay performances pictures effective picture his their character many while performance between

−30 −20 −10 10 20

• more

has than films director will characters

• ne

from there which who much what awful featuring routine dry

• ffered

charlie paris not about movie all would they its have like you was just some

• ut

bad guys watchable its not

• ne

movie least problem unfortunately supposed worse flat dull

Extensions: Correlated Topic Model

Estimate a covariance matrix Σ that parameterizes correlations between topics in a document

Zd,n Wd,n N D K

Σ µ ηd

βk

Noconjugate prior

• n topic proportions

Extensions: Dynamic Topic Models

Track changes in word distributions 
 associated with a topic over time.

AMONG the vicissitudes incident to life no event could have filled me with greater anxieties than that of which the notification was transmitted by your order...

1789

My fellow citizens: I stand here today humbled by the task before us, grateful for the trust you have bestowed, mindful

• f the sacrifices borne by our ancestors...

2009 Inaugural addresses

Extensions: Dynamic Topic Models

D θd Zd,n Wd,n N K α D θd Zd,n Wd,n N α D θd Zd,n Wd,n N α

βk,1 βk,2 βk,T

. . .

Extensions: Dynamic Topic Models

1880 electric machine power engine steam two machines iron battery wire 1890 electric power company steam electrical machine two system motor engine 1900 apparatus steam power engine engineering water construction engineer room feet 1910 air water engineering apparatus room laboratory engineer made gas tube 1920 apparatus tube air pressure water glass gas made laboratory mercury 1930 tube apparatus glass air mercury laboratory pressure made gas small 1940 air tube apparatus glass laboratory rubber pressure small mercury gas 1950 tube apparatus glass air chamber instrument small laboratory pressure rubber 1960 tube system temperature air heat chamber power high instrument control 1970 air heat power system temperature chamber high flow tube design 1980 high power design heat system systems devices instruments control large 1990 materials high power current applications technology devices design device heat 2000 devices device materials current gate high light silicon material technology

Extensions: Dynamic Topic Models

1880 1900 1920 1940 1960 1980 2000

• o o o o o o o
• o o o o o o o o o
• o o o o o o o o o o o o o o
• o
• o o o o
• o o
• o o
• o o

1880 1900 1920 1940 1960 1980 2000

• o o
• o
• o
• o o o
• o o o o o o o o o o
• o o o
• o
• o
• o o o o
• o o
• o o o o o o o o o o o o o
• o o o o o

RELATIVITY LASER FORCE NERVE OXYGEN NEURON "Theoretical Physics" "Neuroscience"

Extensions: Ideal Point Topic Models

θd N D K

βk

α

Wdn Zdn η Xu

Legislator ideal points Observed votes

U

Vud Ad, Bd σ2

u

σ2

d

Bill content (topic model) Bill sentiment variables

Extensions: Ideal Point Topic Models

dod,defense,defense and appropriation,military,subtitle veteran,veterans,bills,care,injury people,woman,american,nation,school producer,eligible,crop,farm,subparagraph coin,inspector,designee,automobile,lebanon bills,iran,official,company,sudan human,vietnam,united nations,call,people drug,pediatric,product,device,medical child,fire,attorney,internet,bills surveillance,director,court,electronic,flood energy,bills,price,commodity,market land,site,bills,interior,river child,center,poison,victim,abuse coast guard,vessel,space,administrator,requires science,director,technology,mathematics,bills computer,alien,bills,user,collection head,start,child,technology,award loss,crop,producer,agriculture,trade bills,tax,subparagraph,loss,taxable cover,bills,bridge,transaction,following transportation,rail,railroad,passenger,homeland security business,administrator,bills,business concern,loan defense,iraq,transfer,expense,chapter medicare,medicaid,child,chip,coverage student,loan,institution,lender,school energy,fuel,standard,administrator,lamp housing,mortgage,loan,family,recipient bank,transfer,requires,holding company,industrial county,eligible,ballot,election,jurisdiction tax credit,budget authority,energy,outlays,tax