Lecture 15 Jan-Willem van de Meent TOPIC MODELS (part 2) Borrowing - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 15

Jan-Willem van de Meent

TOPIC MODELS

(part 2)

Borrowing from:
 David Blei
 (Columbia)

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

βk: Topics (shared) θd: Topic Proportions

Topic Modeling

zd: Assignments (document-specific) xd: Words

zd,n ∼ Discrete(θ d) x d,n | zd,n=k ∼ Discrete(β k)

Calculating the Likelihood for each Word

Word Probabilities

gene 0.04 dna 0.02 genetic 0.01 .,,

β1 β2

life 0.02 evolve 0.01

• rganism 0.01

.,,

…

Topic Probabilities

θd

1 2 K …

How many genes does an

• rganism need to survive?

Vocabulary

1: the … 513: genes 721: organism 918: survive … V: …

Document

x d,6 = 721, zd,6 = 1

Probability that word n is entry v in the vocabulary

p(x d,n=v | β,θ d) =

Probability that word belongs to topic k Probability of word v given topic k

) =

K

X

k=1

p(x d,n=v | β, zd,n=k) p(zd,n=k | θ d)

K

X

X =

K

X

k=1

β k,vθ d,k

Calculating the Likelihood for each Word

Word Probabilities

gene 0.04 dna 0.02 genetic 0.01 .,,

β1 β2

life 0.02 evolve 0.01

• rganism 0.01

.,,

…

Topic Probabilities

θd

1 2 K …

How many genes does an

• rganism need to survive?

Vocabulary

1: the … 513: genes 712: organism 918: survive … V: …

Document

x d,6 = 721, zd,6 = 1

probability of all words n = 1…N in document d (use one-hot trick)

p(x d | β,θ d) =

N d

Y

n=1 V

Y

v=1

p(x d,n | β,θ d)I[x d,n=v] Ç å

take log probability, substitute result from previous slide

YY log p(x d | β,θ d) =

N d

X

n=1 V

X

v=1

I[x d,n = v]log Ç K X

k=1

β k,vθ d,k å

Calculating the Likelihood for each Word

Word Probabilities

gene 0.04 dna 0.02 genetic 0.01 .,,

β1 β2

life 0.02 evolve 0.01

• rganism 0.01

.,,

…

Topic Probabilities

θd

1 2 K …

How many genes does an

• rganism need to survive?

Vocabulary

1: the … 513: genes 712: organism 918: survive … V: …

Document

x d,6 = 721, zd,6 = 1

log probability of all words in document d

YY log p(x d | β,θ d) =

N d

X

n=1 V

X

v=1

I[x d,n = v]log Ç K X

k=1

β k,vθ d,k å

X d,v =

N d

X

n=1

I[x d,n = v]

bag of words

= X d log(θ dβ)

inner product between bag of word vector, and log weighted average over topics

Interpretation as Dimensionality Reduction

  '   112 · 0.91 ... 112 · 0.01 ... 234 · 0.02 ... 234 · 0.86     0.0081 ... 0.0002 ... 0.0001 ... 0.0072  

stocks chairman the wins game stocks game “sports” “finance”

  2 4 8 ... 1 ... 1 7 ... 2 3  

(D x V) (D x K) (K x V) Word Counts

E     X1,1 ... X1,V ... X D,1 ... X D,V    

Topic Counts Topic Word Probabilities

    =   N 1θ 1,1 ... N 1θ 1,K ... N Dθ D,1 ... N Dθ D,K     β1,1 ... β1,V ... β K,1 ... β K,V  

log(p(X d | β,θ d)) = X d log(θ dβ)>

X d,v =

N d

X

n=1

I[x d,n = v]

Bag of Word Vector Log marginal likelihood

Relationship to Latent Semantic Analysis

LSA: Factorize matrix of word counts (using PCA) LSA: Assume Gaussian distribution

⇥

u

⇥ ⇥

(

stocks: 2 · · · · · · · · · 0 chairman: 4 · · · · · · · · · 1 the: 8 · · · · · · · · · 7 · · · . . . · · · · · · · · · . . . wins: 0 · · · · · · · · · 2 game: 1 · · · · · · · · · 3)

u(

0.4 ·· -0.001 0.8 ·· 0.03 0.01 ·· 0.04 . . . ·· . . . 0.002 ·· 2.3 0.003 ·· 1.9 )( z1 . . . zn)

X u

) (

Ud Zk

(V x D) (V x K) (K x D)

Topic Models: Assume mixture of Discrete distributions

PLSI/PLSA(*): EM for Topic Models

E-step: Update assignments Generative Model M-step: Update parameters

*(Probabilistic Latent Semantic Indexing, a.k.a. Probabilistic Latent Semantic Analysis)

Calculate probability that word n in document d belongs to topic k Use assignment probabilities φd to update topics αssignment probabilities θd and topic word 
 probabilities βk

φd,n,k = p(zd,n=k | x d,n,β,θ d)

PLSI/PLSA(*): E-Step

φd,n,k = p(zd,n=k | x d,n = v,β,θ d)

(Apply Bayes’ Rule)

= p(x d,n=v, zd,n=k | β,θ d) p(x d,n=| β,θ d)

(Substitute results from previous slides)

= θ d,kβ k,v PK

l=1 θ d,lβ l,v

PLSI/PLSA(*): EM for Topic Models

Generative Model

*(Probabilistic Latent Semantic Indexing, a.k.a. Probabilistic Latent Semantic Analysis)

E-step: Update assignments M-step: Update parameters

LDA: Topic Models with Priors

Generative Model (with priors)

Review: Conjugacy

Prior on Assignments Prior on Topic Weights Question: What distribution is the posterior?

LDA: Topic Models with Priors

Generative Model (with priors) Maximum a Posteriori E-step: Update assignments M-step: Update parameters

Review: Dirichlet Distribution

Review: Dirichlet Distribution

Common choice in LDA: αk = 0.001 —> Ensures Sparsity

LDA: Summary so far

Core Idea: Model documents as mixtures over topics Model parameters: θd Topic probabilities for each document 
 (K-dimensional vector) βk Word probabilities for each topic
 (V-dimensional vector)
 Relationship to Dimensionality Reduction: Similar to LSA, but assumes Discrete mixture instead of Gaussian distribution on word counts

Next Week: LDA in Practice

MAP estimation is not widely used. LDA 
 implementations in practice use either of:

• 1. Gibbs Sampling
• 2. Variational Inference