Lecture 14 Jan-Willem van de Meent TOPIC MODELS Borrowing from : - - PowerPoint PPT Presentation

Unsupervised Machine Learning 
 and Data Mining

DS 5230 / DS 4420 - Fall 2018

Lecture 14

Jan-Willem van de Meent

TOPIC MODELS

Borrowing from:
 David Blei
 (Columbia)

Word Mixtures

Topics: Words: Idea: Model text as a mixture over words (ignore order)

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

Gaussian Mixtures vs Word Mixtures

Gaussian Mixture Model

zn

xn

Discrete Mixture Model

π

µ,Σ

zn

xn

β

θ

Difference: Replace Gaussian with Discrete

xn | zn=k ∼ Discrete(βk,1,...,βk,V)

xn | zn=k ∼ Normal(µk,Σk)

zn ∼ Discrete(π1,...,πK)

zn ∼ Discrete(θ1,...,θK)

Topic Modeling

Idea: Model corpus of documents with shared topics

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

assignments

Topics (shared) Words in Document (mixture over topics) Topic Proportions (document-specific)

Topic Modeling

• Each topic is a distribution over words
• Each document is a mixture over topics
• Each word is drawn from one topic distribution

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

assignments

Topics (shared) Words in Document (mixture over topics) Topic Proportions (document-specific)

xdn | zdn=k ∼ Discrete(βk)

zdn ∼ Discrete(θd)

gene 0.04 dna 0.02 genetic 0.01 .,, life 0.02 evolve 0.01

• rganism 0.01

.,, brain 0.04 neuron 0.02 nerve 0.01 ... data 0.02 number 0.02 computer 0.01 .,,

assignments

Topics (shared) Words in Document (mixture over topics) Topic Proportions (document-specific)

Each document has Different topic proportions

Topic Modeling

Example Inference

1 8 16 26 36 46 56 66 76 86 96 Topics Probability 0.0 0.1 0.2 0.3 0.4

Example Inference

human evolution disease computer genome evolutionary host models dna species bacteria information genetic

• rganisms

diseases data genes life resistance computers sequence

• rigin

bacterial system gene biology new network molecular groups strains systems sequencing phylogenetic control model map living infectious parallel information diversity malaria methods genetics group parasite networks mapping new parasites software project two united new sequences common tuberculosis simulations

Example Inference

Example Inference

problem model selection species problems rate male forest mathematical constant males ecology number distribution females fish new time sex ecological mathematics number species conservation university size female diversity two values evolution population first value populations natural numbers average population ecosystems work rates sexual populations time data behavior endangered mathematicians density evolutionary tropical chaos measured genetic forests chaotic models reproductive ecosystem

Interpretation as Dimensionality Reduction

(D x V) (D x K) (K x V) Documents bag of words

  '   112 · 0.91 ... 112 · 0.01 ... 234 · 0.02 ... 234 · 0.86     0.0081 ... 0.0002 ... 0.0001 ... 0.0072  

stocks chairman the wins game stocks game “sports” “finance”

  2 4 8 ... 1 ... 1 7 ... 2 3  

E     X1,1 ... X1,V ... X D,1 ... X D,V    

X dv :=

N d

X

n=1

I[x d,n = v]

Word Count per Topic lower-dimensional vector for each document Topics distributions

• ver words

    =   N 1θ 1,1 ... N 1θ 1,K ... N Dθ D,1 ... N Dθ D,K     β1,1 ... β1,V ... β K,1 ... β K,V  

Computing the Likelihood

log(p(x d | β,θ d)) = X

n

log(p(x dn | β,θ d)) X ✓Y ◆

X = X

n

log ✓Y

v

p(x dn = v | β,θ d)I[x dn=v] ◆ X

X ✓Y ◆ = X

n,v

I[x dn = v] log(p(x dn = v | β,θ d)) Ç å

X = X

n,v

I[x dn = v] log ÇX

k

p(x dn = v, zdn=k | β,θ d) å Ç å

X ÇX å = X

n,v

I[x dn = v] log ÇX

k

p(zdn=k | θ d) p(x dn = v | zdn=k,β) å Ç å

X ÇX å = X

n,v

I[x dn = v] log ÇX

k

θ d,k β k,v å

(1 x V) (V x 1)

(words are conditionally independent) (one-hot indexing) (sum rule) (definition of discrete probability distributions) (replace sums 
 with inner products) (replace product in log with sum outside log)