Sunday Homework 3 : an Diniohlet Allocation Model Latent - - PowerPoint PPT Presentation

SLIDE 1

Lecture

11 : Latent

Dirichet

Allocation ( Part

2) Scribes :

Yuan

, Andrea , Jordan Midterm : This Wednesday Homework 3 : Due an

Sunday

SLIDE 2 Latent Diniohlet Allocation : Generative Model Generative model : Pu ~ Diricwetlw ) Yan Bh Oa ~ Dirichlet ( x ) K 2- du~ Discrete ( Od )

• n Ad

Zan W ydnl 7dn=h ~ Discrete ( Pa ) Na D

SLIDE 3

Algorithm

1 :

Collapsed

Gibbs

Sampling

fibbs

Updates

( topic assignments 1 Zdn ~ p ( 7 du 1 y , 7.

FT

f. an = 7 \ { 7dn }

Implementation

Requirement Need to . compute monginals

• ver

O , and B pcy

,7

) =

|dOdB

pcyit , p ,O )

Conjugate

Priori

Bun Dinichlet ( W , , ...

,wv)

Od~ Diniohktld , , ... ,ak )

SLIDE 4 LDA :

Collapsed

Gibbs

Updates

Update far Taste

Assignments

yd \ { gaa

• Zal

{ Fan } ~ ~ PC 7dn= U ( Z . an , Y . an , Yduiv ) L Odh Wwv

• dn

~

• dn

£du = Out

Ndh

whv

= Bv + Nuv e- Sufficient Statistics

Ni

"

"+EBv

. an

q

Nail "

.in?nII7dn=h)Nhv=na.n,=,aIjYdni=YIIZa'nih

]

Nid

"

=T

Ittaneh

)

d 'n' tdh

SLIDE 5 LDA : Expectation Maximization ( MAP ) Objective O* , p* = anymax

log

Pc 9 Bly ) = aongygaxlgply ,9P) , B Lower bound p PLYIQB ) pczly ,0,B )

L( { 0,15 } ,q

)

=

Egan

, ,[

log.pl#TjB1sloypiy.0.p

) =

log

ply ,O,p )

• KL(

qttso ) Hpczly , 0,13 ) ) E . step M

• step

4 = argqmax

[

( 0 ,p,y ) 0,13 = angmaa

LIQB

, 10 ) , B qlz ;¢ ) = 91719,01137 ( will do this )

SLIDE 6 LDA : Expectation Maximization

(

MAP ) angqngax

Ll{

an

} , g) = argampux

Egan, ,[ by

P

"aYIj÷M

]

= angmax # qa ;¢ ) [ log pcyiz ,B ) + leg Palo ) ] t log PIG + leg PIB ) , B Exponential Family , Representation p ( × i y ) = h(x\ exp [ yt tcx )

• acy

) ] Etqcz

;¢|btply

17 , B ) ] =

#gaµ,§

• leg

Bhu . ( II [ yan=v7I[7an=h )

)]

Etgn;¢s[ by pc 7101 ) = Elq , , ,q , [ { log Odu . ( { Ittaneh )))

SLIDE 7 LDA : Expectation Maximization

(

MAP ) Lower bound pcy , 7. 9. B )

L( { 0,13 } ,q )

=

Eqa :p

, t.bg#1sbgply.&m Expectation Step :

Ndu= {

Fla

,#17an=h

] ) =

[

¢ndh ¢ ahh = Ftq , , ,¢ , [ I[7dn=h ) ) = Flpcz , , ,p ,g , [ I[7an=h ) ] Compute expected values

• f

sufficient statistics Maximization Step : ( exploit conjugacu ) Oh

• 1

+ { cfdnh wuu + £I[ ydutu ) ¢ duh

• 1

Odd =

• Phv

=

• {

de . 1 + £ cfdnl [ weu + fnI[ yaiu ]¢dnl

• l

e

SLIDE 8

LDA

: Other

Sampling

Algorithms Sequential

Monte Carlo

Requirement

: Sequence

• f

intermediate densities

Single

document :

p

PC 994in , 7 di :n ) Marginal

• ver

, B yn ( Zd , tin ) = Pcyd , iin , 7dam / B ) ← Marginal aw0

9(

For ,n I Fd , lin

• i )

= PC 7dm I yd , tin

• ,

, 7A , , :n

• a )

"( Analogous to Gibbs sampling ) Example Question : Compute the importance weights was for generations h > 1

SLIDE 9

Sequential

Marte Carlo ( General Formulation

) Assume

: Unnmmalized Densities g. 1 × , ) ... .

jfkt

) First step : Importance

Sampling

x. s ~ 9 ( × , ) ws , :-. ycxsilqkil Subsequent steps : Propose

from

previous samples

at

. , ~ Discrete ( win , ' "

int

' '

}

x. it . , ~

ins

!×"

+ . , )

xst~qcxi.ly?I.7x!t:=x!.x.aiiIi

' xt ~ qcxtix , it . , )

8+45

' it ) Incremental weiswl

vi.

is a

.sn#sqcxilxa?I

, ' ft . ,K

SLIDE 10

LDA

: Other

Sampling

Algorithms Sequential

Monte Carlo

pPlYd

.nl?amil3)p(7d,nlbd,i:n-i,7d.l:n

• i

,) yn ( Fd , 1

:^)

= pcyd , ' in ,7d , '

.nl/B)pCyd.l:nu,7d.l:n.i

)

9(

For ,n I Fd , lin

• i ,&d )

= P ( tap 1 7d,i :n

• i

, Yd ,iin

• i

, P ) Wns = Jul 7h , , :n ) are . ah , Ku ( 7d , , :ni ) 91

Faint

Yd , , ;n . , ,7d ,i:n

• i

) = Plbdn 17in ,B >

PHI

.nl bdy.to?In...lPCyd.i:n%7ad?iin.i1pcydn:n.,#iInilp1pi7'a,nl7IiYnYyd.iin.i.p ) = pcgda -17in , B) = µMpwI[9nn=D Isaiah ) v

SLIDE 11

LDA

: Other

Sampling

Algorithms

Hamiltonian Monte Carlo

Requires

: Gradient

• f

log

joint

Tqp

UC 0,13 ) =

• Vqp log

pcy ,O , R ) to L ( to .pl ,¢ )

¢dnEe

Epa iy ,D,p ) [ Ittndneh ) ) Example Question : Suppose you were to run LDA

• n

all

• f

Wikipedia . Would you recommend using HMC to sample 0,13 x

plop

ly ) ?

SLIDE 12 Hamiltonian

Monte

Carlo :

Algorithm

X. =Is ' '

• .

finlpl

HMC

( Single

step ) 2

!£5

%#

Fi

~

Norm

18 , th )

• ,

Edit

:X

' .

x.

÷ Is ' e. ← .

• It ,p ,

;= LEAPFROG ( RU , I , ,p , ,M,E ,T ) a = min ( l , , expl . HKF ,

• Ft ) ] )

How many times expttlki , -511 ) are we calling Tqp UIO.pl ? u ~ Uniform ( 0,1 ) T times Is ,=

{

It " £4 Is " u > d

SLIDE 13

LDA

: Other

Sampling

Algorithms

Example Question : Suppose you were to run LDA

• n

all

• f

Wikipedia . Would you recommend using HMC to sample 0,13 x

PIQP

ly ) ? Answer ; No . Computing the gradient for each leapfrog step would require a full press

• ver

the data .