Dirichet Allocation Latent Lecture 11 : Jordan Yuan Andrea - PowerPoint PPT Presentation

Dirichet Allocation Latent Lecture 11 : Jordan Yuan Andrea Scribes : , , Exam Next Feb ) ( week Wed 28 : on ) ( Due Homework Man Fri next was :

Midterm Exam focus Open Format book understanding on : , 1 Gandhy ) IEM ) Everything 8 to Lecture Content up ' . ( VBEM Fans 9810 lectures Exponential , , be WE the on be will ) n±+ exam an exam Grading Typically to designed spread maximize : Devi Average Standard :# 5/700 -10 scare . ( 60 lecture Preparation Read ! ) to date notes : pages

Word Mixtures Model topics ( ignore order ) Idea document mixture as : over Zn=k~Dis< Topics Document ( pw ) Disc ( O ) 7n=h ) I PC I Eu yw ~ yn=v e Ryn B , # words Rz vochb Bkd in . - # topics

Word Mixtures Question topics 13h ? learn Can model this : ↳ Problem How figure together which wards cut do we : go Disc ( Bh ) be Disc ( O ) 7n=h ) I Zn= PC I Eu Bwu yw ~ ~ yn=v = B. . B: By

Latent Allocation / Dirichlet PLSA ) and Analysts Probabilistic Senate Latent Idea Infer topics by documents multiple modeling : Disc ( Bu ) Pw~P( 137 Od~plO Disclose ) ) Zdnn KDA ) ( ) ( LPA ) PLSA ydnltaehn ( D ) ( K ) P , 7d , I 0 Oa Bz . : I in 7dNd Nd . PK

LDA Intuition : Not the all Idea documents words will contain same : " " unlikely article genetic sports a - is in " " innings unlikely paper ic in science - a ( top probs ) " " foal Ta to Use cluster documents :c : Od

LDA Example frequent : Topics most of B Bz B3 By , ÷ meal Odz Be % Odh 7 7 > ,

LDA Example frequent : Topics most of

LDA Example frequent : Topics most of

LDA Conditional Independence : Generative model : Pu Dirichetlw ) on Ad ~ Bh Zakyd Yan Oa Dirichlet ( x ) K ~ Discrete ( Od ) 2- dn~ W Zdn Discrete ( Pa ) Na 7dn=h ydnl ~ D 1- Bh | Bh ' 't d. ' 7 d ' ,7 Yd y ¥ a , On 1- Od±d ly , B 1- B Ya I y 'd ' d. 7- # Zd '±d a ,

Algorithm Gibbs Sampling Collapsed 1 : fibbs ( topic Updates assignments 1 FT Priori 7dn } { Zdn ( 7 7. 7 \ f. p 1 y ~ du = an , Requirement Implementation Need monginals B to O compute and over . , |dOdB ) , p ,O ) pcy ,7 pcyit = ,wv) Conjugate ( W Dinichlet Bun Od~ Diniohktld , , ... ) ,ak , , ...

Dirioulet Distribution function Gamma I k Density K .io#Eg.EaIw.*..Bo=rFon M Man ) 1 an i . non E [ Ow ]= g that to sm , ? 9 Eou =L Oi = ( 7.0 ( 0,1 ) A 70 ) ( 0,1 A ) 7.0 9 10 7.0 10 = = , , , , , ,

=L Dirioulet Distribution # [ On ] g. q JEFF an da= 1. 0 law am ,

/ Discrete Multiannual Distribution Distribution Joint Assignments Topic on N ,Ok ) ( 0 Disc Zn ~ . [ ... , Ifh=h7 Nh= - ] n= .lu?qIHn=h , 0 ) MY Pl7i ,7µ ; = ... , E. qE±' II. oath in an = = Distribution Multinemial NKIQN ) MUIHN O ,µ ) ... ,Nh PINI = , ... , , , , II. " on =

; k Has ayy 1/9 Exponential Conjuyacy : Family Forms Pc 717 Observation The riowlet distribution conjugate D: Is : to mulitinemial the lgq " ⇐ / Nh HF ) explnttct ) ] ) = ) = - ftp.expllogtu.NU th OUNH ] = = , II. OF " ' Dinioukt 10 pta as ; = - Hay 's ad hcy ) ( exp ] ) ) ) n p = y - i ) log On - log BK ) [ ( ) au exp = - Bl 1+1 ) In log at Out ) = =

Posterior Distributions Predictive Conjugacy : and Tty ) ] ) ] ytttf 1/7 ) lxp[ [ aci ply plziy ) exp = = . N tu(7l=Nh=[ In log Oh Zn an I nu = . = 4n=dutNh u=i ;§ ) Posterior ply 17 ) pcy = : ~ 9h= - In Jutta + Nh ) = = |dy Marginal pc7 ) ) ) pin pain : [ ACT ac 7 ) ] ) exp = - BC £ )1B(o7 =

LDA Marginal B 0 : over and Nau pl7a\ Eau 4 Observation 113,0 ) Is conjugate 7113,97 ply to : p , B(£)_ Ndw an + = = B 19 N %I[7dn=w ) Nah . D= =L [ III D N ydniv ) Ittdiih ) B ( E ) Eau ) pcylz + : Wu = Blwl - hi , ,

Gibbs Collapsed LDA Updates : Assignments Update far Taste ~ ~ U Yduiv ) ( & PC Z Odh 7dn= Wwv , Y an an . . , + Nuv =T dn Ndh dn whv ~ - - £du Bv An + = = 's E Bu e- " " Ni Sufficient statistics .in?nII7dn=h)Nhv=na,n,=,aIjYdni=YIIZa'nih an q . ] Nail " Hid " Ittaneh ) tdh d 'n'