14 Allocation Dirichlet Latent Lecture : Taheri Sara Scribes - PowerPoint PPT Presentation

14 Allocation Dirichlet Latent Lecture : Taheri Sara Scribes : Chu 4am Exam Man Tue 12 :

Midterm Exam focus Open Format book understanding on : , IEM ) Everything 8 Lecture Content to up ' . ( VBEM 9810 lectures , , be will ) n±t exam an Grading Typically to designed spread maximize : Devi Average Standard :# 5/100 -10 score . lecture Preparation Read ! notes :

Odi Dirichlet IN Dirichlet Discrete ai Allocation Model Latent Generative : Topics between / shared does Generative model : . ,wr ) } Bu A Od ~ Bh yan , . . , Oa Dirichlet 6 ) K .dk ~ , . , . . | Discrete . ,Gdk ) ( 2- du n w 2- an Y , . . \ , Buu ) Na anti ydnl ~ 7 . . . , K topics b { documents D Mixture model for document each distribution component B mixture a a " " b. topic Wends over a a a.

Word Mixtures Model topics ( ignore order ) Idea document mixture as : over Topics Document Disc ( 13h ) be Disc ( O ) 7- n=h ) I 2- pl I 2- Bwv yw ~ n= - yn - v = a - B , Bz . B3 By

Word Mixtures Question topics 13h ? learn Can model this : Problem How together ? ↳ words know which do go we ; Disc ( Bh ) be Disc ( O ) 7- n=h ) I 2- pl I 2- Bwv yw ~ n= - yn - v = a - B , Bz . B 3 By

Latent Allocation / Dirichlet PLSA ) and Idea Infer topics by documents multiple modeling : Disc ( Bu ) ydnltnih Odnplo ) Disclose ) Bu~p( B) Zdnn ~ ( PLSA LDA & ) ( ) LPA ( LDA ) ( k ) ( D ) Adi B , Bz Ydi Td - l - 7dH BK Yan

LDA Intuition : Not the all Idea documents words will contain same : " " unlikely article genetic sports a - is in " " innings unlikely paper ic in science - a ( top probs ) " " foal Ta to Use cluster documents :c : fish ) 70 of be small 1 Want topics to number ~ - - Od

Gamma function Dirioulet Distribution Symmetric ) Chloe i & 4 = = K maB( , - K Density . V d K M Man ) I M 9h " f p( ,ak ) Osa BG ) = = , , ... h a) he p( fgw ) Et [Ow]= ¥ prim 80ns normalized weights an ' OI = ( 7.0 ( 0,1 ) A 70 ) ( 0.1 A ) 7.0 9 10,10 7.0 = = , , , , ,

Dirioulet Distribution :# fsw an D= : 1.0 law no .o ,

LDA Example frequent : Topics most of B Bz B3 By , 9 0dg Od3 Odz 7 7 > di

LDA Example frequent : Topics most of

LDA Example frequent : Topics most of

Gibbs LDA Sampling : Generative Model Bh Yan Bh Dirichlet In .iq ) Odi ai ~ , . . K , Oa ) Dirichlet ( d .dk ~ , , . . . W & Fdn . ,8dk ) Discrete ( d 7- du ~ . , . Nd I , Buu ) Discrete b anti ydnl ~ 7 . . . , a- Gibbs Sampler Conditional Independence , 13 ) pttdulydn , Od Iyetd.to#d,8e-td/pOd~pCOdlyd.2-d 2- Zd yd n du , , given B) , P ) ( documents independent pcphly.tl put I Pu Beth ~ 2- g.

Sufficient Gibbs Statistics Sampling : w ) a ) w ) plot I BI ply ply , B ) plz 103 10 7,0 , p 17 = p , , - h ) cyan Zdn If Ittdn ) yaniv - M I ? Tal p , p ) B ) pl h I I y 2- = Ma v - - - - , n i Pur I Too !bularT wolves documents topics he ] =D Iftar II both I ? =/ ? I ? 13 Ma ur log Phu ( Ea ? [ § f - HI ) ) u ] Iban If exp = yan - - - - number Nhu of times the : " " " he " topic word v appears in

Sufficient Gibbs Statistics Sampling : w ) photo ) ) play B) I pl ply ply PCZIO ) 7,0 17 p = , , , Nur ) exp [ { f leg B ) ply par it = , h ] Iltdh - I ? ) 710 P' ) dn=h I c od = a p Ma Tal ) It Zdih Oda I ? = DM My ( { ) ) ) Cafe log I Ittanh 0dL = exp Number of Ndh : document words in " d " that belong to " topic " U

Sufficient Gibbs Statistics Sampling : w ) photo ) ) play B) I pl ply ply PCZIO ) 7,0 17 p = , , , Nur ) exp [ { f leg par B ) ply iz = , Ndh ) [ § ? log 10 Odu plz ) exp = ( A) Od I lo ) Dirichlet I O p = Ma - i - u 17 one Ma = , a BIO ) ) - log - I ) ) I ? 0dL ( log ou exp =

Sufficient Gibbs Statistics Sampling : w ) w ) photo ) I BI B) ply 10 ply PCZIO ) 7,0 , p 17 p = , , , Nur ) exp [ § f leg par B ) pc I z y = , Ndh ) [ § ? log IO Odu plz ) exp = BIO ) ) - log - t ) ) If ( 0dL log Cola ) ou exp p = - D ) co leg Blunt ) Iwao flog Pw w , pep , exp - =

Gibbs Sampling Conjugacy i w ) w ) photo ) I ply la ply B) PCZIO ) pl 17 7,0 , p p = , , , Nur ) exp I { f leg pc B ) par I y z = , ( flog Pnv If - it ) log Blunt ) I Iw ) pep who exp - = Thu exp I fu ( flog - - leg Blind ) Plz ,w ) - D ) ply But Nhutwuu = , Btwn ) I ) ) expffff.bg/3uufw~w leg = - - Dirichlet ( Wh ) ) Bu PCB I bit ; w = a , ] leg Bluth ) wht B C ( leg exp - w ) pl I Z y = ,

Gibbs Sampling Conjugacy i w ) a ) w ) plot I ply to ply , B ) plz 103 pl 17 7,0 , p p = , , Nur ) exp I { f leg pc B ) par I y z = , exp I fu ( { log Pnv - it ) - by Blunt ) I w ) pep who I = Btwn ) w ) exp I fu ( flog Puufuiw I ) ) leg Plp ly 2- = - - , , I ? ( tog Cwhl ) ] w ) leg Bluth ) B p I y exp I Z - = , ✓ Nur W t hv hv =

Gibbs Sampling Conjugacy i w ) ) photo ) I ply ply play B) PCZIO ) pl 17 7,0 p = , , , Ndh ) [ § ? log plz 10 ) Odu exp = BIO ) ) - log - I ) ) If 0dL ( log Colo ) ou exp p = - log Blot Nd ) ) exp I ? ( Cu log Odle ( Out - I ) Ndh plotz a ) = , I Td ) ( Od Dirichlet Ma = ( log Blotted ) If Bla ) ) ] log ) exp pczio = - Idk Out Ndh =

Gibbs Sampling Collapsed dn¥ ,7lw,o Gibbs ( topic Updates I assignments tan } { 7 7 2- PC 2- I 7 I y = ~ du In du . - , Requirement Implementation Need marginals B to 0 compute and over . , / ) ) do ply dB o pig 13,0 I = 7 w , , , t Exploit Conjugacy Idea : . ) ) ply 1710 17 p w = . ,