Ruiyany shen ; , , Lecture others follow will Notes 3 : - PowerPoint PPT Presentation

Lecture 9 Variation EM : Elizabeth Yao Scribes Ruiyany shen ; , , Lecture others follow will Notes 3 : is up ,

Algorithm Generalized Expectation Maximization : Initialize O ; LIQH log Define Ezra , , ,[logPgYf÷y9I :o) pcy s : i. ... . £ ( O Repeat until threshold , g) below change anymore £10 ( Expectation of Step ) 1 Somewhat misnomer a , y = y ) , r Step Maximization 2 . £10 O ,H = argmax o

Variatimal Inference Today : Posterior Approximate Distribution Idea : Oly of ) 7,0 pc7 ) qc ; I , " ( Htnd calculate be ' to chosen to tractable more Minimise KL Objective by divergence : £19,01 ) Maximizing lower bond a ) ) # ( KL ;g ) 11 p( 7,01g argmin qczo = ¢ £17,01 Analogous ) EM to ← arggnax =

Bound ( Inference Variation Evidence Lower ELBO ) : EM leg likelihood Lower bound : on .gg#j9ofp :o) paly :o) 's ;o , ,[ log £1014 ) Egcz is :o) ) ) Hpctiy logpiy ;D log Kllqttig :o) ply s = - likelihood £0,4 bound Lower the ELBO leg marginal : on losPqYzl.to@oaan.a Add Prior 0 pc O ;D ) 710 ) ) ply 17,0 y z pc ~ ~ : ~ ' 9) Expectation Egg . ;µ[ over ) o i= . KL( go.o.gs//p(z,Oiy ;D ) ) by ply :D log :D ) pcy } =

Variation Maximization Expectation , [ by PYI.LI# ] Lidia # ELBO ) : = ,¢ , , a. % Problem hander compute mud to : 0+7 expectation Writ . factorized Use approximation Idea : Replace point g 0 estimate ; 47 ) 9105401 7,91 ;D ) 9C pc = 7 qlo ;g9 y with diet " ( Analogous to EM Note Pc Oly ;D ) ;D ) 7,01yd ) =/ pizly p( : # Oly z

Define :L Variation Algorithm : Expectation Maximization 40 Initialize ; qc a ,qµ[ loggia 9) @ pcy ,7 ( I. lot ,d° ) ) slog , Et :D ply = . 0,749 -017,100 ) £1,9 , Repeat threshold below until change Expectation 9 Step optimized 1 is n±t , a lot LC 7. Analogous EM to = argyyax step for y Step Maximization 2 . £17,017,100 ) Analogous 010 to EM angmax = go for step O

E for oft Solve step : . lgp%l glom ) ftp.tl ' . 99 ' ' ftp.euamq.am/lgPgYYaYgi@og dep no 7 on . i= ¥ , / . :¢7q( ago )| leg # ) pcyitio + ah y . leg ly 917 ; 47 ) deep 0 no an - = . :p 's # go ;qo ) ( log = off - by qa :¢3 ) # ) ) piynlt qn , 0 = Eg , @ go ) [ log pcy Solution log ) ] canst qc 7:47 : 710 t ± , ( ) optimal ,7lo ) ] ) [ Etqio ;qo)[ log ply Exp qcz ;¢a & ,

Define :L Variation Algorithm : Expectation Maximization 40 Initialize ; ( d. lot ,d° ) qu ,qµ[ logphfj.IT#)sbgpiy Et :D = . -017,010 ) £1,9 , Repeat threshold below until change Expectation Step 1 , pcyitlo ) ] ) Eqio ;q9[ log ( ; 92 ) L qiz exp pcyitiol Imagine theme tnmahweaaj " Step Maximization to 2. . ) ) # get ;qt)[ log I O ; 0101 q( s exp

Families Exponential Intermezzo : An family has form exponential distribution the Depends Only Only depends depends × on on / I and y x an n d hk [ aey ) } yttcxi pcxly ) ) exp = - [ KEY n E leg "EII" its normalize ( , Lebesguc ) ( Canting Base measure ( only depends ) × on

Gaussian Example Uniuaniate ; exp[ yttcx ) ] hcx ) pcxiy ) acy ) = - µ Panama k¥2 ' expf ; ] Dependent " # 2) = = ( zntzj "expft(x2 . z×µ+µ2)/ . ] ( tcx ×2 ) ) × = , ( µ1r2 ) -1/262 n = , log 1262 µ2 6 acy ) + = hcxl 16 1 =

Properties of Families Exponential [ yttcx 1 y ) ] hcx ) ( exp ) ) acy p x = - Derivatives Moments Normalizer Log of . . / = |d× hkiexpfyttki ) ) lacy 1 dx exp pixiy ) → = = gqµg/d× ) ) ) ( yttcx has again exp ' 7) / _.-- # '× P hcxiexplytfki ) taxi ax = = / 1 yitk hki expel alnl ) I exp - ax ) ( hkieaplyttki = |d× . acy ) ))= Epcxiy ,[ ) ] tix tax

Properties Families of Exponential [ yttcx ) ] hcx ) exp pcxiy ) acy ) = - Moments from of computable derivatives acy ) - are a a 'Y tkih ] ) [ # = pain dqn linearly tk independent exponential When - ) are an known family minimal is as family Far minimal and acy ) is convex any - tcxi ] Epa , , ,[ c→ y µ := pcxiy )[ Hxl ] ) from there ( to # to i. 1 mapping n is a .

Conjugate priors Likelihood : yttcy ) ] hcxi I ,yi= ply tly ) acy exp ) pcyiy = - hcyhly Conjugate = hly ) ,hc prior : hiyiexplittcy ads ] pcyii := 9 , ,d ( d ) . ) ) = - := - acqliz ads ) ( y hcy ) exp lytd tly . acyl ) ) - = , , g) exploit In I Joint : . ~ - ) explyt ( " Ii ) - acy ) ( 1+92 ) ) ) ah ) . acyl In acts ) explants ) ] at - , - - pcy.at . ⇒ ) ] I .

Conjugate priors Joint : %) explants hcy ply ,y AH ) ] I 1 pcy.nl ) tk pcyi pcyiy lply - , , = = + , , , I. dzt 1 = Marginal : |dy = hcy , [ acE1 ] aol exp Pay ) - i. 5 nominator by from marginal Com compute Posterior ! pcyl %) , g) ply ) ) = = . family Posterior here Cnjugaey sand prior : as → ) ] J .

Conditionally Families Conjugate Exponential " ) pcyitly qcy 147 II Choose be pcy to to and choose ) conjugate pcy = hlytti pig ,7ly ply tcy " hau 19 to ) the family ) safe as [ ) ] yt ) , } ) exp acy - [ YTI acy )% ) ] pcnli hc ah ) exp y ) = - - , , [ log / y ) ] ) E # qcy ;qn step qlziots a exp 7. - ; , |atexp| , [ log 7. q ) ] ) # qcy ;qn pig , # qiy ;¢n)[ y ]T th " ) qct leg loghcyitit ;¢ ,t ) 't = -9 . terms not that do depend 7 on

Conditionally Families Conjugate Exponential " ) pcyitly qcy 147 II Choose be pcy to to and choose ) conjugate pcy = hlyiti pig ply tcy " hau 19 to ) the family ) safe as 1 y Etqf qcy # [ ) ] yt ) 2- exp acy a ) - , , [ YTI acy )% ) ) pcnld hc ah ) exp y ) = - - , - ,7,y ) ) ) [ log M / ;¢Y ) step exp : . = ; lay . .gg/leyplyn..y ) ] ) exp / Eqn ) ( 1+7 ltlyiti )+d " ) yt ( Ftgn , ) . ) leg qcy :p acy t + , # = ... . 1 No 0/1=7+92 deep 't [ fly ,7 ) ) t I 9 ? Ftqh ;¢ = any ,

Gaussian Mixture Model Next no Lecture : , So ) Normal Inu Wishart ( Ew 7 µu Tom ~ vo . , , , So } I n ) :{ Discrete ( Mo ,%,vo , zn ~ n , ynlznh Normal Gun ,[ ) In } 0 : :{ Mu ~ u , . step E : - ftp.uogltloyknl .tk/hyhmu=o#yu=fhnErnuynNu=&rnu expflognu ) rnu a yawn ] ) - #qµu .eu the '( ( { lyu µu ) - ) . - step M : In fo + Nh Do Nu Vu + v. = = = - ...