Lecture Variational 13 Inference Panini Kaushal Scribes : - - PowerPoint PPT Presentation

Lecture Variational 13 Inference Panini Kaushal Scribes : - Margulies Smedeuranh Niklas

Variational Inference Approximate by posterior Idea maximizing , bound lower Ptt variational fly ply ? ) a , , ) " 9) * = llogpqlcz.co#) 7. O ly go.o.cn/eogPgY:I;:T log pig ) = Eg t 9170101711 o , ) ) KL ( leg party pays = - q L Maximizing lol ) the log is s pay , KL minimizing Same as

Variational Inference Regularization Interpretation as : Equivalent = Ego = Ego Regularized likelihood Interpretation maximum : 917,01 I leg MYTHOS ) Llp ) pcy.t.os-pcyiz.ospcz.co ) Eq , = .gg lol , I ] I log pcyiz.io/-kL(qc7Olos//pl7.os leg pcyiz.at t . o . , ) I log , o , " I I likelihood log make , Oslo ) male got as sure large " possible " similar is to prior as

: Minimizing Intuition KL divergences pcx PCYIX , ,×z ) Ply . ) ) × ,X ,xz / = , , , g) Z ;µ 2) Norm ( qcx ) x = , , ,×z qc ,× , q(× )q( , × ÷ . , /× , ,6 ? ) Norm , ) qcx ;µ ÷ , Normkn ;µz,6i ) qkz ) := approximates LC EaaflogPlgYIlI@ply.x ) aim :-. KL ( q( ) ,×z ) ) , leg Hpcx , ,×zly ply ) ,×ziy ) ,x pcyipix , ) x = = - , , Intuition KL divergence : under variance

: Minimizing Intuition KL divergences 1- > ) pcylx ,×up( P( ) , ,x Yixi ,×z g = , g) z [ ) p ( Norm a. ( ) x x ; = , , ,×z qk qkz qc ,× , ) , , ÷ pagan Norm ( Propagate , ) ,6 × ;µ qlx , ) :-. , , µorm( , G) 91×21 ;µz Xz :-. ,xz ) ) KL( plx 119k . ,xzly ) * , = |d× 9k I ,×z ) ) , KL( 1 log Hpcx dx - ) qlx ,xz ,xzly qk ,xz , ,Xz1y ) . . plx , , , leg 'f ligng log ph lipm , a q o as = = . Intuition whenever ( × , ,xz : q ,xe ) ly )→0 → o PC x ,

Variational Algorithm : Expectation Maximization g C O 's 90 ) ; g) Define 't , O gets ) qcz : Of = "q¥to :L ' I slog , do ) flog Clot Et Objective ply , = qiao , - , 00 ) I left ( change Repeat until smaller threshold ) converges than some Expectation Step 1 . ( oligo ) lot L Analogous to EM argy.mx = step for j Step Maximization z . Updates distribution £ ( 97,010 ) 010 ; go ) angurax instead of = glo 40 estimate O point

( Simplified ) Gaussian Example Mixture : f ! si is :/ I Generative Model Norm ( pro ,d huh ) , d So ~ , EI . ,Yk ) ( YK Discrete 2- - n . . , Norml ) ynl7n=h pea n ,

" Margined Average likelihood Model Selection µ " livelihood Evidence log ldtdoply.z.io ) log £ log pigs I ply ) = I • K=2 I Clusters of Number by keeping Can Intuition model fitting avoid : over highest £ with

Maximization Variational Updates Expectation : lbs , ] - Egypt 119707 ) = , 917145 Eqcyiqn g%YgtYg¢n Eqcziopsqcy 9411091 # gczioftiqcylpn ) ↳ , 7 ) ) depends ,Z Ply ← = an 47 47 and ) ) , flog , I log - 47 depends 47 depends on on ) E LEG , 94144 y step o - : - - exp ( Eqcyipyllogpcyit.nl ftp.E.ge?,q.,fEqcy,qn,llogpcy.t.y7)-bgqczipl ) 9171472 - ¥ y - by M Egm ) ) ) ⇒ I log step pcy.tn : o - - , exp LEqczigzilloyplyit.nl ) ) ) genial a

Derivatives Functional Intermezzo : function derivative an integral Idea Compute of . 't win a : . , fat dy ang y ) ply , 7. ) 8%7 ) ( log ° = gifs amain at = / dy ldy y ) ply , 7. ) gigs ( log !% , acts gig - derivative of integrand take integral drop 2- over , 9in ) ) Egg , flog log log plyit.nl get I t = , - - - , 7. y ) ) Egon , flog leg 9th const ply t = lineation 7- depends ensures henna on

Maximization Variational Updates Expectation : lbs , ] 1197071 = 9171ft , Eqcyiqn g%YofYg¢n Eqcziopsqcy I leg ply Etgcziqz , y ) ) 9411091 depends ,Z Ege ← = an ) 147917197 , 47 47 and ) ) ) , flog , I log - - 47 depends 47 depends on on , 2,77 ) ) exp f Egg , , I log E 917197 x ply ) step : - µ ) ) ) exp I Eg # 147 , flog M 147 ) step qcy L ply .IM : -

Gaussian Mixture Derivation Updates of : Idea Exploit Families Exponential : = log pcztrf 4) t log log + log pin ) 'd pi y pcy.z.ms l 2- ) , 9 9 9 family All exponential these of are acyilIEti-hli.bg E { y log tcyn ) . y I[zn=h pcyiz ) hey , = - h = ? { ytuI[zn=h pctlyt log ) ) =D ! .FR log D ? acyl hints populate ) leg ) t - . pcyt 197 It log Ty hcyz ' ) ) leg t =

Gaussian Mixture Derivation Updates of : E Collect all that terms depend 2- step : - on n n ) ] , [ log qcz.nl pcyn.tn leg lot ) Egon , t = . . - . µ = In Egon , , [ ME , [ yhb ) ) Ayn ) I[7n=h ) I 7n=h ) t t Egon , . . . µ µ values Edyta 9 & E ( y ) Need expected and

Gaussian Mixture Derivation Updates of : M Kiyl+ Collect all that terms depend step : 91414 Ego , ,µz y - an ! 9% on n ) ) , [ log psyn.tn ht tin ! leg z ) t = - . - . can :( YZ = . . - my YZ Qu ¢ h , , - " ( f litton ) tan ) { In y ! log Em , = # t § alga t D ! ! ) lIftn=h ) ) ) Need Eq , expected value ) , , ) ) onbu.ee# Eave , [ Il7n=h

Gaussian Mixture Variational EM : ( ELBO ) Bound Variational Lower Objective Evidence : Eqcziopsqcy g%Y¥Yg¢n lbs , ] 1197071 = , Repeat £10744 ) until converges ¢hI= Nutty ( keeping ) Expectation Update fixed Step qcy ) I get ) ; . # , [ log plynitih 17 ) ) ) L Each exp , ,lIL7n=hl ) Th Eg = = - ely ) ) , [ log f exp pcyn.tn an - ( keeping qcy CZ ) Maximization Step Update fixed ) ) 2 q : . - { loiitisni Nuts ? +9 ? OIL ? ¢ ? ! - - - . .