Jay DeYoung : Hamnett Donald Families Exponential An family - PowerPoint PPT Presentation

Lecture Maximization Expectation It : Scribes Jay DeYoung : Hamnett Donald

Families Exponential An family has form exponential distribution the Depends Only Only depends depend x on on I I and y x an m d hcx ) I YT aey ) ) text y ) pcx I exp = - n E leg ÷ : normalizer ( " , Lebesgue ) ( Canting Base measure ( only depends ) X an

: plx Conjugate priors Likelihood : , ] hey petty tix ) acy exp hlx ) = - lqttcx hey i Conjugate prior : := D , .dz ) ( d is exp IT ' I " act hip payed tch ) ) = - ( y tip - acyl ) , Joint hey ) explyt ( i 9 , ) , yl rt Da ) ) aim ( aids ) i - - - exploit - augier acts ) explant ) acts ] ) hey , - , = - # . I IT ) x ) p C pcyl 7 - aids ) Fa -

Conjugate priors Joint ? : hex 51 ) ] I explored ) d alt fix , pint pcyixi : pox hcx - = = + plx.nl , , Ja Dat I = fdypcx.nl Marginal act ] I ' aol exp , ) = - ptx.gs/plx)=pcy1Qttcxs is p normalizer from marginal leg Com compute Posterior ! Datt ) = , J family Posterior here Conjugacy saz prior : as - ) ) Fa aid -

Homework Gibbs Sampling : - likelihood conjugate Idea Ensure that is to prion : conjugate prior likelihood far h cluster posterior - - ) I Mpcynlfihduh.su/plMh,Eul9h :tn=h3 I y µ ) In , Eh , 2- Pl Mk Iim = n M ) lnizi.az/0lYnl7n=h - - likelihood marginal . I this Derive I 9h the ( y , 7 ) ) , Ch p ( Mu homework t = in

Homework Gibbs ( yn Sampling ynynt ) : . , I fan MVN pcynlzi-h.ly ' in - acyu ) ] ) exp I ) hlyu = hlynlexplyiitlyn - Du algal ) ) ) pi mu ya = I T Lock Wikipedia = II plluu.eu/nl7p/ynlzn=h,,u.IY up " " on El ply , , tlyn.tn ) I ? hlyn ) 11h44 = y ( Du ,tfHynlIEK=h ) ) exp I Gynt 4 ) ) [ acyu ) ( Due E. III. - . h #

Derivatives Moments Normalizer Log of : hcx ) I YT aey ) ) text ) pcx ly exp = - |dx - lax hcxiexpfytki ) Lacy , ) 1 pcxiy exp , → = - = Iq flog lax ( yttcxi ) ) has Iya 'T exp tix ) ) fax ( YT tix ) hcxi 1- exp chain = exp Lacy Rule , ] First I , y , f 't Epix Moment pcxly ) dx y = =

Parameters Moments Natural and hcx ) I YT aey ) ) text ) pcxly exp = - Moments from of computable derivatives acyl - are " than ] at Y d , [ IE ' = pcxiy Fyn linearly to independent exponential When - , are an known family minimal is as family Far minimal and acy ) is convex any - text ) ftp.cx.y , I → y µ i = Etpcxiyjltlxl ) ) from there ( to to I 1. mapping n is a -

Moments Parameters Natural and Distribution Normal Example ; ) t ( ( ,xZ Plxlpe ⇒ ⇒ Efx ] '/y ) µ x x -24 = = , ' ] 62+15=174,24 ( pile ? EL 1/262 ) y x -_ - , Distribution Bernoulli Example ; ' × " - ) µ ) C I yr - = - m ) I leg log exp t I x = tix , acn 9 , Efx ] ⇐ > µ

Maximum Likelihood Estimation Naim likelihood parameters y* The maximum sufficient statistics determined by the are observed of the data It ] text Solve E : : y = I pH 'm ) n Gaussian ( µ 6 ) Example N Xu nil : n → . , , Eia 'Ei ) I tix , = , , ? Yu I ' µ* XI µ* 6*2 ¥ xn = - =

Maximization Expectation * .gg?gxlogpCy11u.E,n * n.ME ) Objective : * a = Repeat trnhynynt ' unchanged ) convergence until ( objective For 1 Ni 7 in n . . . - / dznpcznlyn.io , ) ) IlZn=h Ilan - h ] El ) yuh = - ie he Points # cluster in For W Ki 2. in I , . . . N N Empirical I Nh E t Mu yn ruh yuh :-. = = µ h ' Mean h h 't ! & - pryuht Ih Empirical = , Covariance h Nh IN cluster Fraction nu = in

Maximization Expectation Example : Iteration O :

Maximization Expectation Example : Iteration 1 :

Maximization Expectation Example : 2 Iteration :

Maximization Expectation Example : 3 Iteration :

Maximization Expectation Example : 4 Iteration :

Maximization Expectation Example : 5 Iteration :

Maximization Expectation Example : 6 Iteration :

Maximization Expectation * .gg?gxlogpCy11u.E,n * n.ME ) Objective : * a = Repeat trnhynynt ' unchanged ) convergence until ( objective For 1 Ni 7 in n . . . - / dznpcznlyn.io , ) ) IlZn=h Ilan - h ] El ) yuh = - ie he Points # cluster in For W Ki 2. in I , . . . N N Empirical I Nh E t Mu yn ruh yuh :-. = = µ h ' Mean h h 't ! & - pryuht Ih Empirical = , Covariance h Nh IN cluster Fraction nu = in

6mm Maximum Likelihood Estimation in " observed " Easy Estimate for t : n = & - § acy Am log fly 2- ply o ) 77 ) , , ? Problem pcyly Need marginalize to : over Z ) = ldtplu.tt/y1=n7!/dznpCyn.7nly7oQylogpCy1yl=ptqy,fqnI ldznexplyitly , aint - , throws Integral worms in spanner

Jensen Inequality Intermezzo : 's , Functions Convex Area above fix f- ( tx t ) ) fkn Ci . • t xz curve is a - , . . , set convex • fix t ) flat t s It fix , ) ' t . g - X Xz , Functions Concave Area below •,←¥f f- ( fix flat t ) tx Ci ) t ve is a xz cu - - , set convex - fix , , fix t ) flat t 7 It , ) + s - Xz X , Random Variables Corrolary : t.ci#xnl:Efii.i:iit::::.

Leibler Kullback Divergence Intermezzo lax : - - I 9¥ KL( qcxsll MIX ) ) Measures how by much DX 941 : ' " n deviates from MIN , ,× a Properties ( Positive femi ) ) - definite ) ( a call I KL mix 3 o . " 9¥ KL ( g MIN ) ⇐ galley "g , ) E. kill leg guy - = = , ± log ( E⇐g*l" gift ) lil log o - = - KL ( q C x ) 11171×1 ) 91×1 Mk ) 2 o = a = . I 9kt lax 171×7 , log 94¥ leg dxqix Mix → , o = , ,

Lowen Likelihoods Bounds on Use to Bound Lower Jensen define Idea inequality 's : Marginal fax fax Gaussian 4¥ , 2- goal E ) aix six , = i - = - lost slog It # . I I , 14 L . tog 't E. * : - - - ⇐ " [ Lower bound 7 boy on Mixture Model ; y , PgY¥ = Id 't I 2- I O ) :O ) pig . 't at ;o , act pig ; = = , 7 ;D ) pig log , I log Ez £10,81 , ) pay ol s ; :-. .mg

Algorithm Generalized Expectation Maximization : Egj Llap ) Objective : is , 9175g ) O ; Initialize £10 Repeat , y ) until unchanged : .gg/loyPlY'tt-9/slogpcy;o7 Expectation Step 1 . I ( O y ) y = angngax , Step Maximization 2 . LIO O r ) anymore = ,

KL Bound Lower divergence vs - ;o ) pcyit = :O 'P P 's 't 's :o) Eagan , lloypggjt.sc ] £10,21 = + leg Plaything ) #z~q , ;y)|log :o) pig = on £ Kttdiv depend does not rewrite as log an ;n|log9pYftTo , ] # :o) = pig - ← KL ( ;o ) ) log H :O) paly ply qhsy ) - = a \ Does depend Depends not y on on y Maximizing £ ( O ,y ) Implication equivalent : y is wrt O ) ) to ( KL 11 minimizing ; g) pcttly 917 ; o .

Algorithm Generalized Expectation Maximization : Egj Lto Objective Hi : , . , 's y ) 917 A qc.zi-hsyl-ku-pftn-hlyn.cl : Initialize £10 Repeat , y ) until unchanged : .gg/loyPlYttt-9/slogpcy;o7 IT Expectation Step i , v = argginklfqlt.rs//pCzty d- I O r ) y = angngax , , Step Maximization 2 . \ LIO O r ) anymore = ,

for Families Bound Exponential Lower ] ) Hair Ea , - - , ! tlyn MT § Macy ) leg n ) , 7- ) ply ly 't = - , .gs/logPgY.?IT ) ) , flag # ¥114,8 ) Ig piyitiy = a , , ;g = € - Nj acy ) ' # I tlynitnl ) Egan , [ lay solve this Can as as we can stats sufficient expected compute .

Algorithm Generalized Expectation Maximization : Egj Lto Objective .hr : . , 9175g ) A ; Initialize £10 Repeat , y ) until unchanged : .gg/loyPlY'tt-9/slogpcy;o7 Expectation Step 1 . Computes expected I ( O 8 ) y ang mate = sufficient statistics , r Step Maximization 2 . Maximizes O given L( O 0,8 ) anymore = computed statistics