Ruiyany shen ; , , Lecture others follow will Notes 3 : - - PowerPoint PPT Presentation

SLIDE 1 Lecture 9 : Variation EM Scribes ; Elizabeth , Yao shen , Ruiyany Notes : Lecture 3 is up ,

• thers

will follow

SLIDE 2 Algorithm : Generalized Expectation Maximization Initialize ; O Define : LIQH i. Ezra , ... , ,[logPgYf÷y9I s log pcy :o) . Repeat until £ ( O , g) change below threshold 1 , Expectation Step ( Somewhat

• f

a misnomer ) y = anymore£10 , y ) r 2 . Maximization Step O = argmax £10 ,H

SLIDE 3 Today : Variatimal Inference Idea : Approximate Posterior Distribution pc7 , Oly ) I qc 7,0 ;

• f )

Htnd to ' calculate "( chosen to be more tractable Objective : Minimise KL divergence by Maximizing a lower bond £19,01 ) # = argmin KL ( qczo ;g ) 11 p( 7,01g ) ) ¢ = arggnax £17,01 ) ← Analogous to EM

SLIDE 4 Variation Inference : Evidence Lower Bound ( ELBO) EM : Lower bound

• n

leg likelihood £1014 ) is Egcz ;o, ,[ log

.gg#j9ofp

's :o) paly :o) = log ply :o)

• Kllqttig

) Hpctiy :o) ) s logpiy ;D ELBO : Lower bound

• n

the leg marginal likelihood Add Prior : ~ pc O ;D ) z ~ pc 710 ) y ~ ply 17,0 ) ' 9) Expectation

• ver

£0,4

) i= Egg . ;µ[

losPqYzl.to@oaan.a

• =

log pcy :D ) . KL(go.o.gs//p(z,Oiy ;D )) } by ply :D

SLIDE 5 Variation Expectation Maximization ELBO : Lidia ) = # , , a. ,¢ , [ by PYI.LI# ] % Problem : mud hander to compute expectation Writ . 0+7 Idea : Use factorized approximation g Replace point estimate pc 7,91 y ;D ) = 9C 7 ; 47 ) 9105401 with diet qlo;g9 "( Analogous to EM Note : p( 7,01yd ) =/ pizly ;D ) Pc Oly ;D ) z # Oly

SLIDE 6 Algorithm : Variation Expectation Maximization Initialize ; 40 pcy ,7 , @ 9) Define :L ( I. lot ,d° ) = Et qc a ,qµ[ loggia ) slog ply :D . Repeat until £1,9 ,

• 017,100 )

change below threshold 1 , Expectation Step 9 is n±t

• ptimized

a lot = argyyax LC 7.

0,749

Analogous to EM step for y 2 . Maximization Step 010 = angmax £17,017,100 ) Analogous to EM go step for O

SLIDE 7 E . step : Solve foroft

ftp.tl

' ' . 99 '

ftp.euamq.am/lgPgYYaYgi@og

) no dep

• n

7 . i= ¥ , # ah . :¢7q( ago)| leg pcyitio ) +

lgp%l

/

y no deep an . leg 917 ; 47 )

• ly

glom

= = off , # qn . :p 's # go ;qo ) ( log piynlt ))

• by qa :¢3)

= Solution : log qc 7:47 ± Eg , @ go ) [ log pcy , 710 ) ] t canst (

• ptimal

) qcz ;¢a , & Exp [ Etqio;qo)[log ply ,7lo )] )

SLIDE 8 Algorithm : Variation Expectation Maximization Initialize ; 40 Define :L ( d. lot ,d° ) = Et qu ,qµ[

logphfj.IT#)sbgpiy

:D . Repeat until £1,9 ,

• 017,010 )

change below threshold 1 , Expectation Step qiz ; 92 ) L exp ( Eqio ;q9[log pcyitlo ) ] ) 2. Maximization Step to

Imagine

"

theme

tnmahweaaj . q( O ; 0101 s exp

I

# get ;qt)[ log

pcyitiol

))

SLIDE 9 Intermezzo : Exponential Families An exponential family distribution has the form Only depends

• n

× Depends

• n

/ Only depends I y and x an n d pcxly ) = hk ) exp [ yttcxi

• aey ) }

n E leg normalize

(

[ KEY

"EII" its

Base measure ( Canting , Lebesguc ) ( only depends

• n

× )

SLIDE 10 Example ; Uniuaniate Gaussian pcxiy ) = hcx ) exp[ yttcx )

• acy

) ] =

#2)

' " expf ; k¥2 ] Panama

µ

Dependent = (zntzj"expft(x2 . z×µ+µ2)/ . ] tcx ) = ( × , ×2 ) n = ( µ1r2 ,

• 1/262

) acy ) = µ2 1262 + log 6 hcxl = 1 16

SLIDE 11 Properties

• f

Exponential Families p ( x 1 y ) = hcx ) exp [ yttcx )

• acy

) ] Moments . . Derivatives

• f

Log Normalizer

/

dx pixiy ) = 1 → exp lacy ) ) = |d× hkiexpfyttki

again

= gqµg/d× has exp ( yttcx ) )) / ax taxi hcxiexplytfki ) P '× ' 7) =

_.--

= expel

• alnl )

I / ax hki exp 1 yitk

#

= |d× tax ) ( hkieaplyttki . acy )))= Epcxiy ,[ tix ) ]

SLIDE 12 Properties

• f

Exponential Families pcxiy ) = hcx ) exp [ yttcx )

• acy

) ]

• Moments

are computable from derivatives

• f

acy ) a dqn a 'Y = # pain ) [ tkih ]

• When

tk ) are linearly independent an exponential family is known as minimal

• Far

any minimal family acy ) is convex and µ := Epa , , ,[ tcxi ] c→ y ( there is a i. to . 1 mapping from n to # pcxiy )[ Hxl ] )

SLIDE 13 Conjugate priors Likelihood :

pcyiy

) = hcxi exp I

yttcy

)

• acy

) ] Conjugate prior : pcyii ) = hiyiexplittcy )

• ads ]

9 := ( d , ,d . ) = hcy) exp lytd ,

• acqliz
• ads )

tly )

:=

( y , . acyl ) Joint : In I . ~

• ply

,yi=

hcyhly

) explyt (

tly

) " Ii )

• acy ) ( 1+92 )

. ah ) ) = hly),hc g) exploit ,

• acyl In
• acts ) explants
• at

) ] . pcy.at ⇒ ) ] I .

SLIDE 14 Conjugate priors Joint :

ply ,y

) =

hcy

, , pcyi %) explants

• AH ) ]

I , = 1 , + tk , I. = dzt 1 Marginal : Pay ) i.

|dy

pcy.nl

= hcy , exp [ acE1

• aol

] 5 Com compute marginal from by nominator Posterior !

pcyiy

) = ply , g)

lply

) = pcyl %) . Cnjugaey : Posterior here sand family as prior → ) ] J .

SLIDE 15 Conditionally Conjugate Exponential Families Choose

pcy

II

" ) to be conjugate to

pcyitly

) and choose

qcy 147

) to hau the safe family as

pcy

19 " )

ply ,7ly

) =hlytti exp [ yt

tcy

, } )

• acy

) ] pcnli ) = hc y ) exp [ YTI ,

• acy )%
• ah

) ] E

• step

; qlziots a exp / # qcy ;qn , [ log

pig

, 7. y ) ])

|atexp|

# qcy ;qn , [ log pig ,

• 7. q ) ])

leg

qct

;¢ " ) = loghcyitit # qiy ;¢n)[y ]T th ,t ) 't .

• 9

terms that do not depend

• n

7

SLIDE 16 Conditionally Conjugate Exponential Families Choose

pcy

II

" ) to be conjugate to

pcyitly

) and choose

qcy 147

) to hau the safe family as

pcy

19 " )

ply

, 2- 1 y ) =hlyiti exp [ yt

tcy

, a )

• acy

) ] pcnld ) = hc y ) exp [ YTI ,

• acy )%
• ah

) )

• M

. step :

qcy

;¢Y ) = exp /

Etqf

;

#

[ log

pig

,7,y

) ) ) lay exp / Eqn . .gg/leyplyn..y ) ] ) leg qcy :p " ) = yt ( Ftgn . , # ltlyiti )+d , ) + acy ) ( 1+7 . ) t ... 1 No deep 9 ? = Ftqh ;¢ 't [ fly ,7 )) t I , 0/1=7+92 any

SLIDE 17 Next Lecture : Gaussian Mixture Model µu , Ew ~ Normal Inu Wishart (

no

, 7 . , vo , So ) zn ~ Discrete ( n ) I :{ n , Mo ,%,vo , So } ynlznh ~ Normal Gun ,[ u ) : :{ Mu , In } E . step : a

expflognu

• ftp.uogltloyknl

) rnu the '( yawn ] )

• #qµu .eu

) ( { lyu

• µu )

. M

• step

:

Tom

.tk/hyhmu=o#yu=fhnErnuynNu=&rnu

In = Do + Nu Vu = v. + Nh fo =

• ...