Lecture Variational 13 Inference Panini Kaushal Scribes : - - - PowerPoint PPT Presentation

SLIDE 1 Lecture 13 Variational Inference Scribes : Kaushal Panini Niklas Smedeuranh

• Margulies

SLIDE 2 Variational Inference Idea , Approximate posterior by maximizing a variational lower bound , ply ? Ptt , fly ) " 9) = *

go.o.cn/eogPgY:I;:T

)

• 7. O

ly =

log

pig ) t Eg

• ,

llogpqlcz.co#)

=

leg

pays

• KL (

9170101711

party ) ) q s

log

pay , Maximizing L lol ) is the Same as minimizing KL

SLIDE 3 Variational Inference : Interpretation as Regularization Equivalent Interpretation : Regularized maximum likelihood Llp ) = Eq , .gg , I leg MYTHOS ) pcy.t.os-pcyiz.ospcz.co ) I

917,01

lol =Ego . . o ,

I log

pcyiz.at t leg

]

=Ego , o ,

I log

pcyiz.io/-kL(qc7Olos//pl7.os

) I I " male log likelihood as make sure got , Oslo ) large as possible " is similar to prior "

SLIDE 4 Intuition : Minimizing KL divergences Ply , × , ,X . ) = PCYIX , ,×z ) pcx , ,xz ) Z

/

qcx ) = Norm ( x

;µ

, 2)

g)

qc ,× , ,×z , ÷ q(× , )q( × . , qcx , ) ÷ Norm /× , ;µ , ,6 ? ) qkz ) := Normkn ;µz,6i )

LC

aim ) :-.

EaaflogPlgYIlI@ply.x

, ,×z ) = pcyipix , ,×ziy ) =

leg

ply )

• KL ( q(

x , ,x , ) Hpcx , ,×zly ) ) Intuition : KL divergence under

approximates

variance

SLIDE 5 Intuition : Minimizing KL divergences z

g

P( Yixi ,×z ) = pcylx , ,×up( 1- , ,x > )

• a. (

x ) = Norm ( x ; p , [ )

g)

pagan qc ,× , ,×z , ÷ qk , , qkz ) Propagate qlx , ) :-. Norm ( × , ;µ , ,6 , ) 91×21 :-. µorm( Xz ;µz , G) * KL( plx . ,xzly ) 119k , ,xz )) 9k , ,×z I KL( qlx , ,xz ) Hpcx . ,xzly ) ) = |d× , dx . qk , ,xz 1 log

• plx

, ,Xz1y )

ligng

. q leg 'f =

• lipm ,

a log ph = as Intuition : q ( × , ,xe ) →

• whenever

PC x , ,xz ly )→0

SLIDE 6

Algorithm :

Variational Expectation Maximization Define : qcz , O ; g) = gets Of 't ) g C O 's 90 )

Objective

:L

Clot , do ) = Et qiao ,

flog

'

"q¥to

I slog ply ,

• Repeat

until Ileft , 00 ) converges ( change smaller than some threshold ) 1 . Expectation Step lot = argy.mx L ( oligo) Analogous to EM step for j z . Maximization Step Updates distribution 010 = angurax £ ( 97,010 ) glo ; go ) instead

• f

40 point estimate O

SLIDE 7

Example

: Gaussian Mixture ( Simplified )

Generative

Model

f !

I

si is:/ huh , d ~ Norm ( pro ,d , So ) 2- n

• Discrete

( YK , . . . ,Yk ) ynl7n=h n Norml pea ,

EI

)

SLIDE 8 Model Selection

µ

Margined likelihood " Average Evidence livelihood " £ I log ply )

log

pigs = log ldtdoply.z.io ) K=2

• I

I Number

• f

Clusters Intuition : Can avoid

• ver

fitting by keeping model with highest £

SLIDE 9 Variational Expectation Maximization : Updates 119707 ) =

Eqcziopsqcy

,

lbs

g%YgtYg¢n

, ] = #gczioftiqcylpn ) ↳ Ply ,Z , 7 )) ← depends an 47 and 47

• Egypt

, I log

917145

)

• Eqcyiqn

, flog

9411091

)

depends

• n

47 depends

• n

47 E

• step

:

• ftp.E.ge?,q.,fEqcy,qn,llogpcy.t.y7)-bgqczipl

)

9171472 exp ( Eqcyipyllogpcyit.nl ) M

• step

:

• ¥ y

Egm

,

LEG,

y

⇒ I log pcy.tn ))

• by

94144

)

genial ) a exp LEqczigzilloyplyit.nl))

SLIDE 10 Intermezzo : Functional Derivatives Idea : Compute derivative

• f

an integral win . 't . a function ply , 7. y ) ° = 8%7 , fat dy amain ) (log gifs

)

at ply , 7. y ) = / dy gigs (log gig

)

• ldy

acts

ang

!%,

drop integral

• ver

2- , take derivative

• f

integrand =

• log

get , t Egg , flog plyit.nl

• log

9in ) )

• I

leg 9th = Egon , flog ply , 7. y ) ) t const depends

• n

7- ensures henna lineation

SLIDE 11 Variational Expectation Maximization : Updates 1197071 =

Eqcziopsqcy

,

lbs

g%YofYg¢n

, ] = Ege ,

Ileg ply

,Z , y )) ← depends an

147917197

) 47 and 47

• Etgcziqz

, I log

9171ft

) )

• Eqcyiqn

, flog

9411091

)

depends

• n

47 depends

• n

47 E

• step

: 917197 ) x exp f Egg , µ , I log ply , 2,77 )) M

• step

: qcy 147 ) L exp I Eg # 147 , flog ply .IM ) ) )

SLIDE 12 Gaussian Mixture : Derivation

• f

Updates Idea : Exploit Exponential Families

log

pcy.z.ms = log pi y l 2- ,

4)

t log

pcztrf

) + log pin ) 9 9 9 All

• f

these are exponential family log pcyiz . y

'd

= E { y I[zn=h ) tcyn )

• acyilIEti-hli.bg

hey, h log pctlyt ) = ? { ytuI[zn=h ) leg populate ) =D ! .FR

• D?

. acyl ) t log hints leg

pcyt 197

) =

It

Ty

' t log hcyz )

SLIDE 13 Gaussian Mixture : Derivation

• f

Updates E

• step

: Collect all terms that depend

• n

2- n

leg

qcz.nl

lot ) = Egon , µ , [ log

pcyn.tn

. n ) ] t . .

• = In Egon ,

µ , [ yhb ) I 7n=h ) Ayn ) t Egon , µ , [ME

)

I[7n=h ) t . . . & 9 Need expected

values

E (y and

Edyta

)

SLIDE 14 Gaussian Mixture : Derivation

• f

Updates M

• step

: Collect all terms that depend

• n

y z ht

leg

91414

) = Ego ,,µz , [ log

psyn.tn

. n ) ) t

• .
• YZ

=

can :(

Kiyl+

. .

• YZ

my ¢ h Qu , ,

• log

an! 9% =

{

y ! " (f

In

Em,

litton ) tan )

tin !

Need expected value t § alga )

#

Eq , , , lIftn=h))) t D !! ) Eave , [ Il7n=h ) )

• nbu.ee#

SLIDE 15 Gaussian Mixture : Variational EM Objective : Variational Evidence Lower Bound ( ELBO ) 1197071 =

Eqcziopsqcy

,

lbs g%Y¥Yg¢n

, ] Repeat until £10744 ) converges I . Expectation Step ; Update get ) (keeping qcy ) fixed ) exp L Each , [ log

plynitih

17 ) ) ) = Eg , ,lIL7n=hl ) Th = f exp

#

an , [ log pcyn.tn

• ely ) )

2 . Maximization Step : Update

qcy

) (keeping q CZ ) fixed )

¢hI= Nutty

OIL

?

.

• {

loiitisni

+9? . ¢ ?

!

• Nuts ?