Sampling Gibbs : Maximization Expectation Scribes Jered - - PowerPoint PPT Presentation

SLIDE 1 Lecture 9 : Gibbs Sampling Expectation Maximization Scribes : Jered McInerney Xiongyi 2- hang

SLIDE 2 Gibbs Sampling : Gaussian Mixture ( Homework 2) Grapical Model Generative Model Mh , Eh ~ pipe , I ) let , . . . , K b " In ~ Discretely , . . . it ) n = ' ,

• .

. , N g a ×nlZn=h n Norm fun , fu ) he Gibbs Sampler Updates Local variables 2- u n pl Fnl Yu , µ , I ) n

• I

, . . . ,

N

Global variables 14h , Eun plpele.ch/yiin,Ziix . ) 6--1 ,

• , K

SLIDE 3 Gibbs Sampling : Conditional Independence Local Variables : Kith I ym-tapm.eu 114,2 N ply .im , 71in 114 , E ) = M pcyn ,7n1µ , I ) h = I plyn ,Zn=h I µ , @ )

pl7n=h1yn

, µ , @ ) =

• 9

Can compute I § pcyu ,7u=l I µ , I ) all updates in 044K )

Global

Variables : k Mh .FI/Ue-th.Ie*u/zplyiiu./u,E12-iin)--Mpyuu.Eu)MNonm/ynspu , h

• I

n : 2- n=h

SLIDE 4 Gibbs Sampling : Global Update Idea : Ensure that prion is conjugate to likelihood likelihood conjugate prior posterior far cluster h

• I

Mpcynlzih.in , I )

plmh.su/pCMk,EulyiiN

, 2- n µ ) = In :tn=h3 M

• lnizi.az

P' but h )

• marginal

likelihood

SLIDE 5 Exponential Families An exponential family distribution has the form Only depends

• n

x Depends

• n

I Only depend I y and x an m d pcx I y ) = hcx ) exp I YT text

• aey ) )

n E leg normalizer

(

÷ :

" Base measure ( Canting , Lebesgue ) ( only depends an X )

SLIDE 6 Example ; Univariate Gaussian pcxly ) = hcx ) exp I YT text

• aey ) )

= "

expf.tk#y

Hmmm "

↳

Dependent = " expf.is/x2-zqutM)/ga ] text = ( X , x ' ) y = ( fuld ,

• 1/262

) ally ) = /u2/26 ' t log 6 hcxl = I ITTF

SLIDE 7 Conjugate priors Likelihood :

petty

) = hey exp

lqttcx

)

• acy

, ] Conjugate prior : i D is ( d , .dz ) payed ) = hip exp IT " tch )

• act

' I tip

:=

( y ,

• acyl )

Joint : plx , yl i

• hey

hey) explyt (

tix

) i 9 , )

• aim (

rt Da )

• aids

) =

hlx

) hey, exploit ,

• augier
• acts ) explant )
• acts ]

. I # pcyl IT ) p C x ) 7

• aids )

Fa

SLIDE 8 Conjugate priors Joint :

plx.nl

=

hex

?

pint 51 explored )

• alt

) ] I , = d , + fix , I a = Dat I Marginal :pox ) is

fdypcx.nl

=

hcx

, exp I act '

• aol

] g Com compute marginal from leg normalizer Posterior !

pcyix

) =

ptx.gs/plx)=pcy1Dttcxs

) J Conjugacy : Posterior here saz family as prior

• aids )

Fa

SLIDE 9 Gibbs Sampling : Homework

• Idea

: Ensure that prion is conjugate to likelihood likelihood conjugate prior posterior far cluster h

• I

Mpcynlfihduh.su/plMh,Eul9h

) Pl Mk , Eh I y Iim , 2- n µ ) = In :tn=h3 M

• lnizi.az/0lYnl7n=h

)

• .

marginal likelihood Derive this I in homework = p ( Mu , Ch I 9h t the ( y , 7 ) )

SLIDE 10 Moments : Derivatives

• f

Log Normalizer pcx ly ) = hcx ) exp I YT text

• aey ) )

|dx

pcxiy , = 1 → exp Lacy , )

• lax hcxiexpfytki)

Iya 'T

= Iq flog lax has exp ( yttcxi)) = 1- fax hcxi exp ( YT tix ) ) tix ) chain exp Lacy , ] Rule First = I dx pcxly ) y = Epix , y , f't Moment

SLIDE 11 Moments and Natural Parameters pcxly ) = hcx ) exp I YT text

• aey ) )
• Moments

are computable from derivatives

• f

acyl d " Fyn at Y ' = IE pcxiy , [ than ]

• When

to , are linearly independent an exponential family is known as minimal

• Far

any minimal family acy ) is convex and µ i = ftp.cx.y , I text ) → y ( there is a 1. to

• I

mapping from n to Etpcxiyjltlxl ) )

SLIDE 12 Moments and Natural Parameters Example ; Normal Distribution t ( x ) = ( x ,xZ ) Efx ]

⇒

µ =

• 291g

, y

• _

( pile ?

• 1/262

) EL x ' I

⇒

62+15=274,24

, Example ; Discrete Distribution

pczib

) =l Oh ' " = " = expat

& log On

IG

• 17)

Yu tufts E I Itt

• h ) ] ⇐

Oh = exp I ya )

SLIDE 13 Algorithm : Expectation Maximization * * * Objective : n.ME = a

.gg?gxlogpCy11u.E,n

) Repeat until convergence ' (

• bjective

unchanged ) 1 . For n in 7 , .

• .

Ni yuh ie EL IlZn=h ) ) =

/

dtnpcznlyn.io ) Ilan

• h ]

# Points in cluster he 2. For W in I , . . . Ki N N Mu = I E yuh yn Empirical Nh :-. I ruh µ h h = ' Mean h 't Ih = ,

&

!

trnhynynt

• Iuyuht

Empirical Covariance nu = Nh IN Fraction in cluster h

SLIDE 14 Expectation Maximization : Example Iteration : O

SLIDE 15 Expectation Maximization : Example Iteration : 1

SLIDE 16 Expectation Maximization : Example Iteration : 2

SLIDE 17 Expectation Maximization : Example Iteration : 3

SLIDE 18 Expectation Maximization : Example Iteration : 4

SLIDE 19 Expectation Maximization : Example Iteration : 5

SLIDE 20 Expectation Maximization : Example Iteration : 6

SLIDE 21 Intermezzo , : Jensen 's Inequality Convex Functions Area above f- ( tx , t Ci

• t )

xz ) fix , . . . • fkn curve is a

• convex

set s t fix , ) t It

• t )

flat fix . ' g X , Xz Concave Functions Area below f- ( tx , t Ci

• t )

xz ) fix flat cu

• ve

is a

• ,←¥f
• fix

, convex set 7 t fix , ) + It

• t )

flat s X , Xz Corrolary : Random Variables

t.ci#xnl:Efii.i:iit:::::.

SLIDE 22 Lowen Bounds

• n

Marginal

Likelihoods Idea : Use Jensen 's inequality to define Lower Bound 2- i

• fax

six , = fax gin 4¥, = E goal L :

• E. *

" . I

lost

I

slog

#

* , I It

• . tog

I [ Lower bound

• n

boy 7

Gaussian

Mixture Model 2- I O ) ; = lolz pig . 't :O ) = I at ace ; y , PgY¥ = pig :o) £10,81 :-. Ez . " " , I log "gY÷ I s log pay ;

• l

SLIDE 23 Algorithm : Generalized Expectation Maximization Objective : Llap ) is

Egj

,

.gg/loyPlY'tt-9/slogpcy;o7

9175g )

Initialize

; O Repeat until £10 , y ) unchanged : 1 . Expectation Step y = angngax I ( O , y ) 2 . Maximization Step O = anymore LIO , r )

SLIDE 24 Intermezzo : Kullbach

• Leiber

Divergence 9¥ Measures how much KL(q( × ) H MK ) )

:=

)

dx 91×1 ↳ Y n ' × ' a ,× , deviates from 171×1 Properties 1 . KL ( qcxl 11 nlx ) ) 30 ( Positive Semi . definite) . KL ( qcxllinkl ) = / ax galley "g¥ ' = E# galley "f¥, ) ? lag ( ## , ,× , I 'g¥sl) = log ( i ) =

• 2

. KL( q ( x ) 11171×1) =

• as

qk ) = MK ) 91×1 : : Mk ) |dx qix , by 91¥

• |d×nk, lgYn¥

, =

SLIDE 25 KL divergence vs Lower Bound

• pcyit

;o ) = £10,21 = Eagan , lloypggjt.sc ] P 's :O 'P 't 's :o) = #z~q , ;y)|log pig :o) + leg Plaything ) does not depend

• n £

rewrite as Kttdiv = log pig :o)

• #

← an ;n|log9pYftTo , ] = log ply :O)

• KL (

qhsy ) H paly ;o ) ) a \ Does not depend

• n

y Depends

• n

y Implication : Maximizing £ ( O ,y ) wrt y is equivalent to minimizing KL ( 917 ; g) 11 pcttly ; O ) )

• .

SLIDE 26 Algorithm : Generalized Expectation Maximization Initialize ; O Define : Llan HI . . "

.in/logPgYITjY/slogpcyi07

• Repeat

until an unchanged : 1 . Expectation : act ;D ← petty ;D Yuh i = plan

• h

I yn ; O ) = plush , 2- n

• le

; 07

f

plgn ,Zn=l :O ) 2 . Maximization : Solve for : 2£10 , r ) =

• ( See

next slide )

do

SLIDE 27 Generalized EM : Maximization Step " Generalized Hard K . means " : Update for Me M Ilgply , 7in = [ Ital ) ( yn

• Me

) [ I =

• que

n=i Generalized EM : Update for He N

• f. LLQH

= EffueEganm1logP4gYzIigInI-nT@Eqn.n ;µ|fµe leg plyn , 7h ; 01 ] = n"& Egan ..ru/Ittn=eDlyn.Ne)Ee "

ynh=

II.

ynulyn

• i.

e) Ee "

SLIDE 28 Generalized EM : Maximization Step " Generalized Hard K . means " ; Update for Me N Ilgply , 7in = [ Ital ) ( yn

• Me

)[ I =

• que

no Solution : pie = ten n§ . ,I[7n=l]yn Ne = n§IGn=l] Generalized EM : Update for Me

ftp.LH.r

) = § , kulyn . µe)[ e " =

• N

Solution : me

I

t.eu?.ynhynNe=n&rnu

SLIDE 29 Algorithm : Generalized Expectation Maximization Initialize ; O Define : Llo ,Hi= Eza , ... , ,[logPgYf÷gM s log pcy :o) Repeat until Zn unchanged : 1 , Expectation : qcz ;g ) ← pc Fly ;z ) ynh := plzn=h 1 yn ;O ) = PCY "7n=h ' ' 07 =T±[Ifh=hD

£

plyn ,Zn=l :O) 2 . Maximization : Solve 2210,27/00=0 µh= ,Iun§kuyn [

u=fIa§

. , ynuynyni) . µuµT hit ,Yhg

SLIDE 30