Jay DeYoung : Hamnett Donald Families Exponential An family - - PowerPoint PPT Presentation

SLIDE 1 Lecture It : Expectation Maximization Scribes : Jay DeYoung Donald Hamnett

SLIDE 2 Exponential Families An exponential family distribution has the form Only depends

• n

x Depends

• n

I Only depend I y and x an m d pcx I y ) = hcx ) exp I YT text

• aey ) )

n E leg normalizer

(

÷ :

" Base measure ( Canting , Lebesgue ) ( only depends an X )

SLIDE 3 Conjugate priors Likelihood :

petty

) = hey exp

lqttcx

)

• acy

, ] Conjugate prior : i D is ( d , .dz ) payed ) = hip exp IT " tch )

• act

' I tip

:=

( y ,

• acyl )

Joint : plx , yl i

• hey

hey) explyt (

tix

) i 9 , )

• aim (

rt Da )

• aids

) =

hlx

) hey, exploit ,

• augier
• acts ) explant )
• acts ]

. I # pcyl IT ) p C x ) 7

• aids )

Fa

SLIDE 4 Conjugate priors Joint :

plx.nl

=

hex

?

pint 51 explored )

• alt

) ] I , = d , + fix , Ja = Dat I Marginal :pox ) is

fdypcx.nl

=

hcx

, exp I act '

• aol

] p Com compute marginal from leg normalizer Posterior !

pcyixi

=

ptx.gs/plx)=pcy1Qttcxs

, Datt ) J Conjugacy : Posterior here saz family as prior

• aid

) ) Fa

SLIDE 5 Gibbs Sampling : Homework

• Idea

: Ensure that prion is conjugate to likelihood likelihood conjugate prior posterior far cluster h

• I

Mpcynlfihduh.su/plMh,Eul9h

) Pl Mk , Eh I y Iim , 2- n µ ) = In :tn=h3 M

• lnizi.az/0lYnl7n=h

)

• .

marginal likelihood Derive this I in homework = p ( Mu , Ch I 9h t the ( y , 7 ) )

SLIDE 6 Gibbs Sampling : Homework . ( yn , ynynt ) I fan MVN

pcynlzi-h.ly

) ' =

hlynlexplyiitlyn

)

• acyu )]

pi mu ) =

hlyu

) exp I ya

• Du algal )

T Lock up

• n

Wikipedia I ply ,

in

, El = II plluu.eu/nl7p/ynlzn=h,,u.IY " " tlyn.tn ) = I? hlyn ) 11h44 y . exp I

Gynt

( Du,tfHynlIEK=h) )

• [ acyu ) ( Due E.

III. 4 ) ) h #

SLIDE 7 Moments : Derivatives

• f

Log Normalizer pcx ly ) = hcx ) exp I YT text

• aey ) )

|dx

pcxiy , = 1 → exp Lacy , )

• lax hcxiexpfytki)

Iya 'T

= Iq flog lax has exp ( yttcxi)) = 1- fax hcxi exp ( YT tix ) ) tix ) chain exp Lacy , ] Rule First = I dx pcxly ) y = Epix , y , f't Moment

SLIDE 8 Moments and Natural Parameters pcxly ) = hcx ) exp I YT text

• aey ) )
• Moments

are computable from derivatives

• f

acyl d " Fyn at Y ' = IE pcxiy , [ than ]

• When

to , are linearly independent an exponential family is known as minimal

• Far

any minimal family acy ) is convex and µ i = ftp.cx.y , I text ) → y ( there is a 1. to

• I

mapping from n to Etpcxiyjltlxl ) )

SLIDE 9 Moments and Natural Parameters Example ; Normal Distribution t ( x ) = ( x ,xZ ) Efx ]

⇒

µ =

• 24

'/y

, y

• _

( pile ?

• 1/262

) EL x ' ]

⇒

62+15=174,24

, Example ; Bernoulli Distribution

Plxlpe

) = yr " C I

• µ )

'

• ×

= exp I leg x t log I

• m )

Efx ] ⇐ > µ 9 tix , acn ,

SLIDE 10 Maximum Likelihood Estimation The maximum likelihood parameters y* are determined by the sufficient statistics

• f

the

• bserved

data Solve : y : E It ] = text I n pH 'm ) Example : Gaussian Xu n

Naim

( µ , 6 ) nil , . → N tix , =

I

Eia

,

'Ei )

µ* = Yu I xn 6*2 = ¥ , ? XI

• µ*

'

SLIDE 11 Expectation Maximization * * * Objective : n.ME = a

.gg?gxlogpCy11u.E,n

) Repeat until convergence ' (

• bjective

unchanged ) 1 . For n in 7 , .

• .

Ni yuh ie El IlZn=h ) ) = / dznpcznlyn.io ) Ilan

• h ]

# Points in cluster he 2. For W in I , . . . Ki N N Mu = t E yuh yn Empirical Nh :-. I ruh µ h h = ' Mean h 't Ih = ,

&

!

trnhynynt

• pryuht

Empirical Covariance nu = Nh IN Fraction in cluster h

SLIDE 12 Expectation Maximization : Example Iteration : O

SLIDE 13 Expectation Maximization : Example Iteration : 1

SLIDE 14 Expectation Maximization : Example Iteration : 2

SLIDE 15 Expectation Maximization : Example Iteration : 3

SLIDE 16 Expectation Maximization : Example Iteration : 4

SLIDE 17 Expectation Maximization : Example Iteration : 5

SLIDE 18 Expectation Maximization : Example Iteration : 6

SLIDE 19 Expectation Maximization * * * Objective : n.ME = a

.gg?gxlogpCy11u.E,n

) Repeat until convergence ' (

• bjective

unchanged ) 1 . For n in 7 , .

• .

Ni yuh ie El IlZn=h ) ) = / dznpcznlyn.io ) Ilan

• h ]

# Points in cluster he 2. For W in I , . . . Ki N N Mu = t E yuh yn Empirical Nh :-. I ruh µ h h = ' Mean h 't Ih = ,

&

!

trnhynynt

• pryuht

Empirical Covariance nu = Nh IN Fraction in cluster h

SLIDE 20 Maximum Likelihood Estimation in 6mm Easy : Estimate n for " observed " t

• = &

log ply , 2-

Am

) ? fly , 77

• § acy

)

Problem

: Need to marginalize

• ver

Z

pcyly

) = ldtplu.tt/y1=n7!/dznpCyn.7nly7oQylogpCy1yl=ptqy,fqnI ,

ldznexplyitly

,

• aint

Integral throws spanner in worms

SLIDE 21 Intermezzo , : Jensen 's Inequality Convex Functions Area above f- ( tx , t Ci

• t )

xz ) fix , . . . • fkn curve is a

• convex

set s t fix , ) t It

• t )

flat fix . ' g X , Xz Concave Functions Area below f- ( tx , t Ci

• t )

xz ) fix flat cu

• ve

is a

• ,←¥f
• fix

, convex set , 7 t fix , ) + It

• t )

flat s X , Xz Corrolary : Random Variables

t.ci#xnl:Efii.i:iit::::.

SLIDE 22 Intermezzo : Kullback

• Leibler

Divergence 9¥ Measures how much KL( qcxsll MIX ) ) :

• I

DX 941 by n " ' a ,× , deviates from MIN Properties I . KL ( a call mix ) ) 3

• ( Positive femi
• definite)
• KL ( g

kill MIN ) =

lax

guy leg "9¥ , = E. ⇐ galley "g , ) ± log ( E⇐g*l"gift ) = log lil

• 2

. KL ( q C x ) 11171×1) =

• a

91×1 = Mk ) 9kt 171×7 I dxqix , log 94¥ , → lax Mix , leg , =

SLIDE 23 Lowen Bounds

• n

Marginal

Likelihoods Idea : Use Jensen 's inequality to define Lower Bound 2- i

• fax

six , = fax aix ) 4¥, = E goal L :

• E. *

" . I

lost

I

slog

#

⇐ , 14

It

• . tog

't [ Lower bound

• n

boy 7

Gaussian

Mixture Model 2- I O ) ; = Id 't pig . 't :O ) = I at act ; y , PgY¥ = pig ;o , pig , 7 ;D ) £10,81 :-. Ez .mg , I log , ) s log pay ;

• l

SLIDE 24 Algorithm : Generalized Expectation Maximization Objective : Llap ) is

Egj

,

.gg/loyPlY'tt-9/slogpcy;o7

9175g ) Initialize ; O Repeat until £10 , y ) unchanged : 1 . Expectation Step y = angngax I ( O , y ) 2 . Maximization Step O = anymore LIO , r )

SLIDE 25 KL divergence vs Lower Bound

• pcyit

;o ) = £10,21 = Eagan , lloypggjt.sc ] P 's :O 'P 't 's :o) = #z~q , ;y)|log pig :o) + leg Plaything ) does not depend

• n £

rewrite as Kttdiv = log pig :o)

• #

← an ;n|log9pYftTo , ] = log ply :O)

• KL (

qhsy ) H paly ;o ) ) a \ Does not depend

• n

y Depends

• n

y Implication : Maximizing £ ( O ,y ) wrt y is equivalent to minimizing KL ( 917 ; g) 11 pcttly ; O ) )

• .

SLIDE 26 Algorithm : Generalized Expectation Maximization Objective : Lto , Hi .

Egj

,

.gg/loyPlYttt-9/slogpcy;o7

917 's y ) Initialize : A qc.zi-hsyl-ku-pftn-hlyn.cl Repeat until £10 , y ) unchanged : i , Expectation Step IT v y = angngax d- I O , r ) = argginklfqlt.rs//pCzty , 2 . Maximization Step \ O = anymore LIO , r )

SLIDE 27 Lower Bound for Exponential Families Hair )

• Ea ,

,

.gs/logPgY.?IT

]

leg ply , 't ly ) = MT §

! tlyn

, 7- n )

• Macy )

¥114,8

) = Ig # a , , ;g , flag piyitiy ) ) = € ' Egan , # I tlynitnl )

• Nj acy )

[ Can solve this as lay as we can . compute expected sufficient stats

SLIDE 28 Algorithm : Generalized Expectation Maximization Objective : Lto .hr .

Egj

,

.gg/loyPlY'tt-9/slogpcy;o7

9175g ) Initialize ; A Repeat until £10 , y ) unchanged : 1 . Expectation Step Computes expected y = ang mate I ( O , 8 ) sufficient statistics r 2 . Maximization Step Maximizes O given O = anymore L( 0,8 ) computed statistics

SLIDE 29

SLIDE 30

SLIDE 31

SLIDE 32