Ruiyang Elizabeth : , , Released Homework 4 yesterday : Apr - - PowerPoint PPT Presentation

SLIDE 1 Lecture 17 : Sum

• Product

Algorithm Scribes : Yaoshen , Ruiyang , Elizabeth Homework 4 : Released yesterday Due Fri Apr 13

SLIDE 2 Today : Exact Marginal

• ver

Discrete Variables Goal : Compute Marginal

• f

the form

P(×i=×ii×j=×i

) = קµ= , ;,µ ,.PK '=× ' ,

• ' ×K=×K )

Assumption : All variables Xw are discrete Enables : Calculation

• f

posterior p( X ; =x ; 1 Xj = × ;) = p ( X ;=×i , Xj=×j )

÷

lj=x ;)

SLIDE 3

Example

: Markov Chain Idea : Rearrange Terms in Sum Pla , b , c , d ) = pc al b) pcblcspccld ) pcd ) B C D pca ) = [ [ [ pc a ,b , c. d) Naive : ABCD b= . CI D= ) yd( C) B

• =

[ plalb ) (§ pcblo ) (§ , pccidipld ) )) b= , B = § , pcaib ) ( § pcblc ) ydk ) ) = I pcalbikcb )

• b=

, ya ( b )

SLIDE 4

Example

: Markov Chain pc as

=b¥

, § , a£ .

,p( aiblplbk

)p( cld ) ABCD B B = [ pcaib )

yccb

) ybla ) = § pcalb ) yicb ) b= , = i b AB C

KLB

) = { pcbi c)yd ( c 3

Jdc

4 = [ p ( c i d) pcd ) =L D= , BC Cost : CD Question : What is the Computational Complexity

• f

computing pca ) ? AB + BCTCD

SLIDE 5 Sum . product

• n

Factor Graphs Bayesian Network ( non

• branching )

pla , b , c , d ) = pc a 1 bl PC blc ) plc Id ) p( d) Factor Graph ( non

• branching )

Pla , b , c , d ) = f. I a. b) fz ( b. c) f 3 ( ( ,d ) fald )

SLIDE 6 Sum . product

• n

Factor Graphs Factor Graph ( non

• branching )

Pla , b. c , d ) = f. ( a , b) fzlb , a ) fs ( c. d) f< , (d) Messages : Variable to Variable

µa÷

pea , b. c ) = f. 1 a. b) fzlb ,c ) § . , fzl c. d) fuld ) pca , b ) = f. ( a. b) § , fzlb ,c ) µd→< ( a )

←

→ b 1 b )

SLIDE 7 Sum . product

• n

Factor Graphs Factor Graph ( non

• branching )

Pla , b , c , d ) = f. ( a , b) fzl b. a ) f. ( c. d) f< , (d) Factor Graph ( singly

• connected

) ( a. b. a Tree ) Factors with multiple edges PC a , b , c , d ) = \ Vons with

• f. (

a. b) fz ( b. c. d) q multiple edges f3( c) fald ,e ) fsld )

SLIDE 8 Sum . product

• n

Factor Graphs Factor Graph ( singly

• connected

) Q µh →

• b. (b)

= [ fd b. c. d) µc → f. ( a ) a- ° 'd µd→ f. 1 d ) q Q µc→ f. ( 0 = µf } → c ( c ) gpa as µi3→ do ) = fz ( a ) Ap I µd→ fzldl = µfa→d( d) µfs→d( d ) Mts . → dla ) : fsldl

µk→alal=

{ fald ,e ) µe→ facet µd→f . ( d ) t µfs→d( d) µ fy → d ( d) Messages : Factor as Variable . ~

• pca

, b ) = f. I a. b) {µfzl b. c. d) f3( c) fsla ) { fuld ,e ) i M fz→b ( b )

SLIDE 9 Sum . product

• n

Factor Graphs Advantage

• f

Factor Graphs : Can compute any marginal < p ( e) = { field ,e ) µd→fa( d) s s , t I Md→fdd)= µfz→dldlµfr→dl d) 9 9 ^ µfz→d 1 a ) = [ fzlb , 9 d) My → a ( to ) ' µc

• fz

( C )

SLIDE 10 Sum . product

• n

Factor Graphs helfzl :{ b. c. d } Factors : General torn p ( X ) = M ¢f( Xf ) f held ) a } fz ,fu,fs ) he ( x ) ( factors in which ×

• ccurs

? Xf I nelf ) hecfl ( variables that f depends

• n

) Factor → Variable : µf→× ( × ) = [ Old Xe ) M µ → fly ) ( Sum ) { Xfix }

gene

(f) \{ x } 5 Variable → Factor : µ×→f I × ) = M 9 E ' he ( × ) \ { f } µ9→x K ) ( Product )

SLIDE 11 Belief Propagation : Compute Marginal for All Variables < , a General Form : Marginal L

4 ss

< s

rd

< s s s pkl a M µt→×l× )

f

fehecx ) Algorithm : Compute All Messages 1 . Pich any variable x z , Compute incoming messages 3 Compute

• utgoing

messages

SLIDE 12 Variable ←s Variable vs Factor as Variable Messages Factor Graph ( non

• branching )

< < < < < < < µ < → b ( b ) = I fz( b.cl µ< → f. 1 c )

.

fz → b ( b )

SLIDE 13 Example : Forward

• Backward

Algorithm / HMMS ) Factor Graph Generative Model f , fz fz fg S < s < s < g hi ~ Discrete ( 17 , , . . . , Mk ) g , ^gzn g. ^

gun

htlht . ,

=k~Dik(

Ah , , . . .AkK ) Vt1ht=h ~ pl ✓ + 1h+=k ) Goal : Compute Marginal ,

ytlk

) = p 1 ht IY , . . i , Vt ) Bt( he )

• &

µft→hilhtlµf ,.+,→hdhHµg+→htlht )

• 5

atchl

SLIDE 14 Example : Forward

• Backward

Algorithm / HMMS ) Factor Graph Generative Model f , fz fz fg s Is g g g g > hi ~ Disc ( 17 , , . . . , 17k ) ^ ^ ^ ^ htlht . , =L ~ Disdain , ... ,Ank ) 9 , 92 93 94 Vtlht=h ~ pN+lh+=h ) ' plvtlht = h ) Forward Pass ( outgoing messages ) p

• tlhl

=

µn+→f↳,lht=H

= µg+→h+lht=h\µf+→h+lht=h ) = Mgeshtlhteh )

{

ft ( h ,l ) µh+ .,→f+( heel ) pcht.hlht.pl ) = Ahl de , I b )

SLIDE 15 Example : Forward

• Backward

Algorithm / HMMS ) Factor Graph Generative Model f , fz fz fg < < < 2 ( s < hi ~ Disc ( ni , ... , 17k ) ^ ^ ^ ^ htlht . , =L ~ Disdain , ... ,Ank ) 9 , 92 93 94 ^ ^ ^ n Vtlht=h ~ pN+lh+=h ) Backward Pass ( Incoming Messages ) B , ( h ) = Mf → h.lk ) =

{

fell ,h ) µh++,→f+ll ) t " =

{

fell .h ) µgt+,sh++ , 4 )µft+,→ht+ , " ) Ahl p(✓t+ , lhtnol ) Bttill )

SLIDE 16 Example : Forward

• Backward

Algorithm / HMMS ) ForwardPass d. ( k ) = pch ,=kap( v. 1h ,=k ) ( t.tl K K dt l h ) = Pc Vtlht = h ) { Aeu at . ,ll ) ( t > i ) 1<2 =L Backward Pass OCK + ( t . 1) 1<2 ) < < 0( Kt ) ptlhl = 1 ( t=T ) K k B + 1h ) =

§

Ahe pi✓t+,lht*=l)P++,ll ) ltctl 1<2 it Marginal s ytlh ) a 0+14 Ptlh ) = µf+ . ,→h+l↳µgµhd↳µfphd "

SLIDE 17