Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - PowerPoint PPT Presentation

Algorithm Sum - product Lecture : Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 today : Feb Due I

Today Variables Marginal Discrete Exact over : Marginal Goal form Compute of the : =×n;u§µ± p( , ×K=×K ) , Xj=×j ) PK × " × ' =× ' ' ' - , ; ; , All Assumption variables Xw : discrete are Enables Calculation of posterior : 1 Xj × ;) , Xj=×j ) p ( X p( X x ;=×i ; = = = ; ÷ ;) lj=x

Markov Example Chain : Rearrange Busid Idea Terms Sum : in ABCD b d Pla ) c = , , , ) plc bl blc ) Id pcd pc PC ) 1 a = § D , § pccid [ pcaiblplbic ) )p( terms cld pcas D= i , { yclb plblc ydlc ) ) ) : = ) § = § ) ) ) pcblc ) ( { pcd ) lb pca { ) p " ) 'd 'd 8D ) Pcc ÷ § pcaib ) ( b ) f. =

Markov Example Chain : Naive Sum =b¥ .pl alblplbk , § , a£ ) ABC D )p( cld pca ) . Sun Rearranged 13 [ AB pcaib ( b ) ABTBCTCD ) = ya b= , = § B § Jb Va ( b ) bi ,p( ( 4 ( < ) e) c ) ( ) a ya yb ( i = p c 1 { . Bc < D Question What Computational the Complexity is : ? of computing ) pca

Sum Graphs Factor product on . - branching ) Network ( Bayesian non ) plc Ibl blc b d ) Id d) pla pc PC ) p( c a = , , , - branching ) Graph ( Factor non ) fa f. fz f b d ) b. d ( a ) b) 1 Pla ( d ) 1 c c = a , , , , , ,

Sum Graphs Factor product on . - branching ) Graph ( Factor non f. f< fzl f. b. b. a ) d) (d) b) ( d) Pla ( c. c = a , , , , Message Variable Messages µa÷ µd→c( Variable c) to : : . § fald f. f3( fzlb b. c ) b ) ,c ) ( d) ) pea a. c. = , { f. ( fzl b ) b. b ) c) pca a. = , = { fzl b. c) µd→d4 Message b) µc→b( :

Variable Variable Variable Messages Factor a vs as - branching ) Graph ( Factor non s s s , , s , , { fz ( b c ) ( b ) ( µ , c) µb→f = b . → b c ( < ) µfz→c =

Sum Graphs Factor product on . - branching ) Graph ( Factor non b d Pla ) c = , , , f. fz b b) ( a ) ( a , , f< fs (d) d) ( c. , Graph ( singly ) Factor connected - Difference : b d than ) 2 vws PC c a = more , , , \ factor pen f. 1 a ,b ) fz ( b. d ) c. f 3C fald fs I d ) ) ,e c)

Sum Graphs Factor product on . Pc a. Graph ( singly ) Factor connected - f. ( a. b) lb ,b ) Pla ) < µb→f = , 1- µfz→blH ( bl Mb = < < c g. → n { afz 1 b. µIf,klµd→µld µfz→blbl d) ) = c. < < " µfa→d( dl d) µd→fzld ) µf←→a( t a (d) fn § dldkfshd ( d. e) µe→ fale ) ) Mfa = µf , . → → ( d ) fz µd→ µe→fy( 4 1 = . µfu→d÷ 1 ^ > Mf → Factor Variable Messages . : ←s b) {afz( b. ) § fal d. f fsld f. d ) > ( 4 b ) ( el c. = a.

Sum Graphs Factor product on . = § fu ( d. nel { b. d } f.) fala = ) µfz→d( < ( e) e) c. µd→ p L → lad ) µfz→ddl s s s Md→pyd)= Mfs d . s s ne( d) =3 thfsifis = § fzlb a a) d) ' , , . f. lb ) µb µc fz ( c ) General Form Messages → - : tl ( factors ¢f( Xf ) ( that Variables X ) × ) that ( hecfl ( he p = f ) depends an ) depend × on [ Mµy→f( Variable Factor ¢e( ft ) y ) ( x ) µ → : = , → × { Xfrx ) } yesynecflix } ( Sum M Variable Factor µ×→f 1 × ) ( ) necx )\fzµ9→× = → × : { ( ) Product 9 e

Sum Graphs Factor product on . General Factors torn : 174,1%1 pox ) = f EF F. necfifx EE } { Xf ( x , f) necfl x ' = : EE } ix. f) { f Fx held = : - G ( X , # ) - E = , I Factor Variable M Ott Xe ) → fly ) C x ) µ µ → : = , → × gene S } I flesh ( Sum ) M Variable Factor I µ x ) = → Mg K ) , f × → x → { fg ' new , 9 c- ( Product )

Variables Belief Compute Marginal Propagation All for : < , rd ss 4 f General Marginal Form a : L < s M pkl a µt→×l× ) s s < s fehecx ) Algorithm All Messages Compute : Pich variable 1 x any . Compute incoming 2 messages . 3 Compute outgoing messages .

Variables Belief Compute Marginal Propagation All for : Step Define 1 initial tree rooted at node : d fz fs fe a. a. a. . , b e c f. f , e. a. a

Variables Belief Compute Propagation Marginals All for i f- , man Step 2 tree Walk to : d leaf find nodes T 1- a I a L fo f , fo not we noo , r s r d Propagate d L 3 Step : messages b e c ( incoming ) to parent nodes a a v u f , ma u Step Propagate 4 a : messages ( outgoing ) child to nodes

Belief ( Binary Pseudo Variables ) Propagation : code - Assume Edges E , f ) : C x Potentials OI [ f) d T a t a I a L bp ( ) finna f . def E , OI fo x woo : noooo , r , s r d a u { } Messages µ = b e c I f } f EE for f) ( a x : e ^ i , v u f , f- , ma ma ( µ , f. in E. OI ) µ x sum = - , I f } f EE for f) a ( x : e : , , I - prod ( µ , E x. f ) out µ = , return M

Belief µ ,E ( Binary Pseudo Variables ) Propagation : code - ( × ) def , OI f. in sum : . , EE }s{ { y } x } { y d :( f) yµ = y , , . . . , T at for f. { y } fs fe ↳. ,yµ : ... ye ↳a• . , ... , , r 7 t , E. ¢ y ,f ) ( µ prod in µ = - , b l C ( compute messages children ^ ) from n 1 } for k { a f. may f , ; e ... , , ;yµ=hµ ) § If ](x=h µ[ f. × ][k]=[ yihi , a h ' ' ↳ ' ' " ,f][ µ[ hu ] yn Mn ( updated return return µ parent to ) messages

Belief ( Binary ,E Pseudo Variables ) Propagation : code - - prod ( f ) def , OI ,gµ g. g. g. # in µ x ,gµ : , , }\{ { g :C x. g) f } } { EE d ,f][k] = ... , T at for f. { } fs fe a. : a. a. g e . ... , , r 7 t , E. & x ) ( µ in Sam µ = - , b e c x x for he { } 0,1 f. may f , ; ... ( h ] Mµ[ µ[ x gn = , a return µ

Belief If ( Binary Pseudo Variables ) Propagation : code - Assume Edges E , f ) : C x Potentials OI ] d T at I a L f. tag tag bp ( ) finna f. def E , OI fo x mm. : nooooo , r , s r d a L { } Messages µ = b e c I f } f EE for f) ( a x : e { ^ : , f , messages ( µ , f. E. OI ) in µ × from sum ^ = - , - children f f } f parents EE for f) a to I { x : e : , , I - prod ( µ , E , f ) out messages µ x = , from parents return children to µ

Belief µ ,E ( Binary Pseudo Variables ) Propagation : code - prod ( , # f ) def g. # out x : - ,gµ , , }\{ { g :C x. g) f } } ,f][k] { EE d = ... , Tr T a . at .at f. art Incoming I a L . Lee { } f. for fs fe Message a. 0,1 a. a. : { a . , r , ( h ] Mµ[ µ[ x gn = b e , c yguense § x x for { ,gµ g. } f , : g e g. ... , , E. & x ) ( µ out der µ Sam = - , a from in sum return . µ

f. arg Belief T I I µ ,E ( Binary Pseudo Variables ) Propagation : code - ( x ) def , OI f. out sum : . , EE }s{ { y } x } { y d :( f) yµ = g. , . . . , T t ^ t L ~ i } for Lee { f. fs fo ... b. ... o ; { , . , , ;yµ=hµ ) # If ](x=h µ[f ,×][h]=[ yih b e , c hi , ,hµ ( . . , ,f]( |{ for Mµ[ hu ] yn ]↳. f } { y ibn : ye , ... . , E. ¢ y ,f ) ( µ reverse a prod in µ = - , order fun return µ sum in -

Variables Belief Compute Marginal Propagation All for : < s ÷ a Marginal for variable × 7 L ( I C I M pkl a µp→×l× ) < g fehecx ) 9 Algorithm All Messages Compute : Pich variable n × any } . compute can now Compute outgoing messages z , for marginal , Compute incoming 3 messages x any

dtlhl Algorithm / Example Backward Forward HMMS ) : - Model Generative Graph Factor gf , gk f gk ' ( Discrete n ) ' h , , ~ , , ( Au ) ^ htlht ^ ^ =L Discrete ^ µg+→htM ~ 92 93 9 94 , . , ^ 4 Normal /µu,6u ) ^ ^ Vtlht=h ~ ,h+=hl 9+(4 pivtl h+=h ) ' = µn+→f↳,lht=H Forward ) y ( outgoing Pass messages = → htlhl µtt = , . K ftlhill [ . ,→t+ll µg+→↳M µn+ ) = l = ' ↳ Aeu 1 l ) d+ is , .

Algorithm / Example Backward Forward HMMS ) : - Model Generative Graph Factor fg fz f fz ( Discrete n ) , hi ~ < < 2 < < ( s ( Ah ) htlht ^ ^ =L Discrete ^ ^ ~ 92 93 9 94 , . , ^ ^ ^ ^ Normal ( µu,6u ) |h+=h ✓ , ~ Backward Pass ( Incoming Messages ) Btlh ? Mf+→h+lH [ ft lih ) fell ) ( Mn++ = = → , !llµft+i→ht+ ftllih e§ )µg++,→n " ' = , . . Ahl Bttll ) nt+,=l ✓ + 1 ) pi + ,

/ Algorithm Example Forward Backward HMMS ) : - Forward Pass ( plh ,=h di ( k ) 1h ,=h t=| ) ) plv ) = , { ( t Aeudt > 1) . ,( p(V+lh+=hl dt h ) l ) 1 = Backward Pass ( ) Btlhl t=T 1 = ltctl Ane p++,ll p(✓t+,1ht+,=l 1h ) § B ) ) = + Marginal s " 0+14 . ,→h+l↳µgµhd↳µfphd ytlh Ptlh ) µf+ ) a =