Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - PowerPoint PPT Presentation

Algorithm Sum - product Lecture : , Loopy Belief Propagation Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 today : Feb Due I

Today Variables Marginals Discrete Exact over : Marginals Goal form Compute of the : u¥ , Xj pl - Xie ) Xi Xj Xi ) PK - Xk × " ' - - = ' - . . - , - ,↳ ; . q , All Assumption variables Xu : discrete are Enables Calculation of posterior : I Xj , Xj j ) x ; ) p ( X pcxi x x x - = ; = - - ; - ; - - I x ; ) p ( Xi - -

Markov Example Chain : Rearrange Basic Idea Terms Sum : in b d Pla ) c = , , , [ E E peas = lb ) To : = l E & . E D . c ) C ya = i E = b

Markov Example Chain : " Sm " " I E. pca.bipcbnp.ua , peas = . Sum Rearranged B I b ) b ) I zac ca p = - b = , = § B & 2b b ) Val b Col face c) c ) ( ) ) d I yb pc C I = p c 1 . , I . Question What Computational the Complexity is : ? of computing peas

Sum Graphs Factor product on . - branching ) Network ( Bayesian non ) plc pcalblpcblc b d ) Id pcd pla ) ) c = , , , - branching ) Graph ( Factor non b d Pla ) c = , , ,

Sum Graphs Factor product on . - branching ) Graph ( Factor non ) fa f. fzlb f b a ) d (d) b ) la d Pla ( ) C c = , , , , , , , , Variable Messages Variable to : b c ) pea = , , b ) pca = ,

Variable Variable Variable Messages Factor a vs as - branching ) Graph ( Factor non s s s , , s , , { fz ( b c ) ( b ) ( µ , c) µb→f = b . → b c ( < ) µfz→c =

Sum Graphs Factor product on . - branching ) Graph ( Factor non b d Pla ) c = , , , f. fzlb b) a ) la , , ) fa f (d) d I C , , , Graph ( singly ) Factor connected - b d ) Pla c = , , , fi fz( b. la ,b ) d) c. fuld folds f- e ) C c ) , ,

Sum Graphs Factor product on . General Factors torn : 174,1%1 pox ) = fef } { %= = } { Fx = - G= ( - I ftp.lfxfyc-neltissx3 I Factor Variable M C x ) µ → : = , → × e' ( Sum ) M Variable Factor I µ x ) = → , f × → If } needs g ( Product )

Sum Graphs Factor product on . Graph ( singly ) Factor connected - =L f la , b) , b ) la p < = , 1- l bl µ = < a L b e. → n µfz→blbl L L Cid h fzld ) Md = → ar (d) [ Mfs = , e dldkfshd facet I Yes ) = Mf , → Factor Variable Messages : ) { falafel b) ? f fsld fzlb f. d ) lol b ) I pea c. = a. , , , a

Sum Graphs Factor product on . = ? fu ( d. melfi fala - ) c pie ) e) Iud , L 9 → lad ) Mf S > → old ) → fuld ) Mfs Md = → . , S S = ! nel di I fzlb d ) ^ (d) Mfr d c. - , Mbs falls ) Me fate ) - General Form Messages : tf off ( X , ) CX ) ( x ) hecfl he p = I M Factor Variable x ) C µ → : = , → × ) FX , M } ( yinecflsx Sum - M Variable Factor I µ x ) = → , f × → f 3 I ( ) needs Product g e

Variables Belief Compute Marginal Propagation All for : < , rd ss 4 f General Marginal Form a : L < s M pkl a µt→×l× ) s s < s fehecx ) Algorithm All Messages Compute : Pich variable 1 x any . Compute incoming 2 messages . 3 Compute outgoing messages .

Variables Belief Compute Marginal Propagation All for : Step Define 1 initial tree rooted at node : d fz fs fe a. a. a. . , b e c f. f , e. a. a

Variables Belief Compute Propagation Marginals All for i f- , man Step 2 tree Walk to : d leaf find nodes T 1- a I a L fo f , fo not we noo , r s r d Propagate d L 3 Step : messages b e c ( incoming ) to parent nodes a a v u f , ma u Step Propagate 4 a : messages ( outgoing ) child to nodes

Belief ( Binary Pseudo Variables ) Propagation : code - Assume E OIL : f) d T a 1- a I a L bp ( ) finna f . def E , OI fo x woo : noooo , r , s r d a L { } µ = b e c I f } f EE for f) ( a x : e ^ i , v u f , f- , ma ma ( µ , f. in E. OI ) µ x sum = - , f f } f EE for f) a ( x : e : , , I - prod ( µ , E , f ) out µ x = , return M

Belief µ ,E ( Binary Pseudo Variables ) Propagation : code - ( x ) def , OI f in sum : - , , bn } { y Ifs d = , . . . , T a for f , { y ya } fo : am mm am E y . , . . , , . , r s r , I , f ) ( µ E prod in y µ = - , , b e c A a I } for { he o f. man f , ; e ma , I ( ,x3[ h ] Mf =L , , . . . , a hi - ' IHH , . IT h return M

Belief ( Binary Pseudo Variables ) Propagation : code - - prod ( f ) def E , OI g. in µ x i , , , ,gN .SN } { g d MIX = , . . . , T t a for fo f , { g } fo wa : wa ma g E , . . , . , r s r , I x ) ( µ E in Sam µ = - , , b e c A a I } for k { o f. man f , ; e ma , Eh ] 17 = , h a return In

Belief If ( Binary Pseudo Variables ) Propagation : code - Assume Edges E , f ) : C x Potentials OI ] d T at I a L f. tag tag bp ( ) finna f. def E , OI fo x mm. : nooooo , r , s r d a L { } Messages µ = b e c I f } f EE for f) ( a x : e { ^ : , f , messages ( µ , f. E. OI ) in µ × from sum ^ = - , - children f f } f parents EE for f) a to I { x : e : , , I - prod ( µ , E , f ) out messages µ x = , from parents return children to µ

Belief ( Binary Pseudo Variables ) Propagation : code - prod ( f ) def E , OI out { pix µ x i - , , , I g } d .SN = , . . . , T rd F I I ' T a I C L . f. tag I } { fi f , for he fo ma ma am o : e , , mom h ] 17 = b e , c n A a for { g ,gN } { f , : g E g. , . . . , , arguers , I x ) ( µ E out der µ Sam = - , , a from . prod in return M

' T Belief I I I ( Binary Pseudo Variables ) Propagation : code f. agog - ( x ) def E , OI f out µ sum : - , , , EE }s{ { y I y , } x } d :C y f) ya = , , . . . , T T a J L ✓ I } for le { fo fr fo im ma o : hag e { , , . ;yµ=hµ ) # If ]fx=h h ] pelf ,×][ =L , yah b e , . c hi , kN If . . , , I ? Msn , f) Chu ] f , Boo for { } { y ibn : ye , . . - , , I , f ) ( µ reverse E a prod in y µ = - , , order from return µ sun in -

At Algorithm I HM Example Backward Forward Ms ) : - Model Generative Graph Factor h I s g n 9 I g 9 , ^ hi n a htt h a ~ = , . a a a n the - h Vt n - ] Forward ) ( outgoing Pass messages ( k ) Mfc !ht - ht l k ) tht Mn Mg = = - ht oh f , → → t , , . K ftlh.lt/Uh+..-sf+ll ) I th ) Mg → ↳ = , l - I -

Algorithm I HM Example Backward Forward Ms ) : - Model Generative Graph Factor fg f , f. fr ( Discrete n ) h ~ L L 2 L L L s , ( Ah ) hi ^ n htt =L Discrete ^ n ~ ga 93 g gu , . , ^ ^ " ^ Normal Hun Gu ) Ihr .hn v , , Backward Pass ( Incoming Messages ) th ft Chih ) fell ) ) → h.lk Bt Mf , Mutt ) EE = = → , fell .tn/hg.....s.n!..llMft+ioht ' EE " ' = .

/ Algorithm Example Forward Backward HMMS ) : - Forward Pass ( plh ,=h di ( k ) 1h ,=h t=| ) ) plv ) = , { ( t Aeudt > 1) . ,( p(V+lh+=hl dt h ) l ) 1 = Backward Pass ( ) Btlhl t=T 1 = ltctl Ane p++,ll p(✓t+,1ht+,=l 1h ) § B ) ) = + Marginal s " 0+14 . ,→h+l↳µgµhd↳µfphd ytlh Ptlh ) µf+ ) a =

Variables Belief Compute Propagation Marginals All for i LS 59 as I is General Marginal Form - : L r , M PHI a ftp.sxlxs < s fehecx ) Algorithm All Messages Compute : Pich variable messages } ? ! 1 x any Impute . compute incoming 2 . marginals 3 alt Compute outgoing messages .

Belief Propagation : Problem The With Loops Graph Directed b ; a , b d pca ) c , b d ) p ( b ( Pla ) pcdla CI a) ) = p 1 a ( ,

Belief Propagation : Problem The With Loops Graph Directed b ; a , b. d pca ) c b. p ( b a ) ( Pla ) dla cl d) p( ) = p 1 a ( a. fi fz Graph Factor ; b a. a , b. d Pla ) c f f a. a. 4 b ) fs ( fu( b. fz ( a. 3 d ) f. a. a) ( a) = c. A C

Belief Propagation : Problem The With Loops my fi Graph Factor f- ; z b a ago , b d) Pla c , f f ma my 4 , b ) f , ] a ) fu ( b feta f. d) Ca C as = c. , , d C Marginal pg fi d over ! f- z b wa a , b a ) Pla D , I , f. Ca ) fzla , b) I , d) falls , fzla d ) ago = c , d- I - C

Loopy Belief Propagation Step Messages Initialize 1 : He h ] he{qB µ[ f. x. f) EE ][ 1 = × a. f ' fs fz fu b a. a Step Update 2 : messages a. a. f for SCHEDULE : € a c for nelf ) e : × Update µf→× and µ×→t