Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - - PowerPoint PPT Presentation

▶
fri
SMART_READER_LITE
LIVE PREVIEW

Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - - PowerPoint PPT Presentation

Jul 28, 2023 220 likes •549 views

Algorithm Sum - product Lecture : , Loopy Belief Propagation Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 today : Feb Due I Today Variables Marginals Discrete Exact over : Marginals Goal form Compute of

slide-1
SLIDE 1 Lecture : Sum
  • product

Algorithm

,

Loopy

Belief Propagation Scribes :

Denis

McInerney Sara Taheri Homework 2 : Out today Due

Fri

I Feb
slide-2
SLIDE 2 Today : Exact Marginals
  • ver
Discrete Variables Goal : Compute Marginals
  • f
the form

pl

Xi

  • Xi
, Xj
  • Xj
) = q ,

u¥

. ,↳ ; PK ' ' × " , . .
  • Xk
  • Xie )
Assumption : All variables Xu are discrete Enables : Calculation
  • f
posterior pcxi = x ; I Xj
  • x ; )
= p ( X ;
  • x
; , Xj
  • x
j ) I p ( Xi
  • x ; )
slide-3
SLIDE 3

Example

:

Markov

Chain Basic Idea :

Rearrange

Terms in Sum Pla , b , c , d ) = peas = E E [ To lb ) : = . . E

&

l E

D

ya C c ) i = = E b
slide-4
SLIDE 4

Example

:

Markov

Chain peas = .

I E.

pca.bipcbnp.ua , " " " Sm B Rearranged Sum = I p ca I b ) zac b )
  • b
= , B Val b ) =

&

. . , pc b I c) yb C c ) 2b Col = § p ( c I d ) face ) I 1 Question : What is the Computational Complexity
  • f
computing peas ?
slide-5
SLIDE 5 Sum . product
  • n
Factor Graphs Bayesian Network ( non
  • branching )
pla , b , c , d ) = pcalblpcblc ) plc Id ) pcd ) Factor Graph ( non
  • branching )
Pla , b , c , d ) =
slide-6
SLIDE 6 Sum . product
  • n
Factor Graphs Factor Graph ( non
  • branching )
Pla , b , c , d ) =

f.

la , b ) fzlb , a ) f , ( C , d ) fa , (d) Messages : Variable to Variable pea , b , c ) = pca , b ) =
slide-7
SLIDE 7 Variable a Variable vs Factor as Variable Messages Factor Graph ( non
  • branching )
s , s , , s , s

µ

( c ) = {

fz ( b

, c) µb→f . ( b ) b → c b = µfz→c ( < )
slide-8
SLIDE 8 Sum . product
  • n
Factor Graphs Factor Graph ( non
  • branching )
Pla , b , c , d ) =

f.

la , b) fzlb , a ) f , I C , d ) fa , (d) Factor Graph ( singly
  • connected
) Pla , b , c , d ) = fi la ,b ) fz( b. c. d) f- , C c ) fuld , e ) folds
slide-9
SLIDE 9 Sum . product
  • n
Factor Graphs Factors : General torn pox ) = 174,1%1

fef

%= = { } Fx
  • =
{ } G= ( I Factor → Variable : µ , → × C x ) = I M ( Sum )

ftp.lfxfyc-neltissx3

Variable → Factor , µ × → f I x ) = M ( Product ) g

e'

needs If }
slide-10
SLIDE 10 Sum . product
  • n
Factor Graphs Factor Graph ( singly
  • connected
) < p la , b ) = f , la , b) 1- < a L µ b → e. l bl = n L L µfz→blbl

=L

Cid h Md → fzld ) = Mfs , ar (d) = [ e Yes facet = I Mf , → dldkfshd ) Messages : Factor Variable pea , b ) = f. I a. b) ? a fzlb , c. d ) f , lol fsld ) { falafel
slide-11
SLIDE 11 Sum . product
  • n
Factor Graphs melfi
  • c
pie ) = ? fu ( d. e) Iud , fala ) L > 9 S → Md → fuld ) = Mfs . → lad ) Mf , → old ) S S nel di I ^ Mfr
  • d
(d) = ! fzlb , c. d ) Mbs falls ) Me
  • fate )
Messages : General Form p CX ) =

tf off ( X, )

he ( x ) hecfl Factor → Variable : µ , → × C x ) = I M ( Sum ) FX ,
  • M
yinecflsx } Variable → Factor , µ × → f I x ) = M ( Product ) g e I needs f 3
slide-12
SLIDE 12 Belief Propagation : Compute Marginal for All Variables < , a General Form : Marginal L

4 ss

< s

rd

< s s s pkl a M µt→×l× )

f

fehecx ) Algorithm : Compute All Messages 1 . Pich any variable x 2 . Compute incoming messages 3 . Compute
  • utgoing
messages
slide-13
SLIDE 13 Belief Propagation : Compute Marginal for All Variables Step 1 : Define tree rooted at initial node d fs . a. fz a. fe , a. b c e f. a. f , e. a
slide-14
SLIDE 14 Belief Propagation i Compute Marginals for All Variables Step 2 : Walk tree to d T a 1- find leaf nodes L a I fo we f , not fo , noo s r r L d d Step 3 : Propagate messages b c e a a to parent nodes ( incoming ) u v f- , man f , ma u a Step 4 : Propagate messages to child nodes ( outgoing )
slide-15
SLIDE 15

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) Assume : E d

OIL

f) T a 1- L a I finna f . woo fo , noooo def bp ( E , OI , x ) : s r r L a d b c e µ = { } ^ a for f e I f i ( x , f) EE } : u v f- , ma f , ma

a

µ = in
  • sum
( µ ,
  • E. OI
, f. x ) a for f e f f : ( x , f) EE } : µ =
  • ut
  • prod ( µ
, E , I , x , f ) return M
slide-16
SLIDE 16

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) def in
  • sum
( µ,E , OI , f , x ) : d { y , , . . . bn } = T a

Ifs

. am f , mm fo , am for y E { y , , . . . , ya } : s r r µ = in
  • prod
( µ , E , I , y , f ) b c e a A f.man f , ma for he e {
  • ,
I } ; Mf ,x3[ h ] =L ( , , . . . , I a hi , .
  • ' IHH
IT h return M
slide-17
SLIDE 17

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) def in
  • prod (
µ , E , OI , x , f ) i d { g , , . . .

.SN

} = T a t fo wa f , wa fo , ma for g E { g , , . . .

,gN

} : s r r µ = in
  • Sam
( µ , E , I ,

g.

x ) b c e a A f.man f , ma for k e {
  • ,
I } ;

MIX

, Eh ] = 17 a h return In
slide-18
SLIDE 18

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) Assume : E

Edges

C x , f ) d OI

If

] Potentials T at L a I finna f. mm. fo , nooooo def bp ( E , OI , x ) : s r r L a d b c e µ = { } Messages ^ a for f e I f : ( x , f) EE } : f.tag f ,

tag

messages

{

µ = in
  • sum
( µ ,
  • E. OI
, f. × ) ^ from
  • children
a to parents

{

for f e f f : I x , f) EE } : messages µ =
  • ut
  • prod ( µ
, E , I , x , f ) from parents to children return µ
slide-19
SLIDE 19

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) def
  • ut
  • prod (
µ , E , OI , x , f ) i d I g , , . . .

.SN

} = T a T L C I . fi ma f , ma fo , am for he e {
  • ,
I } :

I ' I

rd

{pix

, h ] = 17 b c e n a A f.tag f ,

mom

arguers ,

{

for g E { g , , . . .

,gN

} :

F

der µ =
  • ut
  • Sam
( µ , E , I ,

g.

x ) a from in . prod return M
slide-20
SLIDE 20

Belief

Propagation :

Pseudo
  • code

( Binary

Variables ) def
  • ut
  • sum
( µ , E , OI , f , x ) : d I y , , . . . ya , } = { y :C y , f) EE }s{ x } T a T L ✓ J fo hag fr im fo , ma for le e {
  • ,
I } :

I

' T

I

{

pelf ,×][ h ] =L # If]fx=h , yah , . .;yµ=hµ) b c e hi , . . , , kN

I

I? Msn , f) Chu ] f.agog f , Boo

If

{

for ye { y , , . .
  • ibn
} : a reverse µ = in
  • prod
( µ , E , I , y , f )
  • rder
from in
  • sun
return µ
slide-21
SLIDE 21

Example

: Forward
  • Backward

Algorithm

I HM Ms ) Factor Graph Generative Model I 9 9 I s g g h , n n a a ^ htt hi . , = h ~ a a n a Vt the
  • h
n ]

Forward

Pass ( outgoing messages )

At

tht = Mn , → f ,

!ht

  • ht
= Mg t → ht ( k ) Mfc .
  • h
, l k ) K = Mg , → ↳

th

) I

ftlh.lt/Uh+..-sf+ll)

l
  • I
slide-22
SLIDE 22

Example

: Forward
  • Backward

Algorithm

I HM Ms ) Factor Graph Generative Model f. fr f , fg L L L 2 L s L h , ~ Discrete ( n ) ^ ^ n n htt hi . , =L ~ Discrete ( Ah ) g , ga 93 gu ^ ^ ^ " v , Ihr .hn Normal Hun , Gu ) Backward Pass

( Incoming

Messages ) Bt th ) =

Mf ,

→ h.lk ) = EE

ft

Chih ) Mutt , →

fell )

' = EE

fell

.tn/hg.....s.n!..llMft+ioht

. " '
slide-23
SLIDE 23

Example

: Forward
  • Backward

Algorithm

/ HMMS ) ForwardPass di ( k ) = plv , 1h ,=h ) plh ,=h ) ( t=| ) dt 1 h ) = p(V+lh+=hl

{

Aeudt . ,( l ) ( t > 1) Backward Pass Btlhl = 1 ( t=T ) B + 1h ) =

§

Ane p(✓t+,1ht+,=l ) p++,ll ) ltctl Marginal s ytlh ) a 0+14 Ptlh ) = µf+ . ,→h+l↳µgµhd↳µfphd "
slide-24
SLIDE 24 Belief Propagation i Compute Marginals for All Variables
  • General
Form : Marginal L

LS 59 as

r , < s

is

PHI a M ftp.sxlxs

I

fehecx ) Algorithm : Compute All Messages 1 . Pich any variable x 2 . compute incoming messages }

! ?

Impute

3 . Compute
  • utgoing
messages alt marginals
slide-25
SLIDE 25

Belief

Propagation :

The Problem With Loops a b Directed Graph ; pca , b , c d ) a ( = Pla) p ( b 1 a) pcdla ) p ( CI b , d )
slide-26
SLIDE 26

Belief

Propagation :

The Problem With Loops a b Directed Graph ; pca , b. c d ) a ( = Pla) p ( b 1 a ) p( dla ) p ( cl b. d)
  • a. fi

fz

Factor Graph ; a a. b f a. a. f Pla , b. c d ) 3 4 =

f.

( a) fz ( a. b ) fs (
  • a. a)
fu( b. c. d ) A C
slide-27
SLIDE 27

Belief

Propagation :

The Problem With Loops my fi f- z Factor Graph ; a ago b f ma my f Pla , b , c d) ] 4 =

f.

C as feta , b ) f , Ca , a ) fu ( b , c. d) d C pg fi

Marginal

  • ver
d ! f- z a wa b Pla , b , a ) D ago

I ,

=
  • f. Ca ) fzla , b)
I fzla , d) falls , c , d ) d-
  • I
C
slide-28
SLIDE 28

Loopy

Belief Propagation Step 1 : Initialize Messages µ[ f. × ][ h ] = 1 He
  • x. f)
EE he{qB
  • a. f '

fz

a a. b Step 2 : Update messages

fs

a. a.

fu

for f € SCHEDULE : a c for × e nelf ) : Update µ×→t and µf→×
slide-29
SLIDE 29

Loopy

Belief Propagation Step 2 : Update messages for f e SCHEDULE : Repeat until for x e he ( f ) : convergence
  • a. f '

fz

a a. b

Yi

, ...

,yµ

= he (f) I { × } f a. a. f 9 ' .
  • ' ,

SM

= necxl \{ f } 3 4 for he { o , 13 : a c µ×→tlkl = .

17µg

→ × ( k ) m h µf→x( k ) = [ ¢ f ( × . . his ,ik , , ... ,Yµ=hn ) b. ,
  • iht
' Mn µ → flkn ) Yn
slide-30
SLIDE 30

Loopy

Belief Propagation Step 2 : Update messages for f E SCHEDULE : Repeat until my fi convergence f- z for x E he ( fl : a wa b Update In , → f and Mf → × f ma ma f ] 4 a c Problems . Unlike EM updates , Loopy BP dates are not guaranteed to converge → designing a good schedule is critical to performance
slide-31
SLIDE 31 Belief Propagation : Compute Marginal for All Variables < s 7 Marginal for variable × L ( I C I

÷

a

< g pkl a M µp→×l× ) 9 fehecx ) Algorithm : Compute All Messages n . Pich any variable ×

}

can now compute z , Compute
  • utgoing
messages marginal , for 3 Compute incoming messages any x