Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - - PowerPoint PPT Presentation

▶
fri
SMART_READER_LITE
LIVE PREVIEW

Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 - - PowerPoint PPT Presentation

Feb 28, 2024 647 likes •990 views

Algorithm Sum - product Lecture : Fri Denis McInerney Scribes : Taheri Sara Out Homework 2 today : Feb Due I Today Variables Marginal Discrete Exact over : Marginal Goal form Compute of the : =n;u p( ,

slide-1
SLIDE 1 Lecture : Sum
  • product

Algorithm

Scribes :

Denis

McInerney Sara Taheri Homework 2 : Out today Due

Fri

I Feb
slide-2
SLIDE 2 Today : Exact

Marginal

  • ver
Discrete Variables Goal : Compute Marginal
  • f
the form

p(

× ; =× ; ,Xj=×j )

=×n;u§µ±

, PK ' ' × " , ' '
  • ,×K=×K )
Assumption : All variables Xw are discrete Enables : Calculation
  • f
posterior p( X ; = x ; 1 Xj = × ;) = p ( X ;=×i , Xj=×j )

÷

lj=x ;)
slide-3
SLIDE 3

Example

:

Markov

Chain Busid Idea :

Rearrange

Terms in Sum Pla , b , c , d ) = pc a 1 bl PC blc ) plc Id ) pcd ) D pcas = § , § , [ pcaiblplbic )p( cld )

ABCD

terms D= i yclb ) : = { plblc ) ydlc ) = § pca lb ) § pcblc ) ( {

pccid

) pcd ) )) =

§

pcaib ) f. ( b ) 8D " ) ÷

{

Pcc 'd ) p 'd )
slide-4
SLIDE 4

Example

:

Markov

Chain Naive Sum pca )

=b¥

, § , a£ .

.pl alblplbk

)p( cld ) ABC D 13 Rearranged Sun = [ pcaib ) ya ( b ) AB ABTBCTCD b= , B Va ( b ) =

§

.

,p(

bi e) yb ( c ) Jb ( 4 = § p ( c i a ) ya ( < ) { 1 Bc < D Question : What is the Computational Complexity
  • f
computing pca ) ?
slide-5
SLIDE 5 Sum . product
  • n
Factor Graphs Bayesian Network ( non
  • branching )
pla , b , c , d ) = pc a Ibl PC blc ) plc Id ) p( d) Factor Graph ( non
  • branching )
Pla , b. c , d ) =

f.

( a , b) fz ( b , a ) f , 1 c , d ) fa 1 d )
slide-6
SLIDE 6 Sum . product
  • n
Factor Graphs Factor Graph ( non
  • branching )
Pla , b. c , d) =

f.

( a , b) fzl b. a ) f. ( c. d) f< , (d) Messages : Variable to Variable Message : µd→c( c) . pea , b. c ) = f. ( a. b ) fzlb ,c )

§

f3(

c. d) fald ) pca , b ) =
  • f. (
a. b )

{

fzl b. c)

µa÷

Message : µc→b( b) = { fzl b. c) µd→d4
slide-7
SLIDE 7 Variable a Variable vs Factor as Variable Messages Factor Graph ( non
  • branching )
s , s , , s , s µ ( c ) = {

fz ( b

, c) µb→f . ( b ) b → c b = µfz→c ( < )
slide-8
SLIDE 8 Sum . product
  • n
Factor Graphs Factor Graph ( non
  • branching )
Pla , b , c , d ) =

f.

( a , b) fz ( b , a ) fs ( c. d) f< , (d) Factor Graph ( singly
  • connected
) Difference : more than 2 vws PC a , b , c , d ) = pen factor \ f. 1 a ,b ) fz ( b. c. d ) f 3C c) fald ,e ) fs I d )
slide-9
SLIDE 9 Sum . product
  • n
Factor Graphs Factor Graph ( singly
  • connected
) < Pla ,b ) = f. ( a. b) µb→f , lb ) 1- < < c Mb → g. ( bl = µfz→blH n < < µfz→blbl =

{

afz

1 b. c. d) µIf,klµd→µld ) " µd→fzld ) t µfa→d( dl µf←→a( d) µf , . → dldkfshd ) Mfa → a (d) = § fn ( d. e) µe→ fale ) µe→fy( 4 = 1 µd→ fz ( d ) . Messages : Factor ←s Variable Mf . → 1 ^ >

µfu→d÷

Pc a. b ) = f. ( a. b) {afz( b. c. d ) f > ( 4 fsld ) § fal d. el
slide-10
SLIDE 10 Sum . product
  • n
Factor Graphs nel f.) = { b. c. d } < p ( e) = § fu ( d. e) µd→ fala ) L s s s d Md→pyd)= Mfs . → lad ) µfz→ddl s s ne( d) =3 thfsifis a

µfz→d(

a) = § . fzlb , ' , d) Messages : General Form µb →
  • f. lb )
µc
  • fz ( c )
p ( X ) =

tl

¢f( Xf ) he ( × ) (factors that hecfl ( Variables that depend
  • n
× ) f depends an ) Factor → Variable : µ , → × ( x ) = [ ¢e( ft )

Mµy→f(

y ) ( Sum ) { Xfrx } yesynecflix } Variable → Factor : µ×→f 1 × ) = M 9 e { necx )\fzµ9→× ( × ) ( Product )
slide-11
SLIDE 11 Sum . product
  • n
Factor Graphs Factors : General torn pox ) = 174,1%1 f EF Xf ' necfl = { x : ( x , f) EE } G = ( X ,

F.

E , # ) Fx
  • held
= { f :
  • ix. f)
EE } Factor → Variable : µ , → × C x ) = I Ott Xe ) M µ → fly ) ( Sum )

necfifx

}

gene

I flesh S Variable → Factor , µ × → f I x ) = M 9 c- ' new , { fg

Mg

→ x K ) ( Product )
slide-12
SLIDE 12 Belief Propagation : Compute Marginal for All Variables < , a General Form : Marginal L

4 ss

< s

rd

< s s s pkl a M µt→×l× )

f

fehecx ) Algorithm : Compute All Messages 1 . Pich any variable x 2 . Compute incoming messages 3 . Compute
  • utgoing
messages
slide-13
SLIDE 13 Belief Propagation : Compute Marginal for All Variables Step 1 : Define tree rooted at initial node d fs . a. fz a. fe , a. b c e f. a. f , e. a
slide-14
SLIDE 14 Belief Propagation i Compute Marginals for All Variables Step 2 : Walk tree to d T a 1- find leaf nodes L a I fo we f , not fo , noo s r r L d d Step 3 : Propagate messages b c e a a to parent nodes ( incoming ) u v f- , man f , ma u a Step 4 : Propagate messages to child nodes ( outgoing )
slide-15
SLIDE 15

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) Assume : E

Edges

C x , f ) d OI [ f) Potentials T a t L a I finna f . woo fo , noooo def bp ( E , OI , x ) : s r r u a d µ = { } Messages b c e ^ a for f e I f i ( x , f) EE } : u v f- , ma f , ma

a

µ = in
  • sum
( µ ,
  • E. OI
, f. x ) a for f e I f : ( x , f) EE } : µ =
  • ut
  • prod ( µ
, E , I ,
  • x. f )
return M
slide-16
SLIDE 16

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) def in . sum ( µ,E , OI , f. × ) : d { y , , . . . yµ } = { y :( y , f) EE }s{ x } T at fs . ↳a• f. ↳. fe , ... for ye { y , , ... ,yµ } : 7 t r µ = in
  • prod
( µ , E. ¢ , y ,f ) b C l ( compute messages n ^ from children ) f.may f , ... for k e { a , 1 } ; µ[ f. × ][k]=[ § If](x=h , yihi , ;yµ=hµ) a h ' ' " ' ' ↳ Mn µ[ yn ,f][ hu ] return µ ( return updated messages to parent )
slide-17
SLIDE 17

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) def in
  • prod (
µ

,E

, OI , x , f ) : d {

g.

, ...

,gµ

} = { g :C x. g) EE }\{ f } T at fs . a. f. a. fe , a. for g e {

g.

, ...

,gµ

} : 7 t r µ = in
  • Sam
( µ , E. & ,

g.

x ) b c e x x f.may f , ... for he { 0,1 } ; µ[ x

,f][k]

= Mµ[ gn ,

#

( h ] a return µ
slide-18
SLIDE 18

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) Assume : E

Edges

C x , f ) d OI

If

] Potentials T at L a I finna f. mm. fo , nooooo def bp ( E , OI , x ) : s r r L a d b c e µ = { } Messages ^ a for f e I f : ( x , f) EE } : f.tag f ,

tag

messages

{

µ = in
  • sum
( µ ,
  • E. OI
, f. × ) ^ from
  • children
a to parents

{

for f e f f : I x , f) EE } : messages µ =
  • ut
  • prod ( µ
, E , I , x , f ) from parents to children return µ
slide-19
SLIDE 19

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) def
  • ut
  • prod (
µ ,E , # , x , f ) : d {

g.

, ...

,gµ

} = { g :C x. g) EE }\{ f } T at L a I . Incoming fs . a. f. a. fe , a. for Lee { 0,1 } : Message

a .

r ,

{

a µ[ x

,f][k]

= Mµ[ gn ,

#

( h ] b c e x x f.art f ,

.at

yguense§ for g e {

g.

, ...

,gµ

} :

Tr

der µ =
  • ut
  • Sam
( µ , E. & ,

g.

x ) a from in . sum return µ
slide-20
SLIDE 20

Belief

Propagation : Pseudo
  • code

( Binary

Variables ) def
  • ut
. sum ( µ ,E , OI , f. x ) : d { y , , . . . yµ } = { y :( g. f) EE }s{ x } T ^ t L ~ t fs . b. f. ... fo , ... for Lee {
  • ,
i } ;

I T

{

µ[f ,×][h]=[ # If](x=h , yih , ;yµ=hµ) b c e hi , . . , ,hµ

I

Mµ[ yn ,f]( hu ]
  • f. arg
f ]↳.

(

|{ for

ye { y , ... . ibn } : a reverse µ = in
  • prod
( µ , E. ¢ , y ,f )
  • rder
fun in
  • sum
return µ
slide-21
SLIDE 21 Belief Propagation : Compute Marginal for All Variables < s 7 Marginal for variable × L ( I C I

÷

a

< g pkl a M µp→×l× ) 9 fehecx ) Algorithm : Compute All Messages n . Pich any variable ×

}

can now compute z , Compute
  • utgoing
messages marginal , for 3 Compute incoming messages any x
slide-22
SLIDE 22 Example : Forward
  • Backward

Algorithm

/ HMMS ) Factor Graph Generative Model f ' ,

gk

,gk ,

gf

' , h , ~ Discrete ( n ) ^ ^ ^ ^ htlht . , =L ~ Discrete ( Au ) 9 , 92 93 94 4 ^ ^ ^ Vtlht=h ~ Normal /µu,6u) '

y

9+(4

,h+=hl

Forward

Pass ( outgoing messages ) = pivtl h+=h )

dtlhl

=µn+→f↳,lht=H =

µg+→htM

µtt . , → htlhl K = µg+→↳M [

ftlhill

µn+ . ,→t+ll ) l = ' ↳ Aeu is d+ . , 1 l )
slide-23
SLIDE 23 Example : Forward
  • Backward

Algorithm

/ HMMS ) Factor Graph Generative Model f , fz fz fg < < < 2 ( s < hi ~ Discrete ( n ) ^ ^ ^ ^ htlht . , =L ~ Discrete ( Ah ) 9 , 92 93 94 ^ ^ ^ ^ ✓ , |h+=h ~ Normal ( µu,6u ) Backward Pass ( Incoming Messages ) Btlh

?

=

Mf+→h+lH

=

[

ft ( lih ) Mn++ , → fell ) = e§ ,

ftllih

)µg++,→n

.

!llµft+i→ht+

. " ' Ahl pi ✓ + + , 1 nt+,=l ) Bttll )
slide-24
SLIDE 24 Example : Forward
  • Backward

Algorithm

/ HMMS ) ForwardPass di ( k ) = plv , 1h ,=h ) plh ,=h ) ( t=| ) dt 1 h ) = p(V+lh+=hl

{

Aeudt . ,( l ) ( t > 1) Backward Pass Btlhl = 1 ( t=T ) B + 1h ) =

§

Ane p(✓t+,1ht+,=l ) p++,ll ) ltctl Marginal s ytlh ) a 0+14 Ptlh ) = µf+ . ,→h+l↳µgµhd↳µfphd "
slide-25
SLIDE 25

Belief

Propagation : The Problem With Loops a b Directed Graph ; pca , b , c d ) a ( = Pla) p ( b 1 a) pcdla ) p ( CI b , d )
slide-26
SLIDE 26

Belief

Propagation : The Problem With Loops a b Directed Graph ; pca , b. c d ) a ( = Pla) p ( b 1 a ) p( dla ) p ( cl b. d)
  • a. fi

fz

Factor Graph ; a a. b f a. a. f Pla , b. c d ) 3 4 =

f.

( a) fz ( a. b ) fs (
  • a. a)
fu( b. c. d ) A C
slide-27
SLIDE 27

Belief

Propagation : The Problem With Loops
  • a. Fi

fz

Factor Graph ; a a. b f a. a. f Pla , b. c d ) 3 4 =

f.

( a) fz ( a. b ) fs (
  • a. a)
fu( b. c. d ) A C
  • a. fi
Marginal
  • ver
d !

fz

[

( a , b. c) a a. b
  • Pla
, b) c) D a.

E

=

f.

(a)

fzl a. b)

[ f3l a. d) falls , c. d) d=l C
slide-28
SLIDE 28

Loopy

Belief Propagation Step 1 : Initialize Messages µ[ f. × ][ h ] = 1 He
  • x. f)
EE he{qB
  • a. f '

fz

a a. b Step 2 : Update messages

fs

a. a.

fu

for f € SCHEDULE : a c for × e nelf ) : Update µ×→t and µf→×
slide-29
SLIDE 29

Loopy

Belief Propagation Step 2 : Update messages for f e SCHEDULE : Repeat until for x e he ( f ) : convergence
  • a. f '

fz

a a. b

Yi

, ...

,yµ

= he (f) I { × } f a. a. f 9 ' .
  • ' ,

SM

= necxl \{ f } 3 4 for he { o , 13 : a c µ×→tlkl = .

17µg

→ × ( k ) m h µf→x( k ) = [ ¢ f ( × . . his ,ik , , ... ,Yµ=hn ) b. ,
  • iht
' Mn µ → flkn ) Yn
slide-30
SLIDE 30

Loopy

Belief Propagation Step 2 : Update messages for f E SCHEDULE : Repeat until my fi convergence f- z for x E he ( fl : a wa b Update In , → f and Mf → × f ma ma f ] 4 a c Problems . Unlike EM updates , Loopy BP dates are not guaranteed to converge → designing a good schedule is critical to performance
slide-31
SLIDE 31 Sum . product
  • n
Factor Graphs Factor Graph ( singly
  • connected
) < Pla ,b ) = f. ( a. b) µb→f , lb ) 1- < < c Mb → g. ( bl = µfz→blH n < < µfz→blbl =

{

afz

1 b. c. d) µIf,klµd→µld ) " µd→fzld ) t µfa→d( dl µf←→a( d) µf , . → dldkfshd ) Mfa → a (d) = § fn ( d. e) µe→ fale ) µe→fy( 4 = 1 µd→ fz ( d ) . Messages : Factor ←s Variable Mf . → 1 ^ >

µfu→d÷

Pc a. b ) = f. ( a. b) {afz( b. c. d ) f > ( 4 fsld ) § fal d. el
slide-32
SLIDE 32 Sum . product
  • n
Factor Graphs nel f.) = { b. c. d } < p ( e) = § fu ( d. e) µd→ fala ) L s s s d Md→pyd)= Mfs . → lad ) µfz→ddl s s ne( d) =3 thfsifis a

µfz→d(

a) = § . fzlb , ' , d) Messages : General Form µb →
  • f. lb )
µc
  • fz ( c )
p ( X ) =

tl

¢f( Xf ) he ( × ) (factors that hecfl ( Variables that depend
  • n
× ) f depends an ) Factor → Variable : µ , → × ( x ) = [ ¢e( ft )

Mµy→f(

y ) ( Sum ) { Xfrx } yesynecflix } Variable → Factor : µ×→f 1 × ) = M 9 e { necx )\fzµ9→× ( × ) ( Product )
slide-33
SLIDE 33 Sum . product
  • n
Factor Graphs Factor Graph ( singly
  • connected
) µ f. → a (a) = µ f 3 → c 1 c ) = f 3 ( c ) T § fi ( a ,b ) µtz→b 1 b ) µ< → fzl c) = µf3→dc ) . L < < s r µfa→d( dl = § fold ,e ) µe→fde) plan →fzld ) < = µk , → dldl µfs→dl al µ f5→d( dl = fsld ) n µfz→b = ? a fz 1 b. c. d) µ< → fd4µd→f , 1 d )

µd→f÷

µf , → ( d ) µfa→d ( d) Messages : Factor ← > Variable
  • pca
, b ) = f. ( a. b ) ?afz( b. c. d ) f > ( ol fsld ) { fal d. el