Graphical models and inference III Milos Hauskrecht milos@pitt.edu - PDF document

CS 3750 Machine Learning Lecture 4 Graphical models and inference III Milos Hauskrecht milos@pitt.edu 5329 Sennott Square, x4-8845 http://www.cs.pitt.edu/~milos/courses/cs3750-Spring2020/ CS 3750 Advanced Machine Learning Clique trees BBNs and MRF can be converted to clique tress: • Optimal clique trees can support efficient inferences MRF (graph) Clique tree A G A A C G G H G B C F C F H C B F B D – . E E D Note: a clique tree = a tree decomposition of an MRF = = junction tree CS 3750 Advanced Machine Learning 1

Algorithms for clique trees Properties • A tree with nodes corresponding to sets of variables • Satisfies: a running intersection property • For every v  G : the nodes in T that contain v form a connected subtree. Inference algorithms for the clique trees exist: • inference complexity is determined by the width of the tree CS 3750 Advanced Machine Learning VE on the Clique tree • Variable Elimination on the clique tree – works on factors • Makes factor a data structure – Sends and receives messages • Graph representing a set of factors, each node i is associated with a subset ( cluster, clique ) C i . CS 3750 Advanced Machine Learning 2

C Clique trees I D • Example clique tree G S L K J H C,D G,I,D G,S,I G,J,S,L S,K H,G,J CS 3750 Advanced Machine Learning Clique tree properties   • S C C Sepset ij i j – separation set (sepset) : Variables X on one side of a sepset are separated from the variables Y on the other side in the factor graph given variables in S • Running intersection property – if C i and C j both contain variable X, then all cliques on the unique path between them also contain X CS 3750 Advanced Machine Learning 3

C Clique trees I D • Running intersection: G E.g. Cliques involving G form S a connected subtree. L K J H C,D G,I,D G,S,I G,J,S,L S,K H,G,J CS 3750 Advanced Machine Learning C Clique trees I D   • S C C Sepsets: G ij i j • Variables X on one side of a sepset are S separated from the variables Y on the L other side given variables in S K J H C,D G,I,D G,S,I G,I D G,S Sepsets G,J,S,L S,K S G,J H,G,J CS 3750 Advanced Machine Learning 4

C Clique trees I D Initial potentials : G Assign factors to cliques and multiply them. S C,D G,I,D G,S,I L K  J 0 ( , , ) G S I H   0 0 ( C , D ) ( G , I , D ) G,J,S,L S,K  0  0 ( G , J , S , L ) i  ( 0 K S , ) H,G,J  0 ( , , ) H G J p ( C , D , G , I , S , J , L , K , H )        0 0 0 0 0 0 ( C , D ) ( G , I , D ) ( G , S , I ) ( G , J , S , L ) ( S , K ) ( H , G , J ) CS 3750 Advanced Machine Learning Message Passing VE • Query for P(J) C Message sent – Eliminate C: I D from [C,D] to [G,I,D] G C,D G,I,D G,S,I     0 S ( D ) [ C , D ]  1 1 D C L K J H G,J,S,L S,K Message received at [G,I,D] -- H,G,J [G,I,D] updates:      0 [ G , I , D ] ( D ) [ G , I , D ] 2 1 2 CS 3750 Advanced Machine Learning 5

Message Passing VE • Query for P(J) C     – ( G , I ) [ G , I , D ] Eliminate D: I 2 2 D D Message sent G C,D G,I,D G,S,I from [G,I,D]  S  to [G,S,I] G,I D L K Message received J H at [G,S,I] -- G,J,S,L SK [G,S,I] updates: H,G,J      0 [ G , S , I ] ( G , I ) [ G , S , I ] 3 2 3 CS 3750 Advanced Machine Learning Message Passing VE • Query for P(J)     – ( G , S ) [ G , S , I ] C Eliminate I: 3 3 I I Message sent D C,D G,I,D G,S,I from [G,S,I]  G  to [G,J,S,L] G,I D i G,S S L K G,J,S,L S,K Message received J H at [G,J,S,L] -- [G,J,S,L] updates: H,G,J !      0 [ G , J , S , L ] ( G , S ) [ G , J , S , L ] 4 3 4 [G,J,S,L] is not ready ! CS 3750 Advanced Machine Learning 6

Message Passing VE • Query for P(J)     – C Eliminate H: ( G , J ) [ H , G , J ] 4 5 H I D Message sent C,D G,I,D G,S,I from [H,G,J] G   to [G,J,S,L] G,I i G,S D S L K G,J,S,L S,K J H h G,J H,G,J        0 [ G , J , S , L ] ( G , S ) ( G , J ) [ G , J , S , L ] 4 3 4 4 And … CS 3750 Advanced Machine Learning Message Passing VE • Query for P(J)     0 ( S ) [ S , K ] – C Eliminate K: 6 K I D C,D G,I,D G,S,I Message sent G   from [S,K] G,I i G,S D S to [G,J,S,L]  L S K G,J,S,L S,K J All messages H received at [G,J,S,L] h G,J [G,J,S,L] updates: H,G,J          0 [ G , J , S , L ] ( G , S ) ( G , J ) ( S ) [ G , J , S , L ] 4 3 4 6 4 And calculate P(J) from it by summing out G,S,L CS 3750 Advanced Machine Learning 7

Message Passing VE • [G,J,S,L] clique potential • … is used to finish the inference C,D G,I,D G,S,I   D G,I i G,S G,J,S,L S,K  S G,J h H,G,J CS 3750 Advanced Machine Learning Message passing VE • Often, many marginals are desired – Inefficient to re-run each inference from scratch – One distinct message per edge & direction • Methods : – Compute ( unnormalized ) marginals for any vertex (clique) of the tree      – Results in a calibrated clique tree i j   C S C S i ij j ij • Recap: three kinds of factor objects – Initial potentials, final potentials and messages CS 3750 Advanced Machine Learning 8

Two-pass message passing VE • Chose the root clique, e.g. [S,K] • Propagate messages to the root C,D G,I,D G,S,I   D G,I i G,S G,J,S,L S,K  S G,J h H,G,J CS 3750 Advanced Machine Learning Two-pass message passing VE • Send messages back from the root   D G,I C,D G,I,D G,S,I G,S h G,J,S,L S,K  S G,J i H,G,J CS 3750 Advanced Machine Learning 9

Message Passing: BP • Belief propagation – A different algorithm but equivalent to variable elimination in terms of the results – Asynchronous implementation CS 3750 Advanced Machine Learning Message Passing: BP • Each node: multiply all the messages and divide by the one that is coming from node we are sending the message to – Clearly the same as VE       i k i        C S  C S k N ( i )   i ij i ij     i j k i     C S k N ( i ) \ j j i j i i ij – Initialize the messages on the edges to 1 CS 3750 Advanced Machine Learning 10

Message Passing: BP  2  1 , 3 A,B B,C C,D B C    0 0 0 ( A , B ) ( B , C ) ( C , D ) 1 2 3         0 Store the last message ( B , C )  2 3 2   on the edge and divide B each passing message by the last stored.           0 0 0 2 3 ( C , D ) ( C , D ) ( C , D ) ( B , C ) 3 3  3 2 2 , 3 B           0 ( B , C ) New message  2 , 3 2 3 2   B CS 3750 Advanced Machine Learning Message Passing: BP         0 ( B , C ) 2 , 3 2   B A,B B,C C,D B C    0 0 ( A , B ) ( B , C ) ( C , D ) 1 2 3         0 0 0 ( C , D ) ( C , D ) ( B , C ) ( C , D ) Store the last message 3 3 2 3 2 , 3 B on the edge and divide     each passing message     ( C , D )   by the last stored. 3 2 3   D   0 ( B , C )                 0 0 0 0 3 2 2 ( B , C ) ( B , C ) ( C , D ) ( C ) ( B , C ) ( C , D ) 2 2   3 2 , 3 2 3 ( C ) ( C ) 2 , 3 2 , 3 D D                0 0 New message ( C , D ) ( C , D ) ( B , C )   2 , 3 3 2 3 3 2   D D B CS 3750 Advanced Machine Learning 11