The Elimination Algorithm Chris Williams School of Informatics, - PowerPoint PPT Presentation

The Elimination Algorithm Chris Williams School of Informatics, University of Edinburgh October 2009 1 / 14

Overview Inference in Belief Networks Elimination in an Example Network The Elimination Algorithm Evidence Potentials Graph Elimination Reconstituted Graph Reading: Jordan chapter 3 2 / 14

Inference in Belief Networks Partition the random variables x into three disjoint subsets x E , x F and x R . We wish to compute the posterior p ( x F | x E ) over the query nodes x F This involves conditioning on the evidence nodes x E and summing out (integrating out) the hidden nodes x R If the joint distribution is simply a huge table this is trivial: select the appropriate indices in the columns corresponding to x E , sum over the columns corresponding to x R , and renormalize the resulting table over x F But what if the distribution is represented by a directed graphical model? 3 / 14

An Example Network X 4 X 2 X 6 X 1 X X 5 3 � ′ p ( x 1 | x 6 ) = p ( x 1 , x 6 ) / p ( x 6 ) = p ( x 1 , x 6 ) / p ( x 1 , x 6 ) x ′ 1 4 / 14

� � � � p ( x 1 , x 6 ) = p ( x 1 ) p ( x 2 | x 1 ) p ( x 3 | x 1 ) p ( x 4 | x 2 ) p ( x 5 | x 3 ) p ( x 6 | x 2 , x 5 ) x 2 x 3 x 4 x 5 � � � � = p ( x 1 ) p ( x 2 | x 1 ) p ( x 3 | x 1 ) p ( x 4 | x 2 ) p ( x 5 | x 3 ) p ( x 6 | x 2 , x 5 ) x 2 x 3 x 4 x 5 � � � = p ( x 1 ) p ( x 2 | x 1 ) p ( x 3 | x 1 ) m 5 ( x 2 , x 3 ) p ( x 4 | x 2 ) x 2 x 3 x 4 � � = p ( x 1 ) p ( x 2 | x 1 ) m 4 ( x 2 ) p ( x 3 | x 1 ) m 5 ( x 2 , x 3 ) x 2 x 3 � = p ( x 1 ) p ( x 2 | x 1 ) m 4 ( x 2 ) m 3 ( x 1 , x 2 ) x 2 = p ( x 1 ) m 2 ( x 1 ) using elimination ordering ( 6 , 5 , 4 , 3 , 2 , 1 ) 5 / 14

Notes on the example x 6 means that x 6 is fixed to a specific value m 5 ( x 2 , x 3 ) = � x 5 p ( x 5 | x 3 ) p ( x 6 | x 2 , x 5 ) etc Note that m 4 ( x 2 ) = 1; why? Key idea 1: push sums inside products Key idea 2: cache subexpressions 6 / 14

Evidence Potentials Elimination uses a book-keeping trick, evidential potentials � g ( x i ) = g ( x i ) δ ( x i , x i ) x i This trick allows us to treat conditioning in the same way as marginalization 7 / 14

Elimination Algorithm, part I ELIMINATE (G,E,F) INITIALIZE (G,F) EVIDENCE (E) UPDATE (G) NORMALIZE (F) INITIALIZE (G,F) choose an ordering O such that F appears last for each node X i in V place potential p ( x i | parents i ) on the active list end for 8 / 14

Elimination Algorithm, part II EVIDENCE (E) for each i in E place potential δ ( x i , x i ) on the active list end for UPDATE (G) for each i in O find all potentials in the active list that reference x i and remove them from the active list Let φ i ( x T i ) denote the product of these potentials Let m i ( x S i ) = � x i φ i ( x T i ) Place m i ( x S i ) on the active list end for NORMALIZE (F) p ( x F | x E ) ← φ F ( x F ) / � x F φ F ( x F ) 9 / 14

Graph Elimination Consider first undirected graphs with p E ( x ) = 1 ψ E � X C ( x c ) Z C where the product is over cliques Start with an elimination ordering O At each step the algorithm eliminates the next node in O , where “eliminate” means removing the node from the graph and connecting the (remaining) neighbours of the node 10 / 14

Moralization There is one additional complexity for directed graphs: parents may not be explicitly connected, but are involved in the same potential function Thus to think of the ELIMINATION algorithm as node removal, we must first connect all the parents of every node and drop the directions of the links: this is known as “moralization” 11 / 14

Graphically the stages for the example are: X 4 X 4 X 4 X 2 X 2 X X 6 2 X 1 X X 6 1 X 1 X X 5 X X 5 3 3 X X 5 (b) (c) 3 X 4 X 2 X 2 X 1 X 1 X 2 X 1 X X 3 3 (d) (e) (f) 12 / 14

Reconstituted Graph The reconstituted graph is the graph whose edge set includes all the original edges as well as any new edges created during elimination In fact, the reconstituted graph is a triangulated graph, see forthcoming lecture on the junction tree algorithm. Elimination is a simple algorithm for triangulating a graph 13 / 14

X 4 X 2 X 6 X 1 X X 5 3 The reconstituted graph; X 2 − X 5 is added via moralization, and X 2 − X 3 is added when eliminating X 5 14 / 14