Belief Propagation Probabilistic Graphical Models Sharif University - PowerPoint PPT Presentation

Sum-Product: Message Passing Belief Propagation Probabilistic Graphical Models Sharif University of Technology Spring 2017 Soleymani

All single-node marginals  If we need the full set of marginals, repeating elimination algorithm for each individual variable is wasteful  It does not share intermediate terms  Message-passing algorithms on graphs (messages are the shared intermediate terms).  sum-product and junction tree  upon convergence of the algorithms, we obtain marginal probabilities for all cliques of the original graph. 2

Tree  Sum-product work only in trees (and we will see it also work on tree-like graphs) Directed tree Undirected tree All nodes have one A unique path between parent expect to the root any pair of nodes 3

Parameterization  Consider a tree 𝒰(𝒲, ℰ)  Potential functions: 𝜚 𝑦 𝑗 , 𝜚(𝑦 𝑗 , 𝑦 𝑘 ) 𝑄 𝒚 = 1 𝜚 𝑦 𝑗 𝜚 𝑦 𝑗 , 𝑦 𝑘 𝑎 𝑗∈𝒲 𝑗,𝑘 ∈ℰ  In directed graphs: 𝑄 𝒚 = 𝑄(𝑦 𝑠 ) 𝑄 𝑦 𝑘 |𝑦 𝑗 𝑗,𝑘 ∈ℰ  𝜚 𝑦 𝑠 = 𝑄(𝑦 𝑠 ) , ∀𝑗 ≠ 𝑠, 𝜚 𝑦 𝑗 = 1  𝜚 𝑦 𝑗 , 𝑦 𝑘 = 𝑄(𝑦 𝑘 |𝑦 𝑗 ) ( 𝑦 𝑗 is the parent of 𝑦 𝑘 )  𝑎 = 1  When we have evidence on variable 𝑦 𝑗 as 𝑦 𝑗 = 𝑦 𝑗 we replace 𝑦 𝑗 in all factors in which it appears by 𝑦 𝑗 4

Sum-product: elimination view  Query node 𝑠  Elimination order: inverse of the topological order  Starts from leaves and generates elimination cliques of size at most two  Elimination of each node can be considered as message- passing (or Belief Propagation):  Elimination on trees is equivalent to message passing along tree branches  Instead of the node elimination, we preserve the node and compute a message from it to its parent  This message is equivalent to the factor resulted from the elimination of that node and all of the nodes in its subtree 5

Messages root … Message that 𝑘 sends to 𝑗 6

Messages on a tree  Messages can be reused to find probabilities on different query variables.  Messages on the tree provide a data structure for caching computations. 𝑌 2 We need 𝑛 32 (𝑦 2 ) to find both 𝑌 1 𝑌 3 𝑄(𝑌 1 ) and 𝑄(𝑌 2 ) 𝑌 4 𝑌 5 7

Messages and marginal distribution Message that X 𝑘 sends to 𝑌 𝑗 𝑛 𝑘𝑗 𝑦 𝑗 = 𝜚 𝑦 𝑘 𝜚 𝑦 𝑗 , 𝑦 𝑘 𝑛 𝑙𝑘 (𝑦 𝑘 ) 𝑦 𝑘 𝑙∈𝒪(𝑘)\𝑗 a function of only 𝑦 𝑗 𝑞 𝑦 𝑠 ∝ 𝜚 𝑦 𝑠 𝑛 𝑙𝑠 (𝑦 𝑠 ) 𝑙∈𝒪(𝑠) 8

Messages and marginal: Example 𝑛 12 𝑦 2 = 𝜚 𝑦 1 𝜚 𝑦 1 , 𝑦 2 𝑦 1 𝑞 𝑦 2 ∝ 𝜚 𝑦 2 𝑛 12 (𝑦 2 )𝑛 32 (𝑦 2 )𝑛 42 (𝑦 2 ) 9

Computing all node marginals  We can compute over all possible elimination order (generating only elimination cliques of size 2) by only computing all possible messages ( 2 ℰ )  T o allow all nodes can be the root, we just need to compute 2 ℰ messages  Messages can be reused  Instead of running the elimination algorithm 𝑂 times  Dynamic programming approach  2-Pass algorithm that saves and uses messages  A pair of messages (one for each direction) have been computed for each edge 10

Messages required to compute all node marginals 11

A two-pass message-passing schedule  Arbitrarily pick a node as the root  First pass: starting at the leaves and proceeds inward  each node passes a message to its parent.  continues until the root has obtained messages from all of its adjoining nodes.  Second pass: starting at the root and passing the messages back out  messages are passed in the reverse direction.  continues until all leaves have received their messages. 12

Asynchronous two-pass message-passing First pass: upward 13 Second pass: downward

Sum-product algorithm: example 𝑛 21 (𝑦 1 ) 𝑛 21 (𝑦 1 ) 14

Sum-product algorithm: example 𝑛 21 (𝑦 1 ) 15

Parallel message-passing  Message-passing protocol: a node can send a message to a neighboring node when and only when it has received messages from all of its other neighbors  Correctness of parallel message-passing on trees  The synchronous implementation is “ non-blocking ”  Theorem: The message-passing guarantees obtaining all marginals in the tree 16

Parallel message passing: Example 17

Tree-like graphs  Sum-product message passing idea can also be extended to work in tree-like graphs (e.g., polytrees) too.  Although the undirected marginalized graphs resulted from polytrees are not tree, the corresponding factor graph is a tree Polytree Moralized Nodes can have graph Factor graph multiple parents 18

Recall: Factor graph 𝜚 𝑦 1 , 𝑦 2 , 𝑦 3 = 𝑔 𝑦 1 , 𝑦 2 , 𝑦 3 𝜚 𝑦 1 , 𝑦 2 , 𝑦 3 = 𝑔 𝑏 (𝑦 1 , 𝑦 2 )𝑔 𝑐 (𝑦 1 , 𝑦 3 )𝑔 𝑑 (𝑦 2 , 𝑦 3 ) 19

Sum-product on factor trees  Factor tree: a factor graph with no loop  Two types of messages:  Message that flows from variable node 𝑗 to factor node 𝑡 : 𝑤 𝑗𝑡 𝑦 𝑗 = 𝜈 𝑢𝑗 (𝑦 𝑗 ) 𝑢∈𝒪 𝑗 −{s}  Message that flows from factor node 𝑡 to variable node 𝑗 : 𝜈 𝑡𝑗 𝑦 𝑗 = 𝑔 𝑡 𝒚 𝒪(𝑡) 𝑤 𝑘𝑡 (𝑦 𝑘 ) 𝒚 𝒪 𝑡 −{𝑗} 𝑘∈𝒪 𝑡 −{𝑗} 20

Sum-product on factor trees  The introduced message-passing schedule for trees can also be used on factor trees  When the messages from all the neighbors of a node is received, the marginal probability will be: 𝑄 𝑦 𝑗 ∝ 𝜈 𝑡𝑗 𝑦 𝑗 𝑡∈𝒪 𝑗 𝑄 𝑦 𝑗 ∝ 𝑤 𝑗𝑡 (𝑦 𝑗 )𝜈 𝑡𝑗 (𝑦 𝑗 ) 𝑡 ∈ 𝒪 𝑗 𝑡 is a factor node that is neighbor of 𝑌 𝑗 21

The relation between sum-product on factor trees and sum-product on undirected trees  Relation of 𝑛 messages of sum-product algorithm for undirected trees and 𝜈 messages of sum-product algorithm for factor trees 𝜈 𝑡𝑗 𝑦 𝑗 = 𝑔 𝑡 𝒚 𝒪(𝑡) 𝑤 𝑘𝑡 (𝑦 𝑘 ) 𝒚 𝒪 𝑡 −{𝑗} 𝑘∈𝒪 𝑡 −{𝑗} = 𝜚(𝑦 𝑗 , 𝑦 𝑘 )𝑤 𝑘𝑡 (𝑦 𝑘 ) 𝑦 𝑘 = 𝜚(𝑦 𝑗 , 𝑦 𝑘 ) 𝜈 𝑢𝑘 (𝑦 𝑘 ) 𝑦 𝑘 𝑢∈𝒪 𝑘 −{s} = 𝜚(𝑦 𝑗 )𝜚(𝑦 𝑗 , 𝑦 𝑘 ) 𝜈 𝑢𝑘 (𝑦 𝑘 ) 𝑢∈𝒪 ′ 𝑘 −{s} 𝑦 𝑘 𝒪 ′ 𝑘 = 𝒪 𝑘 − {factor corresponding to 𝜚(𝑦 𝑘 )} 22

Example 23

References  D. Koller and N. Friedman, “ Probabilistic Graphical Models: Principles and Techniques ” , MIT Press, 2009, Chapter 10.  M.I. Jordan, “ An Introduction to Probabilistic Graphical Models ” , Chapter 4. 24