Probabilistic Graphical Models CMSC 678 UMBC Probabilistic - - PowerPoint PPT Presentation

Probabilistic Graphical Models

CMSC 678 UMBC

Probabilistic Graphical Models

A graph G that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂

Probabilistic Graphical Models

A graph G that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂)

Probabilistic Graphical Models

A graph G that represents a probability distribution

• ver random variables 𝑌1, … , 𝑌𝑂

Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂) Vertices ↔ random variables Edges show dependencies among random variables

Probabilistic Graphical Models

A graph G that represents a probability distribution

• ver random variables 𝑌1, … , 𝑌𝑂

Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂) Vertices ↔ random variables Edges show dependencies among random variables Two main flavors: directed graphical models and undirected graphical models

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗

Benefit: read the independence properties are transparent

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗 A graph/joint distribution that follows this is a

Bayesian network

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

“parents of” topological sort

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

𝑞 𝑦1, 𝑦2, 𝑦3, 𝑦4, 𝑦5 = ???

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, 𝑦4, 𝑦5 = 𝑞 𝑦1 𝑞 𝑦3 𝑞 𝑦2 𝑦1, 𝑦3 𝑞 𝑦4 𝑦2, 𝑦3 𝑞(𝑦5|𝑦2, 𝑦4)

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

exact inference in general DAGs is NP-hard inference in trees can be exact

Directed Graphical Model Notation

𝑦1 𝑦4 𝑦3 5 𝑦2

Shaded nodes are

• bserved R.V.s

Unshaded nodes are unobserved (latent) R.V.s

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Y X Y X Z Y

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Z Y X Z Y X Z Y

• bserving Z blocks

the path from X to Y

• bserving Z blocks

the path from X to Y not observing Z blocks the path from X to Y

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Z Y X Z Y X Z Y

• bserving Z blocks

the path from X to Y

• bserving Z blocks

the path from X to Y not observing Z blocks the path from X to Y 𝑞 𝑦, 𝑧, 𝑨 = 𝑞 𝑦 𝑞 𝑧 𝑞(𝑨|𝑦, 𝑧) 𝑞 𝑦, 𝑧 = ෍

𝑞 𝑦 𝑞 𝑧 𝑞(𝑨|𝑦, 𝑧) = 𝑞 𝑦 𝑞 𝑧

Markov Blanket

x Markov blanket of a node x is its parents, children, and children's parents

𝑞 𝑦𝑗 𝑦𝑘≠𝑗 = 𝑞(𝑦1, … , 𝑦𝑂) ∫ 𝑞 𝑦1, … , 𝑦𝑂 𝑒𝑦𝑗 = ς𝑙 𝑞(𝑦𝑙|𝜌 𝑦𝑙 ) ∫ ς𝑙 𝑞 𝑦𝑙 𝜌 𝑦𝑙 ) 𝑒𝑦𝑗

factor out terms not dependent on xi

factorization

• f graph

= ς𝑙:𝑙=𝑗 or 𝑗∈𝜌 𝑦𝑙 𝑞(𝑦𝑙|𝜌 𝑦𝑙 ) ∫ ς𝑙:𝑙=𝑗 or 𝑗∈𝜌 𝑦𝑙 𝑞 𝑦𝑙 𝜌 𝑦𝑙 ) 𝑒𝑦𝑗

the set of nodes needed to form the complete conditional for a variable xi

(in this example, shading does not show

• bserved/latent)

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

Naïve Bayes argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍)

likelihood prior

argmax𝑍𝑞 𝑍 𝑌)

Apply Bayes rule and take logs

Naïve Bayes argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍) argmax𝑍𝑞 𝑍 𝑌)

Apply Bayes rule and take logs

Represent X is a D-dimensional vector (of features): 𝑌 = (𝑌1, 𝑌2, 𝑌3, … , 𝑌𝐸)

Naïve Bayes argmax𝑍 log 𝑞 𝑌 𝑍) + log 𝑞(𝑍) argmax𝑍𝑞 𝑍 𝑌)

argmax𝑍 ෍

𝑘=1 𝐸

log 𝑞(𝑌

𝑘|𝑍) + log 𝑞(𝑍)

Apply Bayes rule and take logs Naively generate each “feature”

• f X, conditioned on Y

The Bag of Words Representation

Adapted from Jurafsky & Martin (draft)

The Bag of Words Representation

Adapted from Jurafsky & Martin (draft)

The Bag of Words Representation

25

Adapted from Jurafsky & Martin (draft)

Bag of Words Representation

γ( )=c

seen 2 sweet 1 whimsical 1 recommend 1 happy 1 ... ...

classifier classifier

Adapted from Jurafsky & Martin (draft)

Naïve Bayes: A Generative Story

Generative Story

𝜚 = distribution over 𝐿 labels for label 𝑙 = 1 to 𝐿:

global parameters

𝜄𝑙 = generate parameters

𝑞 𝑦𝑗𝑘 𝑧 = 𝑙) 𝑞(𝑧 = 𝑙)

෍

𝑘=1 𝐸

log 𝑞(𝑌𝑗𝑘|𝑍

𝑗) + log 𝑞(𝑍 𝑗)

Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

𝜚 = distribution over 𝐿 labels

y

for label 𝑙 = 1 to 𝐿: 𝜄𝑙 = generate parameters

Choose the label

෍

𝑘=1 𝐸

log 𝑞(𝑌𝑗𝑘|𝑍

𝑗) + log 𝑞(𝑍 𝑗)

Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

𝜚 = distribution over 𝐿 labels for each feature 𝑘 𝑦𝑗𝑘 ∼ F𝑘(𝜄𝑧𝑗)

𝑦𝑗1 𝑦𝑗2 𝑦𝑗3 𝑦𝑗4 𝑦𝑗5

y

for label 𝑙 = 1 to 𝐿: 𝜄𝑙 = generate parameters

local variables Generate each feature based on the label

෍

𝑘=1 𝐸

log 𝑞(𝑌𝑗𝑘|𝑍

𝑗) + log 𝑞(𝑍 𝑗)

Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

𝜚 = distribution over 𝐿 labels

𝑦𝑗1 𝑦𝑗2 𝑦𝑗3 𝑦𝑗4 𝑦𝑗5

y

for label 𝑙 = 1 to 𝐿:

each xij is conditionally independent of one another (given the label)

𝜄𝑙 = generate parameters for each feature 𝑘 𝑦𝑗𝑘 ∼ F𝑘(𝜄𝑧𝑗)

෍

𝑘=1 𝐸

log 𝑞(𝑌𝑗𝑘|𝑍

𝑗) + log 𝑞(𝑍 𝑗)

Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

ℒ 𝜄 = ෍

𝑗

෍

𝑘

log 𝐺

𝑧𝑗(𝑦𝑗𝑘; 𝜄𝑧𝑗) + ෍ 𝑗

log 𝜚𝑧𝑗 s. t.

Maximize Log-likelihood

𝜚 = distribution over 𝐿 labels

𝑦𝑗1 𝑦𝑗2 𝑦𝑗3 𝑦𝑗4 𝑦𝑗5

y

for label 𝑙 = 1 to 𝐿:

෍

𝑙

𝜚𝑙 = 1 𝜄𝑙 is valid for 𝐺

𝑘

𝜄𝑙 = generate parameters for each feature 𝑘 𝑦𝑗𝑘 ∼ F𝑘(𝜄𝑧𝑗)

𝜚𝑙 ≥ 0

Multinomial Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

ℒ 𝜄 = ෍

𝑗

෍

𝑘

log 𝜄𝑧𝑗,𝑦𝑗,𝑘 + ෍

𝑗

log 𝜚𝑧𝑗 s. t.

Maximize Log-likelihood

𝜚 = distribution over 𝐿 labels for each feature 𝑘 𝑦𝑗𝑘 ∼ Cat(𝜄𝑧𝑗)

𝑦𝑗1 𝑦𝑗2 𝑦𝑗3 𝑦𝑗4 𝑦𝑗5

y

for label 𝑙 = 1 to 𝐿:

𝜄𝑙 = distribution over J feature values

෍

𝑙

𝜚𝑙 = 1 ෍

𝑘

𝜄𝑙𝑘 = 1 ∀𝑙 𝜄𝑙𝑘 ≥ 0, 𝜚𝑙 ≥ 0

Multinomial Naïve Bayes: A Generative Story

for item 𝑗 = 1 to 𝑂: 𝑧𝑗 ~ Cat 𝜚

Generative Story

ℒ 𝜄 = ෍

𝑗

෍

𝑘

log 𝜄𝑧𝑗,𝑦𝑗,𝑘 + ෍

𝑗

log 𝜚𝑧𝑗 − 𝜈 ෍

𝑙

𝜚𝑙 − 1 − ෍

𝑙

𝜇𝑙 ෍

𝑘

𝜄𝑙𝑘 − 1

Maximize Log-likelihood via Lagrange Multipliers (≥ 𝟏 constraints not shown)

𝜚 = distribution over 𝐿 labels for each feature 𝑘 𝑦𝑗𝑘 ∼ Cat(𝜄𝑧𝑗,𝑘)

𝑦𝑗1 𝑦𝑗2 𝑦𝑗3 𝑦𝑗4 𝑦𝑗5

y

for label 𝑙 = 1 to 𝐿:

𝜄𝑙 = distribution over J feature values

Multinomial Naïve Bayes: Learning

Calculate class priors For each k:

itemsk = all items with class = k

Calculate feature generation terms For each k:

• bsk = single object containing all

items labeled as k Foreach feature j nkj = # of occurrences of j in obsk

𝑞 𝑙 = |items𝑙| # items 𝑞 𝑘|𝑙 = 𝑜𝑙𝑘 σ𝑘′ 𝑜𝑙𝑘′

Brill and Banko (2001) With enough data, the classifier may not matter

Adapted from Jurafsky & Martin (draft)

Summary: Naïve Bayes is Not So Naïve, but not without issue

Pro

Very Fast, low storage requirements Robust to Irrelevant Features Very good in domains with many equally important features Optimal if the independence assumptions hold Dependable baseline for text classification (but often not the best)

Con

Model the posterior in one go? (e.g., use conditional maxent) Are the features really uncorrelated? Are plain counts always appropriate? Are there “better” ways of handling missing/noisy data?

(automated, more principled)

Adapted from Jurafsky & Martin (draft)

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph Common name: Markov Random Fields

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph Common name: Markov Random Fields Undirected graphs can have an alternative formulation as Factor Graphs

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂

clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!)

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!)

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!) Q: What restrictions should we place on the potentials 𝜔𝐷?

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!) Q: What restrictions should we place on the potentials 𝜔𝐷? A: 𝜔𝐷 ≥ 0 (or 𝜔𝐷 > 0)

Terminology: Potential Functions

𝜔𝐷 𝑦𝑑 = exp −𝐹(𝑦𝐷)

energy function (for clique C) Boltzmann distribution

(get the total energy of a configuration by summing the individual energy functions)

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Ambiguity in Undirected Model Notation

X Y Z

𝑞 𝑦, 𝑧, 𝑨 ∝ 𝜔(𝑦, 𝑧, 𝑨) 𝑞 𝑦, 𝑧, 𝑨 ∝ 𝜔1 𝑦,𝑧 𝜔2 𝑧,𝑨 𝜔3 𝑦,𝑨

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions Edges show what variables are used in which factors

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T

X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables

X Y Z X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions

X Y Z X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions Edges show what variables are used in which factors

X Y Z X Y Z

Different Factor Graph Notation for the Same Graph

X Y Z X Y Z X Y Z

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

𝑞 𝑦1, … , 𝑦4 = 𝑞 𝑦1 𝑞 𝑦2 𝑞 𝑦3 𝑞(𝑦4|𝑦1, 𝑦2, 𝑦3)

x1 x2 x3 x4

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

𝑞 𝑦1, … , 𝑦4 = 𝑞 𝑦1 𝑞 𝑦2 𝑞 𝑦3 𝑞(𝑦4|𝑦1, 𝑦2, 𝑦3)

x1 x2 x3 x4 parents of nodes in a directed graph must be connected in an undirected graph

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Undirected

(e.g., conditional random field [CRF]) z1

w1 w2 w3 w4

z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Undirected as factor graph

(e.g., conditional random field [CRF]) z1 z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain Conditional Random Field

Widely used in applications like part-of-speech tagging

z1 z2 z3 z4

President Obama told Congress …

Noun-Mod Noun Noun Verb

Example: Linear Chain Conditional Random Field

Widely used in applications like part-of-speech tagging and named entity recognition

z1 z2 z3 z4

President Obama told Congress …

Noun-Mod Noun Noun Verb

President Obama told Congress …

Person Person Org. Other

Linear Chain CRFs for Part of Speech Tagging

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 ♣|♢

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|♢

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂 ∝ ෑ

i=1 N

exp( 𝜄 𝑔 , 𝑔

𝑗 𝑨𝑗

+ 𝜄 𝑕 , 𝑕𝑗 𝑨𝑗, 𝑨𝑗+1 )

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂)

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂)

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂)

Feature design, just like in maxent models!

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂) Example: 𝑕𝑘,𝑂→𝑊 zj, zj+1 = 1 (if zj == N & zj+1 == V) else 0 𝑕𝑘,told,𝑂→𝑊 zj, zj+1 = 1 (if zj == N & zj+1 == V & xj == told) else 0

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

X Y

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

Q: What are the cliques?

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

Q: What are the cliques?

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

𝐹 𝑦, 𝑧 = ℎ ෍

𝑗

𝑦𝑗 − 𝛾 ෍

𝑗𝑘

𝑦𝑗𝑦𝑘 − 𝜃 ෍

𝑗

𝑦𝑗𝑧𝑗

xi and yi should be correlated neighboring pixels should be similar allow for a bias

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

𝐹 𝑦, 𝑧 = ℎ ෍

𝑗

𝑦𝑗 − 𝛾 ෍

𝑗𝑘

𝑦𝑗𝑦𝑘 − 𝜃 ෍

𝑗

𝑦𝑗𝑧𝑗

xi and yi should be correlated neighboring pixels should be similar allow for a bias

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

𝐹 𝑦, 𝑧 = ℎ ෍

𝑗

𝑦𝑗 − 𝛾 ෍

𝑗𝑘

𝑦𝑗𝑦𝑘 − 𝜃 ෍

𝑗

𝑦𝑗𝑧𝑗

xi and yi should be correlated neighboring pixels should be similar allow for a bias

Q: Why subtract β and η?

Example: Ising Model

x: original pixel/state y:

• bserved

(noisy) pixel/state Image denoising (Bishop, 2006; Fig 8.30)

• riginal

𝐹 𝑦, 𝑧 = ℎ ෍

𝑗

𝑦𝑗 − 𝛾 ෍

𝑗𝑘

𝑦𝑗𝑦𝑘 − 𝜃 ෍

𝑗

𝑦𝑗𝑧𝑗

xi and yi should be correlated neighboring pixels should be similar allow for a bias

Q: Why subtract β and η? A: Better states → lower energy (higher potential) 𝜔𝐷 𝑦𝑑 = exp −𝐹(𝑦𝐷)

Markov Random Fields with Factor Graph Notation

x: original pixel/state y: observed (noisy) pixel/state

factor nodes are added according to maximal cliques

unary factor variable

factor graphs are bipartite

binary factor

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference

Two Problems for Undirected Models

Finding the normalizer Computing the marginals

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Undirected Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Undirected Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals 𝑎𝑜(𝑤) = ෍

𝑦:𝑦𝑜=𝑤

ෑ

𝑑

𝜔𝑑(𝑦𝑑)

Sum over all variable combinations, with the xn coordinate fixed 𝑎2(𝑤) = ෍

𝑦1

෍

𝑦3

ෑ

𝑑

𝜔𝑑(𝑦 = 𝑦1, 𝑤, 𝑦3 ) Example: 3 variables, fix the 2nd dimension

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Undirected Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals 𝑎𝑜(𝑤) = ෍

𝑦:𝑦𝑜=𝑤

ෑ

𝑑

𝜔𝑑(𝑦𝑑)

Q: Why are these difficult? Sum over all variable combinations, with the xn coordinate fixed 𝑎2(𝑤) = ෍

𝑦1

෍

𝑦3

ෑ

𝑑

𝜔𝑑(𝑦 = 𝑦1, 𝑤, 𝑦3 ) Example: 3 variables, fix the 2nd dimension

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Undirected Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals 𝑎𝑜(𝑤) = ෍

𝑦:𝑦𝑜=𝑤

ෑ

𝑑

𝜔𝑑(𝑦𝑑)

Q: Why are these difficult? A: Many different combinations Sum over all variable combinations, with the xn coordinate fixed 𝑎2(𝑤) = ෍

𝑦1

෍

𝑦3

ෑ

𝑑

𝜔𝑑(𝑦 = 𝑦1, 𝑤, 𝑦3 ) Example: 3 variables, fix the 2nd dimension

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Message Passing: Count the Soldiers

If you are the front soldier in the line, say the number ‘one’ to the soldier behind you. If you are the rearmost soldier in the line, say the number ‘one’ to the soldier in front of you. If a soldier ahead of or behind you says a number to you, add

• ne to it, and say the new

number to the soldier on the

• ther side

ITILA, Ch 16

Message Passing: Count the Soldiers

If you are the front soldier in the line, say the number ‘one’ to the soldier behind you. If you are the rearmost soldier in the line, say the number ‘one’ to the soldier in front of you. If a soldier ahead of or behind you says a number to you, add one to it, and say the new number to the soldier on the other side

ITILA, Ch 16

Message Passing: Count the Soldiers

If you are the front soldier in the line, say the number ‘one’ to the soldier behind you. If you are the rearmost soldier in the line, say the number ‘one’ to the soldier in front of you. If a soldier ahead of or behind you says a number to you, add one to it, and say the new number to the soldier on the other side

ITILA, Ch 16

Message Passing: Count the Soldiers

If you are the front soldier in the line, say the number ‘one’ to the soldier behind you. If you are the rearmost soldier in the line, say the number ‘one’ to the soldier in front of you. If a soldier ahead of or behind you says a number to you, add one to it, and say the new number to the soldier on the other side

ITILA, Ch 16

Sum-Product Algorithm

Main idea: message passing An exact inference algorithm for tree-like graphs Belief propagation (forward-backward for HMMs) is a special case

Sum-Product

𝑞 𝑦𝑗 = 𝑤 = ෑ

𝑦:𝑦𝑗=𝑤

𝑞 𝑦1, 𝑦2, … , 𝑦𝑗, … , 𝑦𝑂

definition of marginal

… …

Sum-Product

𝑞 𝑦𝑗 = 𝑤 = ෑ

𝑦:𝑦𝑗=𝑤

𝑞 𝑦1, 𝑦2, … , 𝑦𝑗, … , 𝑦𝑂

definition of marginal

… …

main idea: use bipartite nature of graph to efficiently compute the marginals

The factor nodes can act as filters

Sum-Product

𝑞 𝑦𝑗 = 𝑤 = ෑ

𝑦:𝑦𝑗=𝑤

𝑞 𝑦1, 𝑦2, … , 𝑦𝑗, … , 𝑦𝑂

definition of marginal

… …

main idea: use bipartite nature of graph to efficiently compute the marginals 𝑠

𝑛→𝑜

𝑠

𝑛→𝑜

𝑠

𝑛→𝑜

Sum-Product

𝑞 𝑦𝑗 = 𝑤 = ෑ

𝑔

𝑠

𝑔→𝑦𝑗(𝑦𝑗) alternative marginal computation

… …

main idea: use bipartite nature of graph to efficiently compute the marginals 𝑠

𝑛→𝑜

𝑠

𝑛→𝑜

𝑠

𝑛→𝑜

Sum-Product

From variables to factors 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

n m

set of factors in which variable n participates

default value of 1 if empty product

Sum-Product

From variables to factors 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜 From factors to variables 𝑠

𝑛→𝑜 𝑦𝑜

= ෍

𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

n m n m

set of variables that the mth factor depends on set of factors in which variable n participates sum over configuration of variables for the mth factor, with variable n fixed

default value of 1 if empty product

Example

Q: What are the variables? 𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

Example

Q: What are the variables? A: 𝑦1, 𝑦2, 𝑦3, 𝑦4 Q: What are the factors? 𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

Example

Q: What are the variables? A: 𝑦1, 𝑦2, 𝑦3, 𝑦4 Q: What are the factors? A: 𝑔

𝑏 𝑦1, 𝑦2 ,

𝑔

𝑐 𝑦2, 𝑦3 ,

𝑔

𝑑(𝑦2, 𝑦4)

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

Q: What is the distribution we’re modeling?

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

Q: What is the distribution we’re modeling?

A: 𝑞 𝑦1, 𝑦2, 𝑦3, 𝑦4 = 𝑔

𝑏 𝑦1, 𝑦2 𝑔 𝑐 𝑦2, 𝑦3 𝑔 𝑑(𝑦2, 𝑦4)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑠

𝑔

𝑏→𝑦2 𝑦2 =? ? ?

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑠

𝑔

𝑏→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦1 = 𝑙, 𝑦2)

𝑠

𝑔

𝑑→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦2, 𝑦4 = 𝑙)

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑠

𝑔

𝑏→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦1 = 𝑙, 𝑦2)

𝑠

𝑔

𝑑→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦2, 𝑦4 = 𝑙)

𝑟𝑦2→𝑔𝑐 𝑦2 =? ? ? 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑠

𝑔

𝑏→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦1 = 𝑙, 𝑦2)

𝑠

𝑔

𝑑→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦2, 𝑦4 = 𝑙)

𝑟𝑦2→𝑔𝑐 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔

𝑑→𝑦2 𝑦2

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root

𝑟𝑦1→𝑔

𝑏 𝑦1 = 1

𝑟𝑦4→𝑔

𝑑 𝑦4 = 1

𝑠

𝑔

𝑏→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦1 = 𝑙, 𝑦2)

𝑠

𝑔

𝑑→𝑦2 𝑦2 = ෍

𝑙

𝑔

𝑏(𝑦2, 𝑦4 = 𝑙)

𝑟𝑦2→𝑔𝑐 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔

𝑑→𝑦2 𝑦2

𝑠

𝑔𝑐→𝑦3 𝑦3 = ෍ 𝑙

𝑔

𝑐(𝑦2 = 𝑙, 𝑦3)

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = ? ? ?

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

We just computed this Q: Where did we compute this?

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

We just computed this Q: Where did we compute this? A: In step 1 (leaves → root)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑦2→𝑔

𝑑 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔𝑐→𝑦2 𝑦2

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑦2→𝑔

𝑑 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔𝑐→𝑦2 𝑦2

𝑠

𝑔

𝑑→𝑦4 𝑦4 = ෍

𝑙

𝑔

𝑑(𝑦2 = 𝑙, 𝑦4)

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves

𝑟𝑦3→𝑔𝑐 𝑦3 = 1 𝑠

𝑔𝑐→𝑦2 𝑦2 = ෍ 𝑙

𝑔

𝑐(𝑦2, 𝑦3 = 𝑙)

𝑟𝑦2→𝑔

𝑏 𝑦2 = 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑦2→𝑔

𝑑 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔𝑐→𝑦2 𝑦2

𝑠

𝑔

𝑑→𝑦4 𝑦4 = ෍

𝑙

𝑔

𝑑(𝑦2 = 𝑙, 𝑦4)

𝑠

𝑔

𝑏→𝑦1 𝑦1 = ෍

𝑙

𝑔

𝑏(𝑦1, 𝑦2 = 𝑙)

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities 𝑞 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities 𝑞 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜 𝑞 𝑦1 = 𝑠

𝑔

𝑏→𝑦1 𝑦1

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities 𝑞 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜 𝑞 𝑦1 = 𝑠

𝑔

𝑏→𝑦1 𝑦1

𝑞 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities 𝑞 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜 𝑞 𝑦1 = 𝑠

𝑔

𝑏→𝑦1 𝑦1

𝑞 𝑦2 = 𝑠

𝑔

𝑏→𝑦2 𝑦2 𝑠

𝑔𝑐→𝑦2 𝑦2 𝑠 𝑔

𝑑→𝑦2 𝑦2

𝑞 𝑦3 = 𝑠

𝑔𝑐→𝑦3 𝑦3

𝑞 𝑦4 = 𝑠

𝑔

𝑑→𝑦4 𝑦4

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities

• 2. Are we done?
• 1. If a tree structure, we’ve converged

2. 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree (𝑦3)
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities

• 2. Are we done?
• 1. If a tree structure, we’ve converged
• 2. If not:
• 1. Either accept the partially

converged result, or… 2. 𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Example

𝑦2 𝑦1 𝑦3 𝑦4 𝑔

𝑏

𝑔

𝑐

𝑔

𝑑

• 1. Select the root, or pick one if a tree
• 1. Send messages from leaves to root
• 2. Send messages from root to leaves
• 3. Use messages to compute marginal

probabilities

• 2. Are we done?
• 1. If a tree structure, we’ve converged
• 2. If not:
• 1. Either accept the partially

converged result, or…

• 2. Go back to (1) and repeat

[Loopy BP]

𝑟𝑜→𝑛 𝑦𝑜 = ෑ

𝑛′∈𝑁(𝑜)\𝑛

𝑠𝑛′→𝑜 𝑦𝑜

𝑠

𝑛→𝑜 𝑦𝑜 = ෍ 𝒙𝑛\𝑜

𝑔

𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

Max-Product (Max-Sum)

Problem: how to find the most likely (best) setting of latent variables Replace sum (+) with max in factor→variable computations

𝑠

𝑛→𝑜 𝑦𝑜 = max 𝒙𝑛\𝑜 𝑔 𝑛 𝒙𝑛

ෑ

𝑜′∈𝑂(𝑛)\𝑜

𝑟𝑜′→𝑛(𝑦𝑜′)

(why max-sum? computationally, implement with logs)

Loopy Belief Propagation

Sum-product algorithm is not exact for general graphs Loopy Belief Propagation (Loopy BP): run sum- product algorithm anyway and hope for the best Requires a message passing schedule

Outline

Directed Graphical Models Naïve Bayes Undirected Graphical Models Factor Graphs Ising Model Message Passing: Graphical Model Inference