Probabilistic Graphical Models CMSC 691 UMBC Two Problems for - - PowerPoint PPT Presentation

Probabilistic Graphical Models

CMSC 691 UMBC

Two Problems for Graphical Models

Finding the normalizer Computing the marginals

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Graphical Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Graphical Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals 𝑎𝑜(𝑤) = ෍

𝑦:𝑦𝑜=𝑤

ෑ

𝑑

𝜔𝑑(𝑦𝑑)

Sum over all variable combinations, with the xn coordinate fixed 𝑎2(𝑤) = ෍

𝑦1

෍

𝑦3

ෑ

𝑑

𝜔𝑑(𝑦 = 𝑦1, 𝑤, 𝑦3 ) Example: 3 variables, fix the 2nd dimension

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Two Problems for Graphical Models

Finding the normalizer 𝑎 = ෍

𝑦

ෑ

𝑑

𝜔𝑑(𝑦𝑑) Computing the marginals 𝑎𝑜(𝑤) = ෍

𝑦:𝑦𝑜=𝑤

ෑ

𝑑

𝜔𝑑(𝑦𝑑)

Q: Why are these difficult? A: Many different combinations Sum over all variable combinations, with the xn coordinate fixed 𝑎2(𝑤) = ෍

𝑦1

෍

𝑦3

ෑ

𝑑

𝜔𝑑(𝑦 = 𝑦1, 𝑤, 𝑦3 ) Example: 3 variables, fix the 2nd dimension

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Probabilistic Graphical Models

A graph G that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂

Probabilistic Graphical Models

A graph G that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂)

Probabilistic Graphical Models

A graph G that represents a probability distribution

• ver random variables 𝑌1, … , 𝑌𝑂

Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂) Vertices ↔ random variables Edges show dependencies among random variables

Probabilistic Graphical Models

A graph G that represents a probability distribution

• ver random variables 𝑌1, … , 𝑌𝑂

Graph G = (vertices V, edges E) Distribution 𝑞(𝑌1, … , 𝑌𝑂) Vertices ↔ random variables Edges show dependencies among random variables Two main flavors: directed graphical models and undirected graphical models

Outline

Directed Graphical Models Undirected Graphical Models Factor Graphs

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗

Benefit: read the independence properties are transparent

Directed Graphical Models

A directed (acyclic) graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes into factors of 𝑌𝑗 conditioned on the parents of 𝑌𝑗 A graph/joint distribution that follows this is a

Bayesian network

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

“parents of” topological sort

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

𝑞 𝑦1, 𝑦2, 𝑦3, 𝑦4, 𝑦5 = ???

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, 𝑦4, 𝑦5 = 𝑞 𝑦1 𝑞 𝑦3 𝑞 𝑦2 𝑦1, 𝑦3 𝑞 𝑦4 𝑦2, 𝑦3 𝑞(𝑦5|𝑦2, 𝑦4)

Bayesian Networks: Directed Acyclic Graphs

𝑦1 𝑦4 𝑦3 5 𝑦2

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = ෑ

𝑗

𝑞 𝑦𝑗 𝜌(𝑦𝑗))

exact inference in general DAGs is NP-hard inference in trees can be exact

Directed Graphical Model Notation

𝑦1 𝑦4 𝑦3 5 𝑦2

Shaded nodes are

• bserved R.V.s

Unshaded nodes are unobserved (latent) R.V.s

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Y X Y X Z Y

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Z Y X Z Y X Z Y

• bserving Z blocks

the path from X to Y

• bserving Z blocks

the path from X to Y not observing Z blocks the path from X to Y

D-Separation: Testing for Conditional Independence

Variables X & Y are conditionally independent given Z if all (undirected) paths from (any variable in) X to (any variable in) Y are d-separated by Z

d-separation

P has a chain with an observed middle node P has a fork with an observed parent node P includes a “v-structure” or “collider” with all unobserved descendants X & Y are d-separated if for all paths P, one of the following is true: X Z Y X Z Y X Z Y

• bserving Z blocks

the path from X to Y

• bserving Z blocks

the path from X to Y not observing Z blocks the path from X to Y 𝑞 𝑦, 𝑧, 𝑨 = 𝑞 𝑦 𝑞 𝑧 𝑞(𝑨|𝑦, 𝑧) 𝑞 𝑦, 𝑧 = ෍

𝑨

𝑞 𝑦 𝑞 𝑧 𝑞(𝑨|𝑦, 𝑧) = 𝑞 𝑦 𝑞 𝑧

Markov Blanket

x Markov blanket of a node x is its parents, children, and children's parents

𝑞 𝑦𝑗 𝑦𝑘≠𝑗 = 𝑞(𝑦1, … , 𝑦𝑂) ∫ 𝑞 𝑦1, … , 𝑦𝑂 𝑒𝑦𝑗 = ς𝑙 𝑞(𝑦𝑙|𝜌 𝑦𝑙 ) ∫ ς𝑙 𝑞 𝑦𝑙 𝜌 𝑦𝑙 ) 𝑒𝑦𝑗

factor out terms not dependent on xi

factorization

• f graph

= ς𝑙:𝑙=𝑗 or 𝑗∈𝜌 𝑦𝑙 𝑞(𝑦𝑙|𝜌 𝑦𝑙 ) ∫ ς𝑙:𝑙=𝑗 or 𝑗∈𝜌 𝑦𝑙 𝑞 𝑦𝑙 𝜌 𝑦𝑙 ) 𝑒𝑦𝑗

the set of nodes needed to form the complete conditional for a variable xi

(in this example, shading does not show

• bserved/latent)

Outline

Directed Graphical Models Undirected Graphical Models Factor Graphs

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph Common name: Markov Random Fields

Undirected Graphical Models

An undirected graph G=(V,E) that represents a probability distribution over random variables 𝑌1, … , 𝑌𝑂 Joint probability factorizes based on cliques in the graph Common name: Markov Random Fields Undirected graphs can have an alternative formulation as Factor Graphs

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂

clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!)

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!)

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!) Q: What restrictions should we place on the potentials 𝜔𝐷?

Markov Random Fields: Undirected Graphs

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

variables part

• f the clique C

maximal cliques global normalization clique: subset of nodes, where nodes are pairwise connected maximal clique: a clique that cannot add a node and remain a clique potential function (not necessarily a probability!) Q: What restrictions should we place on the potentials 𝜔𝐷? A: 𝜔𝐷 ≥ 0 (or 𝜔𝐷 > 0)

Terminology: Potential Functions

𝜔𝐷 𝑦𝑑 = exp −𝐹(𝑦𝐷)

energy function (for clique C) Boltzmann distribution

(get the total energy of a configuration by summing the individual energy functions)

𝑞 𝑦1, 𝑦2, 𝑦3, … , 𝑦𝑂 = 1 𝑎 ෑ

𝐷

𝜔𝐷 𝑦𝑑

Ambiguity in Undirected Model Notation

X Y Z

𝑞 𝑦, 𝑧, 𝑨 ∝ 𝜔(𝑦, 𝑧, 𝑨) 𝑞 𝑦, 𝑧, 𝑨 ∝ 𝜔1 𝑦,𝑧 𝜔2 𝑧,𝑨 𝜔3 𝑦,𝑨

Outline

Directed Graphical Models Undirected Graphical Models Factor Graphs

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions Edges show what variables are used in which factors

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T

X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables

X Y Z X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions

X Y Z X Y Z

MRFs as Factor Graphs

Undirected graphs: G=(V,E) that represents 𝑞(𝑌1, … , 𝑌𝑂) Factor graph of p: Bipartite graph of evidence nodes X, factor nodes F, and edges T Evidence nodes X are the random variables Factor nodes F take values associated with the potential functions Edges show what variables are used in which factors

X Y Z X Y Z

Different Factor Graph Notation for the Same Graph

X Y Z X Y Z X Y Z

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

𝑞 𝑦1, … , 𝑦4 = 𝑞 𝑦1 𝑞 𝑦2 𝑞 𝑦3 𝑞(𝑦4|𝑦1, 𝑦2, 𝑦3)

x1 x2 x3 x4

Directed vs. Undirected Models: Moralization

x1 x2 x3 x4

𝑞 𝑦1, … , 𝑦4 = 𝑞 𝑦1 𝑞 𝑦2 𝑞 𝑦3 𝑞(𝑦4|𝑦1, 𝑦2, 𝑦3)

x1 x2 x3 x4 parents of nodes in a directed graph must be connected in an undirected graph

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Undirected

(e.g., conditional random field [CRF]) z1

w1 w2 w3 w4

z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain

Directed (e.g.,

hidden Markov model [HMM]; generative) z1

w1 w2 w3 w4

z2 z3 z4

Undirected as factor graph

(e.g., conditional random field [CRF]) z1 z2 z3 z4

Directed (e.g..,

maximum entropy Markov model [MEMM]; conditional) z1

w1 w2 w3 w4

z2 z3 z4

Example: Linear Chain Conditional Random Field

Widely used in applications like part-of-speech tagging

z1 z2 z3 z4

President Obama told Congress …

Noun-Mod Noun Noun Verb

Example: Linear Chain Conditional Random Field

Widely used in applications like part-of-speech tagging and named entity recognition

z1 z2 z3 z4

President Obama told Congress …

Noun-Mod Noun Noun Verb

President Obama told Congress …

Person Person Org. Other

Linear Chain CRFs for Part of Speech Tagging

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 ♣|♢

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|♢

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂

A linear chain CRF is a conditional probabilistic model of the sequence of tags 𝑨1, 𝑨2, … , 𝑨𝑂 conditioned on the entire input sequence 𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑞 𝑨1, 𝑨2, … , 𝑨𝑂|𝑦1:𝑂 ∝ ෑ

i=1 N

exp( 𝜄 𝑔 , 𝑔

𝑗 𝑨𝑗

+ 𝜄 𝑕 , 𝑕𝑗 𝑨𝑗, 𝑨𝑗+1 )

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂)

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂)

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂)

Feature design, just like in maxent models!

Linear Chain CRFs for Part of Speech Tagging

z1 z2 z3 z4

𝑔

1

𝑔

2

𝑔

3

𝑔

4

𝑕1 𝑕2 𝑕3 𝑕4

𝑕𝑘: inter-tag features (can depend on any/all input words 𝑦1:𝑂) 𝑔

𝑗: solo tag features

(can depend on any/all input words 𝑦1:𝑂) Example: 𝑕𝑘,𝑂→𝑊 zj, zj+1 = 1 (if zj == N & zj+1 == V) else 0 𝑕𝑘,told,𝑂→𝑊 zj, zj+1 = 1 (if zj == N & zj+1 == V & xj == told) else 0

Outline

Directed Graphical Models Undirected Graphical Models Factor Graphs