Induced Graph Semantics: Another look at the Hammersley-Clifford - - PowerPoint PPT Presentation

Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

• P. Sunehag (with T. Sears)

ANU/NICTA

July 2007

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 1 / 14

Graphical Models

Section

1

Graphical Models

2

Generalizing the exponential family

3

Generalized Factorization

4

Conclusions

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 2 / 14

Graphical Models

Graphical Models

Graph G = (V, E), vertices (or nodes), V, and edges E.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 3 / 14

Graphical Models

Graphical Models

Graph G = (V, E), vertices (or nodes), V, and edges E. A clique, c ⊂ V is a fully connected subgraph of G.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 3 / 14

Graphical Models

Graphical Models

Graph G = (V, E), vertices (or nodes), V, and edges E. A clique, c ⊂ V is a fully connected subgraph of G. Vertices, {xi}M

i=1 correspond to random variables.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 3 / 14

Graphical Models

Graphical Models

Graph G = (V, E), vertices (or nodes), V, and edges E. A clique, c ⊂ V is a fully connected subgraph of G. Vertices, {xi}M

i=1 correspond to random variables.

Missing edges represent conditional independence assumptions.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 3 / 14

Graphical Models

Factorization

Theorem (Hammersley-Clifford) The density of a joint probability distributions satisifies the conditional indepence assumptions represented by a graph G if and only if there are local clique densities ψc such that f (x) = 1 Z

• c

ψc(xc) where x is the full random vector, xc the restriction of x to c, c runs over all cliques and Z is a normalization constant. It is common to use ψc of an exponential family form ψc(xc) = exp(ηc · T(xc) − Zc(ηc)). With such ψc the joint density is also an exponential family distribution.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 4 / 14

Graphical Models

Limitations of the Exponential Family

Exponential family distributions have thin tails and full support.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 5 / 14

Graphical Models

Limitations of the Exponential Family

Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 5 / 14

Graphical Models

Limitations of the Exponential Family

Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for

• ther variables that under more normal circumstaces behave as if they

were close to independent.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 5 / 14

Graphical Models

Limitations of the Exponential Family

Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for

• ther variables that under more normal circumstaces behave as if they

were close to independent. Example: A portfolio of equities is considered to be more correlated in the face of steep market decline.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 5 / 14

Graphical Models

Limitations of the Exponential Family

Exponential family distributions have thin tails and full support. Phenomena with heavy tailed power-law distributions are abundant. We are also interested in situations where an unlikely event for one variable increases the probability for otherwise unlikely events for

• ther variables that under more normal circumstaces behave as if they

were close to independent. Example: A portfolio of equities is considered to be more correlated in the face of steep market decline. The generalizations we consider here can help us model that.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 5 / 14

Generalizing the exponential family

Section

1

Graphical Models

2

Generalizing the exponential family

3

Generalized Factorization

4

Conclusions

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 6 / 14

Generalizing the exponential family

Deformed logarithms

based on φ-logarithms

log(p) = p

1

1 x dx Usual construction

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 7 / 14

Generalizing the exponential family

Deformed logarithms

based on φ-logarithms

logφ(p) = p

1

1 φ(x)dx Deformed Log

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 7 / 14

Generalizing the exponential family

Deformed logarithms

based on φ-logarithms

logφ(p) = p

1

1 φ(x)dx Deformed Log Positive increasing φ.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 7 / 14

Generalizing the exponential family

Deformed logarithms

based on φ-logarithms

logφ(p) = p

1

1 φ(x)dx Deformed Log Positive increasing φ. φ(x) = xq yields the q-logarithm from the non-extensive thermodynamics literature. logq(x) := x1−q − 1 1 − q

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 7 / 14

Generalizing the exponential family

Deformed logarithms

based on φ-logarithms

logφ(p) = p

1

1 φ(x)dx Deformed Log Positive increasing φ. φ(x) = xq yields the q-logarithm from the non-extensive thermodynamics literature. logq(x) := x1−q − 1 1 − q The inverse of the q-logarithm is expq(v) = (1 + (1 − q)v)

1 1−q

+

.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 7 / 14

Generalizing the exponential family

The Generalized Exponential Map

q-Exponential Examples

3 2 1 1 2 3 2 4 6 8 10

expΦ expq

q 1 q 0.5 q 0 q 1.5 Asymptote q 1.5

• q > 1 naturally gives fat

tails. q < 1 truncates the tail.

3 2.5 2 1.5 1 0.5 0.2 0.4 0.6 0.8 1

expΦ expq

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 8 / 14

Generalized Factorization

Section

1

Graphical Models

2

Generalizing the exponential family

3

Generalized Factorization

4

Conclusions

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 9 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti).

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti). Solution: Define a new operation, ⊗q, such that expq(t1) ⊗q expq(t2) = expq(t1 + t2).

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti). Solution: Define a new operation, ⊗q, such that expq(t1) ⊗q expq(t2) = expq(t1 + t2). ⊗q is defined by x ⊗q y = expq(logq(x) + logq(y)) = (x1−q + y1−q − 1)

1 1−q

+

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti). Solution: Define a new operation, ⊗q, such that expq(t1) ⊗q expq(t2) = expq(t1 + t2). ⊗q is defined by x ⊗q y = expq(logq(x) + logq(y)) = (x1−q + y1−q − 1)

1 1−q

+

It is associative, commutative and has 1 as its neutral element.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti). Solution: Define a new operation, ⊗q, such that expq(t1) ⊗q expq(t2) = expq(t1 + t2). ⊗q is defined by x ⊗q y = expq(logq(x) + logq(y)) = (x1−q + y1−q − 1)

1 1−q

+

It is associative, commutative and has 1 as its neutral element. If q = 2 then

1 10 ⊗q 1 10 = 1 19 and 1 100 ⊗q 1 100 = 1 199.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-multiplication

Problem: If q = 1, then expq(ti) is not equal to expq( ti). Solution: Define a new operation, ⊗q, such that expq(t1) ⊗q expq(t2) = expq(t1 + t2). ⊗q is defined by x ⊗q y = expq(logq(x) + logq(y)) = (x1−q + y1−q − 1)

1 1−q

+

It is associative, commutative and has 1 as its neutral element. If q = 2 then

1 10 ⊗q 1 10 = 1 19 and 1 100 ⊗q 1 100 = 1 199. 1 100/ 1 19 = 0.19, 1 10000/ 1 199 = 0.0199.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 10 / 14

Generalized Factorization

q-independence

X1 and X2 with joint density f (x1, x2) are q-independent if there are h1 and h2 such that f (x1, x2) = h1(x1) ⊗q h2(x2).

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 11 / 14

Generalized Factorization

q-independence

X1 and X2 with joint density f (x1, x2) are q-independent if there are h1 and h2 such that f (x1, x2) = h1(x1) ⊗q h2(x2). We prove a generalized Hammersley-Clifford Theorem that guarantees that a joint density q-factorizes over the cliques of a graph if and only if it satisfies the conditional q-independence conditions that correspond to the graph.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 11 / 14

Generalized Factorization

q-independence

X1 and X2 with joint density f (x1, x2) are q-independent if there are h1 and h2 such that f (x1, x2) = h1(x1) ⊗q h2(x2). We prove a generalized Hammersley-Clifford Theorem that guarantees that a joint density q-factorizes over the cliques of a graph if and only if it satisfies the conditional q-independence conditions that correspond to the graph. Thus, q-expontial clique densities yield q-exponential joint density for all the variables.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 11 / 14

Conclusions

Section

1

Graphical Models

2

Generalizing the exponential family

3

Generalized Factorization

4

Conclusions

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 12 / 14

Conclusions

Conclusions and Discussion

We have opened the possibility for using Graphical Models with q-exponential families by proving a q-analogue of the Hammersley-Clifford Theorem.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 13 / 14

Conclusions

Conclusions and Discussion

We have opened the possibility for using Graphical Models with q-exponential families by proving a q-analogue of the Hammersley-Clifford Theorem. This continues a line of work that generalizes key statistical tools to statistics based on a different algebraic structure.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 13 / 14

Conclusions

Conclusions and Discussion

We have opened the possibility for using Graphical Models with q-exponential families by proving a q-analogue of the Hammersley-Clifford Theorem. This continues a line of work that generalizes key statistical tools to statistics based on a different algebraic structure. Tsallis et al. recently proved a new central limit kind of theorem for q-independent variables where the attractor is q-Gaussians, i.e. variables with density on the form expq(ρx2 − βq(ρ)). It is fat tailed (polynomial decay) for q > 1 and has truncated support for q < 1.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 13 / 14

Conclusions

Conclusions and Discussion

We have opened the possibility for using Graphical Models with q-exponential families by proving a q-analogue of the Hammersley-Clifford Theorem. This continues a line of work that generalizes key statistical tools to statistics based on a different algebraic structure. Tsallis et al. recently proved a new central limit kind of theorem for q-independent variables where the attractor is q-Gaussians, i.e. variables with density on the form expq(ρx2 − βq(ρ)). It is fat tailed (polynomial decay) for q > 1 and has truncated support for q < 1. There is a connected q-calculus and q-geometry.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 13 / 14

Conclusions

The Math of Information

Suppose that we are interested in word frequences, say for Information Retrieval purposes.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 14 / 14

Conclusions

The Math of Information

Suppose that we are interested in word frequences, say for Information Retrieval purposes. In IR it is important to understand how informative the appearance of a term is. How surprised are we?

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 14 / 14

Conclusions

The Math of Information

Suppose that we are interested in word frequences, say for Information Retrieval purposes. In IR it is important to understand how informative the appearance of a term is. How surprised are we? If a term has appeared once in a document it is less surprising to see it a second time. Zipf’s law, power-law behaviour of terms.

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 14 / 14

Conclusions

The Math of Information

Suppose that we are interested in word frequences, say for Information Retrieval purposes. In IR it is important to understand how informative the appearance of a term is. How surprised are we? If a term has appeared once in a document it is less surprising to see it a second time. Zipf’s law, power-law behaviour of terms. Is an entropy something that defines a mathematics for the kind of information that you plan to deal with?

• P. Sunehag (with T. Sears) (ANU/NICTA) Induced Graph Semantics: Another look at the Hammersley-Clifford Theorem

MaxEnt 14 / 14