Random Networks, Graphical Models and Exchangeability Alessandro - - PowerPoint PPT Presentation

Exchangeability A finite deFinetti theorem Bidirected graphical models

Random Networks, Graphical Models and Exchangeability

Alessandro Rinaldo Carnegie Mellon University

joint work with Steffen Lauritzen and Kayvan Sadeghi

October 4, 2015 AMS Central Fall Sectional Meeting Loyola University

Special Session on Algebraic Statistics and its Interactions with Combinatorics, Computation, and Network Science

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 1/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Outline

Exchangeability of (infinite) networks. A finite deFinetti theorem and the dissociated property. Exchangeable and extendable finite networks are (mixtures of) bidirected graphical models.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 2/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Statistical Network (Random Graph) Analysis

Let Ln be the set of simple labeled graphs on n nodes: ∣Ln∣ = 2(n

2).

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 3/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Statistical Network (Random Graph) Analysis

Let Ln be the set of simple labeled graphs on n nodes: ∣Ln∣ = 2(n

2).

The nodes represent agents in some population of interest and the edges encode the relationships among them.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 3/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Statistical Network (Random Graph) Analysis

Let Ln be the set of simple labeled graphs on n nodes: ∣Ln∣ = 2(n

2).

The nodes represent agents in some population of interest and the edges encode the relationships among them. Statistical Network Analysis Pose and estimate probability distributions on Ln by modeling the joint occurrence of the (n

2) random edges.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 3/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Motivation: asymptotics of networks

Let L = ⋃n L, be the set of all finite (labeled, simple) graphs. A statistical model for L is a sequence {pn}n∈N of probability distributions, where pn is a probability distribution on Ln. For n < m, let pn

m denote the marginal of pm over Ln.

Consistency and Extendability A statistical model {pn}n∈N on L is consistent when, for any pair n < m, pn = pn

m.

(1) A probability distribution pn on Ln is extendable when (1) holds ∀m > n.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 4/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Motivation: asymptotics of networks

Let L = ⋃n L, be the set of all finite (labeled, simple) graphs. A statistical model for L is a sequence {pn}n∈N of probability distributions, where pn is a probability distribution on Ln. For n < m, let pn

m denote the marginal of pm over Ln.

Consistency and Extendability A statistical model {pn}n∈N on L is consistent when, for any pair n < m, pn = pn

m.

(1) A probability distribution pn on Ln is extendable when (1) holds ∀m > n. Most network models are not consistent!

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 4/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Consistency via Exchangeability

Let L∞ be the set of (countably) infinite lableled, simple graphs. Every probability distrbution on L∞ trivially specifies one consistent model! We impose one further restriction...

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 5/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Consistency via Exchangeability

Let L∞ be the set of (countably) infinite lableled, simple graphs. Every probability distrbution on L∞ trivially specifies one consistent model! We impose one further restriction... Exchangeability A probability distribution on L∞ is exchangeable when all finite isomorphic graphs have the same probabilities.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 5/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Consistency via Exchangeability

Let L∞ be the set of (countably) infinite lableled, simple graphs. Every probability distrbution on L∞ trivially specifies one consistent model! We impose one further restriction... Exchangeability A probability distribution on L∞ is exchangeable when all finite isomorphic graphs have the same probabilities. Exchangeability is a most basic form of invariance, suitable to describe the "shape" of networks (large scale property). Labeled vs unlabeled. The exchangeability assumption is equivalent to define models on Un, the set of unlabaled graohs on n nodes, for all n.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 5/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Exchangeability and graphons

Analytic representation of exchangeable distributions The set of exchangeable distributions, with the topology of weak convergence, is a (Bauer) simplex. Denote its extreme points with E∞. p∞ ∈ E∞ if and only if, for every n and G ∈ Ln pn

∞(G) = ∫[0,1]n

∏

(i,j)∈E(G)

f(zi, zj) ∏

(i,j)/ ∈E(G)

(1 − f(zi, zj))dz1 . . . zn, where f∶ [0, 1]2 → [0, 1] is a (measurable) symmetric function, called a graphon. Graphons are unique up to measure preserving transformations of [0, 1].

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 6/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Exchangeability and graphons

Analytic representation of exchangeable distributions The set of exchangeable distributions, with the topology of weak convergence, is a (Bauer) simplex. Denote its extreme points with E∞. p∞ ∈ E∞ if and only if, for every n and G ∈ Ln pn

∞(G) = ∫[0,1]n

∏

(i,j)∈E(G)

f(zi, zj) ∏

(i,j)/ ∈E(G)

(1 − f(zi, zj))dz1 . . . zn, where f∶ [0, 1]2 → [0, 1] is a (measurable) symmetric function, called a graphon. Graphons are unique up to measure preserving transformations of [0, 1]. Vast literature: Aldous, Hoover, Kallenberg, Diaconis and Freedman, Chayes, Borgs and Lovász, etc ect... Key point: only the finite marginals of p∞ ∈ E∞ can be realized. General exchangeable models are mixtures of such distributions.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 6/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Graphons and homomorphism densities

For G ∈ Ln and H ∈ Lk with k ≤ n, the density homomorphism of H in G is t(H, G) = ∣hom(H, G)∣ nk .

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 7/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Graphons and homomorphism densities

For G ∈ Ln and H ∈ Lk with k ≤ n, the density homomorphism of H in G is t(H, G) = ∣hom(H, G)∣ nk . Convergence of graph sequences = convergence of marginal probabilities A sequence {Gn}n∈N converges if and only if, for some graphon f and each H ∈ L with k nodes, lim

n→∞t(H, Gn) = ∫[0,1]k

∏

(i,j)∈E(H)

f(zi, zj) dz1 . . . zk = P (H ⊆ G′) , G′ a random graph distributed like the pk

∞, p∞ ∈ E∞ defined by f.

The sequence {t(H, f)}H∈L of density homomorphisms uniquely specifies p∞.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 7/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Finite Exchangeability

But real networks are finite! So, what can be said about the set Pn of exchangeable distribution on Ln?

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 8/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Finite Exchangeability

But real networks are finite! So, what can be said about the set Pn of exchangeable distribution on Ln? Finite exchangeability does not yield consistent models Finite exchangeable probability distributions marginalize to but need not be extendable to exchangeable distributions.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 8/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Finite Exchangeability

But real networks are finite! So, what can be said about the set Pn of exchangeable distribution on Ln? Finite exchangeability does not yield consistent models Finite exchangeable probability distributions marginalize to but need not be extendable to exchangeable distributions. Our goal We would like to characterize the distributions in Pn that are extendable. We seek to establish a parametric (finite dimensional) representation of all the distributions {pn

∞, p∞ ∈ E∞}.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 8/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The Möius parametrization

It turns out it is convenient to work with maginal instead of joint probabilities. Möbius parameters For any pn ∈ Pn, let zn the vector with entries indexed by subgraphs H of Kn without isolated nodes of the form zn(H) = P (H ⊆ Gn) , where Gn is the random graph with distribution pn. In particular, zn(∅) = 1.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 9/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The Möius parametrization

It turns out it is convenient to work with maginal instead of joint probabilities. Möbius parameters For any pn ∈ Pn, let zn the vector with entries indexed by subgraphs H of Kn without isolated nodes of the form zn(H) = P (H ⊆ Gn) , where Gn is the random graph with distribution pn. In particular, zn(∅) = 1. Invertible linear transformation: pn(G) = ∑

H∶E(H)⊇E(G)

(−1)E(H)−E(G)zn(H), ∀G ∈ Ln. By exchangeability, zn(H) = zn(H′) if H and H are isomorphic.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 9/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

A finite deFinetti Theorem

We can describe now the relationships among Möbius parameters of consistent finitely exchangeable distributions. A deFinetti’s theorem for finitely exchageable graphs Assume m > n. Let pm an exchangeable distribution on Lm and zn

m the

Möbius parameters corresponding to pn

• m. Then,

max

H

∣zn

m(H) − ∑ G∈Gm

t(H, G)pm(G)∣ ≤ 1 − (m)n mn . A similar guarantee holds for the pn’s. See also Matúš for more general statements.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 10/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The dissociated property

Corollary (The dissociated property) If pn ∈ Pn is extendable to an (infinite) exchangeable distribution in E∞, then it satisfies the dissociated property: zn

∞(H) = zn(H) = zn(H1)zn(H2)

for all subgraphs H = H1 ⊎ H2 of Kn without isolated nodes. Extendable distributions in Pn are mixtures of dissociated distributions in Pn. A distribution p∞ on L∞ is in E∞ if and only if pn

∞ satisfies the

dissociated property for all n.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 11/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The dissociated property

Corollary (The dissociated property) If pn ∈ Pn is extendable to an (infinite) exchangeable distribution in E∞, then it satisfies the dissociated property: zn

∞(H) = zn(H) = zn(H1)zn(H2)

for all subgraphs H = H1 ⊎ H2 of Kn without isolated nodes. Extendable distributions in Pn are mixtures of dissociated distributions in Pn. A distribution p∞ on L∞ is in E∞ if and only if pn

∞ satisfies the

dissociated property for all n. Thus, finitely exchangeable distribution must be dissociated in order to be extendable to extremal distributions in E∞. Result is not new, but derivation via finite exchangeability is. ...so what does dissociated distribution in Pn looks like?

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 11/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Bidirected Graphical Models for Binary Data

Graphical models with bidirected edges, where the nodes of the graph represents the variables and lack of (bidirected) edges among nodes signify marginal independence among the corresponding variables. See Richardson (2003), Drton and Richardson (2008) and Roverato, Luparelli and LaRocca (2013). Global Markov property for bidirected (marginal) graphical models A B ∣ C when every path between A and B has a node outside A ∪ B ∪ C. In particular, C may be empty.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 12/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Bidirected Graphical Models for Binary Data

Example (Drton and Richardson, 2008)

X1 X2 X3 X4 X1 X2 X3 X4

In the undirected graph (left), the global Markov property expresses, e.g., that X1 X4 ∣ {X2, X3}, whereas in the bidirected graph (right) the global Markov property expresses, e.g., that X1 X4 {X1, X2} X4 and X1 X4 ∣ X3.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 12/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Bidirected Graphical Models for Binary Data

Bidirected Markov models arise, e.g., as marginals of directed Markov models with unobserved variables. Example (by S. Lauritzen)

X1 X2 X3 X4 U12 U23 U24

In the graph above, the marginal distribution of (X1, X2, X3) will be bidirected Markov w.r.t. the graph

X1 X2 X3 X4

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 13/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The canonical model for exchangeable and extendable networks

Dissociated property and bidirected graphical models A distribution on Ln is dissociated if and only if it is Markov with respect to the bidirected line graph of Kn.

X12 X23 X24 X34 X13 X14

Contrast this with the Markov graphs of Frank and Strauss (1986) which are Markov w.r.t. the undirected line graph.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 14/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The benfits of Möbius parametrization

Using the Möbius parameters are especially convenient because are marginalizable: for any m > n zn

m(H) = zn(H)

for any ubgraph H of Kn without isolated nodes. expresses the bidirected Markov property in a simple way: (From Drton and Richardson, 2008) A distribution pn ∈ Pn is Markov with respect to the bidirected line graph of Kn if and only if for any H = H1 ⊎ H2 ⊎ . . . ⊎ Hl ∈ Lk without isolated nodes, zn(H) = zn(H1) × ⋯ × zn(Hl).

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 15/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The Möbius parametrization

Polynomial parametrization If a probability pn in Pn is extendable to some p∞ ∈ E∞, then pn(G) = ∑

U∈Un∶E(G)⊆E(U)

(−1)E(U)−E(G)r(G, U) ∏

C∈C(U)

zn(C), G ∈ Ln, where D(U) denotes the maximal connected components of U and r(G, U) are the number of graphs in Ln that contain G as a subgraph and are isomorphic to U ∈ Un. This defines a smooth parametrization, described by a smooth manifold inside Pn specified by polynomial equations. Its dimension is the number

• f connected subgraphs of all unlabeled graphs on n nodes.
• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 16/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The curved exponential family parametrization

Exponential parametrization If a probability pn in Pn is extendable to some p∞ ∈ E∞, then pn(G; ν) = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∑

U∈Un

νUs(U, G) − ψ(ν) ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ , G ∈ Ln, ν ∈ V ⊂ R∣Un∣−1, where s(U, G) is the number of non-empty subgraphs of G isomorphic to U ∈ Un and ψ a normalizing constant.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 17/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

The curved exponential family parametrization

Exponential parametrization If a probability pn in Pn is extendable to some p∞ ∈ E∞, then pn(G; ν) = exp ⎧ ⎪ ⎪ ⎨ ⎪ ⎪ ⎩ ∑

U∈Un

νUs(U, G) − ψ(ν) ⎫ ⎪ ⎪ ⎬ ⎪ ⎪ ⎭ , G ∈ Ln, ν ∈ V ⊂ R∣Un∣−1, where s(U, G) is the number of non-empty subgraphs of G isomorphic to U ∈ Un and ψ a normalizing constant. Duality: the mean value parameters are (sums of) the Möbius parameters. The natural parameters ν are not free to vary, as they need to enforce the dissociated property. These are defined implicitly!

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 17/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Example

Suppose we observe the following graph G:

1 2 3 4

Under the assumed bidirected model, the likelihood under the Mobiüs parametrization is p(G) = z⊵ − 2z<

∣ > + zK4.

and under the curved exponential model is P(G; ν) = exp{4ν− + 5ν∧ + ν∥ + ν△ + ν + 2ν⊓ + ν⊵ − ψ(ν)}.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 18/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Maximum Likelhood Estimation

Given a observation G ∈ Ln, the maximum likelihood estimator of pn is the dissociated point in Pn with positive coordinates that maximizes the likelihood of G.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 19/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Maximum Likelhood Estimation

Given a observation G ∈ Ln, the maximum likelihood estimator of pn is the dissociated point in Pn with positive coordinates that maximizes the likelihood of G. Example 1 For the previous network, the MLE is ˆ z− = 1/2, ˆ z∧ = 5/16, ˆ z∥ = 1/4 ˆ z△ = 3/16, ˆ z = 3/16, ˆ z⊓ = 1/8, ˆ z⊵ = 1/16, This estimate represents a mixture of the uniform distribution of all networks isomorphic to G (there are 12), and the empty network, with weights 3/4 and 1/16, respectively. The MLE does not exist! In fact, we conjecture it never exists.

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 19/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

Maximum Likelhood Estimation

Example 2 When the observed graph G is

1 2 3 4

the likelihood function is maximized for any value of λ satisfying 0 ≤ λ ≤ 1/16 with ˆ z− = 1/2, ˆ z∧ = 3/16, ˆ z∥ = 1/4, ˆ z△ = 1/16 − λ, ˆ z = λ, ˆ z⊓ = 1/16, and all other z’s equal to zero. This corresponds to a random network that has probability 3/4 of being isomorphic to G (12 cases) and the remaining probability mass of 1/4 is distributed arbitrarily between a triangle plus an isolated point (4 cases), and a 3-star (4 cases). The MLE does not exist and is not unique!

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 19/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

An open problem

Does a dissociated exchenagble distribution on Ln always extend to some p∞ ∈ E∞? No! Example 1 shows this not the case. So the dissociated property is

• nly necessary for extendability.
• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 20/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

An open problem

Does a dissociated exchenagble distribution on Ln always extend to some p∞ ∈ E∞? No! Example 1 shows this not the case. So the dissociated property is

• nly necessary for extendability.

Open problem Let EPn ⊂ Pn the set of exchenagble and extendable distributions on Ln and DPn ⊂ Pn the distributions that are exchangeable and dissociated. Then EPn ⊂ DPn. What does DPn ∖ EPn look like?

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 20/21

Exchangeability A finite deFinetti theorem Bidirected graphical models

More open problems...

What are the algebraic and geometric properties of the proposed bidirected model for networks? How do we carry out maximum likelihood estimation in this curved exponential family setting?

• A. Rinaldo

Random Networks, Exchangeability and Graphical Models 21/21