Exponential Random Graph Models and Their Polytopes Johannes Rauh - - PowerPoint PPT Presentation

Exponential Random Graph Models and Their Polytopes

Johannes Rauh

York University (the one in Canada)

AMS Sectional Meeting Chicago 2015

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 1 / 16

Outline

1

Discrete Exponential Families

2

Graphical Models

3

Exponential Random Graph Models

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 2 / 16

Discrete Exponential Families

Given a finite sample space X and a vector of statistics f : X → Rh, we can associate a discrete exponential family

E =

• Pθ : Pθ(x) = 1

Zθ exp(θ, fx)

• Theorem

Let m = EPθ[f]. Then Pθ = argmax

P:EP[f]=m

H(P).

Jaynes’ principle of maximum entropy: If you know nothing about a distribution but its expectation value E[f], you should use Pθ.

Theorem (MLE)

Let P be the empirical distribution of some data set, and suppose that

EP[f] = EPθ[f]. Then Pθ is the unique maximum likelihood estimate (MLE).

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 3 / 16

The Moment Map

Pθ(x) > 0 for all x, θ. However, P(x) = 0 is possible in the closure/boundary

• f E. The closure is denoted by E.

Definition

The map µ : P → EP[f] is the moment map. The image

M = conv fx : x ∈ X.

is the convex support polytope

Theorem µ restricts to a bijection E M.

The inverse will be denoted by µ−1.

Theorem

Suppose m belongs to the face F ⊆ M, and let P = µ−1(m). Then P(x) > 0 if and only if fx ∈ F.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 4 / 16

Example

The independence model of two binary variables: The idea is that E “looks like” M. The combinatorics of M reflect properties of E.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 5 / 16

Generalized MLEs

Theorem (Generalized MLE)

Let P be the empirical distribution, and let m = EP[f]. Then µ−1(m) is the unique maximum likelihood estimate (MLE) within E. The MLE within E is also called the generalized MLE (GMLE) for E. If µ−1(m) belongs to the boundary ∂E := E \ E, one says that “the MLE does not exist.” If the MLE does not exist:

1

not enough data?

2

structural zero?

The GMLE has no parameters, so how do you estimate?

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 6 / 16

Graphical Models

Let G = (V, E) be a graph with n = |V| nodes. To each node v ∈ V associate a finite random variable Xv, taking values in Xv. Thus,

X = ×v∈V Xv. A joint distribution of (Xv)v∈V is a |V|-dimensional tensor (px1,x2,...,xn)x1∈X1,x2∈X2,...,xn∈Xn.

Denote by C(G) the set of cliques of G (i.e. the complete subgraphs).

Definition

The graphical model is the set of all probability distributions P on X of the form

P(x1, . . . , xn) =

• C={i1,...,ik}∈C(G)

φC(xi1, . . . , ik),

where φC is a positive function.

Lemma

The graphical model is a discrete exponential family. The vector of statistics contains the C-marginals for all C ∈ C(G).

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 7 / 16

Marginal Polytopes

The convex support M of a graphical model is a marginal polytope. Marginal polytopes are 0/1-polytopes (each fx is a 0/1-vector) Therefore, each fx is a vertex of M. Moreover, M is a subpolytope of a hypercube. Every fx has the same number of ones (in some parametrization. . . ) Therefore, all fx lie on a sphere. Marginal polytopes are symmetric

(the symmetry group acts transitively on the vertices)

Graphical models have been proven to be a versatile modelling platform in many applications.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 8 / 16

Exponential Random Graph Models

Exponential Random Graph Models (ERGMs) are discrete exponential families where the sample space G is a set of graphs, e.g.

G = Gn := graphs on n nodes.

Any choice of graph statistics defines an ERGM, for example: Subgraph counts (triangles, cycles, k-stars, . . . ) Degree statistics (average degree, degree distribution, degree sequence, . . . )

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 9 / 16

Changing the Number of Nodes

Often, the number of nodes n is fixed in theory; i.e. the sample space is

G = Gn := graphs on n nodes.

In applications it is often a variable. What happens when changing n? Many graph statistics behave “continuously” when changing n. However: Need to take into account scaling of the parameters! For subgraph densities (normalized counts), the convex support polytopes converge (Engstr¨

• m, Nor´

en 2011):

Mn ⊇ Mn+1 ⊇ · · · ⊇ M∞ :=

• n′ Mn′.

For other statistics (e.g. degree sequences), the number of parameters depends on n (Chatterjee, Diaconis 2011)

n plays a role similar to the sample size.

sparse vs. dense graphs: Restrict G or penalize large edge density?

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 10 / 16

ERGMs and Small Samples

When analyzing networks, often there is just a single observation. The MLE might still exist—provided most fG are not vertices. For many ERGMs, this is indeed the case. The “geometry” of such an ERGMs is determined not only by their convex support M, but also by the location of the points fG within M. Example: Edges and 2-stars (Rauh 2012)

n = 6

0.5 1 0.5 1

n = 7

0.5 1 0.5 1

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 11 / 16

An LDP for ERGMs

Chatterjee and Varadhan (2010) proved a Large Deviation Principle that describes the distribution of fG for many statistics for large n. Intuition: The uniform distribution on (labelled!) graphs on n nodes is equivalent to

n

2

• independent random variables.

As a consequence, most fG lie close to the center of mass of M (corresponding to the uniform distribution). Of those fG that don’t lie close to the center of mass, most lies close to some Erd˝

• s-R´

enyi graph.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 12 / 16

Asymptotics of ERGMs and Degeneracy

Diaconis and Chatterjee (2011) showed for certain examples of subgraph counts that for large n and “reasonably scaled parameters” θ, the random graph Pθ lies close to some Erd˝

• s-R´

enyi graph. In principle, you can always escape the Erd˝

• s-R´

enyi by choosing large parameters—but maybe not in all directions!

n = 7

0.5 1 0.5 1

A similar degeneracy has been observed in applications: The MLEs of real networks are often close to Erd˝

• s-R´

enyi.

=⇒

If you are not happy with Erd˝

• s-R´

enyi, you need to change f .

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 13 / 16

Degree Sequences and Partitions

Some ERGMs don’t have interior points: Let fds(G) = the degree sequence. Mds many interior points. Let fdp(G) = the degree sequence ordered by magnitude, larger degrees first (degree partition). Mdp has no interior point.

(reason: every degree partition has a repetition di = di+1, and di ≥ di+1 defines a facet)

Let fdd(G) = the degree distribution ( all k-stars).

Mdd has no interior point. (reason: a graph cannot have both an isolated node and a

fully connected node)

Mdp and Mds are related (Bhattacharya, Sivasubramanian, Srinivasan 2006): Mds consists of n! copies of Mdp, corresponding to all ways of ordering

the nodes. Further example (joint work with K. Sadeghi, T. Short, ´

• E. Czabarka, L. Szekely):

Bi-degree statistics: Find the smallest face containing some fG in its interior.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 14 / 16

• So. . . what’s the difference between graphical models

and ERGMs?

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 15 / 16

• So. . . what’s the difference between graphical models

and ERGMs?

In graphical models, the nodes are random, but in ERGMS, the edges are random.

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 15 / 16

• So. . . what’s the difference between graphical models

and ERGMs?

In graphical models, the nodes are random, but in ERGMS, the edges are random. Technically true, but. . .

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 15 / 16

Comparing graphical models and ERGMs

Both are very different examples of discrete exponential families. Graphical models are a fixed (but very flexible) class of models. Properties of ERGMs depend very much on the chosen statistics. Typically: graphical models ERGMs every fx is a vertex

←→

most fG are not vertices

fx distributed on a sphere ←→ fG cluster at the center of mass

high symmetry

←→

almost no symmetry

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 16 / 16

Comparing graphical models and ERGMs

Both are very different examples of discrete exponential families. Graphical models are a fixed (but very flexible) class of models. Properties of ERGMs depend very much on the chosen statistics. Typically: graphical models ERGMs every fx is a vertex

←→

most fG are not vertices

fx distributed on a sphere ←→ fG cluster at the center of mass

high symmetry

←→

almost no symmetry Actually, there is a way to combine ERGMs and graphical models: Study conditional independences between edges (Lauritzen, Rinaldo, Sadeghi)

• J. Rauh (YorkU)

Exponential Random Graph Models AMS 2015 16 / 16