SLIDE 1
The Hierarchical Structure
- f Networks
Aaron Clauset Santa Fe Institute 4 August 2008 SFI / CAIDA W
- rkshop
Networks and Navigation
The Hierarchical Structure of Networks Aaron Clauset Santa Fe - - PowerPoint PPT Presentation
The Hierarchical Structure of Networks Aaron Clauset Santa Fe - - PowerPoint PPT Presentation
The Hierarchical Structure of Networks Aaron Clauset Santa Fe Institute 4 August 2008 SFI / CAIDA W orkshop Networks and Navigation First, Some Pictures social groups or communities teenage friendships * research collaborations *
SLIDE 2
SLIDE 3
research collaborations teenage friendships
social groups or communities
*image stolen from elsewhere * *
SLIDE 4
metabolites proteins
functional(?) clusters, hierarchies
* * *image stolen from elsewhere
SLIDE 5
amazon.com communities books on politics
co-purchasing (topical?) groups
* *image stolen from elsewhere
SLIDE 6
A Question
How can we extract
- structural patterns
- at many scales
- in a rigorous fashion
from complex networks?
SLIDE 7
What is Structure?
some stylized ideas
SLIDE 8
no structure
SLIDE 9
no structure modular structure
- ne scale
SLIDE 10
no structure modular structure hierarchical structure
- ne scale
multi-scale
SLIDE 11
A Question
How can we extract
- hierarchical structure
- in a rigorous fashion
from complex networks?
→
network data hierarchy ?
SLIDE 12
One Approach
Model-based inference
- 1. describe how to generate hierarchies (a model)
- 2. “fit” model to empirical data
- 3. test “fitted” model
- 4. extract predictions + insight
SLIDE 13
A Model of Hierarchy
SLIDE 14
→
A Model of Hierarchy
probability assortative modules
pr
D, {pr}
SLIDE 15
“inhomogeneous” random graph
→ →
model instance
→
Pr(i, j connected) = pr i j i j = p(lowest common ancestor of i,j)
SLIDE 16
→
SLIDE 17
Model Features
- explicit model = explicit assumptions
- very flexible (many parameters)
- captures structure at all scales
- arbitrary mixtures of assortativity, disassortativity
- learnable directly from data
SLIDE 18
Learning From Data
- We use a Bayesian approach:
- likelihood function
scores quality of model
- sample high quality models via MCMC
- technical details in arXiv : physics/0610051 and
Nature 453, p98 (2008)
L = Pr( data | model )
SLIDE 19
From Graph to Ensemble
SLIDE 20
From Graph to Ensemble
- Given graph
- run MCMC to equilibrium
- then, for each sampled , draw a resampled
graph from ensemble A test: do resampled graphs look like original?
D
G G
SLIDE 21
Grassland species* plant
→ →
herbivore
→
parasite
*thank you: Jennifer Dunne
SLIDE 22
10 10
1
10
!3
10
!2
10
!1
10
a
Degree, k Fraction of vertices with degree k
Degree Distribution
resampled
→
- riginal
→
SLIDE 23
Clustering Coefficient
resampled
→
- riginal
→
0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25
Fraction of graphs with clustering coefficient c Clustering coefficient, c
resampled
→
- riginal
→
SLIDE 24
2 4 6 8 10 10
!3
10
!2
10
!1
10
b
Distance, d Fraction of vertex!pairs at distance d
Distance Distribution
resampled
→
- riginal
→
SLIDE 25
Missing Links
A test: can model predict missing links?
SLIDE 26
Predicting is Hard
- remove edges from
- how easy to guess a missing link?
n = 75 m = 113 pguess ≈ k n2 − m + k = O(n−2) k pguess = k/(2662 + k) G
SLIDE 27
- Given incomplete graph
- run MCMC to equilibrium
- then, over sampled , compute average
for links
- predict links with high values are missing
Test idea via leave-k-out cross-validation perfect accuracy: AUC = 1 no better than chance: AUC = 1/2
(i, j) ∈ G
Predicting Missing Links
D
pr G pr
SLIDE 28
Missing Structure
0.2 0.4 0.6 0.8 1 0.4 0.5 0.6 0.7 0.8 0.9 1 Area under ROC curve Fraction of edges observed, k/m Grassland species network Pure chance Common neighbors Jaccard coeff. Degree product Shortest paths Hierarchical structure
simple predictors
→
hierarchy
→
pure chance
→
AUC
SLIDE 29
0.2 0.4 0.6 0.8 1 0.4 0.5 0.6 0.7 0.8 0.9 1 AUC Fraction of edges observed Terrorist association network
a
Pure chance Common neighbors Jaccard coefficient Degree product Shortest paths Hierarchical structure
Other Networks
0.2 0.4 0.6 0.8 1 0.4 0.5 0.6 0.7 0.8 0.9 1 AUC Fraction of edges observed
- T. pallidum
metabolic network
b
Pure chance Common neighbors Jaccard coefficient Degree product Shortest paths Hierarchical structure
SLIDE 30
Summary
- Many real networks are hierarchically modular
- Hierarchies can
- model multi-scale structure
- generalize a single network
- predict missing links
- Model-based inference is very powerful
Acknowledgments:
- C. Moore, M.E.J. Newman, C.H. Wiggins, and C.R. Shalizi
SLIDE 31
Fin
SLIDE 32
Markov chain Monte Carlo (MCMC)
Given , choose random internal node Choose random reconfiguration of subtrees Recompute probabilities and likelihood Sampling states according to their likelihood
D
three subtree configurations
{pr} L
[ergodicity] [detailed balance] (up to relabeling)
SLIDE 33
Grassland species plant
→ →
herbivore
→
parasite
SLIDE 34
c
SLIDE 35
Graph Resampling
SLIDE 36
- 1. Summary Statistics
degree distribution
1 2 3 4 5 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 Distance, d p(d)distance distribution rich-club distribution short-loop distribution betweenness function degree-degree correlations ... etc.
SLIDE 37
- 1. Summary Statistics
The good
- good for exploratory analysis
- often quick calculations
The bad
- throw away important information
- can make different networks appear similar
- what are right statistics to measure?
- different statistics often highly correlated
- indirect measures of large-scale structure, function
SLIDE 38
- 2. Algorithmic Analysis
global modularity Q
C B B U U
local modularity R network motifs box covering clique covering ... etc.
SLIDE 39
- 2. Algorithmic Analysis
The good
- good for exploratory analysis
- illustrate large-scale structure, heterogeneity
The bad
- often (NP-)hard optimizations
- can be sensitive to noise, uncertainty
- ad hoc or heuristic measures of structure, function
- algorithm = theory
- implied physics often unclear
SLIDE 40
- 3. Statistical Inference
hierarchical random graphs community mixtures latent space models information bottlenecks correlation reconstruction network classification
I(X; Y ) = H(X) − H(X|Y )
SLIDE 41
- 3. Statistical Inference
The good
- model-based measures of structure
- concrete, testable predictions
- better robustness to noise, uncertainty
- well-grounded in computer science, statistics
The bad
- models must be explicit, precise
- often hard computations
- data intensive
SLIDE 42
Two Case Studies
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 101 102 103 104 105 106 107 108 109 110 111 112 113 114 100Zachary’s Karate Club NCAA Schedule 2000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34n = 34 m = 78 n = 115 m = 613
SLIDE 43
Mixing Times
MCMC mixes relatively quickly Equilibrium in steps
!"
!#!"
"!"
#!$$" !$"" !!%" !!&" !!'" !!$" !!"" !%" ()*+,-,.$ /01!/)2+/)3004
, ,!"
!#!"
"!"
#!$$"" !$""" !!%"" !!&"" !!'"" !!$"" !!""" !%"" ()*+,-,.$
, ,2565(+7,.89' :;<<,$"""7,.8!!#
O(n2)
→
equilibrium
→
SLIDE 44
Hierarchies
8 14 3 13 4 20 22 1 8 2 12 5 6 7 11 17 1 9 31 23 15 19 21 32 29 2 8 2 4 27 3 16 3 3 10 34 2 6 2 5
point estimate consensus hierarchy
1 5 6 7 11 17 2 8 1 4 3 4 18 2 22 13 1 2 2 5 26 9 31 29 32 10 15 1 6 1 9 21 2 3 24 2 7 28 30 3 3 3 4
SLIDE 45
Hierarchies
BrighamYoung (0) NewMexico (4) SanDiegoStat (9) Wyoming (16) U t a h ( 2 3 ) N V L a s V e g a s ( 1 4 ) AirForce (93) ColoradoStat (41) NMState (69) A r k a n s a s S t a t ( 2 4 ) NorthTexas (11) BoiseState (28) Idaho (50) UtahState (90) OregonState (108) ArizonaState (8) Arizona (22) C a l i f- r
- n
- u
- r
- r
- l
- (
- n
- w
- u
- C
- l
- r
- r
- m
- u
- l
- u
- r
point estimate consensus hierarchy
LouisianTech (58) LouisianMonr (59) MidTNState (63) LouisianaLaf (97) F l- r
- C
- l
- u
- r
- C
- l
- u
- r
- l
- (
- r
- (
- u
- s
- (
- l
- r
- S
- r
- w
- n
- t
- m
- r