SLIDE 1
Johan Ugander, Cornell University Lars Backstrom, Facebook WSDM ‘13
Balanced Label Propagation for Partitioning Massive Graphs Johan - - PowerPoint PPT Presentation
Balanced Label Propagation for Partitioning Massive Graphs Johan - - PowerPoint PPT Presentation
Balanced Label Propagation for Partitioning Massive Graphs Johan Ugander, Cornell University Lars Backstrom, Facebook WSDM 13 Goal: partition a really big graph Motivation: distributed computation Distributing graph calculations (
SLIDE 2
SLIDE 3
▪ Distributing graph calculations ( ‘sharding a graph’) makes
traversal/aggregation very expensive.
Motivation: distributed computation
...
SLIDE 4
▪ Distributing graph calculations ( ‘sharding a graph’) makes
traversal/aggregation very expensive.
▪ Naive sharding:
ID%n == 0 ID%n == 1 ID%n == 2 ID%n == 3 ID%n == n-4 ID%n == n-3 ID%n == n-2 ID%n == n-1
...
Pr( colocation ) = 1 / n
Motivation: distributed computation
SLIDE 5
▪ Distributing graph calculations ( ‘sharding a graph’) makes
traversal/aggregation very expensive.
▪ Naive sharding: ▪ Intelligent sharding: specify a shard map f() that colocates users with friends
Pr( colocation ) = 1 / n
ID%n == 0 ID%n == 1 ID%n == 2 ID%n == 3 ID%n == n-4 ID%n == n-3 ID%n == n-2 ID%n == n-1
...
f(ID) == 0 f(ID) == 1 f(ID) == 2 f(ID) == 3 f(ID) == n-4 f(ID) == n-3 f(ID) == n-2 f(ID) == n-1
...
Motivation: distributed computation
SLIDE 6
▪ Distributing graph calculations ( ‘sharding a graph’) makes
traversal/aggregation very expensive.
▪ Naive sharding: ▪ Intelligent sharding: specify a shard map f() that colocates users with friends
Pr( colocation ) = 1 / n
ID%n == 0 ID%n == 1 ID%n == 2 ID%n == 3 ID%n == n-4 ID%n == n-3 ID%n == n-2 ID%n == n-1
...
f(ID) == 0 f(ID) == 1 f(ID) == 2 f(ID) == 3 f(ID) == n-4 f(ID) == n-3 f(ID) == n-2 f(ID) == n-1
...
Motivation: distributed computation
Oh... and the algorithm better be FAST.
SLIDE 7
▪ Garey, Johnson, Stockmeyer 1976: Minimum bisection is NP-hard ▪ Karypsis and Kumar 1998:
METIS
▪ Feige and Krautgamer 2000:
O(n1/2 log n)-factor approximation
Partitioning a really big graph: How?
SLIDE 8
▪ Garey, Johnson, Stockmeyer 1976: Minimum bisection is NP-hard ▪ Karypsis and Kumar 1998:
METIS
▪ Feige and Krautgamer 2000:
O(n1/2 log n)-factor approximation
▪ METIS does not scale to 100B+ edges. ▪ Need a principled approach, ideally one that can be Hadoop-ified.
Partitioning a really big graph: How?
SLIDE 9
Basic idea: Label propagation
▪ Iteratively move nodes to be with the plurality of their neighbors:
3 9 2 1 4 5 7 8 6
- Zhu, Ghahramani. CMU Tech Report 2002.
- Raghavan, Albert, Kumara. Phys Rev E 2007.
- Barber, Clark. Phys Rev E 2009.
SLIDE 10
Basic idea: Label propagation
▪ Iteratively move nodes to be with the plurality of their neighbors:
3 3 4 4 4 3 3 3 3
- Zhu, Ghahramani. CMU Tech Report 2002.
- Raghavan, Albert, Kumara. Phys Rev E 2007.
- Barber, Clark. Phys Rev E 2009.
SLIDE 11
Basic idea: Label propagation
▪ Iteratively move nodes to be with the plurality of their neighbors: ▪ But how to maintain balance?
3 3 4 4 4 3 3 3 3
- Zhu, Ghahramani. CMU Tech Report 2002.
- Raghavan, Albert, Kumara. Phys Rev E 2007.
- Barber, Clark. Phys Rev E 2009.
SLIDE 12
Basic idea: Label propagation
▪ Iteratively move nodes to be with the plurality of their neighbors: ▪ But how to maintain balance? ▪ Label = machine
machine machine machine machine machine
V1 V3 V2 V5 V4
SLIDE 13
Basic idea: Label propagation
machine machine machine machine machine
x13 P13
if people want to move from 1 to 3. allow only people move, s.t. flow balance. Modification! Figure out who wants to move.
▪ Iteratively move nodes to be with the plurality of their neighbors: ▪ But how to maintain balance? ▪ Label = machine
V1 V3 V2 V5 V4
SLIDE 14
Basic idea: Label propagation
machine machine machine machine machine
▪ Iteratively move nodes to be with the plurality of their neighbors: ▪ But how to maintain balance? ▪ Label = machine
V1 V3 V2 V5 V4
Constraints
x13 P13
if people want to move from 1 to 3. allow only people move, s.t. flow balance. Modification! Figure out who wants to move.
SLIDE 15
Basic idea: Label propagation
machine machine machine machine machine How do we solve for ? xij
▪ Iteratively move nodes to be with the plurality of their neighbors: ▪ But how to maintain balance? ▪ Label = machine
V1 V3 V2 V5 V4
Constraints
x13 P13
if people want to move from 1 to 3. allow only people move, s.t. flow balance. Modification! Figure out who wants to move.
SLIDE 16
Balance via Linear Program
▪ Greedily maximize edge locality with constraints (max/min sizes Si,Ti):
Cumulative gain from moving people (ordered by co-location gain). Solution: number of people to move from i to j.
xij fij(x) x
xij fij(xij)
(piecewise-linear concave!)
SLIDE 17
max
X
X
i,j
fij(xij) s.t. ⇢
(Maximize the co-location gain of all machine swaps) (Subject to balance) (and the number of people available to move)
Balance via Linear Program
Si − |Vi| ≤ P
j6=i(xij − xji)
≤ Ti − |Vi|, ∀i ≤ xij ≤ Pij, ∀i, j
▪ Greedily maximize edge locality with constraints (max/min sizes Si,Ti):
Cumulative gain from moving people (ordered by co-location gain). Solution: number of people to move from i to j.
xij fij(x) x
SLIDE 18
▪ Greedily maximize edge locality with constraints (max/min sizes Si,Ti): ▪ Linear Program: n=78 machines => 12k variables / 400k constraints
(Maximize the co-location gain of all machine swaps)
max
X,Z
X
i,j
zij s.t. 8 < :
(Subject to balance) (and the number of people available to move)
Balance via Linear Program
max
X
X
i,j
fij(xij) s.t. ⇢ Si − |Vi| ≤ P
j6=i(xij − xji)
≤ Ti − |Vi|, ∀i ≤ xij ≤ Pij, ∀i, j Si − |Vi| ≤ P
j6=i(xij − xji)
≤ Ti − |Vi|, ∀i ≤ xij ≤ Pij, ∀i, j −aijkxij + zij ≤ bijk, ∀i, j, k
Cumulative gain from moving people (ordered by co-location gain). Solution: number of people to move from i to j.
xij fij(x) x
SLIDE 19
▪ Summary of algorithm: ▪ Step 1: Figure out who wants to move ▪ Step 2: Solve LP to decide who can move without breaking balance ▪ Step 3: Move those people
Balance via Linear Program
SLIDE 20
▪ Summary of algorithm: ▪ Step 1: Figure out who wants to move ▪ Step 2: Solve LP to decide who can move without breaking balance ▪ Step 3: Move those people
Step 2 is the contribution compared to ordinary Label Prop.
Balance via Linear Program
SLIDE 21
What about geography?
SLIDE 22
Initialization using geography
▪ Possible to do much better than random with Facebook, using geography.
SLIDE 23
▪ Possible to do much better than random with Facebook, using geography. ▪ Spatial model of small-world networks (for routing): Kleinberg 2000 ▪ Validation: Liben-Nowell et al. 2005; Backstrom, Sun, Marlow 2010. ▪ Friendship probability as a function of rank-distance:
1e-06 1e-05 0.0001 0.001 0.01 0.1 10 100 1000 10000 100000 1e+06 Count Rank Number of Friends at Different Ranks Total connections at ranks Best Fit (rank+104)-0.95
– Backstrom, Sun, Marlow 2010
Initialization using geography
SLIDE 24
▪ IP data reveals geographic location of users: ▪ 1,000,000,000 users mapped to 700,000 cities
Initialization using geography
SLIDE 25
▪ IP data reveals geographic location of users: ▪ 1,000,000,000 users mapped to 700,000 cities ▪ Grow equi-population balloons around population centers.
Initialization using geography
SLIDE 26
▪ IP data reveals geographic location of users: ▪ 1,000,000,000 users mapped to 700,000 cities ▪ Grow equi-population balloons around population centers.
Initialization using geography
SLIDE 27
Results: Iteration convergence
Iteration Fraction of edges local 1 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 Geo Random
* *
* = restrained
▪ Geographic initialization ‘converges’ within 1 step
Facebook ( n=800m, |E|=68b )
SLIDE 28
Results: Iteration convergence
▪ Geographic initialization ‘converges’ within 1 step ▪ Random initialization slow to start when: avg degree > # partitions
Use ‘restraint’: only move big gainers (*s below) Facebook ( n=800m, |E|=68b )
Iteration Fraction of edges local 1 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 Geo Random
* *
* = restrained Iteration Fraction of nodes moving 1 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 Geo Random
* *
* = restrained
SLIDE 29
Results: Iteration convergence
Iteration Fraction of edges local 1 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 Geo Random
* *
* = restrained Iteration Fraction of nodes moving 1 2 3 4 5 6 7 8 0.0 0.2 0.4 0.6 0.8 1.0 Geo Random
* *
* = restrained Iteration Fraction of edges local
2 4 6 8 10 12 0.0 0.2 0.4 0.6 0.8 1.0
20 shards 40 60 80 100 Iteration Fraction of nodes moving
1 3 5 7 9 11 0.0 0.2 0.4 0.6 0.8 1.0
20 shards 40 60 80 100
▪ Geographic initialization ‘converges’ within 1 step ▪ Random initialization slow to start when: avg degree > # partitions
Use ‘restraint’: only move big gainers (*s below)
▪ LJ partitioning quality not so dependent on # partitions:
BLP exploiting primarily local structure. LiveJournal (n=4.8m, |E|=42.8m) Facebook ( n=800m, |E|=68b )
SLIDE 30
Results: Machine adjacency matrix
▪ Random initialization + 8 step prop ▪ Geographic initialization ONLY ▪ Geographic + 1 step prop ▪ Targeting n=78 machines:
2 racks of 39, visible as blocks
Random Initialization + Prop Geographic Initialization Geographic Initialization + Prop
aphic Initialization + Prop
- Local fraction
- 1
10 20 30 40 50 60 70 78 k, shards 0.0 0.2 0.4 0.6 0.8 1.0
- Geo + Prop
- Geo
- Random + Prop
SLIDE 31
‘People You May Know’
▪ PYMK = ‘People You May Know’ ▪ Ranked suggestion of friends-of-friends (FoFs) as friends. ▪ Average user has 40k FoFs, widely distributed. ▪ Ranks 145,000,000 suggestions per second. ▪ Graph distributed across 78 machines
with 72GB RAM each.
Me Friends of Friends
ID 24 ID 54 ID 8 ID 4 ID 30 Me = ID #3 Friends IDs: 24, 54, 4, 30, 8
SLIDE 32
▪ PYMK = ‘People You May Know’ ▪ Ranked suggestion of friends-of-friends (FoFs) as friends. ▪ Average user has 40k FoFs, widely distributed. ▪ Ranks 145,000,000 suggestions per second. ▪ Graph distributed across 78 machines
with 72GB RAM each.
Me Friends of Friends
Me = ID #3 Friends IDs: 24, 54, 4, 30, 8 ID 24 ID 54 ID 8 ID 4 ID 30
Want to shard so that my friends on same machine as me!
‘People You May Know’
SLIDE 33
Results: PYMK request concentration
▪ Median number of machines hit per query reduced from 60 to ?.
Pr( Y = k machines queried) 10 20 30 40 50 60 70 77 k, machines 0.00 0.02 0.04 0.06 0.08 0.10 base balloon propagated
SLIDE 34
Pr( Y = k machines queried) 10 20 30 40 50 60 70 77 k, machines 0.00 0.02 0.04 0.06 0.08 0.10 base balloon propagated
▪ Median number of machines hit per query reduced from 60 to 9.
Results: PYMK request concentration
SLIDE 35
▪ Median number of machines hit per query reduced from 60 to 9. ▪ Query time: What about overhead? Faster or slower?
Results: PYMK request concentration
Pr( Y = k machines queried) 10 20 30 40 50 60 70 77 k, machines 0.00 0.02 0.04 0.06 0.08 0.10 base balloon propagated
SLIDE 36
Results: Query time / network traffic
machine rx, average (MB/s) 6 12 18 24 30 36 42 48 54 60 66 72 hour 10 20 30 40 50
base balloon propagated
machine query time, average (ms) 6 12 18 24 30 36 42 48 54 60 66 72 hour 25 50 75 100 125 150
base balloon propagated
Query time Network traffic
▪ Median number of machines hit per query reduced from 60 to 9. ▪ Query time reduced by 49%, traffic reduced by 63%:
SLIDE 37
Conclusions and Future work
▪ Label propagation is fast, we show it can be constrained ▪ Social networks very clustered, making local algorithms very effective ▪ Geographic metadata very useful ▪ Sharding greatly improves distributed graph computations such as PYMK