Explicit Expanding Expanders as Datacenter Topologies Michael - - PowerPoint PPT Presentation

▶
explicit expanding expanders as datacenter topologies
SMART_READER_LITE
LIVE PREVIEW

Explicit Expanding Expanders as Datacenter Topologies Michael - - PowerPoint PPT Presentation

Feb 28, 2023 37 likes •943 views

Explicit Expanding Expanders as Datacenter Topologies Michael Dinitz Johns Hopkins University Based on joint work with Michael Schapira, Gal Shahaf, and Asaf Valadarsky (Hebrew University of Jerusalem) Outline Question: how should we wire

slide-1
SLIDE 1

Explicit Expanding Expanders as Datacenter Topologies

Michael Dinitz Johns Hopkins University

Based on joint work with Michael Schapira, Gal Shahaf, and Asaf Valadarsky (Hebrew University of Jerusalem)

slide-2
SLIDE 2

Outline

  • Question: how should we wire datacenters?
  • Expanders!
  • 1. Background on expanders as networks
  • 2. Can we build expanders with additional

properties to ease adoption (incremental expansion)?

  • 3. Can other approaches (degree-diameter graphs)

be viewed as just other expanders?

slide-3
SLIDE 3

Expander Graphs as Network Topologies

slide-4
SLIDE 4

Datacenter Topologies

  • What is the “right” topology? Many competing

proposals!

  • Surprising result [Jellyfish: Singla et al, NSDI ’12]:
  • Random graphs outperform all of them, on almost every metric!
  • And have other nice properties (incremental expansion)
  • Practical?
  • Can we get the benefits of random graphs without

randomness?

  • Why are random graphs good? They’re expanders!
slide-5
SLIDE 5

Expander Graphs

slide-6
SLIDE 6
  • Expanders: never get “trapped” in

a subset of vertices

Expander Graphs

slide-7
SLIDE 7
  • Expanders: never get “trapped” in

a subset of vertices

Expander Graphs

G X

slide-8
SLIDE 8
  • Expanders: never get “trapped” in

a subset of vertices

  • Edge expansion:

Expander Graphs

G X

h(G) = min

X⊂V :|X|≤n/2

|E(X, V \ X)| |X|

slide-9
SLIDE 9
  • Expanders: never get “trapped” in

a subset of vertices

  • Edge expansion:
  • Expander: d-regular graph with

expansion Ω(d)

Expander Graphs

G X

h(G) = min

X⊂V :|X|≤n/2

|E(X, V \ X)| |X|

slide-10
SLIDE 10

Expanders: History

  • Widely studied in graph theory / theoretical CS
  • Many, many applications (mostly complexity

theory)

  • Random graphs are (w.h.p) very good expanders
  • Surprisingly difficult to construct expanders

deterministically

slide-11
SLIDE 11

Data Centers

  • Lots of traffic between nodes
  • In a bad topology, might get “stuck”
  • Problem if lots of traffic from one

section to the rest, not much capacity

  • Lots of traffic everywhere, so traffic

proportional to # vertices

  • Really: want large (edge) expansion!
  • Regular graph (# ports at switches)
slide-12
SLIDE 12

Throughput

  • Given graph G and traffic demand matrix T,

throughput is amount we need to scale down all demands to make feasible

  • Max concurrent flow
  • Important special case: T is all 1’s (all-to-all traffic)
  • Not the only metric for network quality, but an

important one

slide-13
SLIDE 13

Throughput: Theory

  • Thm: If T is all-to-all, then any expander has

throughput within O(log d) of the best possible d- regular graph.

  • Thm: For any T, any expander has throughput

within O(log n) of the best possible d-regular graph (for that T).

  • Thm: For any d-regular graph G, there is some T

and d-regular graph G* such that G* has throughput Ω(log n) more than G.

slide-14
SLIDE 14

Throughput: Theory

  • Thm: If T is all-to-all, then any expander has

throughput within O(log d) of the best possible d- regular graph.

  • Thm: For any T, any expander has throughput

within O(log n) of the best possible d-regular graph (for that T).

  • Thm: For any d-regular graph G, there is some T

and d-regular graph G* such that G* has throughput Ω(log n) more than G.

slide-15
SLIDE 15

Incremental Expansion

Explicit Expanding Expanders. Michael Dinitz, Michael Schapira, Asaf Valadarsky. ESA ‘15

slide-16
SLIDE 16

Incremental Expansion

  • So let’s use expanders for our data centers!
  • Data centers grow regularly: more servers and racks

purchased and added

  • Don’t want to completely rewire network every time!
  • Expander on n nodes should have approximately same

edge set as expander on n+1 nodes

slide-17
SLIDE 17

Random [Jellyfish NSDI’12]

slide-18
SLIDE 18

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-19
SLIDE 19

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-20
SLIDE 20

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-21
SLIDE 21

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-22
SLIDE 22

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-23
SLIDE 23

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-24
SLIDE 24

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-25
SLIDE 25

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-26
SLIDE 26

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-27
SLIDE 27

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

slide-28
SLIDE 28

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

  • Works great in simulation — only

theory for uniform random regular graphs (Bollobas)

slide-29
SLIDE 29

Random [Jellyfish NSDI’12]

  • Construct expanders randomly
  • To add node: choose random d/2

matching, remove, connect to new node

  • Works great in simulation — only

theory for uniform random regular graphs (Bollobas)

  • Will companies actually use random datacenters?
  • Can we get same guarantees with deterministic

constructions?

slide-30
SLIDE 30

Expanding Expanders

slide-31
SLIDE 31

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

slide-32
SLIDE 32

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

Gn

slide-33
SLIDE 33

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

Gn

slide-34
SLIDE 34

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

Gn

slide-35
SLIDE 35

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

Gn

slide-36
SLIDE 36

Expanding Expanders

  • Two problems with using existing

deterministic expanders as data centers

  • 1. Need to exist for all n (not just primes,

powers of 2, etc.)

  • 2. Need to handle incremental

expansion

  • Goal: infinite series of d-regular graphs

Gd+1, Gd+2, Gd+3, … where:

  • 1. Gi has i nodes
  • 2. Each Gi has large edge expansion

(approx. d/2)

  • 3. Few edge changes to get from Gi to

Gi+1 (approx. 3d/2)

Gn

slide-37
SLIDE 37

Explicit Expanding Expanders

Main Result: graphs Gi where:

  • Gi has i nodes
  • Expansion approx. d/3
  • At most 5d/2 edge changes from Gi to Gi+1
slide-38
SLIDE 38

Explicit Expanding Expanders

  • Still room for improvement!
  • Technicality: use multiple edges / edge weights

Main Result: graphs Gi where:

  • Gi has i nodes
  • Expansion approx. d/3
  • At most 5d/2 edge changes from Gi to Gi+1
slide-39
SLIDE 39

2-Lifts

slide-40
SLIDE 40

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-41
SLIDE 41

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-42
SLIDE 42

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-43
SLIDE 43

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-44
SLIDE 44

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-45
SLIDE 45

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

slide-46
SLIDE 46

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

  • Two options for each matching,

so 2|E| possible 2-lifts

slide-47
SLIDE 47

2-Lifts

  • Main tool: 2-Lifts [Bilu-Linial]
  • 2-Lift of G: double every vertex,

replace edge by matching

  • Two options for each matching,

so 2|E| possible 2-lifts

  • Thm [BL]: If G an expander,

random matchings gives good expander w.h.p.

  • Can be derandomized!
slide-48
SLIDE 48

Our Approach

slide-49
SLIDE 49

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

slide-50
SLIDE 50

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

slide-51
SLIDE 51

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

slide-52
SLIDE 52

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

slide-53
SLIDE 53

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

slide-54
SLIDE 54

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

slide-55
SLIDE 55

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

d/2

slide-56
SLIDE 56

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

d/2

slide-57
SLIDE 57

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

d/2

slide-58
SLIDE 58

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

d/2

slide-59
SLIDE 59

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

d/2 d/2 - 1

slide-60
SLIDE 60

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

d/2 d/2 - 1 d/2 - 1

slide-61
SLIDE 61

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

  • Split neighbors: replace two weight 1

edges with matching of weight 2 edges

d/2 d/2 - 1 d/2 - 1

slide-62
SLIDE 62

Our Approach

  • “Split” each node one at a time,

rather than all at once

  • Start with d/2-regular expander,

weights 2

  • Inserting new node: split currently

unsplit node

  • Unsplit neighbors: replace weight 2

edge with two weight 1 edges

  • Split neighbors: replace two weight 1

edges with matching of weight 2 edges

  • Nice property: after all nodes split,

have precisely next BL expander

d/2 d/2 - 1 d/2 - 1

slide-63
SLIDE 63

Analysis: Edge Changes

  • Split u into u, u’:
  • Unsplit neighbor v: one edge
  • f weight 2 → 2 edges of

weight 1. Cost 2

  • Split neighbors v, v’: two

edges of weight 1 → two edges of weight 2, decrease {v, v’} by 1. Cost 5.

  • Add {u,u’} of weight (# unsplit

neighbors)

u v v u u’ v’ u u’ v v’ u v k k-1

slide-64
SLIDE 64

Analysis: Edge Changes

  • Split u into u, u’:
  • Unsplit neighbor v: one edge
  • f weight 2 → 2 edges of

weight 1. Cost 2

  • Split neighbors v, v’: two

edges of weight 1 → two edges of weight 2, decrease {v, v’} by 1. Cost 5.

  • Add {u,u’} of weight (# unsplit

neighbors)

u v v u u’ v’ u u’ v v’ u v k k-1

Total cost: 3*(unsplit neighbors) + 5*(split neighbors) Know 2*(unsplit) + 2*(split) = d

slide-65
SLIDE 65

Future Cuts

  • Cut (A, B)

A B S(A) S(B) U(A) U(B)

Split Unsplit

slide-66
SLIDE 66

Future Cuts

  • Cut (A, B)

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

Split Unsplit

  • Future cut (F(A), F(B)) of next BL expander:
  • F(S(･)) = S(･)
  • F(U(･)) = U(･) and splits of U(･)
slide-67
SLIDE 67

Easy expansion

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

slide-68
SLIDE 68

Easy expansion

  • Know that w(F(A), F(B)) large (≈(d/2) |F(A)|) — argue that w(A, B) close to it

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

slide-69
SLIDE 69

Easy expansion

  • Know that w(F(A), F(B)) large (≈(d/2) |F(A)|) — argue that w(A, B) close to it
  • 2 * w(U(A), U(B)) = w(F(U(A), F(U(B)))

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

slide-70
SLIDE 70

Easy expansion

  • Know that w(F(A), F(B)) large (≈(d/2) |F(A)|) — argue that w(A, B) close to it
  • 2 * w(U(A), U(B)) = w(F(U(A), F(U(B)))
  • w(S(A), S(B)) = w(F(S(A)), F(S(B)))

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

slide-71
SLIDE 71

Easy expansion

  • Know that w(F(A), F(B)) large (≈(d/2) |F(A)|) — argue that w(A, B) close to it
  • 2 * w(U(A), U(B)) = w(F(U(A), F(U(B)))
  • w(S(A), S(B)) = w(F(S(A)), F(S(B)))
  • 2 * w(S(A), U(B)) = w(F(S(A)), F(U(B)))

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B))

slide-72
SLIDE 72

Easy expansion

  • Know that w(F(A), F(B)) large (≈(d/2) |F(A)|) — argue that w(A, B) close to it
  • 2 * w(U(A), U(B)) = w(F(U(A), F(U(B)))
  • w(S(A), S(B)) = w(F(S(A)), F(S(B)))
  • 2 * w(S(A), U(B)) = w(F(S(A)), F(U(B)))

A B S(A) S(B) U(A) U(B) F(A) F(B) F(S(A)) F(S(B)) F(U(A)) F(U(B)) Half the weight, so at least half the expansion (d/4)!

slide-73
SLIDE 73

Real expansion

  • Use fact that fewer vertices than in future cut
  • Combine with Expander Mixing Lemma, get

expansion d/3

  • Need to use strong spectral expansion
  • Fact: there are graphs in sequence with 𝝁2 ≈ d/2, so

cannot get expansion bound directly from Cheeger

slide-74
SLIDE 74

Conclusion (Part II)

  • Can get good incremental expansion by

incrementally 2-lifting

  • Can even do this to any starting expander
  • Many open questions
  • Tight or improved bounds?
  • Heterogeneous nodes? Edges?
slide-75
SLIDE 75

Degree-Diameter Graphs as Expanders

Large Fixed-Diameter Graphs are Good Expanders. Michael Dinitz, Michael Schapira, Gal Shahaf. arXiv ‘17

slide-76
SLIDE 76

Different Intuitions

  • Expanders
  • Good because data can’t get “bottlenecked” anywhere
  • Ensure this by making cuts “large”
  • Alternative
  • Long paths are wasteful: flow of size 𝛽 uses (length × 𝛽)

total capacity

  • So minimize distances: try to make diameter small
slide-77
SLIDE 77

Degree-Diameter Graphs

slide-78
SLIDE 78

Degree-Diameter Graphs

  • Three parameters: size n, degree d, diameter k
  • What are the extremal graphs?
  • Fix d, k. What is largest possible value of n?
  • “Degree/Diameter problem”
slide-79
SLIDE 79

Degree-Diameter Graphs

  • Three parameters: size n, degree d, diameter k
  • What are the extremal graphs?
  • Fix d, k. What is largest possible value of n?
  • “Degree/Diameter problem”
  • “Intuitively, the best known degree-diameter topologies should

support a large number of servers with high network bandwidth and low cost (small degree)... Thus, we propose the best- known degree-diameter graphs as a benchmark for comparison.” — Singla et al, NSDI ’12

  • Slim Fly [Besta-Hoefler, SC ’14]: Uses near-optimal degree-diameter

graphs for k=2 (MMS graphs) and k=3 (BDF and Delorme graphs)

slide-80
SLIDE 80

Informal Result

  • So what are the “best” datacenter topologies?
  • Optimal expanders (Ramanujan graphs)? Or optimal degree-

diameter graphs (Moore graphs)?

  • Similar performance in simulation (degree-diameter graphs

slightly worse)

slide-81
SLIDE 81

Informal Result

  • So what are the “best” datacenter topologies?
  • Optimal expanders (Ramanujan graphs)? Or optimal degree-

diameter graphs (Moore graphs)?

  • Similar performance in simulation (degree-diameter graphs

slightly worse)

Informal result: Any sufficiently good degree- diameter graph is a good expander!

slide-82
SLIDE 82

Informal Result

  • So what are the “best” datacenter topologies?
  • Optimal expanders (Ramanujan graphs)? Or optimal degree-

diameter graphs (Moore graphs)?

  • Similar performance in simulation (degree-diameter graphs

slightly worse)

Informal result: Any sufficiently good degree- diameter graph is a good expander!

  • So finding good degree-diameter graphs involves

finding good expanders

  • Expanders already very good — just use them
slide-83
SLIDE 83

Moore Bound

  • Fix d, k. Obvious upper bound on n:

1 d d(d-1) d(d-1)2

slide-84
SLIDE 84

Moore Bound

  • Fix d, k. Obvious upper bound on n:

1 d d(d-1) d(d-1)2

Moore bound:

µd,k = 1 + d + d(d − 1) + d(d − 1)2 + · · · + d(d − 1)k−1 = 1 + d

k−1

X

i=0

(d − 1)i

slide-85
SLIDE 85

Moore Bound

  • Fix d, k. Obvious upper bound on n:

1 d d(d-1) d(d-1)2

Moore bound:

  • Not achievable in general
  • Lots of work by graph theorists
  • Can get arbitrarily close (as n gets

large) for k = 2, 3, 5

µd,k = 1 + d + d(d − 1) + d(d − 1)2 + · · · + d(d − 1)k−1 = 1 + d

k−1

X

i=0

(d − 1)i

slide-86
SLIDE 86

Formal Results

  • Algebraic / spectral expansion: 𝝁(G) = second

largest eigenvalue of adj. matrix

  • Cheeger inequality: h(G) ≥ (d-𝝁(G))/2
slide-87
SLIDE 87

Formal Results

  • Algebraic / spectral expansion: 𝝁(G) = second

largest eigenvalue of adj. matrix

  • Cheeger inequality: h(G) ≥ (d-𝝁(G))/2

Theorem: Any graph with degree d, diameter k, and n ≥ (1-ε) μd,k has 𝝁(G) ≤ O(ε1/k) d Theorem: Any graph with degree d, diameter k, and n ≥ μd,k - O(dk/2) has 𝝁(G) = O(d1/2)

slide-88
SLIDE 88

Techniques

  • Connection to Geronimus Polynomials
  • Formal polynomials Pt(x), where Pt(A) is matrix giving #

“irreducible walks” of length t

Theorem: Let G be graph with degree d, diameter k, and size n. Then for every nontrivial eigenvalue λ,

  • k

X

t=0

Pt(λ)

  • ≤ µd,k − n
slide-89
SLIDE 89

Conclusion (Part III)

  • Expanders and degree/diameter graphs two different

proposals, from different intuitions

  • But and good degree/diameter graph is a good

expander!

  • Suggests that good performance of degree/diameter graphs because
  • f good expansion
  • Expanders easier to construct: just use a good expander
  • Open questions:
  • What is true relationship? No reason to think our bounds tight
  • Moore bound possibly weak: is an optimal degree/diameter graph a

good expander even if not close to Moore?

slide-90
SLIDE 90

Thanks!