Slim Fly: A Cost Effective Low-Diameter Network Topology Images - PowerPoint PPT Presentation

spcl.inf.ethz.ch @spcl_eth T ORSTEN H OEFLER , M ACIEJ B ESTA Slim Fly: A Cost Effective Low-Diameter Network Topology Images belong to their creator!

spcl.inf.ethz.ch @spcl_eth Background I’m an HPC (systems) guy   New to the DC area but very interested and motivated!  Several projects (see last slide)

spcl.inf.ethz.ch @spcl_eth N ETWORKS , L IMITS , AND D ESIGN S PACE  Networks cost 25-30% of a large compute cluster  How much at rack-scale?  Hard limits: network  Router radix radix router radix  Cable length  Soft limits:  Cost  Performance concentration

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth A BRIEF HISTORY OF NETWORK TOPOLOGIES copper cables, small radix switches fiber, high-radix switches Kautz Butterfly Mesh Dragonfly Slim Fly Clos/Benes 2000’s 2008 2014 1980’s ~2005 2007 2008 Hypercube Fat Trees Random Flat Fly Torus Trees ????

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS  Intuition: lower average distance → lower resource needs  A new view as primary optimization target!  Moore Bound [1]: upper bound on the number of routers in a graph with given diameter ( D) and network radix ( k) . [1] M. Miller, J. Siráň . Moore graphs and beyond: A survey of the degree/diameter problem, Electronic Journal of Combinatorics, 2005.

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2  Example Slim Fly design for diameter = 2: MMS graphs [1] (utilizing graph covering) A subgraph with A subgraph with identical groups of routers identical groups of routers [1] B. D. McKay, M. Miller, and J. Siráň . A note on large graphs of diameter two and given maximum degree. Journal of Combinatorial Theory, Series B, 74(1):110 – 118, 1998

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 Groups form a fully-connected bipartite graph

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 E Example : 1 Select a prime power q 50 routers Construct a finite field . 2 network radix : 7 Assuming q is prime: A Slim Fly based on : Number of routers: with modular arithmetic. Network radix:

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 Label the routers 3 Set of routers: E Example: … Routers (0,.,.) Routers (1,.,.) (1,1,.) (1,2,.) (1,3,.) (1,4,.) (0,1,.) (0,2,.) (0,3,.) (0,4,.) (1,0,.) (0,0,.) (1,4,0) (0,0,0) (1,4,1) (0,0,1) (1,4,2) (0,0,2) (1,4,3) (0,0,3) (1,4,4) (0,0,4)

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 E Example: Find primitive element Build Generator Sets 4 5 generates : All non-zero elements of can be written as

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 Intra-group connections 6 E Example: Two routers in one group are connected iff their “vertical Manhattan distance” is an Take Routers element from: (for subgraph 0) (for subgraph 1) (0,0,0) (0,0,1) (0,0,2) (0,0,3) (0,0,4)

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 Intra-group connections 6 E Example: Two routers in one group are connected iff their “vertical Manhattan distance” is an Take Routers element from: (for subgraph 0) (for subgraph 1)

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY C ONNECTING ROUTERS : D IAMETER 2 E Example: Inter-group connections 7 Take Router Take Router Router iff Take Router (1,0,0) (1,1,0)

spcl.inf.ethz.ch @spcl_eth D ESIGNING A N E FFICIENT N ETWORK T OPOLOGY A TTACHING ENDPOINTS : D IAMETER 2  How many endpoints do we attach to each router?  As many to ensure full global bandwidth:  Global bandwidth: the theoretical cumulative throughput if all endpoints simultaneously communicate with all other endpoints in a steady state network radix = 67% of router radix concentration = 33% of router radix

spcl.inf.ethz.ch @spcl_eth C OMPARISON TO O PTIMALITY  How close is the presented Slim Fly network to the Moore Bound? Networks with diameter = 2

spcl.inf.ethz.ch @spcl_eth O VERVIEW OF OUR R ESEARCH Routing and performance Topology design Attaching endpoints Routing Comparison Optimizing towards of optimality Moore Bound Cost, power, resilience analysis Physical layout Comparison targets Cost model Performance, latency, bandwidth Cost & power results Detailed case-study Resilience

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT Mix (pairwise) groups with different cabling patterns to shorten inter-group cables

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT Merge groups pairwise to create drawers

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT Drawers form a fully-connected graph

spcl.inf.ethz.ch @spcl_eth P HYSICAL L AYOUT ~50% fewer SlimFly: Dragonfly: intra-group cables ~33% higher endpoint density 2( q-1) inter-group One inter-group cable between ~25% fewer cable between two groups routers two groups

spcl.inf.ethz.ch @spcl_eth C OST C OMPARISON Assuming COTS material costs and R ESULTS best known layout for each topology! 100 Total cost [millions of $] 75 50 25 0 0 10 20 30 40 50 Number of endpoints [thousands]

spcl.inf.ethz.ch @spcl_eth C OST & P OWER C OMPARISON D ETAILED C ASE -S TUDY  A Rack-Scale Slim Fly with  N = 1,296  k = 22  N r = 162

spcl.inf.ethz.ch @spcl_eth C OST & P OWER C OMPARISON D ETAILED C ASE -S TUDY : HIGH - RADIX TOPOLOGIES Fat tree 3D Torus 5D Torus Fat tree Random Dragfly Dragfly SF 3D Torus 5D Torus Fat tree Dragfly Dfly SF Random

spcl.inf.ethz.ch @spcl_eth O VERVIEW OF OUR R ESEARCH Routing and performance Topology design Attaching endpoints Routing Comparison Optimizing towards of optimality Moore Bound Cost, power, resilience analysis Physical layout Comparison targets Cost model Performance, latency, bandwidth Cost & power results Detailed case-study Resilience

spcl.inf.ethz.ch @spcl_eth P ERFORMANCE & ROUTING  Cycle-accurate simulations [1]  Routing protocols:  Minimum static routing  Valiant routing [2]  Universal Globally-Adaptive Load-Balancing routing [3] UGAL-L: each router has access to its local output queues UGAL-G: each router has access to the sizes of all router queues in the network 3 1 4 2 [1] N. Jiang et al. A detailed and flexible cycle-accurate Network-on-Chip simulator. ISPASS’13 [2] L. Valiant. A scheme for fast parallel communication. SIAM journal on computing, 1982 [3] A. Singh. Load-Balanced Routing in Interconnection Networks. PhD thesis, Stanford University, 2005

spcl.inf.ethz.ch @spcl_eth P ERFORMANCE & ROUTING R ANDOM UNIFORM TRAFFIC

spcl.inf.ethz.ch @spcl_eth S UMMARY Credits Topology design Maciej Besta Optimizing towards (PhD Student the Moore Bound @SPCL) reduces expensive network resources Advantages of SlimFly Cost & power Performance Resilience Avg. distance Diameter Bandwidth Optimization approach Combining mathematical optimization and current technology trends effectively tackles challenges in networking M. Besta , TH: “Slim Fly: A Cost Effective Low - Diameter Network Topology“, SC15