Understanding Understanding Lifecycle Management Lifecycle - PowerPoint PPT Presentation

Understanding Understanding Lifecycle Management Lifecycle Management Complexity of Datacenter Complexity of Datacenter Topologies Topologies Mingyang Zhang (USC) Mingyang Zhang (USC) Radhika Niranjan Mysore (VMware Research) Sucha Supittayapornpong (USC) Radhika Niranjan Mysore (VMware Research) Ramesh Govindan (USC) Sucha Supittayapornpong (USC) Ramesh Govindan (USC) 1

Datacenter topology designs 5-layer Clos Jellyfish [NSDI12] Xpander [CoNEXT16] 2

Previous focus Clos Capacity Jellyfish Xpander Cost ($) 3

Manageability has received very little attention! Clos Capacity Jellyfish Xpander Cost ($) 4

Manageability has received very little attention! Clos Management complexity Capacity Jellyfish Xpander Cost ($) 5

How does the complexity of managing data centers depend on the topology? Our Focus: Lifecycle management 6

Lifecycle management of datacenter topologies Deployment Logical topology Physical topology 7

Lifecycle management of datacenter topologies Deployment Logical topology Physical topology Expansion New added switches 8

Management complexity is important ⎯ Complex deployment stalls the rollout of services for a long time 9

Management complexity is important ⎯ Complex deployment stalls the rollout of services for a long time ⎯ Expensive considering the increasing traffic demand 10 From Singh et al. Sigcomm15

Management complexity is important ⎯ Topology expansion leads to capacity drop due to rewiring ⎯ Complex expansion leads to degraded capacity for a long time New added switches 11

Challenges Contributions 12

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion 13

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned 14

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned New topology Is there a topology family with lower management ⎯ complexity, lower cost and high capacity? FatClique 15

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned New topology Is there a topology family with lower management ⎯ complexity, lower cost and high capacity? FatClique 16

Lifecycle management overview ⎯ Problems: packaging, wiring, placement, rewiring... ⎯ Constraints: switch, rack, patch panel, cable tray... Broadcom Trident 3 Rack Optical patch panel Cable tray 17

Methodology From first principles ⎯ Understand in detail how topologies are deployed and expanded ⎯ Derive metrics that capture the complexity of these operations 18

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned New topology Is there a topology family with lower management ⎯ complexity, lower cost and high capacity? FatClique 19

Packaging Deployment switch server ... data center racks 20

Packaging Deployment switch server ... server rack server rack switch rack switch rack Metric: number of switches 21

Wiring Deployment Intra-rack links: short and cheap switch rack 22

Wiring complexity Deployment Inter-rack links ... ... rack rack 23

Wiring Deployment Inter-rack links over cable trays (expensive) ... ... rack rack Main wiring complexity comes from inter-rack links! 24

Cable bundling Deployment Too many fibers to be handled individually! 25

Cable bundling Deployment Cable bundle ⎯ a fixed number of identical-length fibers between two clusters of network devices. Bundle type ⎯ capacity (# fibers in a bundle) ⎯ length 26

Cable bundling Deployment Bundle type: (bundle capacity, bundle length) Top view of racks w/o bundling 16 individual fibers, 4 types of length 27

Cable bundling Deployment Bundle type: (bundle capacity, bundle length) Top view of racks w/o bundling w/ bundling bundle aggregator 16 individual fibers, 4 types of length 8 equal-length bundles, 1 bundle type Metric: the number of bundle types 28

Cable bundling Deployment It is hard to handle individual fibers with various length! w/o bundling w/ bundling [Singh, et al. Sigcomm15] 29

Role of patch panel in bundling Deployment Aggregator: Patch panel Aggregator 30

Role of patch panel in bundling Deployment Aggregator: Patch panel Manual process Metric: the number of patch panels 31

Deployment complexity metrics 32

Deployment complexity metrics # switches ... 33

Deployment complexity metrics # switches ... # patch panels 34

Deployment complexity metrics # switches ... # patch panels # bundle types 35

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned New topology Is there a topology family with lower management ⎯ complexity, lower cost and high capacity? FatClique 36

Expansion complexity New Metric: # Expansion steps 37

A single expansion step complexity It is hard to move existing links in cable trays 38

A single expansion step complexity Existing links New links Patch panel Patch panel rack rack 39

A single expansion step complexity Existing links New links New Spine Patch panel Patch panel Patch panel Patch panel rack rack rack rack 40

A single expansion step complexity Existing links New links New Spine Patch panel Patch panel Patch panel Patch panel rack rack rack rack Metric: # Rewired links per patch panel rack 41

Metrics Deployment # Switches # Patch panels # Bundle types Expansion # Expansion step # Rewired links per patch panel rack 42

Challenges Contributions Metrics How to characterize the management complexity? ⎯ Deployment ⎯ Expansion Comparison of topologies How does topology structure affect the management ⎯ No topology dominates complexity? ⎯ Principles learned New topology Is there a topology family with lower management ⎯ complexity, lower cost and high capacity? FatClique 43

Topology comparison case study We equalize capacities of topologies 4-layer Clos (Medium) Jellyfish Patch panels Bundle types Switches Re-wired links per patch panel rack Expansion steps 44

Topology comparison case study We equalize capacities of topologies 4-layer Clos (Medium) Jellyfish Patch panels Bundle types Switches Re-wired links per patch panel rack Expansion steps 45

Topology comparison case study We equalize capacities of topologies 4-layer Clos (Medium) Jellyfish Patch panels Bundle types Switches Re-wired links per patch panel rack Expansion steps 46

Topology comparison case study We equalize capacities of topologies 4-layer Clos (Medium) Jellyfish Patch panels Bundle types Switches Re-wired links per patch panel rack Expansion steps No topology dominates by all metrics! 47

Principles learned ⎯ Importance of regularity ⎯ Importance of maximizing intra-rack links ⎯ Importance of fat edge 48

Principle 1: Importance of regularity Jellyfish is a random graph which leads to non-uniform bundles between switch clusters. In large scale, Jellyfish has one order of magnitude more bundle types than Clos! 49

Principle 2: Importance of maximizing intra-rack links Rack in Clos Intra-rack Inter-rack Switch 50

Principle 2: Importance of maximizing intra-rack links Rack in Clos Intra-rack Inter-rack Switch Rack in Jellyfish Intra-rack Inter-rack Switch Most links in Jellyfish are inter-rack links, which leads to more patch panel usage and high wiring complexity! 51

Principle 3: Importance of fat edge Network edge 52

Principle 3: Importance of fat edge Northbound links Switches Southbound links Servers 53

Principle 3: Importance of fat edge Thin Edge North:South = 1:1 54

Principle 3: Importance of fat edge Fat Edge Thin Edge North:South = 1:1 North:South = 2:1 55

Principle 3: Importance of fat edge Residual capacity requirement during expansion: 75% Rewiring leads to capacity drop; Drain traffic before rewiring Draining 25% links --> 25% lose Thin Edge Fat Edge North:South = 1:1 North:South = 2:1 56