Towards Reconfigurable Rack-Scale Networking Tyler Szepesi , Bernard Wong, Tim Brecht, Sajjad Rizvi Cheriton School of Computer Science University of Waterloo April 21, 2015 1
Rack-Scale Computing A B C Traditional Rack: ◮ 10s of servers D ◮ 10s of Gbps per server E F G H 2
Rack-Scale Computing A A C B B D Traditional Rack: ◮ 10s of servers C E G ◮ 10s of Gbps per server D F H E I K Rack-Scale Computing: ◮ 100s of micro-servers J F L ◮ 100s of Gbps per G M O micro-server H N P 3
Rack-Scale Networking A A C B B D A key enabler of rack-scale C E G computing is a network fabric that provides high-bandwidth D F H in a cost effective way. E I K J F L What is the right network fabric? G M O H N P 4
Single Switch Electrical Switch Network Requires hundreds of ports at hundreds of A C E G I K M O Gbps per port J B D F H L N Micro-Servers 5
Oversubscribed Tree Limited bandwidth for many communication A C E G I K M O patterns J B D F H L N 6
Fat-tree (Folded Clos) Costs almost as much for the switching hardware as the micro-servers being A C E G I K M O networked together J B D F H L N 7
Distributed Switching (Torus Networks) A B C D E F G H A tradeoff between long path lengths and high port counts per J I K L micro-server M N O P 8
Reconfigurable Networks Provide bandwidth where it is needed, when it is needed, and minimize over-provisioning 1 2 Optical Circuit Switching IN 3 ◮ High bandwidth 4 1 ◮ Low cost 2 ◮ Low power consumption OUT 3 4 9
Optical Interconnects Optical Circuit Network Most effective when the communication pattern between switch changes slowly A C E G I K M O J B D F H L N 10
Rack-Scale Communication ◮ The expected pattern of communication: ◮ Groups of micro-servers are used for a task ◮ New groups are formed for new tasks ◮ High bandwidth is needed between members of the group ◮ Minimal bandwidth is needed for inter-group communication 11
Optical Interconnects A C E G I K M O J B D F H L N 12
Optical Interconnects A C E G I K M O J B D F H L N 13
Optical Interconnects Groups stay consistent, but the communication pattern among members of the group A C E G I K M O can change rapidly J B D F H L N 14
Group Membership Use optical circuit switch to connect micro-servers to electrical switches A C E G I K M O J B D F H L N 15
Group Membership - Example Allows the formation of arbitrary groups of micro-servers, when connectivity is required A C E G I K M O J B D F H L N 16
Single Optical Circuit Switch Optical circuit switches are not yet available beyond a few hundred ports A C E G I K M O J B D F H L N 17
3 Stage Clos 3 stage Clos provides the same functionality as a single switch A C E G I K M O J B D F H L N 18
3 Stage Clos - Example A C E G I K M O J B D F H L N 19
3 Stage Clos - Example The exact port on the switch is not important A C E G I K M O J B D F H L N 20
2 Stage Clos 2 stage Clos provides sufficient flexibility to create any group A C E G I K M O J B D F H L N 21
2 Stage Clos - Example Any micro-server can reach any port on any switch, using 33% fewer optical ports than a 3 stage Clos A C E G I K M O J B D F H L N 22
Cost Comparison Fattree Opt. Clos OSA Capital Expense OSA requires less ports overall, and is the most cost 4 Cost (Millions) effective for lower bandwidths 3 As the bandwidth moves into 2 the 100s of Gbps, the cost of electrical switching dominates 1 10G 40G 100G 23
Power Comparison Fattree Opt. Clos OSA Operational Expense Operating optical switches is substantially less power 40 intensive than electrical switches Power (kW) 30 20 Green rack-scale computing must consider the impact of networking 10 10G 40G 100G 24
Modular Circuit Switching Perform circuit switching using a distributed set of circuit switches 25
Modular Circuit Switching A B C D Each micro-server is connected to a switch E F G H J I K L M N O P 26
Modular Circuit Switching A B C D Each optical switch is connect to a port on an electrical switch E F G H J I K L M N O P 27
Modular Circuit Switching A B C D E F G H J I K L M N O P 28
Modular Circuit Switching A B C Only deploy the components that are needed E F G J I K M N O 29
Modular Circuit Switching A B C D Supports various electrical switch sizes E F G H J I K L M N O P 30
Direct Connectivity Can extend the concept to direct server to server connections A C E G I K M O J B D F H L N 31
Direct Connectivity Can eliminate some of the electrical switches A C E G I K M O J B D F H L N 32
Direct Connectivity Adding additional ports to micro-servers would allow dynamic construction of server centric networks A C E G I K M O J B D F H L N 33
Summary ◮ What is the right network fabric for rack-scale computing? ◮ Data center networking solutions are not ideal at rack-scale ◮ We propose the use of reconfigurable optics to form groups ◮ The idea extends to dynamically constructing other topologies 34
Recommend
More recommend
