Towards Reconfigurable Rack-Scale Networking Tyler Szepesi , Bernard - - PowerPoint PPT Presentation

▶

Sep 08, 2022 29 likes •373 views

Towards Reconfigurable Rack-Scale Networking Tyler Szepesi , Bernard Wong, Tim Brecht, Sajjad Rizvi Cheriton School of Computer Science University of Waterloo April 21, 2015 1 Rack-Scale Computing A B C Traditional Rack: 10s of servers

SLIDE 1

Towards Reconfigurable Rack-Scale Networking

Tyler Szepesi, Bernard Wong, Tim Brecht, Sajjad Rizvi

Cheriton School of Computer Science University of Waterloo

April 21, 2015

1

SLIDE 2

Rack-Scale Computing

A B C D E F G H

Traditional Rack:

◮ 10s of servers ◮ 10s of Gbps per server

2

SLIDE 3

Rack-Scale Computing

A B C D E F G H A B C D E F G H I J K L M N O P

Traditional Rack:

◮ 10s of servers ◮ 10s of Gbps per server

Rack-Scale Computing:

◮ 100s of micro-servers ◮ 100s of Gbps per

micro-server

3

SLIDE 4

Rack-Scale Networking

A B C D E F G H A B C D E F G H I J K L M N O P

A key enabler of rack-scale computing is a network fabric that provides high-bandwidth in a cost effective way. What is the right network fabric?

4

SLIDE 5

Single Switch

Micro-Servers Electrical Switch Network A B C D E F G H I J K L M N O

Requires hundreds of ports at hundreds of Gbps per port

5

SLIDE 6

Oversubscribed Tree

A B C D E F G H I J K L M N O

Limited bandwidth for many communication patterns

6

SLIDE 7

Fat-tree (Folded Clos)

A B C D E F G H I J K L M N O

Costs almost as much for the switching hardware as the micro-servers being networked together

7

SLIDE 8

Distributed Switching (Torus Networks)

B C D E F G H I J K L M N O P A

A tradeoff between long path lengths and high port counts per micro-server

8

SLIDE 9

Reconfigurable Networks

Provide bandwidth where it is needed, when it is needed, and minimize over-provisioning Optical Circuit Switching

◮ High bandwidth ◮ Low cost ◮ Low power consumption

1 2 3 4 IN 1 2 3 4 OUT

9

SLIDE 10

Optical Interconnects

Optical Circuit Network A B C D E F G H I J K L M N O

Most effective when the communication pattern between switch changes slowly

10

SLIDE 11

Rack-Scale Communication

◮ The expected pattern of communication:

◮ Groups of micro-servers are used for a task ◮ New groups are formed for new tasks

◮ High bandwidth is needed between members of the group ◮ Minimal bandwidth is needed for inter-group communication

11

SLIDE 12

Optical Interconnects

A B C D E F G H I J K L M N O

12

SLIDE 13

Optical Interconnects

A B C D E F G H I J K L M N O

13

SLIDE 14

Optical Interconnects

A B C D E F G H I J K L M N O

Groups stay consistent, but the communication pattern among members of the group can change rapidly

14

SLIDE 15

Group Membership

A B C D E F G H I J K L M N O

Use optical circuit switch to connect micro-servers to electrical switches

15

SLIDE 16

Group Membership - Example

A B C D E F G H I J K L M N O

Allows the formation of arbitrary groups of micro-servers, when connectivity is required

16

SLIDE 17

Single Optical Circuit Switch

A B C D E F G H I J K L M N O

Optical circuit switches are not yet available beyond a few hundred ports

17

SLIDE 18

3 Stage Clos

A B C D E F G H I J K L M N O

3 stage Clos provides the same functionality as a single switch

18

SLIDE 19

3 Stage Clos - Example

A B C D E F G H I J K L M N O

19

SLIDE 20

3 Stage Clos - Example

A B C D E F G H I J K L M N O

The exact port on the switch is not important

20

SLIDE 21

2 Stage Clos

A B C D E F G H I J K L M N O

2 stage Clos provides sufficient flexibility to create any group

21

SLIDE 22

2 Stage Clos - Example

A B C D E F G H I J K L M N O

Any micro-server can reach any port on any switch, using 33% fewer optical ports than a 3 stage Clos

22

SLIDE 23

Cost Comparison

Capital Expense 10G 40G 100G Cost (Millions) Fattree OSA

Opt. Clos

1 2 3 4

OSA requires less ports

verall, and is the most cost

effective for lower bandwidths As the bandwidth moves into the 100s of Gbps, the cost of electrical switching dominates

23

SLIDE 24

Power Comparison

Operational Expense Power (kW) Fattree OSA

Opt. Clos

10G 40G 100G 10 20 30 40

Operating optical switches is substantially less power intensive than electrical switches Green rack-scale computing must consider the impact of networking

24

SLIDE 25

Modular Circuit Switching

Perform circuit switching using a distributed set of circuit switches

25

SLIDE 26

Modular Circuit Switching

A B C D E F G H I J K L M N O P

Each micro-server is connected to a switch

26

SLIDE 27

Modular Circuit Switching

A B C D E F G H I J K L M N O P

Each optical switch is connect to a port on an electrical switch

27

SLIDE 28

Modular Circuit Switching

A B C D E F G H I J K L M N O P

28

SLIDE 29

Modular Circuit Switching

A B C E F G I J K M N O

Only deploy the components that are needed

29

SLIDE 30

Modular Circuit Switching

A B C D E F G H I J K L M N O P

Supports various electrical switch sizes

30

SLIDE 31

Direct Connectivity

A B C D E F G H I J K L M N O

Can extend the concept to direct server to server connections

31

SLIDE 32

Direct Connectivity

A B C D E F G H I J K L M N O

Can eliminate some of the electrical switches

32

SLIDE 33

Direct Connectivity

A B C D E F G H I J K L M N O

Adding additional ports to micro-servers would allow dynamic construction of server centric networks

33

SLIDE 34

Summary

◮ What is the right network fabric for rack-scale computing? ◮ Data center networking solutions are not ideal at rack-scale ◮ We propose the use of reconfigurable optics to form groups ◮ The idea extends to dynamically constructing other topologies