SQR In-network packet loss recovery from link failures for - - PowerPoint PPT Presentation

▶
sqr
SMART_READER_LITE
LIVE PREVIEW

SQR In-network packet loss recovery from link failures for - - PowerPoint PPT Presentation

Mar 24, 2023 442 likes •748 views

SQR In-network packet loss recovery from link failures for high-reliability datacenter networks Ting Qu Raj Joshi Mun Choon Chan 1 2 2 2 2 1 1 Ben Leong Deke Guo Zhong Liu 1 2 Data centers around the world Googles

slide-1
SLIDE 1

1 2

SQR

In-network packet loss recovery from link failures for high-reliability datacenter networks

Ting Qu Raj Joshi Mun Choon Chan Ben Leong Deke Guo Zhong Liu

1，2 1 1 2 2 2

slide-2
SLIDE 2

Data centers around the world

2 Google’s worldwide DC map Facebook DC interior Microsoft’s DC in Dublin, Ireland Global Microsoft Azure DC Footprint

slide-3
SLIDE 3

Low latency is a key requirement

Web search e-commerce database cache

3

Low latency for short messages Better app performance & user experience

slide-4
SLIDE 4

Improve Flow Completion Time (FCT)

  • DCTCP (sigcomm’10)
  • D3 (sigcomm’11)
  • HULL (nsdi’12)
  • pFabric (sigcomm’13)
  • PASE (sigcomm’14)
  • TIMELY (sigcomm’15)
  • FUSO (atc’16)
  • Homa (sigcomm’18)
  • HPCC (sigcomm’19)

…

4

But very few work specifically address how link failures impact FCT

slide-5
SLIDE 5

Link failures are common

  • Gill et al. [1] reported:
  • Link failure are common and can cause loss of a large number of

small packets.

  • The 95th percentile value of link failure is 136 times per day

during their measurement period.

[1] Phillipa Gill, Navendu Jain, and Nachiappan Nagappan. 2011. Understanding network failures in data centers: measurement, analysis, and implications. In Proceedings of SIGCOMM.

5

slide-6
SLIDE 6

Link failure management

Link failure management Link failure detection (e.g., F10) Route recovery Protection (e.g., Conga, Hula, SPIDER) Restoration (e.g. Sharebackup) Packet loss recovery Host-based (e.g., TCP)

Host-based pkt loss recovery can lead to much longer flow completion time (FCT) for short flows

6

slide-7
SLIDE 7

Link failure case

7

Link detection time 30us (F10, NSDI’13) route reconfiguration time 730us (ShareBackup, sigcomm’18) 760us + =

slide-8
SLIDE 8

Long FCT under link failure

Hundreds of µs Tens of ms to 1s

8

Host based recovery is a major contributor to the large increase in FCT

slide-9
SLIDE 9

Why does host-based recovery increase FCT significantly?

  • Packet losses in the TCP three-way handshake
  • Wait at least 1s and retransmit
  • Packet losses in the middle of a cwnd
  • Fast retransmission: 1RTT (100s of us)
  • Packet losses at the tail of a cwnd
  • Retransmission timeout: several ms

1 3 ACK1 ACK2 2 3 SYN ACK SYN, ACK SYN 1 1 2 3 ACK1 ACK1 2 ACK1 4

Can we keep FCT low under link failure for latency-sensitive flows?

9

slide-10
SLIDE 10

Our solution: SQR

Link failure management Link failure detection (e.g., F10) Route recovery Protection (e.g., Conga, Hula, SPIDER) Restoration (e.g., Sharebackup) Packet loss recovery Host-based (e.g., TCP) Host-based (e.g., TCP) In-network (SQR)

10

The network is the “right” place to perform packet loss recovery

slide-11
SLIDE 11

How does SQR keeps FCT low when there is link failure？

Objective:

  • Mask the effect of packet loss from the end-points

during link failure detection time and route reconfiguration time (route failure time). Key idea:

  • Continuously cache recently sent packet in

the switch for a duration equal to the route failure time

11

slide-12
SLIDE 12

Is it feasible to cache pkts on switch?

Buffer size

PortLand (65ms) SIGCOMM’09 F10 (1ms) NSDI’13 ShareBackup (760us) SIGCOMM’18 9MB Tridernt +’10 16MB Trident 2 +’15 22MB Tomahawk +’16 42MB Tomahawk 2 +’17

Route failure time

12

+ availability of dataplane programming (e.g. P4)

slide-13
SLIDE 13

Where and how to cache?

Challenges

  • The default FIFO queues send out packets as fast as

possible.

  • No BQE today readily provides the queuing discipline

required to realize packets caching with a fixed time.

  • BQE does not support custom packet scheduling algorithms.
  • In a switch dataplane, the packets can only be stored in the

packet buffer within the buffer & queuing engine (BQE).

13

slide-14
SLIDE 14

Solution

 Keep recent copies of transmitted packets by cloning and then recirculating cloned packets to BQE.  Supported by the Portable Switch Architecture (PSA)  Packets are cached for durations sufficiently long to detect link failure and perform route recovery.  Resend cached packets to new route when it is available.

14

slide-15
SLIDE 15

Challenges

“Aging” of packets

Load balancing of circulating packets

Handle packet reordering

15

slide-16
SLIDE 16

Delay timer

. . .

... BQE

Caching queue

Egress pipeline Is delay duration is enough?

CurrentEgressTstamp − StartEgressTstamp; Make a copy

17

Packet is dropped if it has been cached greater than link detection time Transmit packet if this is the first/original packet

slide-17
SLIDE 17

Dynamic queue selection

...

... BQE

Caching queue

Egress pipeline

Caching queue

...

LeastLoadedPort 1 LeastUtilization Port Utilization 1 2 100 … … Port 1 Port 2 （backup path） 50 80 50 80 link down? No Yes mirroring

18

Packets from same flow can be cached on different queues

slide-18
SLIDE 18

Packet order logic

... BQE

Caching queue

Egress pipeline

Pkt tag counter 5 6 PktTag = 5 Backup port

...

Caching queue

19

slide-19
SLIDE 19

Packet order logic

... BQE

Caching queue

Egress pipeline

NextPktTag Compare PktTag with NextPktTag Same 8 9

...

Caching queue

Backup port

20

slide-20
SLIDE 20

Packet order logic

...

... BQE

Caching queue

Egress pipeline

NextPktTag Compare PktTag with NextPktTag larger 8

Caching queue

Backup port

21

slide-21
SLIDE 21

Why it works

  • No packet loss

✓ Cache a copy of sent packets for a duration at least equal to the route failure time ✓ Pkt is sent to backup port if new route is ready

  • Packets in order

✓ Recover lost pkts based on pkt tag

  • Minimize egress processing delays on other flows going

through the switch ✓ Select caching queue from multiple ports ✓ Dynamic least loaded port selection

  • Complements existing methods of link failure detection and

route reconfiguration

22

slide-22
SLIDE 22

Evaluation

  • Hardware Testbed
  • Barefoot Tofino switch
  • Intel Xeon servers equipped with Intel X710 NICs
  • Trace
  • Web search
  • Data mining
  • Schemes compared (SQR implemented in P4)
  • SB’ (simple ShareBackup, 760us route failure time)
  • SB’ + SQR
  • LRR (30us route failure time)
  • LRR + SQR

23

slide-23
SLIDE 23

SQR masks link failures from end-point transport

24

slide-24
SLIDE 24

SQR achieves low FCT under link failure

25

2ms 2ms

slide-25
SLIDE 25

Overhead: Buffer size

28

Steady-state packet buffer consumption with 30us link failure detection time

slide-26
SLIDE 26

Conclusion

  • Design SQR an In-Network packet loss recovery

method which keeps FCT low for latency-sensitive flows when there is link failure.

  • Eliminate packet loss during link failures and

enables handing-off flows seamlessly to alternative paths.

  • SQR can be implemented on any programmable

ASIC based on Portable Switch Architecture (PSA)

29

slide-27
SLIDE 27
slide-28
SLIDE 28

Impact of SQR Traffic

slide-29
SLIDE 29

Overhead: Egress processing

32