BurstRadar Practical Real-time Microburst Monitoring for Datacenter - - PowerPoint PPT Presentation

▶
burstradar
SMART_READER_LITE
LIVE PREVIEW

BurstRadar Practical Real-time Microburst Monitoring for Datacenter - - PowerPoint PPT Presentation

Jun 22, 2023 130 likes •432 views

BurstRadar Practical Real-time Microburst Monitoring for Datacenter Networks Raj Joshi 1 , Ti Ting Qu 2 , Mun Choon Chan 1 , Ben Leong 1 , Boon Thau Loo 3 1 2 3 Microbursts (bursts) Events of intermittent congestion lasting 10s or

slide-1
SLIDE 1

Raj Joshi1, Ti Ting Qu2, Mun Choon Chan1, Ben Leong1, Boon Thau Loo3

BurstRadar

Practical Real-time Microburst Monitoring for Datacenter Networks

1 2 3

slide-2
SLIDE 2

Microbursts (µbursts)

Events of intermittent congestion lasting 10’s or 100’s of µs

  • Common Causes: TCP Incast

Bursty UDP traffic TCP segment offloading

BurstRadar (APSys ‘18) 2

  • Intermittent increase in latency  variability
  • Network jitter and Packet loss

Senders …… Receiver

slide-3
SLIDE 3

Modern Datacenter Networks

Small amounts of queueing (microbursts):

BurstRadar (APSys ‘18) 3

> 10 Gbps 10’s of µs

Latency

Performance

slide-4
SLIDE 4

Modern Datacenter Networks

BurstRadar (APSys ‘18) 4

Detect the occurrence of µbursts & identify the contributing flows!

slide-5
SLIDE 5

Detecting & characterizing µbursts is hard

Measurement study from FB’s datacenter

  • Last for less than 200 µs
  • Occur unpredictably

Traditional sampling-based techniques

  • Cannot even detect microbursts

Commercial Solutions

  • Can detect the occurrence of microbursts
  • Provide no information about the cause

BurstRadar (APSys ‘18) 5

slide-6
SLIDE 6

New Advancements

In-band Telemetry (INT)

  • Adds queuing telemetry info into packets & exports it

to monitoring servers from the last-hop switches

BurstRadar (APSys ‘18) 6

Programmable dataplanes and dataplane telemetry

slide-7
SLIDE 7

Challenges: Effective & real-time monitoring

BurstRadar (APSys ‘18) 7

Using INT to detect µbursts is wasteful

  • Need to capture and export/process telemetry data for

all packets

Since µbursts are unpredictable

  • Expensive computation and delay

Correlate monitoring data from different points in the network

slide-8
SLIDE 8

Solution: Out-band

BurstRadar (APSys ‘18) 8

Key Insight: Key Idea:

  • We can detect the microburst directly on the

switch where it happens

Egress Port Queues Switch’s Queuing Engine

µbursts are localized to a switch’s egress port queue

slide-9
SLIDE 9

Solution: egress pipeline

BurstRadar (APSys ‘18) 9

  • Switching ASIC’s “Buffer and Queuing Engine” (BQE)

does not provide any support to peek into the contents of any queue

slide-10
SLIDE 10

Egress Processing Pipeline

BurstRadar Overview

BurstRadar (APSys ‘18) 10

Egress Port Queues

Ring Buffer Courier Pkt Generator Snapshot Algorithm

Egress Ports Egress Deparser Queuing Telemetry (metadata) Markbit (metadata)

slide-11
SLIDE 11

Egress Processing Pipeline

BurstRadar Overview

BurstRadar (APSys ‘18) 11

Egress Port Queues

Ring Buffer Courier Pkt Generator Snapshot Algorithm

Egress Ports Egress Deparser Mirror Port Queue Courier Packet

slide-12
SLIDE 12

Egress Processing Pipeline

BurstRadar Overview

BurstRadar (APSys ‘18) 12

Egress Port Queues

Ring Buffer Courier Pkt Generator Snapshot Algorithm

Egress Ports Egress Deparser Mirror Port Queue Courier Packet

slide-13
SLIDE 13

Egress Processing Pipeline

BurstRadar Overview

BurstRadar (APSys ‘18) 13

Egress Port Queues

Ring Buffer Courier Pkt Generator Snapshot Algorithm

Egress Ports Egress Deparser Telemetry Info:

  • Pkt 5-tuple
  • Queuing telemetry data

Mirror Port Courier Packet Mirror Port Queue

slide-14
SLIDE 14

Egress Processing Pipeline

BurstRadar Overview

BurstRadar (APSys ‘18) 14

Egress Port Queues

Ring Buffer Courier Pkt Generator Snapshot Algorithm

Egress Ports Egress Deparser Mirror Port

slide-15
SLIDE 15

BurstRadar (APSys ‘18) 15

Courier Pkt Generator Snapshot Algorithm Ring Buffer

slide-16
SLIDE 16

BurstRadar (APSys ‘18) 16

Snapshot Algorithm

“Snapshot” the telemetry info of only the

packets involved in µbursts

Telemetry info: 5-tuple (packet header) ingress/egress timestamps enqueue/dequeue queue depths

(metadata)

slide-17
SLIDE 17

BurstRadar (APSys ‘18) 17

Snapshot Algorithm

Latency Increase Threshold

  • Eg. RTT = 50µs, threshold = 30%, i.e., maximum delay = 15µs

Queue Snapshots Snapshot Algorithm

  • Each packet reports deqQdepth
  • if deqQdepth > threshold, then mark pkt  snapshot
  • Track remaining bytes in the queue
  • elif tracked bytes still remaining then mark pkt  snapshot
slide-18
SLIDE 18

BurstRadar (APSys ‘18) 18

Courier Pkt Generator Snapshot Algorithm Ring Buffer

slide-19
SLIDE 19

BurstRadar (APSys ‘18) 19

Courier Pkt Generator

“Courier” Packets transport the telemetry

info via the switch’s mirror port (out-of-band)

All the data stays together  Avoids the expensive correlation on the monitoring servers

slide-20
SLIDE 20

BurstRadar (APSys ‘18) 20

Courier Pkt Generator

Each marked packet  generate new courier packet clone egress to egress, clone_e2e

  • Copy of the exiting marked packet
  • Place it in the egress queue of the mirror port
slide-21
SLIDE 21

BurstRadar (APSys ‘18) 21

Courier Pkt Generator Snapshot Algorithm Ring Buffer

slide-22
SLIDE 22

BurstRadar (APSys ‘18) 22

Ring Buffer

“Ring Buffer” temporarily stores the

telemetry info of marked packets until they can be copied into the courier packets.

slide-23
SLIDE 23

Evaluation

BurstRadar (APSys ‘18) 23

slide-24
SLIDE 24

Evaluation Setup

Hardware Testbed

  • About 550 lines of p4 code

Generated µburst Traffic Traces

  • µbursts data for “web” and “cache” traffic [IMC ‘17]

Compare BurstRadar against

  • In-band Telemetry (INT)  dataplane-based solution
  • “Oracle” Algorithm  ground truth (exact pkts in µbursts)

BurstRadar (APSys ‘18) 24

BurstRadar Prototype Send/Receive µburst Traffic

slide-25
SLIDE 25

Efficiency

BurstRadar (APSys ‘18) 25

5

5% RTT  10 times less packets compared to INT

No Note: e: 5% RTT ≈ 1.25µs of queuing @10Gbps in our testbed

slide-26
SLIDE 26

Handling Concurrent µbursts

BurstRadar (APSys ‘18) 26

300 entries (8.7KB SRAM) 10 concurrent µbursts (< 0.5%)

No Note: e: 1000 entries (29KB SRAM) fully handle 10 concurrent µbursts

0.5

slide-27
SLIDE 27

Dataplane Resource Utilization

BurstRadar (APSys ‘18) 27

Reso source ce switch ch.p4* p4* Match Crossbar 50.13% Hash Bits 32.35% SRAM 29.79% TCAM 28.47% VLIW Actions 34.64% Stateful ALUs 15.63%

Tofino Resource Utilization (Ring Buffer = 1000 entries)

* resource utilization of a fully-featured datacenter ToR switch

Very little resources  combined with switch.p4

BurstRadar 3.39% 4.83% 4.06% 0.69% 4.69% 12.5% Combined 53.52% 37.18% 33.85% 29.16% 39.33% 28.13%

slide-28
SLIDE 28

Conclusion

BurstRadar (APSys ‘18) 28

  • Microburst monitoring is important
  • High impact on performance
  • BurstRadar can detect and identify Microbursts

effectively and continuously

  • Capture and report the telemetry information of only the

packets involved in microbursts

  • BurstRadar demonstrates that modern programmable

ASICs have made it practical to detect and characterize microbursts at multi-gigabit line rates in high-speed datacenter networks.

slide-29
SLIDE 29

BurstRadar (APSys ‘18) 29