High-resolution Measurement of Data Center Microbursts Qiao Zhang - - PowerPoint PPT Presentation

▶
high resolution measurement of data center microbursts
SMART_READER_LITE
LIVE PREVIEW

High-resolution Measurement of Data Center Microbursts Qiao Zhang - - PowerPoint PPT Presentation

Sep 07, 2022 971 likes •1.39k views

High-resolution Measurement of Data Center Microbursts Qiao Zhang (University of Washington) Vincent Liu (University of Pennsylvania) Hongyi Zeng (Facebook) Arvind Krishnamurthy (University of Washington) Networks are Fast, Measurements are

slide-1
SLIDE 1

High-resolution Measurement of Data Center Microbursts

Qiao Zhang (University of Washington) Vincent Liu (University of Pennsylvania) Hongyi Zeng (Facebook) Arvind Krishnamurthy (University of Washington)

slide-2
SLIDE 2

Networks are Fast, Measurements are not...

Data center networks are getting faster

  • 100Gbps, ~100 ns to process a packet, 10-100 μs RTT

But measurement frameworks are not keeping up

  • SNMP counters (e.g. bytes sent or drops) typically

collected every couple minutes

  • Packet sampling (sFlow or iptables) typically at low

sampling rate, e.g. 1/30k

slide-3
SLIDE 3

Networks are Fast, Measurements are not...

Data center networks are getting faster

  • 100Gbps, ~100 ns to process a packet, 10-100 μs RTT

But measurement frameworks are not keeping up

  • SNMP counters (e.g. bytes sent or drops) typically

collected every couple minutes

  • Packet sampling (sFlow or iptables) typically at low

sampling rate, e.g. 1/30k

Too coarse-grained !

slide-4
SLIDE 4

The Case for High Resolution

  • Packet drop correlates poorly with utilization at 4 minute granularity
  • 4 minute granularity hides short-term traffic spikes
  • Need high-resolution to reveal finer-grained behaviors
slide-5
SLIDE 5

The Case for High Resolution

  • Packet drop correlates poorly with utilization at 4 minute granularity
  • 4 minute granularity hides short-term traffic spikes
  • Need high-resolution to reveal finer-grained behaviors

drop rate generally very low

slide-6
SLIDE 6

The Case for High Resolution

  • Packet drop correlates poorly with utilization at 4 minute granularity
  • 4 minute granularity hides short-term traffic spikes
  • Need high-resolution to reveal finer-grained behaviors

drop rate generally very low unusual drop rates at both low and high utilization

slide-7
SLIDE 7

Roadmap

Mechanism

  • It is possible to do high resolution measurements on

today's switches Results

  • Many if not most traffic bursts are very short-lived
slide-8
SLIDE 8

High-resolution Counter Collection Framework

We designed a high-resolution counter collection framework

  • Switch CPUs poll ASIC registers with microsecond level latency
  • Sample fast (~25 μs) while keeping sampling loss below 1%

We focus on three kinds of counters

  • 1. Byte count: cumulative and used to compute utilization
  • 2. Packet size: a histogram of packet sizes
  • 3. Peak buffer occupancy: for single port and shared pool
slide-9
SLIDE 9

Deployment

  • One of the largest data centers at Facebook with a 3-tier

Clos network

  • Only collect from ToRs due to deployment constraints
  • 10Gbps server links and 4x40Gbps ToR uplinks
slide-10
SLIDE 10

Workload and Methodology

  • Mostly single-role racks
  • Web: handle user request, lookup with cache
  • Cache: handle k-v lookups, respond to Web servers
  • Hadoop: handle batched processing
  • 30 racks in total: 10 racks for each app, over 24 hours
  • Sample a random 2-minute interval per hour, for 1TB+
slide-11
SLIDE 11

Microburst Measurements

  • How long do they last and how often do they occur?
  • How much of congestion is caused by microbursts?
  • Does network behavior differ significantly inside a burst?
  • Are there synchronized behaviors during bursts?

Microburst: a period of short-term high utilization (e.g. >50%)

slide-12
SLIDE 12

Distribution of Link Utilization

25 μs

slide-13
SLIDE 13

Distribution of Link Utilization

a lot of intervals with almost nothing happening 25 μs

slide-14
SLIDE 14

Distribution of Link Utilization

a lot of intervals with almost nothing happening 25 μs a few intervals have ~100% utilization

slide-15
SLIDE 15

Distribution of Link Utilization

a lot of intervals with almost nothing happening insensitive to 50% threshold 25 μs a few intervals have ~100% utilization

slide-16
SLIDE 16

Bursts are Short

  • Burst: an unbroken sequence of hot samples (> 50% util)

25 μs

slide-17
SLIDE 17

Bursts are Short

  • Burst: an unbroken sequence of hot samples (> 50% util)

many bursts last at most 25 μs 25 μs

slide-18
SLIDE 18

Bursts are Short

  • Burst: an unbroken sequence of hot samples (> 50% util)

many bursts last at most 25 μs 90pct at 200 μs 25 μs

slide-19
SLIDE 19

Bursts are Short

  • Burst: an unbroken sequence of hot samples (> 50% util)

many bursts last at most 25 μs 90pct at 200 μs Almost all congestion is short-lived 25 μs

slide-20
SLIDE 20

Time between Bursts

25 μs

slide-21
SLIDE 21

Time between Bursts

For Web/ Hadoop, 50% < 1 RTT

25 μs

slide-22
SLIDE 22

Time between Bursts

For Web/ Hadoop, 50% < 1 RTT Even for cache, median is < 10x RTT

25 μs

slide-23
SLIDE 23

Time between Bursts

  • Some predictability: a burst is likely to

be followed by another relatively soon

  • Potential for re-balance between bursts

For Web/ Hadoop, 50% < 1 RTT Even for cache, median is < 10x RTT

25 μs

slide-24
SLIDE 24

Packet Size Distribution

Inside Burst Outside Burst

100 μs

slide-25
SLIDE 25

Packet Size Distribution

Inside Burst Outside Burst

Bigger packets inside bursts for Web/Cache 100 μs

slide-26
SLIDE 26

Packet Size Distribution

Inside Burst Outside Burst

Bigger packets inside bursts for Web/Cache Burst are correlated with app-level behaviors (e.g. sending bigger responses

  • r scatter-gather/incast)

100 μs

slide-27
SLIDE 27

Directionality of Bursts

300 μs

slide-28
SLIDE 28

Directionality of Bursts

More bursts towards servers due to high fan-in

300 μs

slide-29
SLIDE 29

Directionality of Bursts

More bursts towards servers due to high fan-in

Cache see more bursts on uplinks as responses are typically bigger than requests 300 μs

slide-30
SLIDE 30

Directionality of Bursts

More bursts towards servers due to high fan-in

Cache see more bursts on uplinks as responses are typically bigger than requests 300 μs Bursts are correlated with app behaviors

slide-31
SLIDE 31

Efficacy of Network Load Balancing

  • 4 ToR Uplinks: compute mean absolute deviation (MAD) for each polling interval
  • MAD = mean( |u - u̅| / u̅ ), so MAD=0 means perfect load balancing

40 μs

slide-32
SLIDE 32

Efficacy of Network Load Balancing

  • 4 ToR Uplinks: compute mean absolute deviation (MAD) for each polling interval
  • MAD = mean( |u - u̅| / u̅ ), so MAD=0 means perfect load balancing

links well balanced at 1s scale 40 μs

slide-33
SLIDE 33

Efficacy of Network Load Balancing

  • 4 ToR Uplinks: compute mean absolute deviation (MAD) for each polling interval
  • MAD = mean( |u - u̅| / u̅ ), so MAD=0 means perfect load balancing

links well balanced at 1s scale links are highly unbalanced at 40 μs scale 40 μs

slide-34
SLIDE 34

Efficacy of Network Load Balancing

  • 4 ToR Uplinks: compute mean absolute deviation (MAD) for each polling interval
  • MAD = mean( |u - u̅| / u̅ ), so MAD=0 means perfect load balancing

links well balanced at 1s scale links are highly unbalanced at 40 μs scale Implications for design of network, e.g. for low latency and loss 40 μs

slide-35
SLIDE 35

Conclusions

  • Deployed a microsecond-scale measurement framework

in production

  • Demonstrated it is possible to do high-resolution

measurement on today's switches

  • Microbursts are real, short, correlated, and related to

application behaviors

  • Future work to correlate with end-host measurements to

better understand causes for microbursts