Understanding Network Failures in Data Centers: Measurement, - - PowerPoint PPT Presentation

▶
understanding network failures in data centers
SMART_READER_LITE
LIVE PREVIEW

Understanding Network Failures in Data Centers: Measurement, - - PowerPoint PPT Presentation

Mar 04, 2024 168 likes •474 views

SIGCOMM 2011 Toronto, ON Aug. 18, 2011 Understanding Network Failures in Data Centers: Measurement, Analysis and Implications Phillipa ipa Gill Navendu ndu Jain in & & Nachia chiapp ppan an Nag agappan appan Microsof osoft

slide-1
SLIDE 1

Understanding Network Failures in Data Centers: Measurement, Analysis and Implications

Phillipa ipa Gill

Uni Univer ersity ity of Toron

  • nto

Navendu ndu Jain in & & Nachia chiapp ppan an Nag agappan appan

Microsof

  • soft

t Resear search

1

SIGCOMM 2011 Toronto, ON

  • Aug. 18, 2011
slide-2
SLIDE 2

Motivation

2

slide-3
SLIDE 3

Motivation

$5,600 per minu nute We need to understand failures to prevent and mitigate them!

3

slide-4
SLIDE 4

Overview

Our goal: : Improve reliability by understanding network failures

  • 1. Failure charact

acteriz erization ation

– Most failure prone components – Understanding root cause

  • 2. What is the impact

act of failure?

  • 3. Is redun

undanc ancy effective? Our r cont ntri ribution: bution: First st large ge-sc scale ale empir iric ical al study udy of net networ

  • rk

k failure ures across ss multi ltiple ple DCs

  • Methodology to extract failures from noisy data sources.
  • Correlate events with network traffic to estimate impact
  • Analyzing implications for future data center networks

4

slide-5
SLIDE 5

Road Map

Motivation ivation Backgr ground und & M Met ethodo dology

  • gy

Res esults lts

  • 1. Characterizing failures
  • 2. Do current network redundancy strategies help?

Conclusion lusions

5

slide-6
SLIDE 6

Internet

Data center networks overview

Servers Top of Rack (ToR) switch Aggregation “Agg” switch Load balancers Access routers/network “core” fabric

6

slide-7
SLIDE 7

Internet

Data center networks overview

7

How effective is redundancy? What is the impact of failure? Which components are most failure prone? What causes failures? ?

slide-8
SLIDE 8

Failure event information flow

  • Failure is logged in numerous data sources

LINK DOWN! Syslog, SNMP traps/polling Network event logs Troubleshooting Tickets Troubleshooting LINK DOWN! Ticket ID: 34 LINK DOWN! Diary entries, root cause Network traffic logs 5 min traffic averages on links

8

slide-9
SLIDE 9

Data summary

  • One year of event logs from Oct. 2009-Sept. 2010

– Network event logs and troubleshooting tickets

  • Network event logs are a combination of Syslog, SNMP

traps and polling

– Caveat: may miss some events e.g., UDP, correlated faults

  • Filtered by operators to acti

tionable

  • nable events

– … still many warnings from various software daemons running

9

Key challenge: How to extract failures of interest?

slide-10
SLIDE 10

Network event logs

Extracting failures from event logs

  • Defi

finin ning failu lures res – Device failure: device is no longer forwarding traffic. – Link failure: connection between two interfaces is down.

Detected by monitoring interface state.

  • Dealing

ing wi with inconsist nsisten ent t data: – Devices:

  • Correlate with link failures

– Links:

  • Reconstruct state from logged messages
  • Correlate with network traffic to determine impact

10

slide-11
SLIDE 11

Reconstructing device state

  • Devices may send spurious DOWN messages
  • Verify at least one link on device fails within five minutes

– Conservative to account for message loss (correlated failures)

DEVICE DOWN! Top-of-rack switch Aggregation switch 1 Aggregation switch 2 LINK DOWN! LINK DOWN!

This sanity check reduces device failures by 10x

11

slide-12
SLIDE 12

Reconstructing link state

  • Inconsistencies in link failure events

– Note: our logs bind each link down to the time it is resolved

What we expect

UP DOWN Link state LINK DOWN! LINK UP! time

12

slide-13
SLIDE 13
  • Inconsistencies in link failure events

– Note: our logs bind each link down to the time it is resolved

Reconstructing link state

What we sometimes see.

UP DOWN Link state LINK DOWN 1! LINK UP 1! time LINK DOWN 2! LINK UP 2!

? ?

How to deal with discrepancies?

  • 1. Take the earliest of the down times
  • 2. Take the earliest of the up times

13

slide-14
SLIDE 14

Identifying failures with impact

  • Summar

ary y of impa pact ct:

– 28.6% of failures impact network traffic – 41.2% of failures were on links carrying no no traf affic ic

  • E.g., scheduled maintenance activities
  • Cavea

veat: Impact is only on network traffic not ne neces essari arily ly ap appl plic icat ation ions! s!

– Redundancy: Network, compute, storage mask outages

Network traffic logs

LINK DOWN LINK UP BEFORE RE DURING AFTER

Correlate link failures with ne netwo work k traffic

time

Only consider events where traffic decreases ses

14

𝒖𝒔𝒃𝒈𝒈𝒋𝒅 𝒆𝒗𝒔𝒋𝒐𝒉 𝒖𝒔𝒃𝒈𝒈𝒋𝒅 𝒄𝒇𝒈𝒑𝒔𝒇 < 𝟐

slide-15
SLIDE 15

Road Map

Motiv tivati ation Background & Methodology

Res esul ults ts

1. 1. Charact racter eriz izing ing failur lures – Distribution of failures over measurement period. – Which components fail most? – How long do failures take to mitigate? 2. Do current network redundancy strategies help?

Conclu nclusio sions ns

15

slide-16
SLIDE 16

All Failures 46K

Visualization of failure panorama: Sep’09 to Sep’10

2000 4000 6000 8000 10000 12000 Oct-09 Dec-09 Feb-10 Apr-10 Jul-10 Sep-10

Links s sorted ed by data a center er Time (binne ned d by day) Widespread failures Long lived failures.

16

Link Y had failure on day X.

slide-17
SLIDE 17

All Failures 46K

2000 4000 6000 8000 10000 12000 Oct-09 Dec-09 Feb-10 Apr-10 Jul-10 Sep-10

Links s sorted ed by data a center er Time (binne ned d by day)

Visualization of failure panorama: Sep’09 to Sep’10

Failures with Impact 28% Load balancer update (multiple data centers) Component failure: link failures on multiple ports

17

slide-18
SLIDE 18

Internet

Which devices cause most failures?

18

?

slide-19
SLIDE 19

38% 28% 15% 9% 4% 4% 2% 18% 66% 5% 8% 0.4%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% LB-1 LB-2 ToR-1 LB-3 ToR-2 AggS-1

Percentag centage Device ce type pe

failures downtime

Which devices cause most failures?

Top of rack switches have few failures… (annu nual al pro rob. . of fail ilure e <5%) Load balancer 1: very little downtime relative to number of failures.

19

…but a lot of downtime! Load Balan ancer cer 1 Load Balan ancer cer 2 Top of Rack k 2 Aggregati egation

  • n

Switch tch Top of Rack k 1 Load Balan ancer cer 3

Device e Type

slide-20
SLIDE 20

Internet

How long do failures take to resolve?

20

slide-21
SLIDE 21

How long do failures take to resolve?

Load balancer 1: short-lived transient faults Correlated failures on ToRs connected to the same Aggs.

21

Median time to repair: 4 mins Median time to repair: ToR-1: 3.6 hrs ToR-2: 22 min Median time to repair: 5 minutes Mean: 2.7 hours Load Balan ancer cer 1 Load Balan ancer cer 2 Top of Rack ck 1 Lo Load Balance ncer r 3 Top of Rack ck 2 Aggregati egation

  • n Switch

ch Overa erall

slide-22
SLIDE 22

Summary

  • Data center networks are highly reliable

– Majority of components have four 9’s of reliability

  • Low-cost top of rack switches have highest reliability

– <5% probability of failure

  • …but most downtime

– Because they are lower priority component

  • Load balancers experience many short lived faults

– Root cause: software bugs, configuration errors and hardware faults

  • Software and hardware faults dominate failures

– …but hardware faults contribute most downtime

22

slide-23
SLIDE 23

Road Map

Motivation ivation Background & Methodology Res esults lts

  • 1. Characterizing failures

2.

  • 2. Do
  • curre

rrent nt net network

  • rk re

redu dund ndancy ancy strat rategie gies s he help? p? Conclusion lusions

23

slide-24
SLIDE 24

Is redundancy effective in reducing impact?

Internet

Redundant devices/links to mask failures

  • This is expensive! (management overhead + $$$)

Goal: l: Reroute traffic along available paths How effective is this in practice?

24

X

slide-25
SLIDE 25

Measuring the effectiveness of redundancy Ide dea: compare traffic before and during failure

Measure traffic on links:

  • 1. Before failure
  • 2. During failure
  • 3. Compute “normalized traffic” ratio:

Compare normalized traffic over redundancy groups to normalized traffic

  • n the link that failed

Agg. switch (primary) Agg. switch (backup)

  • Acc. router

(primary)

  • Acc. router

(backup)

X

25

𝒖𝒔𝒃𝒈𝒈𝒋𝒅 𝒆𝒗𝒔𝒋𝒐𝒉 𝒖𝒔𝒃𝒈𝒈𝒋𝒅 𝒄𝒇𝒈𝒑𝒔𝒇 ~𝟐

slide-26
SLIDE 26

Is redundancy effective in reducing impact?

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% All Top of Rack to Aggregation switch Aggregation switch to Access router Core Norma malized ized traffic ic during ng fa failure ure (media dian) n) Per link Per redundancy group

Core link failures have most impact… … but redundancy masks it

Less impact lower in the topology Redundancy is least effective for AggS and AccR

Overall increase of 40% in terms of traffic due to redundancy

26

Internet

slide-27
SLIDE 27

Road Map

Motivation ivation Background & Methodology Res esults lts

  • 1. Characterizing failures
  • 2. Do current network redundancy strategies help?

Con

  • ncl

clusi usion

  • ns

27

slide-28
SLIDE 28

Conclusions

  • Goal: Unde

derstand tand failure ures s in da data center r net etworks ks

– Empirical study of data center failures

  • Key obs

bservati ations:

  • ns:

– Data center networks have high reliability – Low-cost switches exhibit high reliability – Load balancers are subject to transient faults – Failures may lead to loss of small packets

  • Future

re di dire rections: tions:

– Study application level failures and their causes – Further study of redundancy effectiveness

28

slide-29
SLIDE 29

Thanks!

Contact: phillipa@cs.toronto.edu

Proje

  • ject

ct page: e:

http://research.microsoft.com/~navendu/netwiser

29