Predicting intermittent network device failures based on network - - PowerPoint PPT Presentation

▶

Oct 24, 2023 153 likes •468 views

Predicting intermittent network device failures based on network metrics from multiple data sources Supervisors: Authors: P. Boers H.P.M. van Doorn M. Kaat C.H.J. Kuipers SURFnet University of Amsterdam Tuesday 3 juli RP91 1 I

SLIDE 1

Predicting intermittent network device failures based on network metrics from multiple data sources

Authors: H.P.M. van Doorn C.H.J. Kuipers

University of Amsterdam

Supervisors:

P. Boers
M. Kaat

SURFnet

Tuesday 3 juli RP91

SLIDE 2

~690 Million Device Events ~163 Billion Device Metrics

Introduction

Collected Data over 2 years

SLIDE 3

Failures impacting connectivity

Introduction

Relevance

SLIDE 4

Introduction

Research question

To what extent is it possible to predict intermittent network device failures based on network metrics from multiple data sources?

SLIDE 5

Introduction

Sub questions

Which metrics are relevant?
Patterns between failures?
Correlation between data sources?

SLIDE 6

Introduction

Fault vs Failure

Source: Salfner et al. “A Survey of Online Failure Prediction Methods”.

SLIDE 7

Methodology

Identifying outages

Startingpoint: Big outages in the past 2 years: Big: multiple customers losing connectivity Based on:

Ticketing System
Network operators

SLIDE 8

Methodology

Categorizing outages

Intermittent failure

(Spontaneous reboots)

Permanent failure

(Line-card malfunctioning)

SLIDE 9

Switch chassis metrics

CPU and Memory utilization
Temperature
Uptime

Metrics per interface:

Throughput
Unicast packets
Multicast packets
Broadcast packets
In/Out Errors

Methodology

Metrics at hand

SLIDE 10

Device Metrics: Device Events:

Data Sources

Overview Device Data

SLIDE 11

SLIDE 12

Methodology

Line-card failure

Line-card Bor malfunctioning

SLIDE 13

Findings

Line Card fault

SLIDE 14

Some Charts

Results

Packet CRC Error at core router [BOR] # Interface Input Errors

SLIDE 15

Findings

Interface Input errors 11-09-2017 [TRUUS] # Interface Input Errors

SLIDE 16

Findings

Loss of throughput

SLIDE 17

Findings

Spontaneous throughput loss (1)

SLIDE 18

Findings

Spontaneous throughput loss (2)

Syslog event

2018 May 24 09:50:33 active.5410-01.Asd001A.dcn.surf.net DATAPLANE-4-FLOOD_CONTAINMENT_THRESHOLD: chassis(1): :Flood Containment Threshold Event Container LIMIT_2

n l2-ucast EXCEEDED

SLIDE 19

Findings

Spontaneous throughput loss (3)

So is this a real problem?

Roughly 21.000 events for this switch alone

Events M a y h

i d a y s ?

SLIDE 20

Findings

Spontaneous throughput loss (2)

Syslog event

2018 May 24 09:50:33 active.5410-01.Asd001A.dcn.surf.net DATAPLANE-4-FLOOD_CONTAINMENT_THRESHOLD: chassis(1): :Flood Containment Threshold Event Container LIMIT_2

n l2-ucast EXCEEDED

SLIDE 21

Findings

Validating our hypothesis

PDU/sec Time (s) 15 45 Transmitted PDUs Received PDUs Opposite link

SLIDE 22

Identified:

2 cases of permanent line-card faults
Thousands of flood containment events

Challenges:

Data inconsistencies
Measurement errors
No labeled dataset

Discussion

SLIDE 23

Dataset not (yet) suitable for automated predictions
No data that could indicate failure beforehand
Proved link between two datasets
Validated hypothesis

Conclusion

SLIDE 24

Normalizing datasets
Create labeled dataset
Other areas:
Capacity Management
Service Level Specification

Future Work

SLIDE 25

Questions?

SLIDE 26

Backup slides

SLIDE 27

Bonus

Spontaneous throughput loss

SLIDE 28

So is this a real problem?

Bonus

Spontaneous throughput loss

SLIDE 29

Bonus

Spontaneous throughput loss

SLIDE 30

Bonus

Spontaneous reboots