perfSONAR Deployment on ESnet Brian Tierney ESnet ISMA 2011 - - PowerPoint PPT Presentation

▶
perfsonar deployment on esnet
SMART_READER_LITE
LIVE PREVIEW

perfSONAR Deployment on ESnet Brian Tierney ESnet ISMA 2011 - - PowerPoint PPT Presentation

Dec 01, 2022 356 likes •689 views

perfSONAR Deployment on ESnet Brian Tierney ESnet ISMA 2011 AIMS-3 Workshop on Active Internet Measurements Feb 9, 2011 Why does the Network seem so slow ? U.S. Department of Energy | Office of Science Lawrence Berkeley National

slide-1
SLIDE 1

perfSONAR Deployment on ESnet

Brian Tierney ESnet

ISMA 2011 AIMS-3 Workshop on Active Internet Measurements Feb 9, 2011

slide-2
SLIDE 2

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Why does the Network seem so slow ?

slide-3
SLIDE 3

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Where are common problems?

Source Campus Backbone Regional D S Destination Campus NREN

Congested or faulty links between domains Latency dependant problems inside domains with small RTT

slide-4
SLIDE 4

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Local testing will not find all problems

Source Campus R&E Backbone Regional D S Destination Campus Regional

Performance is good when RTT is < 20 ms Performance is poor when RTT exceeds 20 ms Switch with small buffers

slide-5
SLIDE 5

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Soft Network Failures

Soft failures are where basic connectivity functions, but high performance is not possible. TCP was intentionally designed to hide all transmission errors from the user:

  • “As long as the TCPs continue to function properly and the internet

system does not become completely partitioned, no transmission errors will affect the users.” (From IEN 129, RFC 716)

Some soft failures only affect high bandwidth long RTT flows. Hard failures are easy to detect & fix

  • soft failures can lie hidden for years!

One network problem can often mask others

slide-6
SLIDE 6

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Common Soft Failures

Small Queue Tail Drop

  • Switches not able to handle the long packet trains prevalent in long

RTT sessions and local cross traffic at the same time

Un-intentional Rate Limiting

  • Processor-based switching on routers due to faults, acl’s, or mis-

configuration

  • Security Devices
  • E.g.: 10X improvement by turning off Cisco Reflexive ACL

Random Packet Loss

  • Bad fibers or connectors
  • Low light levels due to amps/interfaces failing
  • Duplex mismatch
slide-7
SLIDE 7

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Building a Global Network Diagnostic Framework

slide-8
SLIDE 8

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Addressing the Problem: perfSONAR perfSONAR - an open, web-services-based framework for:

  • running network tests
  • collecting and publishing measurement results

ESnet is:

  • Deploying the framework across the science community
  • Encouraging people to deploy ‘known good’

measurement points near domain boundaries

  • “known good” = hosts that are well configured, enough memory

and CPU to drive the network, proper TCP tuning, clean path, etc.

  • Using the framework to find and correct soft network

failures.

slide-9
SLIDE 9

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

perfSONAR Architecture The perfSONAR framework:

  • Is middleware.
  • Is distributed between domains.
  • Facilitates inter-domain performance information sharing.

perfSONAR services ‘wrap’ existing measurement tools.

slide-10
SLIDE 10

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Lookup Service

  • gLS – Global lookup service used to find services
  • hLS – Home lookup service for registering local perfSONAR metadata

Measurement Archives (data publication)

  • SNMP MA – Interface Data
  • pSB MA -- Scheduled bandwidth and latency data

PS-Toolkit includes these measurement tools:

  • BWCTL: network throughput
  • OWAMP: network loss, delay, and jitter
  • PINGER: network loss and delay

PS-Toolkit includes these Troubleshooting Tools

  • NDT (TCP analysis, duplex mismatch, etc.)
  • NPAD (TCP analysis, router queuing analysis, etc)

perfSONAR Services

slide-11
SLIDE 11

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

ESNet PerfSONAR Deployment

slide-12
SLIDE 12

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

ESnet Deployment Activities

ESnet has deployed OWAMP and BWCTL servers next to all backbone routers, and at all 10Gb connected sites

  • 30 locations deployed, ~20 more planned
  • Full list of active services at:
  • http://stats1.es.net/perfSONAR/directorySearch.html
  • Instructions on using these services for network

troubleshooting: http://fasterdata.es.net These services have proven extremely useful to help debug a number of problems

slide-13
SLIDE 13

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

http://weathermap.es.net

slide-14
SLIDE 14

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Global PerfSONAR-PS Deployments

Based on “global lookup service” (gLS) registration, Feb 2011: currently deployed in over 80 locations

  • ~ 80 bwctl and owamp servers
  • ~ 125 active probe measurement archives
  • ~ 20 SNMP measurement archives
  • Countries include: USA, Australia, Hong Kong, Argentina, Brazil,

Uruguay, Guatemala, Japan, China, Canada, Netherlands, Switzerland

  • Many more deployments behind firewalls

US Atlas Deployment

  • Monitoring all “Tier 1 to Tier 2” connections

For current list of public services, see:

  • http://stats1.es.net/perfSONAR/directorySearch.html

14

slide-15
SLIDE 15

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

SAMPLE results

slide-16
SLIDE 16

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Results

16

Heavily used path: probe traffic is “scavenger service”

Asymmetric Results: different TCP stacks?

slide-17
SLIDE 17

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Results: Finding/Fixing soft failures

Rebooted router with full route table Gradual failure of

  • ptical line card
slide-18
SLIDE 18

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Results: Latency/Loss Data

XXXX

slide-19
SLIDE 19

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Network Research Using perfSONAR data

slide-20
SLIDE 20

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

perfSONAR workshop series

ESnet and Internet2 are actively encouraging researcher use of the data we are collecting NSF, DOE, and LSN sponsored a workshop to discuss the research uses of perfSONAR in Washington DC last summer.

  • 90 attendees!

“The goal of the workshop is to use perfSONAR as a focus to cross- fertilize ideas from the network research community and the needs

  • f the research and education networks around the world,

documenting open areas and best practices.” Workshop Website:

  • http://www.internet2.edu/workshops/perfSONAR/

Workshop Report:

  • http://www.internet2.edu/workshops/perfSONAR/201007perfSONA

R-Workshop-Report.pdf

2/9/2011 20

slide-21
SLIDE 21

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Accessing Archived Results

All results are stored in the perfSONAR “Measurement Archive” (MA)

  • Periodic bwctl tests (throughput)
  • Ongoing owamp tests (latency, loss, jitter)
  • Periodic traceroute tests
  • SNMP results for all router interface, including virtual interfaces
  • ESnet topology

All results are publically accessible Simple Web-service model Easy to use Perl API to query for results See: http://fasterdata.es.net/fasterdata/perfSONAR/client-api/

slide-22
SLIDE 22

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Project: Malathi Veeraraghavan, Univ of Virginia

One-way Active Measurement Protocol(OWAMP) Packet interval: 0.1 sec

  • 10 packets per sec
  • 600 packets per minute

Use perl programs provided by perfSONAR Sample columns of the OWAMP data file:

  • endTime, loss, maxError, max_delay

min_delay, sent startTime

  • one report per minute

22

Zhenzhen Yan and M. Veeraraghavan, University of Virginia

slide-23
SLIDE 23

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Results: PerfSONAR OWAMP data analysis

Max delay plot:

  • ELPA-BOIS
  • ALBU-DENV

Overlapping paths Data traffic not host issues?

23

Zhenzhen Yan and M. Veeraraghavan, University of Virginia

slide-24
SLIDE 24

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Sample Results: Dependence on day of week

24

IQR (max-delay) in sec Day of week SUNN-BOST

(min-delay = 0.036)

KANS-CHIC (min-delay = 0.005) Sunday 0.08876 0.077011 Monday 0.12059 0.136785 Tuesday 0.10407 0.128747 Wednesday 0.11138 0.091315 Thursday 0.12504 0.231436 Friday 0.13171 0.128005 Saturday 0.10733 0.198049

Zhenzhen Yan and M. Veeraraghavan, University of Virginia

slide-25
SLIDE 25

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Another Sample Project: Constantine Dovrolis, Georgia Tech

Pythia: Detection, Localization and Diagnosis of Performance Problems using perfSONAR (DOE-funded) Pythia will be a data-analysis tool

  • Processing data collected from PerfSONAR (owamp)
  • Focusing on performance problems

Detection:

  • “noticeable lossrate between ORNL and AARNet at 10:54:02 GMT”

Localization:

  • “it happened at PNW-AARnet link”

Diagnosis:

  • “it was a high-loss event due to insufficient router buffering”

2/9/2011 25

slide-26
SLIDE 26

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

How to Participate

Deploy perfSONAR!

  • Using the “NP Toolkit” takes < 15 minutes to configure

Use perfSONAR to find & correct the hidden performance problems in your networks. Help write analysis and visualization tools

  • There is a huge amount of data publicly available ready to be mined
  • E.g.: look for correlations between active probes and passively

collected SNMP data

slide-27
SLIDE 27

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

More Information

Information on downloading/installing perfSONAR

  • http://psps.perfsonar.net/
  • http://fasterdata.es.net/fasterdata/perfSONAR/

Plot ESnet perfSONAR data:

  • http://stats1.es.net/

perfSONAR Client API:

  • http://fasterdata.es.net/fasterdata/perfSONAR/client-api/

email: BLTierney@es.net

slide-28
SLIDE 28

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Extra Slides

28

slide-29
SLIDE 29

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Globally accessible measurement services

  • Support for both active probes and passive results (SNMP)
  • In particular: throughput testing servers
  • Recommended tool for this is bwctl
  • http://www.internet2.edu/performance/bwctl/
  • Includes controls to prevent DDOS attacks

Services must be registered in a globally accessible lookup service Open to the entire R&E network community

  • Ideally using light-weight authentication and authorization

Components of a Global Diagnostic Service

slide-30
SLIDE 30

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Typical Campus Deployment

30

slide-31
SLIDE 31

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Support adhoc network measurements for troubleshooting and infrastructure verification.

  • By ESnet Staff within the backbone
  • By ESnet sites to the backbone
  • By ESnet customers across the backbone

Support regularly scheduled tests for diagnosing problems, demonstrating capabilities and monitoring.

  • Between ESnet POPS
  • Between ESnet sites & peers to ESnet POPs
  • From sites across ESnet.

Visualization

  • Allow ESnet user community to better understand our network & it's capabilities.
  • Allow ESnet users to understand how their use impacts the backbone.

Alarming

  • Automated analysis of regularly scheduled measurements to raise alerts.

Providing Diagnostic Services to ESnet Users