UMAMI: A Recipe for Generating Meaningful Metrics through Holistic - - PowerPoint PPT Presentation

▶
umami a recipe for generating meaningful metrics through
SMART_READER_LITE
LIVE PREVIEW

UMAMI: A Recipe for Generating Meaningful Metrics through Holistic - - PowerPoint PPT Presentation

Jan 03, 2023 481 likes •745 views

UMAMI: A Recipe for Generating Meaningful Metrics through Holistic I/O Performance Analysis Glenn K. Lockwood, Shane Snyder, Wucherl Yoo, Kevin Harms, Zachary Nault, Suren Byna, Philip Carns, Nicholas J. Wright October 27, 2017 - 1 -

slide-1
SLIDE 1

UMAMI: A Recipe for Generating Meaningful Metrics through Holistic I/O Performance Analysis

  • 1 -

Glenn K. Lockwood, Shane Snyder, Wucherl Yoo, Kevin Harms, Zachary Nault, Suren Byna, Philip Carns, Nicholas J. Wright October 27, 2017

slide-2
SLIDE 2
  • 2 -

Understanding I/O today is hard

  • Storage hierarchy is

ge1ng more complicated

IO Nodes, BB Nodes Compute Nodes Storage Servers

slide-3
SLIDE 3

HDF5

Custom Binary Format

ES

Custom Binary Format .txt

  • 3 -

Understanding I/O today is hard

IO Nodes, BB Nodes Compute Nodes Storage Servers

  • Storage hierarchy is

ge1ng more complicated

  • Currently monitor

each component separately is standard prac:ce

slide-4
SLIDE 4

HDF5

Custom Binary Format

ES

Custom Binary Format .txt

  • 4 -

IO Nodes, BB Nodes Compute Nodes Storage Servers

Expert Knowledge Understanding I/O today is hard

  • Storage hierarchy is

ge1ng more complicated

  • Currently monitor

each component separately is standard prac:ce

I/O expert (Phil Carns) from ATPESC: hHps://insidehpc.com/2017/10/hpc-io-computa:onal-scien:sts/

slide-5
SLIDE 5
  • 5 -

Total Knowledge of I/O with holistic analysis

  • Can we augment expert

knowledge?

  • Using exis:ng tools?

IO Nodes, BB Nodes Compute Nodes Storage Servers

slide-6
SLIDE 6

HDF5

Custom Binary Format

ES

Custom Binary Format .txt

IO Nodes, BB Nodes Compute Nodes Storage Servers

Total Knowledge of I/O (TOKIO)

  • Can we augment expert

knowledge?

  • Using exis:ng tools?
  • Combine, index, and

normalize their metrics

  • Provide a holis:c view
  • 6 -

I/O expert (Phil Carns) from ATPESC: hHps://insidehpc.com/2017/10/hpc-io-computa:onal-scien:sts/

Total Knowledge of I/O with holistic analysis

slide-7
SLIDE 7

What is possible with holistic I/O analysis?

  • Run four different I/O workloads every day for

a month

– Jobs scaled to achieve > 80% of peak fs performance – Exercise file-per-proc, shared file, big and small xfers

  • Run on ALCF Mira (IBM BG/Q) and NERSC

Edison (Cray XC)

– One GPFS file system on Mira (gpfs-mira) – Two Lustre file systems on Edison (lustre-reg and lustre-bigio)

  • Use data from producYon monitoring tools at

ALCF and NERSC

– Darshan for applica:on-level I/O profiling – GPFS and Lustre-specific server-side monitoring tools

  • 7 -
slide-8
SLIDE 8

Defining performance variation

  • 8 -
  • "Frac%on of Peak Performance"

is rela:ve to max performance for that app on that file system

  • Normalizes out the effects of

applica:on I/O paHerns and peak file system performance

gpfs (Mira)

slide-9
SLIDE 9

Variation due to application I/O pattern

  • 9 -
  • "Bad I/O paHerns" can cause

– bad performance – bad performance varia%on

  • Some applica:on paHerns are

more suscep:ble to high amounts of varia:on!

gpfs (Mira)

slide-10
SLIDE 10

Variation across file system architectures

  • 10 -

ApplicaYon I/O paZerns are not the only contributor to performance variaYon

lustre-bigio (Edison) gpfs (Mira)

slide-11
SLIDE 11

Variation between Lustre configurations

  • 11 -

Significant differences even on similar Lustre file systems—

  • ther factors (configuraYon, workload) also maZer!

lustre-reg (Edison) lustre-bigio (Edison)

slide-12
SLIDE 12

What does this tell us about variation?

  • 12 -

lustre-bigio (Edison) gpfs (Mira) lustre-reg (Edison)

Performance variaYon a funcYon of

  • applica:on I/O paHerns (cf. HACC, VPIC)
  • architecture (cf. gpfs, lustre-bigio)
  • other factors (cf. lustre-bigio, lustre-reg)
slide-13
SLIDE 13

What does this tell us about variation?

  • 13 -

lustre-bigio (Edison) gpfs (Mira) lustre-reg (Edison)

File systems have their own "I/O climate" (like Berkeley vs. Argonne) Understanding these "other factors" (climate) holisYcally is essenYal to understanding performance variability!

slide-14
SLIDE 14

Exploring I/O weather and climate

  • 14 -

lustre-reg (Edison)

Let's look at a few cases of bad performance using a Unified Monitoring and Metrics Interface (UMAMI) What can a holis:c view (climate) tell us about performance (weather)?

slide-15
SLIDE 15

Case Study #1:" HACC write performance on lustre-reg

  • 15 -
  • Is this a snowy day at Argonne or a snowy day at

Berkeley?

  • Quan:ta:vely define "bad" based on quar:les
  • Use UMAMI to determine which aspects of

weather were "bad"

slide-16
SLIDE 16

Case Study #1:" First guess: blame someone else

  • 16 -

Coverage Factor = how much global bandwidth was consumed by my job?

slide-17
SLIDE 17

Case Study #1:" Add Coverage Factor to UMAMI

  • 17 -

Most jobs get exclusive access to Lustre bandwidth (CFbw ≈ 1.0)

slide-18
SLIDE 18

Case Study #1:" Add Coverage Factor to UMAMI

  • 18 -

Bad performance coincided with low CF Performance varia:on caused by bandwidth conten:on

slide-19
SLIDE 19

Case Study #2:" VPIC/GPFS: when bandwidth contention isn't the issue

  • 19 -

Bad performance did not coincide with low CF Either use expert knowledge or sta:s:cal analysis to add more metrics

slide-20
SLIDE 20
  • 20 -

Sta:s:cally "bad" levels of conten:on for metadata IOPS Performance loss affected by file system implementa:on

Case Study #2:" VPIC/GPFS: when bandwidth contention isn't the issue

slide-21
SLIDE 21

Case Study #3: " HACC/lustre-bigio: effects of "I/O climate change"

  • 21 -

Abnormally good performance revealed a long-term bad I/O climate Bandwidth conten:on was not the culprit

slide-22
SLIDE 22

Case Study #3: " HACC/lustre-bigio: effects of "I/O climate change"

  • 22 -
  • Moderate nega:ve

correla:on with OSS CPU load

  • Strong nega:ve

correla:on with file system fullness

  • Result of Lustre

block alloca:on at >90% fullness

slide-23
SLIDE 23

Conclusions

  • 23 -
  • Performance variability is a funcYon of file system climate:

– file system architecture – overall system workload – file system configura:on (default striping, etc) and health

  • No single metric predicts variaYon universally; many factors

can affect I/O weather:

– bandwidth conten:on – metadata op conten:on (GPFS) – file system fullness (Lustre)

  • A holisYc view of the storage subsystem is essenYal to

understand performance on complex I/O architectures

slide-24
SLIDE 24

HDF 5 Custom Binary Format ES Custom Binary Format .txt IO Nodes, BB Nodes Compute Nodes Storage Servers Total Knowledge of I/O (TOKIO)

  • Incorporate machine learning

– Cluster similar I/O mo:fs to define I/O climates – Infer cri:cal metrics to remove expert from the loop

  • Join the TOKIO effort!

– Open source & development – contribu:ons welcome! – hHps://github.com/nersc/pytokio/ – Support for new component-level tools being added regularly

  • 24 -

Closer to Total Knowledge

slide-25
SLIDE 25

This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contracts DE-AC02-05CH11231 and DE-AC02-06CH11357 (Project: A Framework for Holistic I/O Workload Characterization, Program manager: Dr. Lucy Nowell). This research used resources and data generated from resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE- AC02-05CH11231 and the Argonne Leadership Computing Facility, a DOE Office of Science User Facility supported under Contract DE- AC02-06CH11357.

  • 25 -