The State and Needs of IO Performance Tools Scalable Tools Workshop - - PowerPoint PPT Presentation

▶

Nov 06, 2023 285 likes •675 views

The State and Needs of IO Performance Tools Scalable Tools Workshop Lake Tahoe, CA Elsa Gonsiorowski August 612, 2017 Greg Becker LLNL-PRES-735910 This work was performed under the auspices of the U.S. Department of Energy by Lawrence

SLIDE 1

The State and Needs of IO Performance Tools

Scalable Tools Workshop Lake Tahoe, CA August 6–12, 2017 Elsa Gonsiorowski Greg Becker

LLNL-PRES-735910

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

SLIDE 2

Outline

Motivating Example IO vs Compute Performance History Measuring I/O Performance The I/O Stack Questions from Applications

LLNL-PRES-735910 2

SLIDE 3

Simulation Output

LLNL-PRES-735910 3

SLIDE 4

Simulation Output

LLNL-PRES-735910 4

SLIDE 5

Simulation Output

I/O Performance hasn’t changed

LLNL-PRES-735910 4

SLIDE 6

Motivation

As computation performance increases I/O must be re-evaluated.

LLNL-PRES-735910 5

SLIDE 7

Top-500 History

LLNL-PRES-735910 6

SLIDE 8

Initial IO-500 Effort

Peak IO Site PFLOPs (GiB/s)1 KAUST, SAU 7.2 1955.78 JCAHPC, JP 24.91 1918.52 RIKEN, JP 10.62 1510.85 NCSA, US 13.4 1158 LLNL, US 20.1 1000 NSCG, CN 59.6 1000 ORNL, US 27.1 1000

1vi4io.org LLNL-PRES-735910 7

SLIDE 9

Challenges for IO-500

Which metrics matter?

LLNL-PRES-735910 8

SLIDE 10

Challenges for IO-500

Storage capacity

LLNL-PRES-735910 9

SLIDE 11

Challenges for IO-500

Storage capacity Storage hierarchy

LLNL-PRES-735910 9

SLIDE 12

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth

LLNL-PRES-735910 9

SLIDE 13

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size

LLNL-PRES-735910 9

SLIDE 14

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance

LLNL-PRES-735910 9

SLIDE 15

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance

LLNL-PRES-735910 9

SLIDE 16

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance Easy to "game" the system

LLNL-PRES-735910 9

SLIDE 17

Draft IO-500

Two workloads: IO and Metadata Two measurements: Easy and Hard

LLNL-PRES-735910 10

SLIDE 18

Draft IO-500

IO-Easy: IOR Large, sequential IO on unique POSIX files IO-Hard: IOR Small, random IO on a shared POSIX file MD-Easy: mdtest Unique directories, empty files MD-Hard: MD-REAL-IO Complex metadata, 3900 byte file

LLNL-PRES-735910 11

SLIDE 19

Measuring I/O Performance

Benchmarking Proxy Applications Profiling

LLNL-PRES-735910 12

SLIDE 20

Benchmarking

IOR mdtest IO_Bench MPI Tile IO b_eff_io SPIOBENCH iozone MADbench2 Mainly testing POSIX interface, with some MPI-IO.

LLNL-PRES-735910 13

SLIDE 21

Proxy Applications

MACSio HACC_IO / GenericIO

LLNL-PRES-735910 14

SLIDE 22

Profiling

Darshan Vampir

LLNL-PRES-735910 15

SLIDE 23

The I/O Stack

LLNL-PRES-735910 16

John Bent, Seagate

SLIDE 24

The I/O Stack

LLNL-PRES-735910 17

SLIDE 25

The I/O Stack

LLNL-PRES-735910 18

SLIDE 26

The I/O Stack

LLNL-PRES-735910 19

SLIDE 27

The I/O Stack

LLNL-PRES-735910 20

John Bent, Seagate

SLIDE 28

Questions from Applications

1. Where do we fall in the I/O envelope?
2. Parameters to achieve best performance?
3. How do we best use new storage tiers?

Current examples and some unposed questions

LLNL-PRES-735910 21

SLIDE 29

Where do we fall in the I/O Envelope?

Given: Peak system I/O performance Current application performance I/O pattern or trace . . . other details? Answer: Where is the application losing performance? What will gains can be made?

LLNL-PRES-735910 22

SLIDE 30

Where do we fall in the I/O Envelope?

Current Examples Use IOR and mdtest to measure peak system performance I/O Specific proxy application Lots of work

LLNL-PRES-735910 23

SLIDE 31

Where do we fall in the I/O Envelope?

Unposed Questions What is the point of this I/O? Could this use-case be achieved in a more efficient way? How do we enable in-situ or co-situ processes? High-level questions

LLNL-PRES-735910 24

SLIDE 32

Parameters to achieve best performance?

Given: Tuning of peak performing benchmark Current application I/O Answer: What file system settings need to be tuned? Is metadata a bottleneck / file locking?

LLNL-PRES-735910 25

SLIDE 33

Parameters to achieve best performance?

Current Examples None. Validation of simulation models with counters, no analysis of real applications

LLNL-PRES-735910 26

SLIDE 34

Parameters to achieve best performance?

Unposed Questions Can any of this be detected at a lower level? Automatic tuning of the file system during a workload How can this drive future procurements? Lower level and inter-level questions

LLNL-PRES-735910 27

SLIDE 35

How do we best use new Storage Tiers?

Given: Scientific need System limitations Answer: Which I/O patterns perform best Resiliency models

LLNL-PRES-735910 28

SLIDE 36

How do we best use new Storage Tiers?

Current Examples Defensive I/O Assumption

Optimal checkpoint interval SCR with system-specific configuration

Lossy compressions

HDF5 ZFP Compression

LLNL-PRES-735910 29

SLIDE 37

How do we best use new Storage Tiers?

Unposed Questions Interactions between resource schedulers and application

pre-stage / post-stage dynamic job allocation resources

What is the scientific need? How much precision is needed? Work flows to manage data movement Questions requiring full-stack knowledge

LLNL-PRES-735910 30

SLIDE 38

Thank you

LLNL-PRES-735910 31