The State and Needs of IO Performance Tools Scalable Tools Workshop - - PowerPoint PPT Presentation

the state and needs of io performance tools
SMART_READER_LITE
LIVE PREVIEW

The State and Needs of IO Performance Tools Scalable Tools Workshop - - PowerPoint PPT Presentation

The State and Needs of IO Performance Tools Scalable Tools Workshop Lake Tahoe, CA Elsa Gonsiorowski August 612, 2017 Greg Becker LLNL-PRES-735910 This work was performed under the auspices of the U.S. Department of Energy by Lawrence


slide-1
SLIDE 1

The State and Needs of IO Performance Tools

Scalable Tools Workshop Lake Tahoe, CA August 6–12, 2017 Elsa Gonsiorowski Greg Becker

LLNL-PRES-735910

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

slide-2
SLIDE 2

Outline

Motivating Example IO vs Compute Performance History Measuring I/O Performance The I/O Stack Questions from Applications

LLNL-PRES-735910 2

slide-3
SLIDE 3

Simulation Output

LLNL-PRES-735910 3

slide-4
SLIDE 4

Simulation Output

LLNL-PRES-735910 4

slide-5
SLIDE 5

Simulation Output

I/O Performance hasn’t changed

LLNL-PRES-735910 4

slide-6
SLIDE 6

Motivation

As computation performance increases I/O must be re-evaluated.

LLNL-PRES-735910 5

slide-7
SLIDE 7

Top-500 History

LLNL-PRES-735910 6

slide-8
SLIDE 8

Initial IO-500 Effort

Peak IO Site PFLOPs (GiB/s)1 KAUST, SAU 7.2 1955.78 JCAHPC, JP 24.91 1918.52 RIKEN, JP 10.62 1510.85 NCSA, US 13.4 1158 LLNL, US 20.1 1000 NSCG, CN 59.6 1000 ORNL, US 27.1 1000

1vi4io.org LLNL-PRES-735910 7

slide-9
SLIDE 9

Challenges for IO-500

Which metrics matter?

LLNL-PRES-735910 8

slide-10
SLIDE 10

Challenges for IO-500

Storage capacity

LLNL-PRES-735910 9

slide-11
SLIDE 11

Challenges for IO-500

Storage capacity Storage hierarchy

LLNL-PRES-735910 9

slide-12
SLIDE 12

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth

LLNL-PRES-735910 9

slide-13
SLIDE 13

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size

LLNL-PRES-735910 9

slide-14
SLIDE 14

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance

LLNL-PRES-735910 9

slide-15
SLIDE 15

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance

LLNL-PRES-735910 9

slide-16
SLIDE 16

Challenges for IO-500

Storage capacity Storage hierarchy Performance / bandwidth In-system memory size Metadata performance Easy to "game" the system

LLNL-PRES-735910 9

slide-17
SLIDE 17

Draft IO-500

Two workloads: IO and Metadata Two measurements: Easy and Hard

LLNL-PRES-735910 10

slide-18
SLIDE 18

Draft IO-500

IO-Easy: IOR Large, sequential IO on unique POSIX files IO-Hard: IOR Small, random IO on a shared POSIX file MD-Easy: mdtest Unique directories, empty files MD-Hard: MD-REAL-IO Complex metadata, 3900 byte file

LLNL-PRES-735910 11

slide-19
SLIDE 19

Measuring I/O Performance

Benchmarking Proxy Applications Profiling

LLNL-PRES-735910 12

slide-20
SLIDE 20

Benchmarking

IOR mdtest IO_Bench MPI Tile IO b_eff_io SPIOBENCH iozone MADbench2 Mainly testing POSIX interface, with some MPI-IO.

LLNL-PRES-735910 13

slide-21
SLIDE 21

Proxy Applications

MACSio HACC_IO / GenericIO

LLNL-PRES-735910 14

slide-22
SLIDE 22

Profiling

Darshan Vampir

LLNL-PRES-735910 15

slide-23
SLIDE 23

The I/O Stack

LLNL-PRES-735910 16

John Bent, Seagate

slide-24
SLIDE 24

The I/O Stack

LLNL-PRES-735910 17

slide-25
SLIDE 25

The I/O Stack

LLNL-PRES-735910 18

slide-26
SLIDE 26

The I/O Stack

LLNL-PRES-735910 19

slide-27
SLIDE 27

The I/O Stack

LLNL-PRES-735910 20

John Bent, Seagate

slide-28
SLIDE 28

Questions from Applications

  • 1. Where do we fall in the I/O envelope?
  • 2. Parameters to achieve best performance?
  • 3. How do we best use new storage tiers?

Current examples and some unposed questions

LLNL-PRES-735910 21

slide-29
SLIDE 29

Where do we fall in the I/O Envelope?

Given: Peak system I/O performance Current application performance I/O pattern or trace . . . other details? Answer: Where is the application losing performance? What will gains can be made?

LLNL-PRES-735910 22

slide-30
SLIDE 30

Where do we fall in the I/O Envelope?

Current Examples Use IOR and mdtest to measure peak system performance I/O Specific proxy application Lots of work

LLNL-PRES-735910 23

slide-31
SLIDE 31

Where do we fall in the I/O Envelope?

Unposed Questions What is the point of this I/O? Could this use-case be achieved in a more efficient way? How do we enable in-situ or co-situ processes? High-level questions

LLNL-PRES-735910 24

slide-32
SLIDE 32

Parameters to achieve best performance?

Given: Tuning of peak performing benchmark Current application I/O Answer: What file system settings need to be tuned? Is metadata a bottleneck / file locking?

LLNL-PRES-735910 25

slide-33
SLIDE 33

Parameters to achieve best performance?

Current Examples None. Validation of simulation models with counters, no analysis of real applications

LLNL-PRES-735910 26

slide-34
SLIDE 34

Parameters to achieve best performance?

Unposed Questions Can any of this be detected at a lower level? Automatic tuning of the file system during a workload How can this drive future procurements? Lower level and inter-level questions

LLNL-PRES-735910 27

slide-35
SLIDE 35

How do we best use new Storage Tiers?

Given: Scientific need System limitations Answer: Which I/O patterns perform best Resiliency models

LLNL-PRES-735910 28

slide-36
SLIDE 36

How do we best use new Storage Tiers?

Current Examples Defensive I/O Assumption

Optimal checkpoint interval SCR with system-specific configuration

Lossy compressions

HDF5 ZFP Compression

LLNL-PRES-735910 29

slide-37
SLIDE 37

How do we best use new Storage Tiers?

Unposed Questions Interactions between resource schedulers and application

pre-stage / post-stage dynamic job allocation resources

What is the scientific need? How much precision is needed? Work flows to manage data movement Questions requiring full-stack knowledge

LLNL-PRES-735910 30

slide-38
SLIDE 38

Thank you

LLNL-PRES-735910 31