SCR and Preparing for Burst Buffers DOE COE Performance Portability - - PowerPoint PPT Presentation

▶

Aug 31, 2022 178 likes •525 views

SCR and Preparing for Burst Buffers DOE COE Performance Portability Meeting August 23, 2017 Elsa Gonsiorowski LLNL-PRES-737156 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National

SLIDE 1

SCR and Preparing for Burst Buffers

DOE COE Performance Portability Meeting August 23, 2017 Elsa Gonsiorowski

LLNL-PRES-737156

This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

SLIDE 2

Outline

Burst Buffer Technologies SCR Overview Burst Buffers and SCR Additional Software Projects

LLNL-PRES-737156 2

SLIDE 3

Burst Buffer Technologies

Type Technology Location Node Local IBM BBAPI LLNL (Sierra) Machine Global Cray Datawarp LANL (Trinity)

LLNL-PRES-737156 3

SLIDE 4

Burst Buffer Technologies

Type Technology Location Node Local IBM BBAPI LLNL (Sierra) Machine Global Cray Datawarp LANL (Trinity) How can an application utilize this layer for I/O workloads?

LLNL-PRES-737156 3

SLIDE 5

Burst Buffers Use Case

Relies on integration with resource scheduler Different for machine-global vs. node-local storage Does not address inter-job data movement

LLNL-PRES-737156 4

SLIDE 6

Burst Buffers Use Case

Perfect for Checkpoint/Restart

LLNL-PRES-737156 5

SLIDE 7

Checkpoint Restart

a.k.a. Defensive I/O

LLNL-PRES-737156 6

SLIDE 8

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory

LLNL-PRES-737156 6

SLIDE 9

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

LLNL-PRES-737156 6

SLIDE 10

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

Which may change over time

LLNL-PRES-737156 6

SLIDE 11

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

Which may change over time

Creating a checkpoint may not be as efficient as recomputing

LLNL-PRES-737156 6

SLIDE 12

SCR Goal

Enable checkpointing applications to take advantage of system storage hierarchies

LLNL-PRES-737156 7

SLIDE 13

SCR Goal

Enable checkpointing applications to take advantage of system storage hierarchies Efficient file movement between storage layers Data redundancy operations

LLNL-PRES-737156 7

SLIDE 14

SCR Components

LLNL-PRES-737156 8

SLIDE 15

SCR Component: Backend Library

Redirect application files Synchronous & asynchronous flush operations

Hardware specific capabilities

Data redundancy Support for both checkpoint & output data

LLNL-PRES-737156 9

SLIDE 16

SCR Component: Backend Library

int rc = MyApp_Checkpoint(path);

LLNL-PRES-737156 10

SLIDE 17

SCR Component: Backend Library

SCR_Route_file(path, newpath); int rc = MyApp_Checkpoint(newpath);

LLNL-PRES-737156 10

SLIDE 18

SCR Component: Backend Library

SCR_Start_output("dataset name", flags); SCR_Route_file(path, newpath); int rc = MyApp_Checkpoint(newpath); SCR_Complete_output(rc);

LLNL-PRES-737156 10

SLIDE 19

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart

LLNL-PRES-737156 11

SLIDE 20

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart

LLNL-PRES-737156 11

SLIDE 21

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart During Execution Manage datasets

LLNL-PRES-737156 11

SLIDE 22

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart During Execution Manage datasets Resource Scheduler Integration Pre- and post-stage data movement

LLNL-PRES-737156 11

SLIDE 23

SCR Component: Configurations

Define the levels of the hierarchy Define modes/groups of failure Define checkpointing and data residency needs

LLNL-PRES-737156 12

SLIDE 24

SCR Component: Configurations

Define the levels of the hierarchy Define modes/groups of failure Define checkpointing and data residency needs Machine Portability

LLNL-PRES-737156 12

SLIDE 25

Burst Buffers Use Case

Checkpoint Restart

LLNL-PRES-737156 13

SLIDE 26

Burst Buffers & SCR: Prestage

Machine Global Solved

Global access from CNs to storage

Node Local Requires new softwares

Requires deep integration with resource scheduler Most useful for DATs or half+ system jobs

LLNL-PRES-737156 14

SLIDE 27

Burst Buffers & SCR: Poststage

Unaddressed Concerns

Applications without checkpointing Shared Files Arbitrary data movement

Machine-learning use case

LLNL-PRES-737156 16

SLIDE 29

VELOC

Combining two codes: FTI and SCR FTI: variable-based checkpointing scheme Will support existing FTI and SCR applications

LLNL-PRES-737156 17

SLIDE 30

UnifyCR

User-level file system Shared namespace across distributed burst buffers I/O interception layer

LLNL-PRES-737156 18

SLIDE 31

MPI File Utils

Use parallel processes to perform file operations Executed within a job allocation dbcast: broadcast from PFS to node-local storage dcp: multiple file copy in parallel drm: delete files in parallel many more https://github.com/hpc/mpifileutils

LLNL-PRES-737156 19

SLIDE 32

SCR Team

https://github.com/llnl/scr Kathryn Mohror Adam Moody Greg Becker Elsa Gonsiorowski

LLNL-PRES-737156 20

SLIDE 33

SCR and Preparing for Burst Buffers

DOE COE Performance Portability Meeting August 23, 2017 Elsa Gonsiorowski

Outline

Burst Buffer Technologies SCR Overview Burst Buffers and SCR Additional Software Projects

Burst Buffer Technologies

Type Technology Location Node Local IBM BBAPI LLNL (Sierra) Machine Global Cray Datawarp LANL (Trinity)

Burst Buffer Technologies

Type Technology Location Node Local IBM BBAPI LLNL (Sierra) Machine Global Cray Datawarp LANL (Trinity) How can an application utilize this layer for I/O workloads?

Burst Buffers Use Case

Relies on integration with resource scheduler Different for machine-global vs. node-local storage Does not address inter-job data movement

Burst Buffers Use Case

Perfect for Checkpoint/Restart

Checkpoint Restart

a.k.a. Defensive I/O

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

Checkpoint Restart

a.k.a. Defensive I/O Related to the size of system memory Depends on resiliency of machine

Creating a checkpoint may not be as efficient as recomputing

SCR Goal

Enable checkpointing applications to take advantage of system storage hierarchies

SCR Goal

Enable checkpointing applications to take advantage of system storage hierarchies Efficient file movement between storage layers Data redundancy operations

SCR Components

SCR Component: Backend Library

Redirect application files Synchronous & asynchronous flush operations

Data redundancy Support for both checkpoint & output data

SCR Component: Backend Library

int rc = MyApp_Checkpoint(path);

SCR Component: Backend Library

SCR_Route_file(path, newpath); int rc = MyApp_Checkpoint(newpath);

SCR Component: Backend Library

SCR_Start_output("dataset name", flags); SCR_Route_file(path, newpath); int rc = MyApp_Checkpoint(newpath); SCR_Complete_output(rc);

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart During Execution Manage datasets

SCR Component: Frontend Scripts

On Startup Locate most recent checkpoint and fetch for restart Within Allocation Detect application crash or system failures and trigger restart During Execution Manage datasets Resource Scheduler Integration Pre- and post-stage data movement

SCR Component: Configurations

Define the levels of the hierarchy Define modes/groups of failure Define checkpointing and data residency needs

SCR Component: Configurations

Define the levels of the hierarchy Define modes/groups of failure Define checkpointing and data residency needs Machine Portability

Burst Buffers Use Case

Checkpoint Restart

Burst Buffers & SCR: Prestage

Machine Global Solved

Node Local Requires new softwares

Burst Buffers & SCR: Poststage

Similar solution for both BB types Take advantage of vendor APIs asynchronous operations Decouples burst buffer usage from compute usage

Unaddressed Concerns

Applications without checkpointing Shared Files Arbitrary data movement

VELOC

Combining two codes: FTI and SCR FTI: variable-based checkpointing scheme Will support existing FTI and SCR applications

UnifyCR

User-level file system Shared namespace across distributed burst buffers I/O interception layer

MPI File Utils

Use parallel processes to perform file operations Executed within a job allocation dbcast: broadcast from PFS to node-local storage dcp: multiple file copy in parallel drm: delete files in parallel many more https://github.com/hpc/mpifileutils

SCR Team

https://github.com/llnl/scr Kathryn Mohror Adam Moody Greg Becker Elsa Gonsiorowski