mpiFileUtils Parallel File Utilities for HPC November 13, 2017 - - PowerPoint PPT Presentation
mpiFileUtils Parallel File Utilities for HPC November 13, 2017 Danielle Sikich LLNL-PRES-740981 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract
LLNL-PRES-740981
This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Parallel File Utilities for HPC
Danielle Sikich November 13, 2017
LLNL-PRES-740981
2
§ What is mpiFileUtils?
—Suite of MPI-based suite of file utilities built on a common
library for handling large HPC datasets
—Similar to current unix file commands such as cp, rm, cmp,
etc.
§ Why mpiFileUtils?
—Applications that produce and consume data are highly-
parallel and frequently optimized for HPC file systems
—UNIX file management utilities are single-process solutions
LLNL-PRES-740981
3
§ Currently available for use:
1.
dchmod – change permissions & group access
2.
dcp – copy files
3.
dcmp – compare files
4.
drm – remove files
5.
dstripe – restripe files
6.
dwalk – list files
LLNL-PRES-740981
4
Tool 1 Proc 64 Procs 256 Procs cp 12 hours dcp 1x (12hr) 37x (19m) 51x (14m) rsync 6 hours dcmp 0.5x (12hr) 51x (7m) 60x (6m) rm 15 minutes drm 0.5x (28m) 3x (5m) 5x (3m)
LLNL-PRES-740981
5
§ Easy for contributors to add new parallel file utilities
based on a common API
Your tool here!
LLNL-PRES-740981
6
§ Result of multi-institutional collaboration: LANL, LLNL,
ORNL, DataDirect Networks, Red Hat, and Australian National University
§ We are in Spack
—spack install mpifileutils
§ Link to GitHub site:
—https://github.com/hpc/mpifileutils