mpifileutils

mpiFileUtils Parallel File Utilities for HPC November 13, 2017 - PowerPoint PPT Presentation

mpiFileUtils Parallel File Utilities for HPC November 13, 2017 Danielle Sikich LLNL-PRES-740981 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract


  1. mpiFileUtils Parallel File Utilities for HPC November 13, 2017 Danielle Sikich LLNL-PRES-740981 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

  2. Motivation § What is mpiFileUtils? — Suite of MPI-based suite of file utilities built on a common library for handling large HPC datasets — Similar to current unix file commands such as cp, rm, cmp, etc. § Why mpiFileUtils? — Applications that produce and consume data are highly- parallel and frequently optimized for HPC file systems — UNIX file management utilities are single-process solutions 2 LLNL-PRES-740981

  3. Tools § Currently available for use: dchmod – change permissions & group access 1. dcp – copy files 2. dcmp – compare files 3. drm – remove files 4. dstripe – restripe files 5. dwalk – list files 6. 3 LLNL-PRES-740981

  4. 1 Million File Experiment Tool 1 Proc 64 Procs 256 Procs cp 12 hours dcp 1x (12hr) 37x (19m) 51x (14m) rsync 6 hours dcmp 0.5x (12hr) 51x (7m) 60x (6m) rm 15 minutes drm 0.5x (28m) 3x (5m) 5x (3m) 4 LLNL-PRES-740981

  5. Common Library § Easy for contributors to add new parallel file utilities based on a common API Your tool here! 5 LLNL-PRES-740981

  6. Acknowledgements & Download § Result of multi-institutional collaboration: LANL, LLNL, ORNL, DataDirect Networks, Red Hat, and Australian National University § We are in Spack — spack install mpifileutils § Link to GitHub site: — https://github.com/hpc/mpifileutils 6 LLNL-PRES-740981

Recommend


More recommend