Binary packaging for HPC with Spack HPC, Big Data, and Data Science Devroom at FOSDEM 2018 Brussels, Belgium Todd Gamblin Center for Applied Scientific Computing Feburary 4, 2018 LLNL LLNL-PRES-745747 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC
Spack is a general purpose, from-source package manager § Inspired somewhat by homebrew and nix Spack § Targets HPC and scientific computing — Community is growing! https://spack.io § Goals: — Facilitate experimenting with performance options — Flexibility. Make these things easy: • Build packages with many different: – compilers/versions/build options • Change compilers and flags in builds (keep provenance) • Swap implementations of ABI-incompatible libraries – MPI, BLAS, LAPACK, others like jpeg/jpeg-turbo, etc. — Build software stacks for scientific simulation and analysis — Run on laptops, Linux clusters, and some of the largest supercomputers in the world github.com/spack @spackpm 2 LLNL-PRES-745747
Spec CLI syntax makes it easy to install different ways $ spack install mpileaks unconstrained $ spack install mpileaks@3.3 @ custom version $ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler $ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option $ spack install mpileaks@3.3 cflags="-O3 –g3" setting compiler flags $ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ dependency constraints § Each expression is a spec for a particular configuration — Each clause adds a constraint to the spec — Constraints are optional – specify only what you need. — Customize install on the command line! § Spec syntax is recursive — ^ (caret) adds constraints on dependencies github.com/spack @spackpm 3 LLNL-PRES-745747
Spack packages are templates : they define how to build a spec Simple Python DSL Packages are classes (ala homebrew) — from spack import * Directives use the same spec syntax — class Dyninst(Package): """API for dynamic binary instrumentation.""” Metadata at the class level homepage = "https://paradyn.org" url = "http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz" version('8.2.1', 'abf60b7faabe7a2e’) Versions version('8.1.2', 'bf03b33375afa66f’) version('8.1.1', 'd1a04e995b7aa709’) depends_on("cmake", type="build") Dependencies depends_on("libelf", type="link") depends_on("libdwarf", type="link") Patches, variants, resources, conflicts, etc. depends_on("boost @1.42: +multithreaded") (not shown) def install(self, spec, prefix): with working_dir('spack-build', create=True): cmake('-DBoost_INCLUDE_DIR=‘ + spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=‘ + spec['boost'].prefix.lib, Install logic in instance methods '-DBoost_NO_SYSTEM_PATHS=TRUE’ '..') make() make("install") github.com/spack @spackpm 4 LLNL-PRES-745747
Depend on interfaces (not implementations) with virtual dependencies mpi is a virtual dependency § mpi § Install the same package built with two mpileaks libdwarf different MPI implementations: callpath dyninst libelf Virtual dependencies can be versioned: $ spack install mpileaks ^mvapich class Mpileaks(Package): dependent depends_on("mpi@2:") $ spack install mpileaks ^openmpi@1.4: class Mvapich(Package): provider § Virtual deps are replaced with a valid provides("mpi@1” when="@:1.8") provides("mpi@2” when="@1.9:") implementation at resolution time. — If the user didn’t pick something and there class Openmpi(Package): provider are multiple options, Spack picks. provides("mpi@:2.2" when="@1.6.5:") github.com/spack @spackpm 5 LLNL-PRES-745747
Spack builds packages with compiler wrappers Similar to homebrew “shims” ▪ Spack do_install() Process Forked build process isolates environment for each build ▪ Use compiler wrappers to add include, lib, and RPATH flags ▪ RPATHs ensure that the correct dependencies are found ▪ … Install dep1 Install dep2 Install package automatically at runtime. icc icpc ifort Fork Build Set up environment Process Compiler wrappers CC = spack/env/intel/icc SPACK_CC = /opt/ic-15.1/bin/icc ( spack- cc, spack-c++, spack-f77, spack-f90 ) CXX = spack/env/intel/icpc SPACK_CXX = /opt/ic-15.1/bin/icpc F77 = spack/env/intel/ifort SPACK_F77 = /opt/ic-15.1/bin/ifort FC = spack/env/intel/ifort SPACK_FC = /opt/ic-15.1/bin/ifort -I /dep1-prefix/include -L /dep1-prefix/lib PKG_CONFIG_PATH = ... PATH = spack/env:$PATH -Wl,-rpath=/dep1-prefix/lib CMAKE_PREFIX_PATH = ... LIBRARY_PATH = ... make install configure make install() github.com/spack @spackpm 6 LLNL-PRES-745747
Hashes handle combinatorial software complexity. § Each unique dependency graph is a unique Dependency DAG configuration . mpi mpileaks libdwarf § Each configuration installed in a unique directory. callpath dyninst libelf — Configurations of the same package can coexist. Installation Layout Hash § Hash of directed acyclic graph (DAG) metadata is spack/opt/ appended to each prefix linux-rhel7-x86_64/ gcc-4.7.2/ — Note: we hash the metadata, not the artifact. mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ § Installed packages automatically find dependencies bgq/ xl-12.1/ — Spack embeds RPATHs in binaries. hdf5-1-8.16-fqb3a15abrwx/ — No need to set LD_LIBRARY_PATH ... — Things work the way you built them github.com/spack @spackpm 7 LLNL-PRES-745747
Spack’s dependency model centers around “concretization” User input: abstract spec § Similar to other dependency resolvers, but solves for more than just package and version. mpileaks ^callpath@1.0+debug ^libelf@0.8.11 mpileaks@2.3 § Full spec is stored in a file in the installation directory mpileaks %gcc@4.7.3 =linux-ppc64 — Can reinstall same build with: spack install –f spec.yaml callpath@1.0 callpath@1.0 %gcc@a4.7.3+debug +debug =linux-ppc64 spec.yaml spec.yaml spec: - mpileaks: arch: linux-x86_64 compiler: name: gcc mpich@3.0.4 dyninst@8.1.2 Store version: 4.9.2 Concretize mpi dyninst %gcc@4.7.3 %gcc@4.7.3 dependencies: =linux-ppc64 =linux-ppc64 adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4 callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77 mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh variants: {} version: '1.0' libdwarf@20130729 - adept-utils: libdwarf %gcc@4.7.3 arch: linux-x86_64 =linux-ppc64 compiler: name: gcc version: 4.9.2 dependencies: boost: teesjv7ehpe5ksspjim5dk43a7qnowlq mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 libelf@0.8.11 hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4 libelf@0.8.11 %gcc@4.7.3 variants: {} =linux-ppc64 version: 1.0.1 ... Concrete spec is fully constrained Abstract , normalized spec Detailed provenance is stored and can be built. with some dependencies. with the installed package github.com/spack @spackpm 8 LLNL-PRES-745747
Source installs are great, but they’re slow § Most people prefer using a binary package manager — Binary packages typically use portable code — Binary installs are typically a lot slower than what you get from building from source § We’d like to have the best of both worlds: — Optimized buids for specific machine models (skylake, haswell, ivy bridge, etc.) — Binary packages available without having to build from source § What’s needed? Binary packaging capability 1. Metadata describing architecture-specific builds 2. Good dependency resolution to select optimized or generic versions of packages 3. github.com/spack @spackpm 9 LLNL-PRES-745747
https://github.com/spack/spack/releases/ We recently released Spack v0.11 § 2,178 packages (up from 1,114 a year ago) § Big features for users: — Relocatable binary packages (spack buildcache) — Full support for Python 3 — Improved module support; custom module templates using jinja2 § Many improvements for packagers: — Multi-valued variants — Test dependency type — Packages can patch their dependencies (not just themselves) § Many speed improvements (to Spack itself) github.com/spack @spackpm 10 LLNL-PRES-745747
Binary packaging in Spack v0.11 § Spack v0.11 has a new spack buildcache command: spack buildcache create <spec> # create a new binary package spack buildcache list # list available binaries spack buildcache install # install a binary package (specifically) § Typically, install is not needed; you can just do: spack install --use-cache # prefer binaries if available § We don’t enable binaries by default yet — We’ll make –use-cache default when we start hosting stable binaries § Thanks to our collaboration with Fermilab, CERN, and Kitware for this feature! github.com/spack @spackpm 11 LLNL-PRES-745747
Recommend
More recommend
