Fourteen years of Cactus Community Frank L¨ offler Center for Computation and Technology Louisiana State University, Baton Rouge, LA September 6th 2012 Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Outline Motivation scenario from Astrophysics Cactus structure technically Cactus structure socially Future directions Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Challenging Astrophysics Problems Black Holes and Neutron Stars Supernovae Cosmology Gravitational Wave Data Analysis Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Gravitational Wave Physics Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Solving Einstein’s Equations Einstein equations: G µν = 8 π T µν 12 fully 2nd order PDE evolution equations 4 coupled constraint equations 4 gauge conditions GR hydrodynamics, MHD, radiation transport Fully numerical 3D models needed Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Computational Requirements Unigrid scales to hundreds of thousands of cores Productions runs use ≈ 10 levels of mesh refinement, nested grids of size ≈ 60 x 60 x 60 Current mesh refinement runs scale up to ≈ 10k cores Runtime from weeks to few months Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Challenge Many scientific/engineering components Physics Mathematics CFD Many numerical algorithm components Finite difference, finite volume, spectral methods Structured or unstructured meshes, mesh refinements Multipatch and multimodel Many different computational components Parallelism (MPI, OpenMP, ...) Parallel I/O (e.g. Checkpointing) Visualization Challenge Defining good abstractions to bring these together in a unified, scalable framework, enabling science Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus Framework Structure Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus Core: The Flesh ANSI-C and Perl, C++ Independent of all other components Unified error handling Build system Parameter parsing/steering Global variable management Rule-based scheduler Extensible APIs Parallel Operations Input/Output Reduction Interpolation Timers Functionality provided by (swappable) components Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus Components: Thorns C, C++, Fortran 77, Fortran 90 Typically not implementing grid setup and memory allocation input/output interpolation, reductions Encapsulating some functionality initial data boundary conditions evolution systems equations of state remote steering (e.g. https server) Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Basis Module Overview Basis for scalable algorithm development Most used: finite differences on structured meshes Parallel driver components Simple Unigrid Carpet: Multipatch, Mesh-refinement Method of lines Interfaces to external Libraries/Tools Interface to elliptic solvers (e.g. PETSc, Lorene) Input/Output: HDF5 Visualization: VisIt, OpenDX, Vish Other: PAPI, Hypre, Saga, Flickr, Twitter Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Convenience Tools GetComponents Simfactory Formaline Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus as growing project Cactus Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus as growing project Cactus Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus as growing project Cactus Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus as growing project Cactus Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Cactus as growing project Cactus Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Social structure Few Core Members: Gabrielle Allen, Steven R. Brandt, Frank L¨ offler, Erik Schnetter, ... Developers: about 50 worldwide Many more users Cactus Community Open Source Yearly releases Mailing lists Issue tracker IRC support channel Tutorials Web, HPC allocations, Repositories, ... Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Guiding Principles Open, community-driven software development Separation of physics software and computational infrastructure Stable interfaces, allowing extensions Simplify usage where possible: Doing science >> Running a simulation Students need to know a lot about physics (meaningful initial conditions, numerical stability, accuracy/resolution, have patience, have curiosity, develop a “gut feeling” for what is right ...) Cactus Toolkit cannot give that, however : Open codes that are easy to use allow to concentrate on these things! Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Credits, Citations In academics: citations, citations, citations! In Cactus: Open and free source No requirement to cite anything However: requested to cite a few publications Which publications: Few for the Cactus framework Some components list a few as well List published on website and manage through publication database Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Future From certain to more speculative: Multiblock techniques GPU support Requirement-based scheduling Discontinuous Galerkin instead of finite differences Support for unstructured grids Completely requirement-based programming (MPI → ParalleX?) Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Tools: GetComponents Task: Collect software from various repositories at different sites Example simulation assembly: Cactus Flesh and Toolkit (svn.cactuscode.org) Core Einstein Toolkit (svn.einsteintoolkit.org) Carpet AMR (carpetcode.org, hg) Tools, Parameter Files and Data (svn.einsteintoolkit.org) Group Modules (x.groupthorns.org) Individual Modules (x.mythorns.org) x: cvs, svn, darcs, git, hg, http Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Tools: Simulation Factory http://www.simfactory.org/ Task: Provide support for common, repetitive steps: Access remote systems, synchronize source code trees Configure and build on different systems semi-automatically Provide maintained list of supercomputer configurations Manage simulations (follow “best practices”, avoid human errors) Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Tools: Formaline Task: Ensure that simulations are and remain repeatable, remember exactly how they were performed Take snapshots of source code, system configuration; store it in executable and/or git repository Tag all output files Frank L¨ offler Fourteen years of Cactus Community 2012-09-06
Recommend
More recommend
