FLASH and Its Research Communities D. Q. Lamb Flash Center for - - PowerPoint PPT Presentation

Flash Center for Computational Science The University of Chicago

NSF Community Code Workshop Chicago, IL 6 September 2012

• D. Q. Lamb

Flash Center for Computational Science University of Chicago

FLASH and It’s Research Communities

Flash Center for Computational Science The University of Chicago

Outline of Talk

 Role of community codes  Properties community codes need to have  Related needs  FLASH basics  FLASH capabilities  Code contribution policy and major contributions  FLASH research communities:

 Astrophysics and cosmology (examples: thermo- nuclear-powered supernovae and galaxy cluster mergers)  High energy density physics (example: shock- generated magnetic fields)  Computational fluid dynamics

Flash Center for Computational Science The University of Chicago

Role of community codes

Community codes serve their scientific communities by providing:

 Extensive physics capabilities that would otherwise not be available and that can be used to address cutting edge research problems in science and engineering.  A framework with existing capabilities into which new open or proprietary solvers can be imported.  A means of maintaining solvers over time and migrating them to new platforms, which is particularly important given the radical new architectures expected in the next generation of platforms.  An opportunity to support the education and training of the applied mathematicians, computer scientists, and computational scientists needed to develop the algorithms and codes needed to run at scale on these architectures.

Flash Center for Computational Science The University of Chicago

Properties community codes need to have

 It must be fully modular, extensible, and open — only if this is the case can it support creative approaches to algorithms and solvers, facilitate prototyping, and enable individual researchers to use its capabilities while adding their own solvers, whether the researchers make them open or retain proprietary control of them for shorter or longer periods of time.  It must be professionally written and documented – this is essential to maintaining the code over time, and making it possible for members of the user community to change it.  It must be subject to continuous, rigorous verification, including unit and regression testing – this is essential if it is to be robust.

The extensive experience the Flash Center has had with designing, implementing, disseminating, and maintaining FLASH has shown the importance of a community code having certain properties in order to successfully serve a large research community:

Flash Center for Computational Science The University of Chicago

Related needs

 Having more than one community code for a research community is usually desirable – one size/ shape usually does not fit all.  Development of algorithms, solvers, and codes by individuals and small groups also needs to be supported – this encourages the development of innovative new algorithms and solvers, many of which will end up enhancing the capabilities of the code.

Flash Center for Computational Science The University of Chicago

FLASH code contributors

 Current Flash Center Contributors :

 John Bachan, Sean Couch, Chris Daley, Milad Fatenejad, Norbert Flocke, Carlo Graziani, Cal Jordan, Dongwook Lee, Min Long, Prateeti Mohapatra, Anthony Scopatz, Petros Tzeferacos, Klaus Weide

 Current External Contributors:

 Paul Ricker, John Zuhone, Marcos Vanella, Mats Holmstrom, Seyit Hocuk, Chris Orban, Igor Golovkin, Tommaso Vinci, William Gray, Cristoph Federrath, Richard Wunsch

 Past Major Contributors:

 Katie Antypas, Alan Calder, Jonathan Dursi, Robert Fisher, Bruce Fryxell, Murali Ganapathy, Shravan Gopal, Nathan Hearn, Timur Linde, Bronson Messer, Kevin Olson, Tomek Plewa, Lynn Reid, Katherine Riley, Andrew Siegel, Dan Sheeler, Frank Timmes, , Dean Townsley, Natalia Vladimirova, Greg Weirs, Mike Zingale

Flash Center for Computational Science The University of Chicago

FLASH basics

 FLASH is a multi-physics finite-volume Eulerian code and framework implemented on a block-structured mesh with adaptive mesh refinement (AMR)  FLASH scales to well over a hundred thousand processors; it uses a variety of parallelization techniques including domain decomposition, mesh replication, and threading to best utilize hardware resources  FLASH is extremely portable and can run on a variety

• f platforms from laptops to supercomputing systems

such as the IBM BG/P and BG/Q

Flash Center for Computational Science The University of Chicago

FLASH basics (continued)

 FLASH is composed of interoperable units/modules; particular modules are combined to run individual simulations, so only the code relevant to a particular problem is included when FLASH is compiled, allowing important compile-time optimizations.  FLASH is professionally managed software with daily, automated regression testing on a variety of platforms, version control, coding standards, extensive documentation, user support, and integration of code contributions from external users.  FLASH has been downloaded more than 3,000 times; more than 1,000 scientists around the world have now used FLASH, and more than 700 papers have been published that directly use it.

Flash Center for Computational Science The University of Chicago

A Snapshot of FLASH Simulations

Cellular detonation Helium burning on neutron stars Richtmyer-Meshkov instability Laser-driven shock instabilities Nova outbursts on white dwarfs Rayleigh-Taylor instability Wave breaking on white dwarfs

Shortly: Relativistic accretion onto NS

Orzag/Tang MHD vortex Gravitationally confined detonation Intracluster interactions Magnetic Rayleigh-Taylor Turbulent Nuclear Burning Shock-generated B-field experiment

Flash Center for Computational Science The University of Chicago

A Snapshot of FLASH Simulations

Cellular detonation Helium burning on neutron stars Richtmyer-Meshkov instability Laser-driven shock instabilities Nova outbursts on white dwarfs Rayleigh-Taylor instability Wave breaking on white dwarfs

Shortly: Relativistic accretion onto NS

Orzag/Tang MHD vortex Gravitationally confined detonation Intracluster interactions Magnetic Rayleigh-Taylor Turbulent Nuclear Burning Shock-generated B-field experiment

FLASH capabilities include:

• Adaptive mesh refinement (AMR) on a block-structured mesh
• Multiple state-of-the-art hydrodynamic solvers (1T and 3T)
• State-of-the-art magnetohydrodynamics (currently 1T)
• Implicit solvers for diffusion using the HYPRE library (currently

being used to model thermal conduction, radiation diffusion, and viscosity)

• Many physics modules relevant to astrophysics and cosmology,

including gravity and nuclear burning

• Many physics modules relevant to high energy density physics,

including laser drive, 3T, electron conduction, multi-group radiation diffusion, tabular EOS and opacities

• Incompressible Navier-Stokes solver
• Generic, highly scalable parallel particles framework (currently

used for PIC simulations, laser ray tracing, dark matter, Lagrangian tracer particles, fluid-structure interactions)

Flash Center for Computational Science The University of Chicago

Flash Center code contribution policy

 The code contribution policy is designed with both the intellectual property rights of the contributors and the benefits to other researchers in mind.  Often researchers want to add some functionality to the code that they may not wish to share in a public release.  The Flash Center's policy is to allow individual contributors a pre- negotiated time when their code can exist in the main FLASH source tree without being included in the public release.  During this period the testing and maintenance of the code is carried out as an integral part of the code without any cost to the contributor, as long as the contributor provides supporting documentation and test problems for verification.  At the end of this period, the contributor agrees to release the code with the public distribution.  In return, the Flash Center assumes the responsibility for maintaining the contributed code and migrating it to new platforms.

Flash Center for Computational Science The University of Chicago

Major code contributions so far

 Already in the code:

 2D staggered mesh unsplit MHD solver – contributed by Dongwook Lee before he joined the Center  Huang Greengard method-based multigrid solver – contributed by Paul Ricker after he left the Center  Ionization – Salvatore Orlando  Radiation transfer using hybrid characteristics – Eric Jan Rijkhorst  Several direct solvers for uniform grid – Marcos Vanella  Hybrid particle-in-cell capability – Mats Holmstrom

 In the next release (FLASH 4.0 in mid-September):

 Sink particles – Christoph Federrath, Robi Banerjee, Martin Schron,  Barnes-Hut tree – Richard Wunsch  Primordial chemistry – William Gray

Flash Center for Computational Science The University of Chicago

FLASH research communities

 Astrophysics and cosmology (examples: thermonuclear-powered supernovae and mergers of galaxy clusters)  High energy density physics (example: shock-generated magnetic fields)  Computational fluid dynamics

Flash Center for Computational Science The University of Chicago

“Pulsationally assisted” gravitationally confined detonation model of thermonuclear-powered supernova

Flash Center for Computational Science The University of Chicago

Flash Center for Computational Science The University of Chicago

Merger of two galaxy clusters

Flash Center for Computational Science The University of Chicago

Flash Center for Computational Science The University of Chicago

FLASH 2D hydrodynamics simulation

• f the Vulcan foil experiment

2D FLASH hydro simulation of a laser-driven jet breaking out of a 50 micron-thick Carbon target.

Flash Center for Computational Science The University of Chicago

Al Slab 100 μm thick 25 mm wide Jet, v~35 km/s Induction Coil 3 cm t = 400 ns

Vulcan foil experiment

Schlieren shadowgraphy image1 Bdot probe traces1

1Images courtesy Jena Meinecke and Gianluca Gregori.

Flash Center for Computational Science The University of Chicago

MHD simulation of the Vulcan foil experiment

MHD simulation of the Vulcan foil experiment in which a magnetic field is generated by the Biermann battery effect with no magnetic dissipation in a 2D cylindrical (R-Z) computational

• domain. A 10 mm x 0.1 mm carbon

foil inside an Argon-filled chamber is illuminated from below by a single laser beam of 448 Joules for 1 nsec. A breakout shock propagates through the foil, producing a jet-like flow and generating a field of the order of a few

• kG. The image shows the magnitude
• f the B-field superposed on the

density, and reveals that magnetic field generation is stronger immediately downstream of the shock, in its wake.

Flash Center for Computational Science The University of Chicago

Summary

 Community codes contribute to research communities in many important ways  Community codes need to have certain properties  FLASH is a highly capable, fully modular and extensible code with a broad user base  Users have made major contributions to it  FLASH primarily serves three research communities:

 Astrophysics and cosmology (examples: thermo-nuclear- powered supernovae and galaxy clusters)  High energy density physics (example: shock-generated magnetic fields)  Computational fluid dynamics