Advancing first-principle symmetry-guided nuclear modeling for - - PowerPoint PPT Presentation

Advancing first-principle symmetry-guided nuclear modeling for studies of nucleosynthesis and fundamental symmetries in nature

NCSA Blue Waters Symposium for Petascale Science and Beyond, 2019

Students & Postdocs Collaborators

Nuclear Physics Nuclear Physics

Residual strong force between quarks → highly complex two-, three- and four-body forces

100,000 fm 1 fm proton neutron 0.8 fm

Atom Nucleus

1 fm Nuclear force holds nucleons together

Nucleus

contains nearly all mass of the atom extremely tiny compared to the size of the atom made up of protons and neutrons nucleons [protons and neutrons] made up of quarks

Ab initio Ab initio Approaches to Nuclear Structure and Reactions Approaches to Nuclear Structure and Reactions

Many-body dynamics Nuclear reactions Realistic nuclear potential models Nuclear interaction

Energy

3H 3H 4He 2H

n

wave functions nuclear properties reaction rates cross sections

solve the Schrodinger equation for a system of interacting nucleons

• 2. Compute Hamiltonian matrix
• 3. Find lowest-lying eigenvalues and eigenvectors

1+ 2+ 0+ 4+

• 1. Choose physically relevant model space and construct its basis

Input:

Nuclear Hamiltonian – operator of energy Lanczos algorithm eigenvalues eigenvectors

Fundamental task:

Solving Nuclear Problem Solving Nuclear Problem

Key Challenge: Scale Explosion Key Challenge: Scale Explosion

Limits application of ab initio studies to lightest nuclei Use partial symmetries of nuclear collective motion to adopt smaller physically relevant model spaces

Computational Scale Explosion Why symmetry-adapted approach? Why Blue Waters?

Large aggregate memory and amount of memory per node (64GB) High peak memory bandwidth (102.4 GB/s)

[courtesy of Pieter Maris]

Many-nucleon basis natural for description of many-body dynamics of nuclei

N Sp Sn S L

number of harmonic oscillator excitations total proton, total neutron and total intrinsic spins deformation rotation

Many-nucleon basis natural for description of many-body dynamics of nuclei

Symmetry-Adapted No-Core Shell Model Symmetry-Adapted No-Core Shell Model

MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model

Mapping of Hamiltonian matrix to MPI processes

Computational effort: 90 % - computing matrix elements 10% - solving eigenvalue problem

Implementation

C++/Fortran code parallelized using hybrid MPI/OpenMP Open source: https://sourceforge.net/p/lsu3shell/home/Home/

Excellent scalability

Original density structure

• f Hamiltonian matrix

15 processes 378 processes 37,950 processes

Leads to load balanced computations Round-robin distribution of basis states among MPI processes

MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model MPI/OpenMP Implementation of Symmetry-Adapted No-Core Shell Model

(0 0) (1 1) (0 3) (3 0) (2 2) (1 4) (4 1) (3 3) (0 6) (6 0) (2 5) (5 2) (4 4) (7 1) (6 3) (9 0) (8 2) (10 1) (12 0) 0.0% 0.3% 0.5% (0 1) (2 0) 0% 60% (1 0) (0 2) (2 1) (4 0) 0% 7% 14% (0 0) (1 1) (0 3) (3 0) (2 2) (4 1) (6 0) 0% 5% 10% (0 1) (2 0) (1 2) (3 1) (0 4) (2 3) (5 0) (4 2) (6 1) (8 0) 0% 2% 4% (1 0) (0 2) (2 1) (1 3) (4 0) (3 2) (0 5) (2 4) (5 1) (4 3) (7 0) (6 2) (8 1) (10 0) 0.00% 0.75% 1.50%

( 1 ) ( 2 ) ( 1 2 ) ( 3 1 ) ( 4 ) ( 2 3 ) ( 5 ) ( 4 2 ) ( 1 5 ) ( 3 4 ) ( 6 1 ) ( 7 ) ( 5 3 ) ( 2 6 ) ( 4 5 ) ( 8 ) ( 7 2 ) ( 6 4 ) ( 9 1 ) ( 8 3 ) ( 1 1 ) ( 1 2 ) ( 1 2 1 ) ( 1 4 )

0.00% 0.06% 0.11%

remaining Sp Sn S Sp=1/2 Sn=3/2 S=2 Sp=3/2 Sn=1/2 S=2 Sp=3/2 Sn=3/2 S=3 Sp=1/2 Sn=1/2 S=1

60.77% 18.82% 11.63% 5.37% 2.28% 0.85% 0.27%

6Li : 1+ gs

Discovery: Emergence of Simple Patterns in Complex Nuclei Discovery: Emergence of Simple Patterns in Complex Nuclei

Dytrych, Launey, Draayer, et al., PRL 111 (2013) 252501

Low spin Large deformation

Key features of nuclear structure Model space truncation

SA-NCSM on BlueWaters: reaching towards medium mass nuclei SA-NCSM on BlueWaters: reaching towards medium mass nuclei

Nuclear density

Complete space: Symmetry-adapted space: Quadrupole moment

SA-NCSM on BlueWaters: reaching towards medium mass nuclei SA-NCSM on BlueWaters: reaching towards medium mass nuclei

Complete space: Symmetry-adapted space: Number of BW nodes: Basis construction: Matrix calculation: Solving eigenproblem: 10 s 1518 s 113 s Size of Hamiltonian matrix: 3335 20 TB Total: Performance on BW system 1641 s

SA-NCSM on BlueWaters: reaching towards medium mass nuclei SA-NCSM on BlueWaters: reaching towards medium mass nuclei

Ruotsalainen et al., PRC 99, 051301 (R) (2019) B(E2) transition strengths B(E2) transition strengths

Calculation of reaction rates Calculation of reaction rates

SA-NCSM

Probability to find cluster structure

Nuclear reaction:

Calculation of reaction rates Calculation of reaction rates

Blue Waters

Probability to find cluster structure astrophysical simulation

Nuclear reaction:

Response function Response function

Nucleus response to external probe (photon, neutrino, etc ..)

SA-NCSM SA-NCSM

New approach: SA-NCSM + Lorentz Integral Transform Method

Response functions for neutrino studies Response functions for neutrino studies

Response functions – input for neutrino experiments Nuclear input - 2nd largest source of uncertainties SA-NCSM + LIT: preliminary results : component of neutrino detectors

Baseline implementation became bottleneck for heavier nuclei and large Nmax spaces

Accelerating basis construction algorithm Accelerating basis construction algorithm

Unable to utilize threads as the algorithm was inherently sequential Workaround: precompute basis segments; store on disk; read during initial step

Accelerating basis construction algorithm Accelerating basis construction algorithm

New algorithm: two orders of magnitude speedup Good scalability

• D. Langr, et al., Int. J. High Perform. Comput. Appl. 33 (2019)

Code optimizations Code optimizations

Dynamic memory allocation optimizations

Dynamic allocation – slow and dependend on malloc implementation.

malloc replacement

tcmalloc (Google), jemalloc (Facebook), tbbmalloc (Intel), litemalloc, LLAlloc, SuperMalloc Matrix construction involves lot of concurrent small allocations tcmalloc – best performance & memory footprint decrease

Code optimizations Code optimizations

Resulting speedup

20Ne J=0 20Ne J=2 16O J=0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 legacy code

• ptimized code

speedup

Memory pooling allocating large number of small objects of constant size is inneficient Solution: memory pooling Boost.Pool – best performance Small buffer optimizations small static buffer for a small number of elements, and dynamic memory over the specified threshold. https://dspace.cvut.cz/handle/10467/80473 For more results see Martin Kocicka's MSc thesis:

Summary Summary

Key challenges

Description of 99.9% mass of the Universe

Why it matters

Ultimate source of energy in the Universe Aggregate memory and high memory bandwidth

Why Blue Waters

Many papers in top journals and reaching beyond what competitives theories could accomplish

Accomplishments Blue Waters team contributions

Excellent support and guidance as needed

Broader impacts

Training students in using HPC resources

Shared Data

Codes and results publicly available

Products

https://sourceforge.net/p/lsu3shell/home/Home/