Sustained Petascale Performance of Seismic Simulations with SeisSol - - PowerPoint PPT Presentation

Technische Universit¨ at M¨ unchen

SIAM EX 14 – Workshop on Exascale Applied Mathematics Challenges and Opportunities

Sustained Petascale Performance of Seismic Simulations with SeisSol

• M. Bader, A. Breuer, A. Heinecke, S. Rettenberger
• C. Pelties, A.-A. Gabriel

Technische Universit¨ at M¨ unchen, Ludwig-Maximilians-Universit¨ at M¨ unchen

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 1

Technische Universit¨ at M¨ unchen

HPC Meets Geoscience

Alexander Alice-Agnes Alexander Christian Sebastian Breuer Gabriel Heinecke Pelties Rettenberger

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 2

Technische Universit¨ at M¨ unchen

Overview and Agenda

SeisSol:

• dynamic rupture and seismic wave propagation
• unstructured tetrahedral meshes
• high-order ADER-DG discretisation

Optimisation for Heterogeneous Petascale Platforms:

• code generation to optimize element-local matrix kernels
• hybrid MPI/OpenMP parallelisation
• offload scheme to address multiphysics

Performance on Tianhe-2, Stampede and SuperMUC:

• weak scaling of wave propagation component
• strong scaling for 1992 Landers M7.2 earthquake
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 3

Technische Universit¨ at M¨ unchen

Dynamic Rupture and Earthquake Simulation

Tohoku subduction zone: CAD model and tetrahedral mesh (C. Pelties)

Use of Adaptive Tetrahedral Meshes:

• curved subduction zones that meet surface at shallow angles

→ high impact on uplift for tsunamigenic earthquakes

• complicated fault systems with multiple branches

→ non-linear multiphysics dynamic rupture simulation

• goal: automated meshing process (incl. CAD generation)
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 4

Technische Universit¨ at M¨ unchen

Dynamic Rupture and Earthquake Simulation

Landers fault system: simulated ground motion and tetrahedral mesh

Use of Adaptive Tetrahedral Meshes:

• curved subduction zones that meet surface at shallow angles

→ high impact on uplift for tsunamigenic earthquakes

• complicated fault systems with multiple branches

→ non-linear multiphysics dynamic rupture simulation

• goal: automated meshing process (incl. CAD generation)
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 4

Technische Universit¨ at M¨ unchen

Seismic Wave Propagation with SeisSol

Elastic Wave Equations: (velocity-stress formulation) qt + Aqx + Bqy + Cqz = 0 with q = (σ11, σ22, σ33, σ12, σ23, σ13, u, v, w)T



A =               −λ − 2 µ −λ −λ −µ −µ −ρ−1 −ρ−1 −ρ−1                

    

    B =               −λ −λ − 2 µ −λ −µ −µ −ρ−1 −ρ−1 −ρ−1                

• high order discontinuous Galerkin discretisation
• ADER-DG: high approximation order in space and time:
• additional features: local time stepping, high accuracy of

earthquake faulting (full frictional sliding) → Dumbser, K¨ aser et al. [3,5]

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 5

Technische Universit¨ at M¨ unchen

SeisSol in a Nutshell – ADER-DG

Qn+1

k

= Qk−|Sk| |Jk| M−1

• 4

X

i=1

F−,iI(tn, tn+1, Qn

k)Nk,iA+ k N−1 k,i

+

4

X

i=1

F+,i,j,hI(tn, tn+1, Qn

k(i))Nk,iA− k(i)N−1 k,i

• +M−1KξI(tn, tn+1, Qn

k)A∗ k

+M−1KηI(tn, tn+1, Qn

k)B∗ k

+M−1KζI(tn, tn+1, Qn

k)C∗ k

Update scheme

I(tn, tn+1, Qn

k) = J

X

j=0

(tn+1 − tn)j+1 (j + 1)! ∂j ∂tj Qk(tn) (Qk)t = −M−1 (Kξ)TQkA∗

k + (Kη)TQkB∗ k + (Kζ)TQkC∗ k

• Cauchy

Kovalewski

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 6

Technische Universit¨ at M¨ unchen

Optimisation of Sparse Matrix Operations

Apply sparse matrices to multiple DOF-vectors Qk

Code Generator for Sparse Kernels: (Breuer et al. [1])

• avoid overhead of CSR (or similar) data structures;

store CSR elements vector, only

• full “unrolling” of all element operations using a code generator
• use intrinsics and apply blocking to improve vectorisation
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 7

Technische Universit¨ at M¨ unchen

Optimisation of Sparse Matrix Operations

Apply sparse matrices to multiple DOF-vectors Qk

Dense vs. Sparse Kernels: (Breuer et al. [2])

• switch to dense kernels depending on achieved time to solution
• for sparse and dense kernels:

exploit zero-blocks generated during recursive CK computation

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 7

Technische Universit¨ at M¨ unchen

Mesh Generation and Partitioning

Mesh Generation:

• high-quality meshes required

(shallow subduction zones, complicated fault structures)

• with 108–109 grid cells
• using SimModeler by Simmetrix

(http://simmetrix.com/) Two-stage approach to provide parallel mesh partitions:

• graph-based partitioning (ParMETIS)
• create customised parallel format (based on netCDF) for mesh

partitions

• highly scalable mesh input via netCDF/MPI-IO in SeisSol
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 8

Technische Universit¨ at M¨ unchen

Optimization for Intel Xeon Phi Platforms

Offload Scheme:

• to address load

imbalances of multiphysics simulation

• hides communication

with Xeon Phi and between nodes OpenMP parallelisation:

• to address manycore

parallelism with 1–3 coprocessors

• careful parallelisation
• f all loops

Host PCIe Xeon Phi

MPI comm., receiver output dynamic rupture fluxes, fault output plot wave field (if required) download cells for receivers, DR, MPI upload MPI- received cells upload dynamic rupture updates download all data (if required) time integration of non-MPI cells, volume integration time integration of MPI boundary cells wave propagation fluxes apply dynamic rupture updates, pack transfer data

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 9

Technische Universit¨ at M¨ unchen

Supercomputing Platforms

SuperMUC @ LRZ, Munich

• 9216 compute nodes (18 “thin node” islands)

147,456 Intel SNB-EP cores (2.7 GHz)

• Infiniband FDR10 interconnect (fat tree)
• #12 in Top 500: 2.897 PFlop/s

Stampede @ TACC, Austin

• 6400 compute nodes, 522,080 cores

2 SNB-EP (8c) + 1 Xeon Phi SE10P per node

• Mellanox FDR 56 interconnect (fat tree)
• #7 in Top 500: 5.168 PFlop/s

Tianhe-2 @ NSCC, Guangzhou

• 8000 compute nodes used, 1.6 Mio cores

2 SNB-EP (12c) + 3 Xeon Phi 31S1P per node

• TH2-Express custom interconnect
• #1 in Top 500: 33.862 PFlop/s
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 10

Technische Universit¨ at M¨ unchen

Weak Scaling of Wave Propagation

85 87.5 90 92.5 95 97.5 100 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• goal: test scalability towards large problem sizes
• cubic domain, uniformly refined tetrahedral cells
• weak scaling: 400,000 elements per card/node
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 11

Technische Universit¨ at M¨ unchen

Weak Scaling of Wave Propagation

85 87.5 90 92.5 95 97.5 100 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• more than 90 % parallel efficiency on Tianhe-2 and Stampede
• 87 % on full SuperMUC (no overlapping)
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 11

Technische Universit¨ at M¨ unchen

Weak Scaling of Wave Propagation

85 87.5 90 92.5 95 97.5 100 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes SuperMUC, classic Stampede SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• 8.6 PFlop/s on Tianhe-2 (8000 nodes)
• 2.3 PFlop/s on Stampede (6144 nodes)
• 1.6 PFlop/s on SuperMUC (9216 nodes)
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 11

Technische Universit¨ at M¨ unchen

Weak Scaling – Peak Efficiency

10 20 30 40 50 60 % peak: hardware SuperMUC, gr. buff. Stampede Tianhe-2 SuperMUC, classic 5 10 15 20 25 30 1 2 4 8 16 32 64 128 256 512 1024 2048 4096 6144 9216 % peak: non-zero # nodes

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 12

Technische Universit¨ at M¨ unchen

1992 Landers M7.2 Earthquake

• multiphysics simulation of dynamic rupture and resulting ground

motion of a M7.2 earthquake

• fault inferred from measured data, regional topography from

satellite data, physically consistent stress and friction parameters

• 1D velocity structure, low velocity near surface
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 13

Technische Universit¨ at M¨ unchen

Strong Scaling of Landers Scenario

40 50 60 70 80 90 100 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• 191 million tetrahedrons; 220,982 element faces on fault
• 6th order, 96 billion degrees of freedom
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 14

Technische Universit¨ at M¨ unchen

Strong Scaling of Landers Scenario

40 50 60 70 80 90 100 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• more than 85 % parallel efficiency on Stampede and Tianhe-2

(when using only one Xeon Phi per node)

• multiple-Xeon-Phi performance suffers from MPI communication
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 14

Technische Universit¨ at M¨ unchen

Strong Scaling of Landers Scenario

40 50 60 70 80 90 100 256 512 1024 2048 4096 6144 9216 % parallel efficiency # nodes Stampede SuperMUC, classic SuperMUC, gr. buff. Tianhe-2, 1 card Tianhe-2, 2 cards Tianhe-2, 3 cards

• 3.3 PFlop/s on Tianhe-2 (7000 nodes)
• 2.0 PFlop/s on Stampede (6144 nodes)
• 1.3 PFlop/s on SuperMUC (9216 nodes)
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 14

Technische Universit¨ at M¨ unchen

Landers Strong Scaling – Peak Efficiency

10 20 30 40 50 % peak: hardware SuperMUC, gr. buff. Stampede Tianhe-2 SuperMUC, classic 5 10 15 20 25 256 512 1024 2048 4096 6144 9216 % peak: non-zero # nodes

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 15

Technische Universit¨ at M¨ unchen

Landers Earthquake – Production Run

On SuperMUC:

• 7 h 15 min computing time
• fault output and 23 receivers
• 1.25 PFLOPS sustained

performance Observations:

• complex rupture dynamics

(fault branching, etc.)

• high-frequency signals (up to

10 Hz) from rupture propagate directly into wave field

• ground shaking in the

engineering frequency band

• 42 s simulated time
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 16

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

Multiphysics Dynamic Rupture Simulation

• spontaneous rupturing due to exceeded stress limits
• including rupture jumps, fault branching, etc.
• tackles fundamental questions on the dynamics of earthquakes

in natural fault zone systems

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 17

Technische Universit¨ at M¨ unchen

SeisSol: Earthquake Simulation @ Petascale

Multiphysics Dynamic Rupture Simulations with SeisSol:

• high-order ADER-DG on unstructured adaptive meshes
• focus on complicated geometries and rupture physics
• non-linear interaction of rupture process and seismic waves

Petascale Performance on Heterogeneous Platforms:

• exploits high computational intensity of ADER-DG
• requires careful tuning of the entire simulation pipeline
• code generation to accelerate element kernels
• scalable mesh input for 200M cells on 147k cores
• offload-scheme for multiphysics with Xeon Phi

http://seissol.geophysik.uni-muenchen.de/

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 18

Technische Universit¨ at M¨ unchen

I Would Like to Thank . . .

• Intel:

– Mikhail Smelyanskiy, Karthikeyan Vaidyanathan – Pradeep Dubey

• all colleagues from the three supercomputing centers, esp.:

– Arndt Bode (Leibniz Supercomputing Centre) – William Barth (Texas Advanced Computing Centre) – Xiang-Ke Liao (NCSS and NUDT)

• for financial support:

– Volkswagen Foundation (project ASCETE) – KONWIHR

• the entire SeisSol team and all contributors
• You!
• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 19

Technische Universit¨ at M¨ unchen

References

[1] A. Breuer, A. Heinecke, M. Bader, C. Pelties: Accelerating SeisSol by generating vectorized code for sparse matrix operators. In: Advances in Parallel Computing 25, IOS Press, 2014. Proceedings of ParCo 2013 [2] A. Breuer, A. Heinecke, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties: Sustained Petascale Performance of Seismic Simulations with SeisSol on

• SuperMUC. In: Supercomputing, LNCS 8488, p. 1–18. PRACE ISC Award 2014.

[3] M. Dumbser, M. K¨ aser: An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes – II. The three-dimensional isotropic case.

• Geophys. J. Int. 167(1), 2006.

[4] A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A.-A. Gabriel, C. Pelties, A. Bode, W. Barth, X.-K. Liao, K. Vaidyanathan, M. Smelyanskiy, P . Dubey: Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous

• Supercomputers. Gordon Bell Prize Finalist 2014.

[5] M. K¨ aser, M. Dumbser, J. de la Puente, H. Igel: An arbitrary high-order Discontinuous Galerkin method for elastic waves on unstructured meshes –

• III. Viscoelastic attenuation. Geophys. J. Int. 168(1), 2007.

[6] C. Pelties, J. de la Puente, J.-P . Ampuero, G. B. Brietzke, M. K¨ aser: Three-dimensional dynamic rupture simulation with a high-order discontinuous Galerkin method on unstructured tetrahedral meshes. J. Geophys. Res.: Solid Earth, 117(B2), 2012.

• M. Bader et al.: Sustained Petascale Performance of Seismic Simulations with SeisSol

SIAM EX 14, July 7, 2014 20