Burning on the GPU: Fast and Accurate Chemical Kinetics GPU - - PowerPoint PPT Presentation

LLNL-PRES-687782

This work was performed under the auspices of the U.S. Department

• f Energy by Lawrence Livermore National Laboratory under Contract

DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Burning on the GPU: Fast and Accurate Chemical Kinetics

GPU Technology Conference Russell Whitesides

April 7, 2016

Session 6195

Funded by: U.S. Department of Energy Vehicle Technologies Program Program Manager: Gurpreet Singh & Leo Breton

Lawrence Livermore National Laboratory

To make it go faster?

+ Why?

Lawrence Livermore National Laboratory

Why?

We burn a lot of gasoline.

• Transportation efficiency
• Chemistry is vital to predictive simulations
• Chemistry can be > 90% of simulation time.

Lawrence Livermore National Laboratory

National lab compute power and industry need.

Supercomputing @ DOE labs: Strong investment in GPUs with eye towards exascale OEM engine designers: Require fast turnaround with desktop class hardware

Why?

Lawrence Livermore National Laboratory

“Colorful Fluid Dynamics” YO2 Temperature

“Typical” engine simulation w/ detailed chemistry

Lawrence Livermore National Laboratory

Detailed Chemistry in Reacting Flow CFD:

Each cells is treated as an isolated system for chemistry. Operator Splitting Technique: Solve independent set of ordinary differential equations (ODEs) in each cell to calculate chemical source terms for species and energy advection/diffusion equations.

t t+∆t

Lawrence Livermore National Laboratory

CPU (un-coupled) chemistry integration

Each cells is treated as an isolated system for chemistry.

t t+∆t

Lawrence Livermore National Laboratory

GPU (batched) chemistry integration

On the GPU we solve chemistry in batches of cells simultaneously.

t t+∆t

Lawrence Livermore National Laboratory

See also Whitesides & McNenly, GTC 2015; McNenly & Whitesides, GTC 2014

Previously at GTC:

Lawrence Livermore National Laboratory

n_gpu = 0;

Note: most CFD simulations are done on distributed memory systems

rank0 rank1 rank2 rank3 rank4 rank6 rank7 rank5

CPU CPU CPU CPU CPU CPU CPU CPU

Lawrence Livermore National Laboratory

++n_gpu; //now what?

Note: most CFD simulations are done on distributed memory systems

rank0 rank1 rank2 rank3 rank4 rank6 rank7 rank5

CPU CPU CPU CPU CPU CPU CPU CPU

Lawrence Livermore National Laboratory

Here CPU is a single core.

Ideal CPU-GPU Work-sharing

SGPU = walltime(CPU) walltime(GPU)

Lawrence Livermore National Laboratory

Let’s make use of the whole machine.

Ideal CPU-GPU Work-sharing

§ # CPU cores = NCPU § # GPU devices = NGPU

Stotal = NCPU + NGPU SGPU −1

( )

( )

NCPU

1 2 3 4 5 6 7 8 1 2 3 4 Stotal NGPU

SGPU = 8

NCPU=4 NCPU=8 NCPU=16 NCPU=32

* *

* TITAN (1.4375) * surface (1.8750)

SGPU = walltime(CPU) walltime(GPU)

Lawrence Livermore National Laboratory

Distribute based on number of cells and give more to GPU.

Good performance in simple case with both CPU and GPU doing work

100 1000 10000 1 2 4 8 16 Chemistry Time (seconds) Number of Processors

CPU Chemistry GPU Chemistry (std work sharing)

Lawrence Livermore National Laboratory

Distribute based on number of cells and give more to GPU.

Good performance in simple case with both CPU and GPU doing work

100 1000 10000 1 2 4 8 16 Chemistry Time (seconds) Number of Processors

CPU Chemistry GPU Chemistry (std work sharing) GPU Chemistry (custom work sharing)

SGPU = 7 Stotal = 1.7 (SGPU = 6.6)

Lawrence Livermore National Laboratory

Let’s go!

First attempt @ engine calculation on GPU+CPU

Lawrence Livermore National Laboratory

What happened?

First attempt @ engine calculation on GPU+CPU

§ 2x Xeon E5-2670 (16 cores) => § 2x Xeon E5-2670 + 2 Tesla K40m => § Stotal = 21.2/17.6 = 1.20

21.2 hours 17.6 hours (SGPU = 2.6)

Lawrence Livermore National Laboratory

Integrator performance when doing batch solution

If the systems are not similar how much extra work needs to be done?

vs.

Lawrence Livermore National Laboratory

Batches of dissimilar reactors will suffer from excessive extra steps

What penalty do we pay when batching?

Lawrence Livermore National Laboratory

Batches of dissimilar reactors will suffer from excessive extra steps

What penalty do we pay when batching?

Lawrence Livermore National Laboratory

Batches of dissimilar reactors will suffer from excessive extra steps

Possibly a lot of extra steps.

Lawrence Livermore National Laboratory

Sort reactors by how many steps they took to solve on the last CFD step

Easy as pie? n_steps

>100 1

batch3 batch2 batch1 batch0

Lawrence Livermore National Laboratory

Have to manage the sorting and load- balancing in distributed memory system

Not so fast. rank0 rank7 rank5 rank6 rank4 rank1 rank2 rank3

Lawrence Livermore National Laboratory

Load balance based on expected cost and expected performance.

MPI communication to re-balance for chemistry. rank0 rank7 rank5 rank6 rank4 rank1 rank2 rank3

Lawrence Livermore National Laboratory

Let’s go again!

Second attempt @ engine calculation on GPU+CPU

Lawrence Livermore National Laboratory

How much does difference does it make?

Total steps significantly reduced by batching appropriately

Lawrence Livermore National Laboratory

J

Engine results with improved work- sharing and reactor sorting

9.1 hrs 7.6 hrs 13.0 hrs ~40 % reduction in chemistry time; ~36% reduction in overall time Stotal =1.7 SGPU =6.6

Lawrence Livermore National Laboratory

§ Improve SGPU

• Derivative kernels
• Matrix operations

§ Extrapolative integration methods

• Less “startup” cost when re-initializing
• Potentially well suited for GPU

§ Non-chemistry calc’s on GPU

• Multi-species transport
• Particle spray

Future directions

Possibilities for significant further improvements.

Lawrence Livermore National Laboratory

§ Much improved CFD chemistry work-

sharing with GPU

§ ~40% reduction in chemistry time

for real engine case (~36% total time)

§ Working on further improvement

Summary

Thank you!

+