Acceleration of Time Integration edited version, with extra images - - PowerPoint PPT Presentation

▶
acceleration of time integration
SMART_READER_LITE
LIVE PREVIEW

Acceleration of Time Integration edited version, with extra images - - PowerPoint PPT Presentation

Nov 10, 2023 516 likes •1.01k views

Acceleration of Time Integration edited version, with extra images removed Rick Archibald, John Drake, Kate Evans, Doug Kothe, Trey White, Pat Worley Research sponsored by the Laboratory This research used resources of the National Directed

slide-1
SLIDE 1

Rick Archibald, John Drake, Kate Evans, Doug Kothe, Trey White, Pat Worley

Acceleration of Time Integration

1 Research sponsored by the Laboratory Directed Research and Development Program

  • f Oak Ridge National Laboratory (ORNL),

managed by UT-Battelle, LLC for the U. S. Department of Energy under Contract No. DE- AC05-00OR22725. This research used resources of the National Center for Computational Sciences at Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract

  • No. DE-AC05-00OR22725.

edited version, with extra images removed

slide-2
SLIDE 2

2

Lawrence Buja, NCAR Climate Models: From IPCC to Petascale Keynote for 2007 NCCS User Meeting

Motivation:

Higher resolution

slide-3
SLIDE 3

3

300 km current climate models

slide-4
SLIDE 4

4

http://www.geo-prose.com/projects/pdfs/petascale_science.pdf

“More importantly, because the assumptions that are made in the development of parameterizations of convective clouds and the planetary boundary layer are seldom satisfied, the atmospheric component model must have sufficient resolution to dispense with these parameterizations. This would require a horizontal resolution of 1 km.”

slide-5
SLIDE 5

5

1 km

slide-6
SLIDE 6

6

TIME BARRIER

Current climate models use explicit time integration I f r e s

  • l

u t i

  • n

g

  • e

s u p the time step must go down!

slide-7
SLIDE 7

7

300 km current climate models 20 minutes

slide-8
SLIDE 8

8

1 km 4 seconds!

slide-9
SLIDE 9

9

Extreme-scale systems will provide unprecedented parallelism!

slide-10
SLIDE 10

10

But

performance of individual processes has stagnated

4-second time step... multi-century simulation?

slide-11
SLIDE 11

Fast Forward

Overcoming the time barrier

11

  • Fully implicit time integration
  • Stable for big time steps
  • Parallel in time
  • Time is the biggest dimension
  • New discretizations
  • Better time accuracy
  • ne metaphor just isn’

t enough

slide-12
SLIDE 12

How to build a new climate model

12

  • 1. Start with shallow-water equations on the sphere

They mimic full equations for atmosphere and ocean

slide-13
SLIDE 13
  • 2. Prove yourself on standard tests

13

Defined by Williamson, Drake, Hack, Jakob, and Swarztrauber in 1992 (148 citations)

How to build a new climate model

slide-14
SLIDE 14

How to build a new climate model

14

  • 3. Proceed to 3D tests and inclusion in a full model

That’ s all there is to it!

slide-15
SLIDE 15

Fast Forward

Overcoming the time barrier

15

  • Fully implicit time integration
  • Stable for big time steps
  • Parallel in time
  • Time is the biggest dimension
  • New discretizations
  • Better time accuracy
  • ne metaphor just isn’

t enough

slide-16
SLIDE 16

Explicit versus implicit

16

y State of simulation at current time Values are known y′ State of simulation at next time step Values are unknown y′ = f(y) Explicit Compute unknown directly from known

slide-17
SLIDE 17

Explicit good and bad

  • Good
  • Highly parallel
  • Nearest-neighbor communication
  • Bad
  • Numerically unstable (blows up) for ∆t > O(∆x)
  • Increase resolution → decrease ∆x → decrease ∆t

17

slide-18
SLIDE 18

Explicit versus implicit

18

y State of simulation at current time Values are known y′ State of simulation at next time step Values are unknown y′= ay+f(y′) Implicit Solve a (nonlinear) system of equations

slide-19
SLIDE 19

Implicit bad and good

  • Bad
  • Must solve a (nonlinear) system of equations
  • Good
  • Numerically stable for arbitrary time steps
  • Ugly
  • Still need to worry about accuracy (for big time steps)

19

slide-20
SLIDE 20

Implicit + shallow water

(Kate Evans)

  • Start with HOMME shallow-water code
  • Convert explicit formulation to implicit
  • Solve with Trilinos

20

slide-21
SLIDE 21

HOMME

  • High-Order-Method Modeling Environment
  • Principal developers
  • NCAR: John Dennis, Jim Edwards, Rory Kelly, Ram

Nair, Amik St-Cyr

  • Sandia: Mark Taylor
  • Cubed-sphere grid
  • Spectral-element formulation (and others)
  • Shallow-water equations (and others)

21

image courtesy of Mark Taylor

slide-22
SLIDE 22

Jacobian-Free Newton Krylov (JFNK)

22

slide-23
SLIDE 23

Jacobian-Free Newton Krylov

  • What we want: F(y)=0
  • What we have: F(y)≠0
  • Find the change in F

as y changes

  • Jacobian, J, derivative of a vector
  • Approximate correction: F(y+∆y)=0

0=F(y+∆y)≈F(y)+J∆y F(y)=-J∆y

  • Solve the linear system for ∆y and add to y
  • Repeat until F(y)≈0

23

slide-24
SLIDE 24

Jacobian-Free Newton Krylov

  • F(y)=-J∆y
  • Solve for ∆y using an

iterative linear solver

  • Krylov subspace methods
  • Take a guess at ∆y
  • Calculate how bad it is (residual)
  • Use residual to improve guess
  • Iterate, using past residuals and

Russian Navy know-how to improve guess

  • Stop when residual is small (guess is good)

24

slide-25
SLIDE 25

Jacobian-Free Newton Krylov

  • Don’t compute the Jacobian
  • Approximate it using finite

differences J∆y≈(F(y+ε∆y)-F(y))/ε ε is a small number

  • Can be much cheaper to

calculate, only need F

25

slide-26
SLIDE 26

Test case 1: cosine bell initial condition

26

slide-27
SLIDE 27

Test case 1: cosine bell explicit solver with “hyperviscosity”

27

slide-28
SLIDE 28

Test case 1: cosine bell implicit solver, no preservatives

28

slide-29
SLIDE 29

Test case 1: cosine bell implicit versus explicit

  • Implicit takes many iterations per time step
  • But 2-hour time step instead of 2-minute
  • Similar error at the end
  • 40% shorter runtime (no preconditioner)

29

Performance result #1!

slide-30
SLIDE 30

Test case 2: steady state

  • 12 simulated days
  • Explicit
  • 4-minute time step
  • 28s runtime
  • Implicit
  • 12-day time step
  • 3.6s runtime

30

Performance result #2!

slide-31
SLIDE 31

Fast Forward

Overcoming the time barrier

31

  • Fully implicit time integration
  • Stable for big time steps
  • Parallel in time
  • Time is the biggest dimension
  • New discretizations
  • Better time accuracy
  • ne metaphor just isn’

t enough

slide-32
SLIDE 32

Parareal

(my interest)

  • Algorithm published in 2001 by

Jacques-Louis Lions, Yvon Maday, and Gabriel Turinici

  • Variants successful for range
  • f applications
  • Navier-Stokes
  • Structural dynamics
  • Reservoir simulation

32

slide-33
SLIDE 33

Parareal

33

to t1 t2 t3 Y(x) tN . . . . . .

Solve serially at coarse time steps

slide-34
SLIDE 34

Parareal

34

to t1 t2 t3 Y(x) tN . . . . . .

Compute fine time integrations between coarse steps in parallel

slide-35
SLIDE 35

Parareal

35

to t1 t2 t3 Y(x) ∆1 ∆

2

∆

3

tN . . . . . . ∆N

Propagate and accumulate fine-time corrections at coarse scale

slide-36
SLIDE 36

Parareal

36

  • Iterate until corrections are negligible
  • Published results by others: 2-3 iterations
slide-37
SLIDE 37

My parareal experience

37

  • Numerically unstable for pure advection
  • Confirms theoretical result by Maday and

colleagues

  • Should work for Burgers’ equation

∂u ∂t + v ∂u ∂x = 0 ∂u ∂t + u∂u ∂x − ν ∂2u ∂x2 = 0

X ?

slide-38
SLIDE 38

Fast Forward

Overcoming the time barrier

38

  • Fully implicit time integration
  • Stable for big time steps
  • Parallel in time
  • Time is the biggest dimension
  • New discretizations
  • Better time accuracy
  • ne metaphor just isn’

t enough

slide-39
SLIDE 39

Curvelets

(Rick Archibald)

  • Compact in space

(like finite elements)

  • Preserve shape

(like Fourier waves)

  • Might allow Δt ~ Δx1/2

39

slide-40
SLIDE 40

Curvelets

40

But they require a periodic domain

slide-41
SLIDE 41

Multi-wavelets

(Rick Archibald)

  • Adaptive
  • Designed for refinement
  • Strong error bounds
  • Control refinement and coarsening
  • Requires integral formulation
  • Translation: more theoretical work to do
  • Work just getting started

41

slide-42
SLIDE 42

Finite differences

(my interest)

42

slide-43
SLIDE 43

High-order single-step time integration

43

Consider advection:

∂u ∂t + v ∂u ∂x = 0

Time integration using a Taylor series in small ∆t

u = u − ∆t∂u ∂t + ∆t2 2 ∂2u ∂t2 − ∆t3 6 ∂3u ∂t3 + O(∆t4)

Implicit

slide-44
SLIDE 44

High-order single-step time integration

  • Replace time derivatives with space

derivatives

  • Why?

Many grid points in space, few in time (2) So you can form high-order space derivatives

  • How?

Use the governing equation

44

∂u ∂t + v ∂u ∂x = 0 ∂u ∂t = −v ∂u ∂x

→

slide-45
SLIDE 45

High-order single-step time integration

  • Got high-order space derivatives?
  • Get high accuracy in time for free*!
  • Just 2 points in time: this one and next one
  • Save memory
  • Save I/O storage space and bandwidth
  • Easy startup from initial condition

45

* Since flops are free.

u = u + v∆t∂u ∂x + v2∆t2 2 ∂2u ∂x2 + v3∆t3 6 ∂3u ∂x3 + O(∆t4)

slide-46
SLIDE 46

High-order single-step time integration

  • Explicit and implicit work for advection
  • Explicit works for Burgers’ equation
  • Implicit and semi-implicit for Burgers’ under

development

  • Goal is shallow-water equations

46

slide-47
SLIDE 47

Would you believe I cut some topics from the talk?

  • High-order methods for compact stencils
  • Single-cycle multi-level linear solvers

47