Parallel-in-Time Optimization with the General-Purpose XBraid - - PowerPoint PPT Presentation

▶
parallel in time optimization with the general purpose
SMART_READER_LITE
LIVE PREVIEW

Parallel-in-Time Optimization with the General-Purpose XBraid - - PowerPoint PPT Presentation

Jun 07, 2023 341 likes •612 views

Parallel-in-Time Optimization with the General-Purpose XBraid Package Stefanie Gnther, Jacob Schroder , Robert Falgout, Nicolas Gauger 7th Workshop on Parallel-in-Time methods Wednesday, May 3 rd , 2018 LLNL-PRES-750301 This work was performed

slide-1
SLIDE 1

Lawrence Livermore National Laboratory

LLNL-PRES-750301

1

LLNL-PRES-750301

This work was performed under the auspices of the U.S. Department

  • f Energy by Lawrence Livermore National Laboratory under Contract

DE-AC52-07NA27344. Lawrence Livermore National Security, LLC

Parallel-in-Time Optimization with the General-Purpose XBraid Package

Stefanie Günther, Jacob Schroder, Robert Falgout, Nicolas Gauger

7th Workshop on Parallel-in-Time methods Wednesday, May 3rd, 2018

slide-2
SLIDE 2

Lawrence Livermore National Laboratory

LLNL-PRES-750301

2

§ Motivation: parallelizing optimization problems § Parallel-in-time overview: Multigrid reduction in time (MGRIT) § XBraid-adjoint code interface : Open source implementation

  • Non-intrusively uses existing user code
  • Example: user-interface for simple scalar ODE problem

Outline

slide-3
SLIDE 3

Lawrence Livermore National Laboratory

LLNL-PRES-750301

3

Example1

§ Objective: Lift maximization § Design: Amplitudes of actuation

Runtimes

§ Simulation:

2.5h

§ Optimization: 1,152h

Motivation: PDE constrained optimization

Base flow Optimized flow

54% lift increase

  • 1. Ötzkaya, Nemili et al., 2015
slide-4
SLIDE 4

Lawrence Livermore National Laboratory

LLNL-PRES-750301

4

§ Optimize object function J, with a design variable u

(continuous)

§ While satisfying constraint of the forward in time process, (continuous)

with state variable y and initial condition

Problem description: Optimization with unsteady PDEs

min 1 T Z T J(y(t), u)dt δy(t) δt + c(y(t), u) = 0, ∀t ∈ (0, T) y(0) = g g

slide-5
SLIDE 5

Lawrence Livermore National Laboratory

LLNL-PRES-750301

5

§ Optimize object function J, with a design variable u

(discrete)

§ While satisfying constraint of the forward in time process, (discrete)

with state variable y and initial condition

yn = Φ(yn−1, u), n = 1, . . . , N y0 = g

Problem description: Optimization with unsteady PDEs

g min 1 N

N

X

i=1

J(yi, u) Jn := J(yn, u)

slide-6
SLIDE 6

Lawrence Livermore National Laboratory

LLNL-PRES-750301

6

Form Lagrangian

1.

State equations:

2.

Adjoint equations:

3.

Design equation:

¯ yn = rynJn + (δyΦn)T ¯ yn, n = N, . . . , 1 ¯ yN+1 = 0 yn = Φ(yn−1, u ), n = 1, . . . , N y0 = g

First Order Optimality Conditions

L =

N

X

i

(Jn + (¯ yn)T (Φn−1 − yn))

N

X

n=1

  • ruJn + (δuΦn−1)T ¯

yn−1 = 0

slide-7
SLIDE 7

Lawrence Livermore National Laboratory

LLNL-PRES-750301

7

1.

State equations solve:

2.

Adjoint equations solve:

3.

Design update:

yn = Φ(yn−1, ui), n = 1, . . . , N y0 = g

Nested Optimization Approach

Initial design ui For i = 1, 2, ...

yn ¯ yn

Time Parallel! Time Parallel!

ui+1 = ui B−1

i

N X

n=1

(ruJn + (δuΦn−1)T ¯ yn) ! ¯ yn = rynJn + (δyΦn)T ¯ yn, n = N, . . . , 1 ¯ yN+1 = 0

slide-8
SLIDE 8

Lawrence Livermore National Laboratory

LLNL-PRES-750301

8

§ Motivation: parallelizing optimization problems § Parallel-in-time overview: Multigrid reduction in time (MGRIT) § XBraid-adjoint code interface : Open source implementation

  • Non-intrusively uses existing user code
  • Example: user-interface for simple scalar ODE problem

Outline

slide-9
SLIDE 9

Lawrence Livermore National Laboratory

LLNL-PRES-750301

9

Parallel-in-time overview: Approach leverages spatial multigrid research

§

Motivation: Clock speeds are stagnate but speedup still possible through more concurrency à Parallel-in-time is needed

Error on the fine grid Error approximated on a smaller coarse grid restriction prolongation (interpolation)

The Multigrid V-cycle

smoothing (relaxation)

slide-10
SLIDE 10

Lawrence Livermore National Laboratory

LLNL-PRES-750301

10 10

§ General one-step method for a forward evolution process § In the linear setting (for simplicity), sequential marching forward solve

Multigrid reduction in time (MGRIT)

§ We solve this system iteratively with multigrid reduction

(MGR 1979)

  • Replace sequential O(N) method with O(N) parallel alternative
  • Coarsens only in time

≡

à non-intrusive, i.e. is "arbitrary"

Φ yn = Φ(yn−1, ui), n = 1, . . . , N Ay ≡      I −Φ I ... ... −Φ I           y0 y1 . . . yN      =      g0 g1 . . . gN      ≡ g

slide-11
SLIDE 11

Lawrence Livermore National Laboratory

LLNL-PRES-750301

11 11

§ General one-step method for a backward evolution process § In the linear setting (for simplicity), sequential marching forward solve

¯ A¯ y ⌘      I ΦT I ΦT ... ... I ΦT           ¯ y0 ¯ y1 . . . ¯ yN      =      ry0J0 ry1J1 . . . ryN JN     

Multigrid reduction in time (MGRIT)

≡

§ We solve this system iteratively with multigrid reduction

(MGR 1979)

  • Replace sequential O(N) method with O(N) parallel alternative
  • Coarsens only in time à non-intrusive, i.e. is "arbitrary"

Φ ¯ yn = rynJn + (δyΦn)T ¯ yn, n = N, . . . , 1

slide-12
SLIDE 12

Lawrence Livermore National Laboratory

LLNL-PRES-750301

12 12

§ Relaxation is highly parallel

  • Alternates between F-points and C-points
  • F-relaxation = block Jacobi on each coarse time interval

MGRIT for forward solve

t0 t1 t2 t3 …

T0 T1

dt

DT = mdt

tN F-point (fine grid only) C-point (coarse & fine grid)

F-relaxation

§ Coarse system approximates fine system

  • Approximate impractical with (e.g., time rediscretization with )
  • Apply recursively for multilevel hierarchy

Φ∆ Φm ∆T

A∆ =      I −Φm I ... ... −Φm I      ⇒ B∆ =      I −Φ∆ I ... ... −Φ∆ I     

slide-13
SLIDE 13

Lawrence Livermore National Laboratory

LLNL-PRES-750301

13 13

§

Expose concurrency in the evolution dimension with multigrid

§

Non-intrusive, with unchanged fine-grid problem

§

Converges to same solution as sequential marching

A broader summary of MGRIT

     I −Φ I ... ... −Φ I     

§

Only store C-points to minimize storage

§

Optimal for variety of parabolic problems

  • Converges in ~10 iterations for any coarsening factor

t0 t1 t2 t3 … dt tN

§

Extends to nonlinear problems with FAS formulation

§

In specialized two-level setting, MGRIT Parareal

≡

§

Large speedups available, but in a new way

  • Time stepping is already O(N)
  • Useful only beyond a crossover
  • More time steps à more speedup potential
  • Example strong scaling of the 2D heat eqn

§

Much future work to do...

slide-14
SLIDE 14

Lawrence Livermore National Laboratory

LLNL-PRES-750301

14 14

§ Motivation: parallelizing optimization problems § Parallel-in-time overview: Multigrid reduction in time (MGRIT) § XBraid-adjoint code interface : Open source implementation

  • Non-intrusively uses existing user code
  • Example: user-interface for simple scalar ODE problem

Outline

slide-15
SLIDE 15

Lawrence Livermore National Laboratory

LLNL-PRES-750301

15 15

XBraid-adjoint: open source & non-intrusive

§ Solve for (reduced gradient of J w.r.t. design u) § Wrap existing user code to obtain time parallelism

Standard XBraid: forward in time solve

  • 1. Step:
  • 2. Clone
  • 3. Sum:
  • 4. SpatialNorm:
  • 5. Buf[Un]Pack
  • 6. Init
  • 7. Free
  • 8. Access

yn = Φ(yn−1, u) ym = αyn + βym kynk

Iteration k of XBraid:

yk+1 ← XBraid(yk, uk) J ← J(yk, uk)

y, ¯ y, ¯ u := (δJ/δu)T

slide-16
SLIDE 16

Lawrence Livermore National Laboratory

LLNL-PRES-750301

16 16

XBraid-adjoint: open source & non-intrusive

§ Solve for (reduced gradient of J w.r.t. design u) § Wrap existing user code to obtain time parallelism ui+1 = ui B−1

i

N X

n=1

(ruJn + (δuΦn−1)T ¯ yn) !

¯ yn = rynJn + (δyΦn)T ¯ yn+1

Functions 2 and 3 allow XBraid to compute Iteration k of XBraid-adjoint:

¯ yk+1 ← XBraid adjoint(yk, ¯ yk, uk) ¯ u ← δJ(yk, uk) δu

y, ¯ y, ¯ u := (δJ/δu)T

Reduced gradient: ¯

u

XBraid-Adjoint

  • 1. ObjectiveT:
  • 2. Step_diff:
  • 3. ObjectiveT_diff:

J(yn, u)

¯ yn = (δyΦn)T ¯ yn+1 ¯ u +=(δuΦn)T ¯ yn+1

¯ yn += rynJn ¯ u += ruJn

slide-17
SLIDE 17

Lawrence Livermore National Laboratory

LLNL-PRES-750301

17 17

§ Solve for the reduced gradient (later use in optimization) § Begin with simple problem: § First, user must define objects: App and Vector

XBraid example: ex-01-adjoint.c

¯ u := (δJ/δu)T J(y, λ) = 1/T Z T kykdt s.t. yt = λy y(0) = 1

slide-18
SLIDE 18

Lawrence Livermore National Laboratory

LLNL-PRES-750301

18 18

§ Step():

XBraid example: ex-01-adjoint.c

yn = Φ(yn−1, u)

...

slide-19
SLIDE 19

Lawrence Livermore National Laboratory

LLNL-PRES-750301

19 19

§ Step_diff():

XBraid example: ex-01-adjoint.c

...

¯ yn = (δyΦn)T ¯ yn+1 ¯ u +=(δuΦn)T ¯ yn+1

slide-20
SLIDE 20

Lawrence Livermore National Laboratory

LLNL-PRES-750301

20 20

§ ObjectiveT() and ObjectiveT_diff() are similar § Initialize and run XBraid-adjoint:

XBraid example: ex-01-adjoint.c

...

¯ u ← (δJ/δu)T

slide-21
SLIDE 21

Lawrence Livermore National Laboratory

LLNL-PRES-750301

21 21

§ Run it!

XBraid example: ex-01-adjoint.c

slide-22
SLIDE 22

Lawrence Livermore National Laboratory

LLNL-PRES-750301

22 22

§ XBraid-adjoint solver

  • Generically solves adjoint equations, backwards in time
  • Generically computes reduced gradients

à Designed to work with many optimization methods!

§ Example: ex-01-optimization.c

  • Implements simple reduced space optimization loop

(This is a template for implementing your favorite optimization method)

  • Requires:
  • Global objective evaluation function (and derivative)
  • Design update function
  • Halting metric (e.g., gradient norm)
  • Again, the goal is to wrap existing user code!

XBraid example: ex-01-optimization.c

¯ u

slide-23
SLIDE 23

Lawrence Livermore National Laboratory

LLNL-PRES-750301

23 23

§

Advection dominated, with Van Der Pol oscillator on left boundary

§

Tracking objective: Minimize difference

  • f space-time averaged

solution to preset value

§

When used with

  • ne-shot strategies,

the max speedup is 25x

Model Problem: 1D advection-diffusion

0.25 0.5 1 2 4 8 2 4 8 16 32 64 128 256 4 8 16 32 64 128

N = 60000 N = 120000 N = 240000

runtime primal (sec) runtime adjoint (sec) cores primal adjoint

Scaling of primal (solid lines) and adjoint (dashed lines) XBraid solvers.

yt + yx − ✏yxx = 0, ✏ = 10−5

slide-24
SLIDE 24

Lawrence Livermore National Laboratory

LLNL-PRES-750301

24 24

Summary and Conclusions

§ Parallel time integration is needed on future architectures,

especially for adjoint-based optimization techniques

§ New XBraid-adjoint non-intrusively couples with existing codes

  • Available in May, 2018 release
  • http://llnl.gov/casc/xbraid

§ References

  • 1. Günther, Gauger, and Schroder. A Non-Intrusive Parallel-in-Time

Approach for Simultaneous Optimization with Unsteady PDEs. Optimization Methods and Software, (submitted), (2018).

  • 2. Günther, Gauger, and Schroder. A Non-Intrusive Parallel-in-Time Adjoint

Solver with the XBraid Library. CVS, Springer, (accepted), (2017).

PARALLEL MULTIGRID IN TIME

BRAID BRAID

slide-25
SLIDE 25

Lawrence Livermore National Laboratory

LLNL-PRES-750301

25 25

Thank You! Any Questions?

A good read, Parallel Time Integration with Multigrid, SIAM J. Sci. Comp. Open Source XBraid Code

§

http://llnl.gov/casc/xbraid

Tzanio Kolev Rob Falgout Jacob Schroder

Our Team

PARALLEL MULTIGRID IN TIME

BRAID BRAID

Collaborators

CU Boulder (Manteuffel, McCormick, O'Neill), Red Deer (Howse), U Stuttgart (Roehrle, Hessenthaler), Kaiserslautern (Günther, Gauger), LLNL (Woodward, Top)

Matthieu Lecouvez Scott MacLachlan Hans De Sterck Stephanie Friedhoff Stefanie Günther

slide-26
SLIDE 26

Lawrence Livermore National Laboratory

LLNL-PRES-750301

26 26

Done