What can we expect from grid point AROME ? Thomas Burgot (PhD - - PowerPoint PPT Presentation

What can we expect from grid point AROME ?

Thomas Burgot (PhD Student, CNRM/GMAP/ALGO) Supervisors : Ludovic Auger, Pierre Bénard Dynamic Day 28/05/2019

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Problem

Two issues :

• Scalability
• Steep slopes

A common solution ?

• Grid Point approach

1/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

AROME 2D - presentation

• ICI constant coefficient, SL, A-grid, mass-based coordinate, etc
• No physics, idealised test cases
• Spectral or grid point versions

Grid point version :

• Explicit diffusion
• Krylov solver

Stop criterion : ε =

• (Ax −b)T(Ax −b)

√ bbT

2/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Density current test case

Spectral vs Grid Point 4th order, Niter ≈ 10

3/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Hypothesis

Parallelisation :

• No MPI, no Open-MP

Geometry :

• 500 x 500 pts (vs 1536 x 1440 pts in operational AROME)

Spectral computations :

• From Dt+∆t to Ut+∆t and V t+∆t
• Implicit diffusion
• RHS

Grid point computations :

• Derivatives in the linear operator
• Krylov solver

4/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Spectral vs Grid Point

43°N 44°N 45°N 46°N 47°N 3°E 4°E 5°E 6°E 7°E 8°E 9°E 10°E

Spectral AROME

266.0 269.1 272.2 275.3 278.4 281.5 284.6 287.7 290.8 293.9 297.0 43°N 44°N 45°N 46°N 47°N 3°E 4°E 5°E 6°E 7°E 8°E 9°E 10°E

Grid Point AROME

266.0 269.1 272.2 275.3 278.4 281.5 284.6 287.7 290.8 293.9 297.0

T80, δt = 50 s, T = 2 h, ∆x = 1.3 km, Niter ≈ 13, 8th order

5/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Spectral vs Grid Point

43°N 44°N 45°N 46°N 47°N 3°E 4°E 5°E 6°E 7°E 8°E 9°E 10°E

Difference

1.5 1.2 0.9 0.6 0.3 0.0 0.3 0.6 0.9 1.2 1.5

δ(T80), δt = 50 s, T = 2 h, ∆x = 1.3 km, Niter ≈ 13, 8th order

6/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Sensitivity test case

43°N 44°N 45°N 46°N 47°N 3°E 4°E 5°E 6°E 7°E 8°E 9°E 10°E

Difference

0.20 0.16 0.12 0.08 0.04 0.00 0.04 0.08 0.12 0.16 0.20

δ(T80), δt = 50 s, T = 2 h, ∆x = 1.3 km One more iteration in the ICI

7/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Sensitivity test case

43°N 44°N 45°N 46°N 47°N 3°E 4°E 5°E 6°E 7°E 8°E 9°E 10°E

Difference

0.5 0.4 0.3 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5

δ(T80), δt = 50 s, T = 2 h, ∆x = 1.3 km Random noise at the bottom of the atmosphere (σ = 0.01 K) at t = 0

8/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Comparisons

• Comparisons between AROME and some observations : nearly

identical scores after 4 hours (not shown) Perspectives :

• To extend study to 8∗24 hours forecast

9/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Solver for constant coefficient SI

• 1−δt2B∆
• Dt+δt = D••

B non-symmetric matrix (boundary conditions) : GMRES method

10/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Solver for constant coefficient SI

By projecting in the eigenspace of B :

• 1−δt2bm∆
• QDt+δt = QD••

where bm ∈ [10−2,105] m2s−2 (

• bm ∈ [0.1,320] ms−1)

10/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Solver for constant coefficient SI

By projecting in the eigenspace of B :

• 1−δt2bm∆
• QDt+δt = QD••

where bm ∈ [10−2,105] m2s−2 (

• bm ∈ [0.1,320] ms−1)

cond ≈ 1+∆t2bmπ2/∆x2 1+∆t2bm4π2/L2 ≈ 1+δt2bm π2 ∆x2 = 1+C 2

∗

where C∗ is the CFL number

10/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Convergence behaviour

ε = 10−8

10 20 30 40 50 60 70 80 90 Vertical mode index 20 40 60 80 100 120 140 160 Number of iterations required 11/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Numerical cost

AROME operational configuration (2019)

• 170 nodes on Bull SX supercomputer
• output bandwidth from a node : 7 Go/s
• network latency : 0.864 ms

12/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Numerical cost

AROME operational configuration (2019)

• 170 nodes on Bull SX supercomputer
• output bandwidth from a node : 7 Go/s
• network latency : 0.864 ms

Without projection on vertical modes (150 iterations required) : "Total" cost : 27.5+0.1 ≈ 27.6 s With projection on vertical modes (13 iterations required) : "Total" cost : 2.4+1 ≈ 3.4 s

12/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Comparison with an HEVI model

Cost of 1 iteration in the solver ≈ Cost of 1 acoustic time step

13/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Comparison with an HEVI model

Cost of 1 iteration in the solver ≈ Cost of 1 acoustic time step δt = 50 s, ∆x = 1300 m, c = 350 m/s HEVI model (if we suppose CFL < 1) : ∆t ≈ 4 s

13/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Comparison with an HEVI model

Cost of 1 iteration in the solver ≈ Cost of 1 acoustic time step δt = 50 s, ∆x = 1300 m, c = 350 m/s HEVI model (if we suppose CFL < 1) : ∆t ≈ 4 s SI grid point model : ∆τ = δt 2Niter = 50 2∗13 ≈ 2 s

13/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Conclusion

Results

• Non exact derivatives –> Order ≥ 6
• Convergence –> Niter ≈ 13
• Technical viability
• Simulated computational cost seems low

14/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations without orographic terms

∂U′ ∂t =−RT ∗ ∂q′ ∂x − RT ∗ π∗

S

∂π′

S

∂x −R

1

η

m∗ π∗ ∂T ′ ∂x dη′ +RT ∗

1

η

m∗ π∗ ∂q′ ∂x dη′ ∂d′ ∂t =− g rH∗ ∂ ∗(∂ ∗ +1)q′ ∂q′ ∂t =− Cp Cv ∂U′ ∂x +d′

• + 1

π∗

η

0 m∗ ∂U′

∂x dη′ ∂T ′ ∂t =− RT ∗ Cv ∂U′ ∂x +d′

• ∂π′

S

∂t =−

1

0 m∗ ∂U′

∂x dη

15/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations σ-coor with orographic terms

∂U′ ∂t =...+ RT ∗ π∗2

S

∂π∗

S

∂x π′

S + 1

T ∗ ∂φ ∗ ∂x T ′ − ∂φ ∗ ∂x q′ − π∗ m∗ ∂φ ∗ ∂x ∂q′ ∂η ∂d′ ∂t =... ∂q′ ∂t =− Cp Cv

• ...+

1 RT ∗ π∗ m∗ ∂φ ∗ ∂x ∂U′ ∂η

• − 1

π∗ ∂π∗ ∂x U′ +... ∂T ′ ∂t =− RT ∗ Cv

• ...+

1 RT ∗ π∗ m∗ ∂φ ∗ ∂x ∂U′ ∂η

• ∂π′

S

∂t =...−

1

0 U′ ∂m∗

∂x dη

16/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations η-coor with orographic terms

∂U′ ∂t =...+RdT ∗

1

η

∂ ∂x m∗ π∗

• qdη′ −Rd

1

η

∂ ∂x m∗ π∗

• T ′dη′

∂d′ ∂t =... ∂q′ ∂t =... ∂T ′ ∂t =... ∂π′

S

∂t =...

17/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations η-coor with orographic terms

In general : ∂X ∂t = L(X) With a 2-TL discretisation :

• I − δt

2 L

• X + = X •

17/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

I A B C U+ d+ q+ T + π+

S

= U• d• q• T • π•

S 18/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

I A B C U+ Φ+ = U• Φ•

18/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• We can reduce the problem to only one equation :

• I −AC −1B
• U+ = U• −AC −1Φ•

19/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• In constant coefficient approach :

• I −A∗C −1B∗∆
• U+ = U• −AC −1Φ•

19/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• With orography in σ-coordinate :

• I −AC −1B
• U+ = U• −AC −1Φ•
• Orographic idealised test cases
• Identify the instability contribution of each orographic term

19/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• With orography in η-coordinate :

• I −AC −1B
• U+ = U• −AC −1Φ•

19/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Conclusion & perspectives

Scalability :

• Grid point approach seems viable in AROME
• Grid point approach seems competitive

Perspectives

• To test some preconditioners ?
• To remove completely spectral computations in AROME ?

Steep slopes : Perspectives

• To test it in AROME 2D
• To measure the interest/potential

20/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

End

Thank you for your attention ! Do you have some questions ?

21/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linearisation

Constant coefficient approach : π(x,η) = π∗(η)+π′(x,η) where : π∗(η) = A(η)+B(η)π∗

S

Linearisation around a state which contains orography : π(x,η) = π∗(x,η)+π′(x,η) where : π∗(x,η) = A(η)+B(η)π∗

S(x)

In σ-coordinate : A(η) = 0, hence : m∗ π∗ = ∂ηB(η)π∗

S(x)

B(η)π∗

S(x)

= ∂ηB(η) B(η)

22/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations without orographic terms

∂U′ ∂t =−RT ∗ ∂q′ ∂x − RT ∗ π∗

S

∂π′

S

∂x −R

1

η

m∗ π∗ ∂T ′ ∂x dη′ +RT ∗

1

η

m∗ π∗ ∂q′ ∂x dη′ ∂d′ ∂t =− g rH∗ ∂ ∗(∂ ∗ +1)q′ ∂q′ ∂t =− Cp Cv ∂U′ ∂x +d′

• + 1

π∗

η

0 m∗ ∂U′

∂x dη′ ∂T ′ ∂t =− RT ∗ Cv ∂U′ ∂x +d′

• ∂π′

S

∂t =−

1

0 m∗ ∂U′

∂x dη

23/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations σ-coor with orographic terms

∂U′ ∂t =...+ RT ∗ π∗2

S

∂π∗

S

∂x π′

S + 1

T ∗ ∂φ ∗ ∂x T ′ − ∂φ ∗ ∂x q′ − π∗ m∗ ∂φ ∗ ∂x ∂q′ ∂η ∂d′ ∂t =... ∂q′ ∂t =− Cp Cv

• ...+

1 RT ∗ π∗ m∗ ∂φ ∗ ∂x ∂U′ ∂η

• − 1

π∗ ∂π∗ ∂x U′ +... ∂T ′ ∂t =− RT ∗ Cv

• ...+

1 RT ∗ π∗ m∗ ∂φ ∗ ∂x ∂U′ ∂η

• ∂π′

S

∂t =...−

1

0 U′ ∂m∗

∂x dη

24/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations η-coor with orographic terms

∂U′ ∂t =...+RdT ∗

1

η

∂ ∂x m∗ π∗

• qdη′ −Rd

1

η

∂ ∂x m∗ π∗

• T ′dη′

∂d′ ∂t =... ∂q′ ∂t =... ∂T ′ ∂t =... ∂π′

S

∂t =...

25/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Linear equations η-coor with orographic terms

In general : ∂X ∂t = L(X) With a 2-TL discretisation :

• I − δt

2 L

• X + = X •

25/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• We can reduce the problem to only one equation :

• I −AC −1B
• U+ = U• −AC −1Φ•

26/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• In constant coefficient approach :

• I −A∗C −1B∗∆
• U+ = U• −AC −1Φ•

26/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• With orography in σ-coordinate :

• I −AC −1B
• U+ = U• −AC −1Φ•
• Orographic idealised test cases
• Identify the instability contribution of each orographic term

26/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

SI scheme

U+ +AΦ+ = U• BU+ +CΦ+ = Φ• With orography in η-coordinate :

• I −AC −1B
• U+ = U• −AC −1Φ•

C is a block diagonal matrix that contains NXNY ≈ 106 blocks of matrix of size NL = 90 Solution ? Some blocks are so similar that we can suppose they are identical, then we have to invert only few (1000 ?) blocks

26/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Accélération de la convergence

Ax = b est équivalent à : PAx = Pb P est un bon préconditionneur si P ≈ A−1 ie :

• cond(PA) << cond(A)
• coût CPU faible du produit PA
• coût communication faible du produit PA

27/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Méthode point de grille

cond(A) = 262 A A−1

28/ 21

Introduction AROME 2D AROME 3D Steep slopes Conclusion

Préconditionneur

cond(A) = 262 cond(PA) = 31, α = 47% cond(PA) = 55, α = 7%

29/ 21