* Dr. Axel Voigt (voigt@caesar.de) research center caesar crystal - - PowerPoint PPT Presentation

▶
dr axel voigt voigt caesar de research center caesar
SMART_READER_LITE
LIVE PREVIEW

* Dr. Axel Voigt (voigt@caesar.de) research center caesar crystal - - PowerPoint PPT Presentation

Mar 06, 2023 554 likes •809 views

AMDiS Adaptive Multidimensional simulations: Parallel Concepts Parallel Concepts Christina Stcker (stoecker@caesar.de), Dr. Angel Ribalta (ribalta@caesar.de), Simon Vey (vey@caesar.de), * Dr. Axel Voigt (voigt@caesar.de) research

slide-1
SLIDE 1

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Parallel Concepts

Christina Stöcker (stoecker@caesar.de),

  • Dr. Angel Ribalta (ribalta@caesar.de),

Simon Vey (vey@caesar.de),

  • Dr. Axel Voigt (voigt@caesar.de)

research center caesar – crystal growth group Ludwig-Erhard-Allee 2 53175 Bonn, Germany

*

slide-2
SLIDE 2

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

What is AMDiS?

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion

AMDiS (Adaptive MultiDimensional Simulations)

C++ library for the numerical solution of PDEs of 2nd order

∂tu−∇⋅A ∇ ub⋅∇ ucu= f

Enables Computation in 1d, 2d, 3d Coupling of problems (same or different dimension) Solution of PDE systems Implements the finite element method (FEM) Local mesh adaption based on local error estimates Design goals: high level of abstraction, generality, extensibility, efficiency

real world process

  • math. model

domain triangulation basis functions FEM simulation approx. solution

slide-3
SLIDE 3

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Adaptivity

Pre-processing

assemble solve estimate adapt

Post-processing stationary adaption loop

mesh, eq.sys. solution

  • perators

adapted mesh error indicators solution mesh,

Idea

Reach given error tolerance with minimal effort

  • 1. Start with a coarse macro triangulation
  • 2. Calculate solution (assemble, solve)
  • 3. Estimate local and global error indicators
  • 4. If tolerance not reached
  • > refine elements with large errors
  • > Goto 2

Bisectioning of simplices:

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-4
SLIDE 4

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Hierarchical mesh structure

Example instance diagram:

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-5
SLIDE 5

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Classical adaptive Domain Decomposition

assemble solve estimate adapt

stationary adaption loop

adapted mesh

DD

rank i

assemble solve estimate adapt

stationary adaption loop

adapted mesh

DD

rank j

  • DD tries to balance the load for current mesh
  • Each processor computes on it's domain only

=> Communication at domain boundaries => Adaption of many software components

  • After mesh adaption -> new load balanced DD!

=> Communication needed

  • to calculate DD
  • to redistribute mesh elements
  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-6
SLIDE 6

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Parallel adaptive meshing in AMDiS

“A new paradigm for parallel adaptive meshing algorithms”,

  • R. E. Bank and M. Holst, SIAM Rev. 2003

Literature:

  • Solve problem on rel. coarse mesh.
  • Estimate element errors.
  • Create partitions with approx. equal error.
  • Each processor solves problem on whole domain.
  • Refinement allowed only in own partition.
  • Create composite mesh.
  • Create global solution.

(1) (2) (3) init DD adapt

mesh sync.

collect solutions build global solution

init DD adapt

mesh sync. send solution

init DD adapt

mesh sync. send solution

init DD adapt

mesh sync. send solution

rank 0: rank 1: rank 2: rank 3:

assemble solve estimate adapt

parallel_init parallel_exit

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-7
SLIDE 7

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Error-weighted spectral bisectioning

Problem: S – set of mesh elements, n = |S|, - weights Find sets and s.t.

P1 P2

(a) (b) |C| is minimized with C = {( , ) | and are neighbour elements and belong to different partitions} (c) and are connected (d)

P1∪P2=S ∧ P1∩P2=∅ i  j i  j P1 P2 w1⋅error P1 ≈ w2⋅errorP2 w1,w2

spectral bisectioning

w1=2 3 ,w2=1 3

P1 P2

spectral bisectioning

w1=1 2 ,w2=1 2

P1 P2

final partitions

Example for 3 partitions:

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-8
SLIDE 8

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Mesh structure code

1 2 3 4 5 6 7 8 9 10 11

Pre-order traverse: 4 8 9 5 1 2 3 6 7 10 11 1 1 1 1 Has children? decimal: 24 20

1 2 3 4 5 6 7 8 9 1011

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-9
SLIDE 9

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Mesh synchronization

Merging codes:

1 1 1 1 1 1 1 1 1 0 =

∪

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-10
SLIDE 10

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Mesh synchronization (2)

rank 0

mesh 0

  • comp. mesh

code code code 1 code 2 code 3 merged code

rank 1

mesh 1

  • comp. mesh

code 1 code code 1 code 2 code 3 merged code

rank 2

mesh 2

  • comp. mesh

code 2 code code 1 code 2 code 3 merged code

rank 3

mesh 3

  • comp. mesh

code 3 code code 1 code 2 code 3 merged code

MPI::Allgatherv

  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
slide-11
SLIDE 11

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Partition of Unity

{i} ui i i 

: open covering of : approximate solution on local domain : continuous partition of unity subordinate to the covering {i}

ix≥0 ix=0

∑ ix=1

∀ x∈ ∀ x∉i ∀ x∈ uGx=∑ ixuix

Building global solution by Partition of Unity method: Global solution:

  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
slide-12
SLIDE 12

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Partition of Unity (2)

H i ∖i

: edge size of mesh on Theorem:

i hi i

: edge size of mesh on

i H :=max H i h:=max hi u∈H

2

Assume , then

∥u−uG∥H

1 ≤ ChH

2

In particular, if then

H≤h ∥u−uG∥H

1 ≤ Ch

  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
slide-13
SLIDE 13

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

adaption loop adaption loop

Time dependent problems

adaption loop parallel init parallel exit

time += timestep

adaption loop parallel init parallel exit Repartitioning: (1) Store composite mesh structure code of last timestep (2) Go back to macro triangulation (3) Do initial iterations for the new timestep (4) Compute new DD (5) Refine processors partition according to structure code

time += timestep

after every timestep: after every i-th timestep:

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-14
SLIDE 14

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Example repartitioning

final mesh rank 0, timestep 1 mesh sync. composite mesh timestep 1 repartitioning refine DD timestep 2 start mesh rank 0, timestep 2

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-15
SLIDE 15

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Example problem

macro triangulation

−ux= f x x∈ ux=gx x∈

f x=−400 x

2−20⋅dime −10 x

2

gx=e

−10 x

2

ux=e

−10 x

2

Poisson equation:

Example problem

Solution: solution for dim = 2

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-16
SLIDE 16

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Partitioning

mesh after 6 initial iterations macro triangulation

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-17
SLIDE 17

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Partitioning

partitioning after 6 initial iterations final rank0 mesh

1 2 3

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-18
SLIDE 18

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Composite mesh and corresonding sequential mesh

after mesh synchronization sequential mesh

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-19
SLIDE 19

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Pointwise error

2.67⋅10

−2

3.69⋅10

−3

after PU after final solve

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion

total error:

slide-20
SLIDE 20

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

L2 error

2.67⋅10

−4

7.01⋅10

−5

after PU after final solve

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion

total error:

slide-21
SLIDE 21

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

H1 error

1.19⋅10

−2

1.15⋅10

−2

after PU after final solve

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion

total error:

slide-22
SLIDE 22

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Speed-up and load balance

DD Adaption loop Mesh sync. Value collect. PU

“sequential part” “parallel part”

#procs Time [sec] Speed-Up 1 358 1 2 235 1.52 4 147 2.44 8 120 2.98

  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion
slide-23
SLIDE 23

AMDiS – Adaptive Multidimensional simulations: Parallel Concepts Simon Vey vey@caesar.de DD17

  • St. Wolfgang/Strobl July 3-7 2006

http://www.caesar.de/amdis.html

crystal growth group crystal growth group

Conclusion

  • Parallel adaptive meshing succesfully implemented in AMDiS
  • Partitioning by error-weighted spectral bisectioning
  • Global solution built by Partition of Unity
  • Time dependent problems manageable
  • Speed-up and load balance not yet optimal

Further work:

  • Performance improvement
  • Goal-oriented error estimates
  • 1. Introduction
  • 2. Partitioning
  • 3. Building global solution
  • 4. Time dependent problems
  • 5. Examples
  • 6. Conclusion