A Parallel Strategy for a Level Set Simulation of A Parallel Strategy - - PowerPoint PPT Presentation

▶
a parallel strategy for a level set simulation of a
SMART_READER_LITE
LIVE PREVIEW

A Parallel Strategy for a Level Set Simulation of A Parallel Strategy - - PowerPoint PPT Presentation

Feb 07, 2023 351 likes •534 views

A Parallel Strategy for a Level Set Simulation of A Parallel Strategy for a Level Set Simulation of Droplets Moving in a Liquid Medium Oliver Fortmeier, H. Martin Bcker VECPAR10 June 24 th VECPAR 10, June 24 Berkeley, CA, USA Motivation

slide-1
SLIDE 1

A Parallel Strategy for a Level Set Simulation of A Parallel Strategy for a Level Set Simulation of Droplets Moving in a Liquid Medium

Oliver Fortmeier, H. Martin Bücker VECPAR’10 June 24th VECPAR 10, June 24 Berkeley, CA, USA

slide-2
SLIDE 2

2

Motivation – Experimental Setup

  • E. Bertakis, S. Groß, J. Grande, O. Fortmeier, A. Reusken, and A. Pfennig. Validated simulation of droplet sedimentation with

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

finite‐element and level‐set methods. Chemical Engineering Science, 65(6):2037‐2051, 2010

slide-3
SLIDE 3

3

Outline

  • Mathematical model for simulating droplets as two‐phase

flows flows P ll l i l ti f d l t

  • Parallel simulation of droplets

– Hierarchy of tetrahedral grids D i i f d i l fi d h d l id – Decomposition of adaptively refined tetrahedral grids – Distributed unknowns

  • Performance results up to 256 MPI processes

– Load‐balancing – Strategies to place MPI processes on nodes

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-4
SLIDE 4

4

Mathematical Model

  • Level set functions describe two phases 1 (e.g., droplet) and

2 (e.g., surrounding water)

2D example for a level‐set function 1D example for a level‐set function

  • Coupling the level set equation and the Navier‐Stokes equations

d ib t h fl describes two‐phase flows

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-5
SLIDE 5

5

Typical Serial Simulation

Solution Solution Solution

Evolving Grid‐

for time ti for time ti+1 for time ti+1

in time mod.

Vertical slice through computational domain A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

Vertical slice through computational domain

slide-6
SLIDE 6

6

Need for Load‐Balancing

Solution Solution Solution

Evolving Grid

Solution

Load

for time ti for time ti+1 for time ti+1

in time mod.

for time ti+1

balancing

Vertical slice through computational domain A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

Vertical slice through computational domain

slide-7
SLIDE 7

7

Distributed Hierarchy

Serial hierarchy of triangulations

s.

Serial hierarchy of triangulations

r esian Grids Lecture ger. gorithm fo ve to Carte 5976 of L

  • 010. Sprin

Parallel Alg

  • m Adaptiv

a, volume , Berlin, 20

Level 0 Level 1

Distributed hierarchy of triangulations

A Hybrid P nctions fro ON, Canada ges 48-61,

  • M. Bücker.

lement Fun Kingston, O cience, pag er and H. M ng Finite El CS 2009, K mputer Sc . Fortmeie ransformin n Proc HPC

  • tes in Com

Level 0 Level 1

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th O Tr In No

slide-8
SLIDE 8

8

Representing the Hierarchy by Graphs

  • Dual, reduced Graph:

– Vertices = Family of tetrahedra T(v) i e all children Vertices = Family of tetrahedra T(v), i.e. all children

  • f a parent tetrahedra

– Edges = Vertices are adjacent if corresponding Edges = Vertices are adjacent if corresponding tetrahedron families share a common face – Vertex weight = Number of children or one if no children exists Vertex weight = Number of children, or one if no children exists – Edge weight = Number of common faces

2 4 1

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-9
SLIDE 9

9

Updating the Hierarchy of Triangulations

efin- ing Re i

refine coarse

Graph 2 1 4 2 1 4

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-10
SLIDE 10

10

Updating the Hierarchy of Triangulations

a eling data d and 2010. efin- ing

+Unknowns

  • ker. Mode

weighted Sytems

  • 38. VDE,

Re i

refine

  • M. Bück

blems by n Parallel ages 31-3

coarse

Graph

+Unknowns

h, and H. flow prob kshop on many, pa 2

  • T. Henrich
  • -phase f

23rd Work

  • ver, Germ

1 4 tmeier, T

  • n for two

n Proc., 2 s, Hanno 2

+Unknowns

2] O. Fort istributio

  • raphs. In

lgorithms 1 4

+Unknowns A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

[ 2 di gr A

slide-11
SLIDE 11

11

Handling of Distributed Unknowns

U k d i d

  • Unknowns are stored on vertices and

edges of the tetrahedra.

  • We distinguish between local and

distributed unknowns.

  • Distributed unknowns are can be either

– Partially assembled, i.e., each process only h ld f h l holds a part of the value – Accumulated, i.e., each process holds the “global” value

P0 P1

global value

  • For linear algebra operations

transforming partially assembled to transforming partially assembled to accumulated unknowns is needed

  • Transformation involves neighbor

P3 P2

  • Transformation involves neighbor

communication which can be

  • verlapped by computations

local unknowns di ib d k

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

  • verlapped by computations

distributed unknowns

slide-12
SLIDE 12

12

Techniques to Solve Two‐Phase Flows

  • Software

– DROPS (http://www.igpm.rwth‐aachen.de/drops)

  • Computational domain

/ – Hierarchy of tetrahedral grids for adaptively refining/coarsening – Domain decomposition for parallelization

  • Numerical techniques:

– Level set method to represent phases and to capture the interface Level set method to represent phases and to capture the interface – Finite elements (quadratic for velocity and level set function, linear and X‐FEM for pressure) p ) – Time integration is based on a linear theta scheme – Continuum surface force (CSF) for describing the surface tension – Parallel Krylov subspace methods (and in serial: iterative solvers based on multigrid methods)

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-13
SLIDE 13

13

Is Updating a Bottleneck?

  • Problem size:

– 875k tetrahedra, 30k k

Speedup:

– 4 130k unknowns – radius 2mm, 5 levels

  • Total time:

– 5 822s on 32 procs. 893 2 – 893s on 256 procs.

  • Updating the triangulation:

– 12.6s (=0.22%) on 32 procs. – 3.7s (=0.41%) on 256 procs.

 Right now updating is  Right now, updating is not a bottleneck

Harpertown processors InfiniBand network Intel MPI library Intel compiler

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

Harpertown processors, InfiniBand network, Intel MPI library, Intel compiler

slide-14
SLIDE 14

14

Placing MPI processes on nodes

Schematic view of two “Harpertown” processors on a node Schematic view of two “Harpertown” processors on a node:

Core0 Core1 Core2 Core3 Core4 Core5 Core6 Core7

ssor0 ssor1

L2 L2 L2 L2

Proces Proces Bus Memory

Two strategies to place MPI processes:

4 process per processor 1 process per processor

compact scatter A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

p

slide-15
SLIDE 15

15

Effect of Placement

1 node 4 nodes 16 nodes 32 nodes

Problem size:

2 MPI 2104 s 8 MPI 533 s 32 MPI 277 s 64 MPI 75 s 8 MPI 32 MPI 128 MPI 256 MPI

– 155k tetrahedra – 725k unknowns – Radius 2mm

8 MPI 1713 s 32 MPI 445 s 128 MPI 155 s 256 MPI 119 s Factor 1.2 1.2 1.8 0.6

– Radius 2mm – 4 level

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-16
SLIDE 16

16

Conclusion and Outlook

  • Conclusion:

– Parallel strategy to simulate a raising drop – Parallel strategy to simulate a raising drop – Updating the triangulation is not a bottleneck – Data movement is expensive for Harpertown processors Data movement is expensive for Harpertown processors – Now, focus on optimizing parallel algorithms

  • Outlook:

Outlook:

– Parallel iterative mulitgrid solvers for the linear equation systems – Advanced techniques to “re‐initialize” the level set function Advanced techniques to re initialize the level set function – Hybrid MPI/OpenMP parallelization

A Parallel Strategy for a Level Set Simulation

  • f Droplets Moving in a Liquid Medium

Oliver Fortmeier

VECPAR‘10, June 24th

slide-17
SLIDE 17

Thank you for your attention!

I am looking forward to I am looking forward to questions and discussions!

http://www.sc.rwth‐aachen.de/fortmeier http://www igpm rwth‐aachen de/drops fortmeier@sc.rwth‐aachen.de http://www.igpm.rwth‐aachen.de/drops