PaMPA : Parallel Mesh Partitioning and Adaptation C edric Lachat - - PowerPoint PPT Presentation

▶
pampa parallel mesh partitioning and adaptation
SMART_READER_LITE
LIVE PREVIEW

PaMPA : Parallel Mesh Partitioning and Adaptation C edric Lachat - - PowerPoint PPT Presentation

Nov 02, 2023 351 likes •772 views

PaMPA : Parallel Mesh Partitioning and Adaptation C edric Lachat & Franc ois Pellegrini June 14, 2012 Introduction Contents Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian

slide-1
SLIDE 1

C´ edric Lachat & Franc ¸ois Pellegrini June 14, 2012

PaMPA : Parallel Mesh Partitioning and Adaptation

slide-2
SLIDE 2

Introduction

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 2

slide-3
SLIDE 3

Introduction

Middleware

◮ PaMPA: Parallel Mesh Partitioning and Adaptation ◮ Middleware library managing the parallel repartitioning and

remeshing of unstructured meshes modeled as interconnected valuated entities

◮ The user can focus on his/her “core business”:

◮ Solver ◮ Sequential remesher ◮ Coupling with MMG3D provided for tetrahedra

Physics, solver

Remeshing and redistribution

PT-Scotch

  • Seq. Qual.

Measurement

Sequential remesher

MMG3D

PaMPA

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 3

slide-4
SLIDE 4

Common needs of solvers regarding meshes

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 4

slide-5
SLIDE 5

Common needs of solvers regarding meshes

Common needs of solvers regarding meshes:

◮ Handling of mesh structures ◮ Distribution of meshes across the processors of a parallel

architecture

◮ Handling of load balance

◮ Data exchange across neighboring entities ◮ Iteration on mesh entities

◮ Entities of any kind: e.g. elements, faces, edges, nodes, . . . ◮ Entity sub-classes: e.g. regular or boundary faces, . . . ◮ Inner or frontier entities with respect to neighboring processors ◮ Maximization of cache effects thanks to proper data reordering

◮ Dynamic modification of mesh structure

◮ Dynamic redistribution

◮ Adaptive remeshing

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 5

slide-6
SLIDE 6

Data structures

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 6

slide-7
SLIDE 7

Data structures

Constraints

◮ Distributed data ◮ Mesh data should not be replicated for the sake of scalability ◮ Minimization of data exchanges ◮ Abstraction from actual data structure implementations

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 7

slide-8
SLIDE 8

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Top-level mesh entity May bear some data (volume, pressure, etc.)

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-9
SLIDE 9

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

May bear some data (geometry, etc.)

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-10
SLIDE 10

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

May bear some data (flux, etc.)

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-11
SLIDE 11

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Regular mesh edge

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-12
SLIDE 12

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Boundary mesh edge

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-13
SLIDE 13

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

What all entities are in fact. . .

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-14
SLIDE 14

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Subset of edges between vertices belonging to prescribed entity types

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-15
SLIDE 15

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Subset of vertices bearing the same data

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-16
SLIDE 16

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Subset of entity vertices that may bear specific data

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-17
SLIDE 17

Data structures

Definitions

◮ Element ◮ Node ◮ Edge

◮ Internal ◮ Boundary

◮ Vertex ◮ Relation ◮ Entity ◮ Sub-entity ◮ Enriched graph

Whole set of vertices and relations Every vertex belongs to one and only

  • ne entity (and sub-entity)

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 8

slide-18
SLIDE 18

Data structures

Global vue

baseval enttglbnbr proccnttab procvrttab 1 3 3 4 3 1 4 8 11

5 1 2 9 7 4 3 6 8 10

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 9

slide-19
SLIDE 19

Data structures

Local vision of process 0

vertgstnbr vertlocnbr edgelocnbr ventloctab edgeloctab vertloctab vendloctab 6 3 7 3 1 1 1 1 2 3 8 1 2 3 4 2 3

4 1 2 3 1 2

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 10

slide-20
SLIDE 20

Data structures

Local vision of process 1

vertgstnbr vertlocnbr edgelocnbr ventloctab edgeloctab vertloctab vendloctab 7 4 3 1 3 2 13 1 2 3 6 1 2 1 3 2 1 2 1 3 1 14 12 3 3

2 3 1 1 2 1 3

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 11

slide-21
SLIDE 21

Data structures

Per-entity data (1/2)

E.g. entity 1 on processor 0:

vertgstnbr vertlocnbr 2 1

4 1 2 3 1 2

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 12

slide-22
SLIDE 22

Data structures

Per-entity data (2/2)

E.g. node neighbors of element 1 on process 0:

vertloctab edgeloctab vendloctab 4 1 1 8 4 3 2 1 2

4 1 2 3 1 2

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 13

slide-23
SLIDE 23

Data structures

Data linking

5 1 2 9 7 4 3 6 8 10

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 14

slide-24
SLIDE 24

Version 0.1

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 15

slide-25
SLIDE 25

Version 0.1

Entities

◮ Based (for C and Fortran interfaces) ◮ Simple ◮ Stable with respect to sub-entities

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 16

slide-26
SLIDE 26

Version 0.1

Partitioning

◮ Performed with respect to top-level entity

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 17

slide-27
SLIDE 27

Version 0.1

Overlap

◮ Of size 1 at the time being ◮ Extensible to any distance n ◮ Requires top-level relations

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 18

slide-28
SLIDE 28

Version 0.1

Local vue of process 0

◮ Halo of size 1 ◮ Overlap of size 1

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 19

slide-29
SLIDE 29

Version 0.1

Local vue of process 1

◮ Halo of size 1 ◮ Overlap of size 1

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 20

slide-30
SLIDE 30

Version 0.1

Iterators

◮ Entities - sub-entities ◮ Decrease of cache misses ◮ Debugging tools

PAMPA dmeshItInitStart ( mesh , ventlocnum , PAMPA VERT ANY, &i t v e r t ) ; PAMPA dmeshItInit ( mesh , ventlocnum , nentlocnum , &i t n g h b ) ; while ( PAMPA itHasMore(& i t v e r t )) { vertlocnum = PAMPA itCurEnttvertlocnum(& i t v e r t ) ; p r i n t f ( ” vertlocnum : %d\n” , vertlocnum ) ; PAMPA itStart(& it nghb , vertlocnum ) ; while ( PAMPA itHasMore(& i t n g h b )) { PAMPA Num sngblocnum , engblocnum , mngblocnum , nsublocnum ; engblocnum = PAMPA itCurEnttvertlocnum(& i t n g h b ) ; mngblocnum = PAMPA itCurMeshvertlocnum(& i t n g h b ) ; sngblocnum = PAMPA itCurSubEnttvertlocnum(& i t n g h b ) ; nsublocnum = PAMPA itCurnsublocnum(& i t n g h b ) ; p r i n t f ( ”[%d ] ngb

  • f %d

with e n t i t y %d : %d ( sub : %d ) %d ( ent : %d ) %d ( glb )\n” , rank , vertlocnum , ventlocnum , sngblocnum , nsublocnum , engblocnum , nentlocnum , mngblocnum ) ; PAMPA itNext(& i t n g h b ) ; } } C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 21

slide-31
SLIDE 31

Example: Laplacian equation using finite element method

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 22

slide-32
SLIDE 32

Example: Laplacian equation using finite element method

All steps

! P a r a l l e l Mesh Computations ! − − − − − − − − − − − − − − − − − − − − − − − − − − − ! P r o c e s s o r

  • nly :

CALL ReadMesh () ! Read mesh on f i l e CALL GraphPampa () ! Build graph

  • f

v e r t e x n e i g h b o r s f o r PAMPA CALL PampaCentralizedMesh () ! Build PaMPA g l o b a l non d i s t r i b u t e d mesh ! On a l l p r o c e s s o r s : CALL PampaDistributedMesh () ! Build PAMPA d i s t r i b u t e d mesh : ! 1− S c a t t e r c e n t r a l i z e d mesh on a l l p r o c e s s o r s ! 2− C a l l PAMPA mesh p a r t i o n n e r ! 3− R e d i s t r i b u t e d i s t r i b u t e mesh CALL ElementVolume () CALL I n i t i a l i z e M a t r i x C S R () ! S o l u t i o n computation ! − − − − − − − − − − − − − − − − − − − − − CALL I n i t S o l () CALL F i l l M a t r i x () CALL SolveSystem () C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 23

slide-33
SLIDE 33

Example: Laplacian equation using finite element method

Solve system: Jacobi (1/2)

UaPrec = 0. ! Suppose A = L + D + U, system to s o l v e : A x = b DO i r e l a x = 1 , Nrelax r e s = 0. DO i s = 1 , MatCSR%Nin r e s 0 = RHS( i s ) ! r e s 0 = b i 1 = MatCSR%I a ( i s ) I 1 F i n = MatCSR%I a ( i s + 1) −1 DO i v = I1 , I 1 F i n j s = MatCSR%Ja ( i v ) r e s 0 = r e s 0 − MatCSR%Vals ( i v ) ∗ UaPrec ( j s ) ! r e s 0 = b − (L + U) xˆn END DO Ua( i s ) = r e s 0 / MatCSR%Diag ( i s ) ! xˆn+1 = ( b − (L + U) xˆn )/D r e s 0 = r e s 0 − MatCSR%Diag ( i s ) ∗ UaPrec ( i s ) ! r e s 0 = ( b − (L + D + U) xˆn ) r e s 0 = r e s 0 ∗ r e s 0 ! r e s 0 = ( b − (L + D + U) xˆn ) r e s = r e s + r e s 0 END DO CALL PAMPAF dmeshHaloValue (dm, & & enttnum = PAMPA ENTITY NODE, & & tagnum = INT(PAMPA TAG SOL, PAMPAF NUM) , & & r e t v a l = s t a t u t ) C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 24

slide-34
SLIDE 34

Example: Laplacian equation using finite element method

Solve system: Jacobi (2/2)

deltaU = 0. deltaU = DOT PRODUCT( UaPrec ( : MatCSR%Nin ) − Ua ( : MatCSR%Nin ) , UaPrec ( : MatCSR%Nin ) − Ua ( : MatCSR%Nin ) ) r e s 0 = 0. r e s 0 = DOT PRODUCT( RHS ( : MatCSR%Nin ) , RHS ( : MatCSR%Nin ) ) UaPrec = Ua d el t a U g l b = 0. CALL MPI ALLREDUCE( deltaU , deltaUglb , 1 , t y p e r e a l , MPI SUM ,MPI COMM WORLD, code ) d el t a U g l b = s q r t ( de l ta U gl b ) r e s g l b = 0. CALL MPI ALLREDUCE( res , r e s g l b , 1 , t y p e r e a l , MPI SUM , MPI COMM WORLD, code ) r e s g l b = s q r t ( r e s g l b ) r e s 0 g l b = 0. CALL MPI ALLREDUCE( res0 , re s 0g l b , 1 , t y p e r e a l , MPI SUM , MPI COMM WORLD, code ) r e s 0 g l b = s q r t ( r e s 0 g l b ) r e s g l b = r e s g l b / r e s 0 g l b END DO ! end loop

  • n

i r e l a x C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 25

slide-35
SLIDE 35

Some results

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 26

slide-36
SLIDE 36

Some results

First results (1/2)

50 100 150 200 250 300 350 400 450 500 550 1 2 3 4 5 6 7 8 9 10 Time (in seconds) Number of processors (x1000) Time for distributing centralized mesh and partitionning distributed mesh Scatter Part

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 27

slide-37
SLIDE 37

Some results

First results (2/2)

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 1 2 3 4 5 6 7 8 9 10 Time (in seconds) Number of processors (x1000) Time for distributing centralized mesh and partitionning distributed mesh Redistribution

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 28

slide-38
SLIDE 38

Upcoming features

Contents

Introduction Common needs of solvers regarding meshes Data structures Version 0.1 Example: Laplacian equation using finite element method Some results Upcoming features

C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 29

slide-39
SLIDE 39

Upcoming features

New features

◮ Main idea:

◮ Parallel mesh adaptation

◮ Others:

◮ Parallel I/O with HDF5 ◮ Overlap greater than 1 ◮ Unbreakable relations C´ edric Lachat & Franc ¸ois Pellegrini - PaMPA June 14, 2012- 30

slide-40
SLIDE 40

Upcoming features

Merci

Fran ¸

  • is Pellegrini

June 14, 2012