libCEED Finite Element Library Development Update and Examples - - PowerPoint PPT Presentation

▶

Mar 07, 2024 24 likes •278 views

libCEED Finite Element Library Development Update and Examples Jeremy L Thompson Valeria Barra, Jed Brown University of Colorado Boulder jeremy.thompson@colorado.edu Sept 25, 2019 Jeremy L Thompson (CU Boulder) libCEED Finite Element Library

SLIDE 1

libCEED Finite Element Library Development Update and Examples

Jeremy L Thompson Valeria Barra, Jed Brown

University of Colorado Boulder jeremy.thompson@colorado.edu

Sept 25, 2019

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 1

SLIDE 2

libCEED Team

Developers: Jed Brown1, Jeremy Thompson1 Thilina Rathnayake2, Jean-Sylvain Camier3, Tzanio Kolev3, Veselin Dobrev3, Valeria Barra1, Yohann Doudouit3, David Medina4, Tim Warburton5, & Oana Marin6 Grant: Exascale Computing Project (17-SC-20-SC)

1: University of Colorado, Boulder 2: University of Illinois, Urbana-Champaign 3: Lawrence Livermore National Laboratory 4: OCCA 5: Virginia Polytechnic Institute and State University 6: Argonne National Laboratory

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 2

SLIDE 3

Overview

libCEED is an extensible library that provides a portable algebraic interface and optimized implementations of high-order operators We have optimized implementations for CPU and GPU We have new performance optimizations, development in our example suite, and research in preconditioning strategies

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 3

SLIDE 4

Overview

1

Introduction

2

libCEED

3

Example Suite

4

Current Efforts

5

Future Work

6

Questions

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 4

SLIDE 5

Introduction

Center for Efficient Exascale Discretizations

DoE exascale co-design center Design discretization algorithms for exascale hardware that deliver significant performance gain over low order methods Collaborate with hardware vendors and software projects for exascale hardware and software stack Provide efficient and user-friendly unstructured PDE discretization component for exascale software ecosystem

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 5

SLIDE 6

Introduction

Tensor Product Elements

Using an assembled matrix forgoes performance optimizations for hexahedral elements

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 6

SLIDE 7

libCEED

libCEED Design

libCEED design approach: Avoid global matrix assembly Optimize basis operations for all architectures Single source user quadrature point functions Easy to parallelize across hetrogeneous nodes

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 7

SLIDE 8

libCEED

libCEED Backends

CPU GPU Pure C MAGMA AVX LIBXSMM Pure CUDA OCCA libCEED CPU GPU MFEM Nek5000 PETSc ...

libCEED provides multiple backend implementations

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 8

SLIDE 9

libCEED

libCEED Operator Decomposition

AL = G TBTDBG G - CeedElemRestriction, local gather/scatter B - CeedBasis, provides basis operations such as interp and grad D - CeedQFunction, representation of PDE at quadrature points AL - CeedOperator, aggregation of Ceed objects for local action of operator

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 9

SLIDE 10

libCEED

Laplacian Example

Solving the 2D Poisson problem: −∆u = f Weak Form:

∇v∇u =
vf

General libCEED Operator AL = G TBTDBG Laplacian Operator AL = G TBT

Grad2DDBGrad2DG

where D is block diagonal by quadrature point: Di = (wi det Jgeo) J−1

geoJ−T geo and Jgeo =

∂x

∂r ∂x ∂s ∂y ∂r ∂y ∂s

x, y physical coords; r, s reference coords

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 10

SLIDE 11

libCEED

Basis Optimization

Solving the 2D Poisson problem: −∆u = f Weak Form:

∇v∇u =
vf

General libCEED Operator AL = G TBTDBG Laplacian Operator AL = G TBT

Grad2DDBGrad2DG

Computationally Efficient Form AL = G T BT

G ⊗ BT I

BT

I ⊗ BT G

BG ⊗ BI BI ⊗ BG

BI - 1D Interpolation BG - 1D Gradient

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 11

SLIDE 12

libCEED

Basis Optimization

Solving the 2D Poisson problem: −∆u = f Weak Form:

∇v∇u =
vf

General libCEED Operator AL = G TBTDBG Laplacian Operator AL = G TBT

Grad2DDBGrad2DG

Computationally Efficient Form AL = G T BT

I ⊗ BT I

ˆ BT

G ⊗ I2

I2 ⊗ ˆ BT

G

ˆ BG ⊗ I2 I2 ⊗ ˆ BG

(BI ⊗ BI)G

where ˆ BG = BGBI

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 12

SLIDE 13

libCEED

Operator Definition

General libCEED Operator: vL = ALuL AL = GTBTDBG Laplacian Operator Code:

CeedOperatorCreate (ceed , qf_apply , NULL , NULL , &op_apply); CeedOperatorSetField (op_apply , "du", e r e s t r i c t u , CEED_TRANSPOSE , basisu , CEED_VECTOR_ACTIVE ); CeedOperatorSetField (op_apply , "geo",erestrictqdi ,CEED_NOTRANSPOSE , CEED_BASIS_COLLOCATED , geo); CeedOperatorSetField (op_apply , "dv", e r e s t r i c t u , CEED_TRANSPOSE , basisu , CEED_VECTOR_ACTIVE ); ... CeedOperatorApply (op_apply , uloc , vloc , CEED_REQUEST_IMMEDIATE );

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 13

SLIDE 14

libCEED

QFunction Definition

General libCEED QFunction: vq = Duq 2D Laplacian QFunction: dv0 dv1

D00 D01 D01 D11 du0 du1

2D Laplacian QFunction Code:

CeedQFunctionCreateInterior (ceed , 1, Poisson2D , Poisson2D_loc , &qf_apply); CeedQFunctionAddInput (qf_apply , "du", 2, CEED_EVAL_GRAD ); CeedQFunctionAddInput (qf_apply , "geo", 3, CEED_EVAL_NONE ); CeedQFunctionAddOutput (qf_apply , "dv", 2, CEED_EVAL_GRAD );

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 14

SLIDE 15

libCEED

QFunction Definition

Single Source QFunctions for all backends: C/C++ code, compiled with main for CPU, JiT for GPU

int Poisson2D(void ctx , const CeedInt Q, const CeedScalar const in , CeedScalar const out) { // Inputs and Outputs const CeedScalar du = in [0]; CeedScalar geo = out [0], dv = out [1]; // Quadrature Point Loop CeedPragmaSIMD // For CPU vectorization for (CeedInt i=0; i<Q; i++) { dv[i+Q0] = geo[i+Q0]du[i+Q0] + geo[i+Q2]du[i+Q1]; dv[i+Q1] = geo[i+Q2]du[i+Q0] + geo[i+Q1]du[i+Q1]; } // End of Quadrature Point Loop return 0; }

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 15

SLIDE 16

libCEED

libCEED Performance

Benchmark performance across multiple implementations Benchmark Problem 1/2: Mu = f L2 projection problem Benchmark Problem 3/4: Ku = f Poisson problem 3D scalar problem (BP 1/3) or 3D vector problem (BP 2/4) Unpreconditioned CG, maximum of 20 iterations

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 16

SLIDE 17

libCEED

GPU Performance

Substantial performance increase with Single Source QF + JiT +/- 10% performance of tuned kernels in libParanumal

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 17

SLIDE 18

libCEED

CPU Performance

101 102 103 104 105 106 Points per compute node 1 2 3 4 5 [DOFs x CG iterations] / [compute nodes x seconds] 1e8

4 nodes × 24 ranks, /cpu/self/xsmm/serial, PETSc BP3

p=1 p=2 p=3 p=4 p=5 p=6 p=7 p=8 p=9 p=10 p=11 p=12 101 102 103 104 105 106 Points per compute node 1 2 3 4 5 [DOFs x CG iterations] / [compute nodes x seconds] 1e8

4 nodes × 24 ranks, /cpu/self/xsmm/blocked, PETSc BP3

p=1 p=2 p=3 p=4 p=5 p=6 p=7 p=8 p=9 p=10 p=11 p=12

RMACC Summit, 4 x Intel Xeon E5-2680 v3

External vectorization important at lower order Order we see performance ’switch’ problem dependent

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 18

SLIDE 19

Example Suite

Navier-Stokes Example

State Variables:

ρ - Mass density U - Momentum density E - Total Energy density

3D Compressible Navier-Stokes:

∂ρ ∂t + div (U) = 0 ∂U ∂t + div (ρ (u × u) + PI3) + ρg ˆ

k = div (Fu)

∂E ∂t + div ((E + P) u) = div (Fe)

Viscous and Thermal Stresses: Fu = µ

∇u + (∇u)T + λdiv (u) I3
Fe = uFu + k∇T

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 19

SLIDE 20

Example Suite

QFunction Assembly

User QFunction:

--- Fuvisc

const CeedInt Fuviscidx [3][3] = {{0, 1, 2}, {1, 3, 4}, {2, 4, 5}}; for (CeedInt j=0; j <3; j++) for (CeedInt k=0; k <3; k++) dv[k][j+1][i] -= wJ*(Fu[Fuviscidx[j][0]]* dXdxdXdxT[k][0] + Fu[Fuviscidx[j][1]]* dXdxdXdxT[k][1] + Fu[Fuviscidx[j][2]]* dXdxdXdxT[k][2]);

Assembly:

dv[k][j+1][i] -= wJ*(Fu[Fuviscidx[j][0]]* dXdxdXdxT[k][0] + b08d: c5 7d 28 d0 vmovapd %ymm0 ,% ymm10 Fu[Fuviscidx[j][1]]* dXdxdXdxT[k][1] + b091: c4 42 c5 b8 d3 vfmadd231pd %ymm11 ,%ymm7 ,% ymm10 b096: c5 fd 28 84 24 c8 04 vmovapd 0x4c8 (% rsp) ,%ymm0 b09d: 00 00 dv[k][j+1][i] -= wJ*(Fu[Fuviscidx[j][0]]* dXdxdXdxT[k][0] + b09f: c4 62 f5 ac 14 07 vfnmadd213pd (%rdi ,%rax ,1) ,%ymm1 ,% ymm10 b0a5: c5 7d 11 14 07 vmovupd %ymm10 ,(%rdi ,%rax ,1) Fu[Fuviscidx[j][1]]* dXdxdXdxT[k][1] + b0aa: c5 7d 59 94 24 68 04 vmulpd 0x468 (% rsp) ,%ymm0 ,% ymm10 b0b1: 00 00 ... Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 20

SLIDE 21

Current Efforts

Example Suite

Ongoing development in example suite PHASTA investigating porting to libCEED

SUPG stabilization Primitive variable formulation Implicit time integrator

Initial development of shallow water equations example

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 21

SLIDE 22

Current Efforts

Preconditioning

Iterative solvers require preconditioning Especially with high-order finite element operators Operator Diagonal

Diagonally dominant operators

P-Multigrid

Elliptic operators

BDDC with FDM

In development

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 22

SLIDE 23

Future Work

Further performance enhancements (GPU and CPU) Improved mixed mesh and operator composition support Expanded non-linear and multi-physics examples Preconditioning based on libCEED operator decomposition Algorithmic differentiation of user quadrature functions We invite contributors and friendly users

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 23

SLIDE 24

Questions

Questions?

Advisors : Jed Brown1 & Daniel Appel¨

Collaborators: Valeria Barra1, Oana Marin2, Tzanio Kolev3, Jean-Sylvain Camier3, Veselin Dobrev3, Yohann Doudouit3, Tim Warburton4, David Medina5, & Thilina Rathnayake6 Grant: Exascale Computing Project (17-SC-20-SC)

1: University of Colorado, Boulder 2: Argonne National Laboratory 3: Lawrence Livermore National Laboratory 4: Virginia Polytechnic Institute and State University 5: OCCA 6: University of Illinois, Urbana-Champaign

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 24

SLIDE 25

Questions

libCEED Finite Element Library Development Update and Examples

Jeremy L Thompson Valeria Barra, Jed Brown

University of Colorado Boulder jeremy.thompson@colorado.edu

Sept 25, 2019

Jeremy L Thompson (CU Boulder) libCEED Finite Element Library Sept 25, 2019 24