Finite Element Multigrid Solvers for PDE Problems on GPUs and GPU - - PowerPoint PPT Presentation

Finite Element Multigrid Solvers for PDE Problems

• n GPUs and GPU Clusters

Robert Strzodka Integrative Scientific Computing Max Planck Institut Informatik www.mpi$inf.mpg.de/ ~strzodka Dominik Göddeke Institute for Applied Mathematics Technical University of Dortmund www.mathematik.tu$dortmund.de/ ~goeddeke

• Structure of Double Lecture 2 x 90 min

PART 1 Parallelism Grid Discretization Multigrid & Smoothers Mixed$Precision Data Layout PART 2 FEM on GPU Clusters New MPI Application (Rewrite) Legacy MPI Code (Accelerate)

• Overview

Levels of Parallelism Grid Discretizations of PDEs Multigrid and Strong Smoothers Mixed Precision Iterative Refinement Layout of Multi$valued Data

• Parallelism

Sequential execution is an illusionary software concept

All transistors always do something in parallel !

Billions of transistors in modern CPUs(>0.5) & GPUs(>2) Old: Implicit parallelism with caches, ILP, speculation

• diminishing returns, power constraints

New: Explicit parallelism on multiple levels

• much more efficient & natural

• SIMD Parallelism

It is impossible to execute just one instruction

(add, nop, nop, nop, >)

Penalty for ignoring SIMD

!"#$%&'()* "#! +, instructions

"- . #

• /

" + "- . #

• /

" +

• Load/store

Global Memory

Execution Manager

Input Assembler Host Cache

Cache Cache Cache Cache Cache Cache Cache Cache Local Memory Local Memory Local Memory Local Memory Local Memory Local Memory Local Memory Local Memory

Load/store Load/store Load/store Load/store Load/store

Many$Core Parallelism

Penalty for ignoring many$cores

! /0! $()* "+!/+,

"- . #

• /

" + "- . #

• /

" +

• Intra$Node Parallelism (multiple CPUs/GPUs per PC)

Penalty for ignoring intra$node parallelism

! $! ! $!,

" " /

• ,"

,0 ,/ , !

• "

Inter$Node Parallelism in a Cluster

• Penalty for ignoring inter$node parallelism

123$23 114

• Bandwidth in a CPU$GPU System

#

• 5!"0
• (

6 !31

• +,*7
• (

6 !31

• +,*7
• (

6 !31

• +,*7
• (

6 !31

• 40 GB/s

20 GB/s

343!2,5 28

• 200 GB/s

2 TB/s

( 6 ( 6 ( 6 ( 6 !31

• !31
• !31
• !31
• 4 GB/s

2 GB/s

• Overview

Levels of Parallelism Grid Discretizations of PDEs Multigrid and Strong Smoothers Mixed Precision Iterative Refinement Layout of Multi$valued Data

• ℜ

⊆

• ℜ

→

• Generalized Poisson Problem
• =
• ∂
• 2
• =

∂ν

• 923"020:311;

,88$28"'' 80':2$<

• =
• −
• =

∇ −

• Discretization Grids

=242

14<1 4<1 <direct

,>2!1242

14<1 4<1 <direct

• Discretization Grids

242

14<explicit 4<1 <2

21842

14<explicit 4<171 <33'14

• nD Arrays

,>2!1242

14<1 4<171 <direct

• simple 1

1148precious 42:23 :$$!311 42?@>

• 145A242$$

$=1:$8

• Deformation Adaptivity

This grid is a tensor$product ! Easier to accelerate in hardware than resolution adaptive grids Anisotropy level determines

• ptimal solver

• nD Arrays

2187242

14<explicit 4<171 <2

• 43$24

2$local 2 82418some data locality

• 1824$1478

48:$3 2$$322341

• Hash

2187242

14<explicit 4<171 <2

• 43$24

2$arbitrary global 2 perfect hashes 38'342?@

• 1824$1478

33212$!4222 1$33418$234

• Tree

21842

14<explicit 4<171 <2

• :$$213

124$global 1478 :dynamic changes 1

• 822

2$$?@1

• #

Structured and Unstructured Sparse MatVec

3$B*2,20++CD

• Overview

Levels of Parallelism Grid Discretizations of PDEs Multigrid and Strong Smoothers Mixed Precision Iterative Refinement Layout of Multi$valued Data

• ℜ

⊆

• ℜ

→

• Generalized Poisson Problem
• =
• ∂
• 2
• =

∂ν

• 923"020:311;

,88$28"'' 80':2$<

• =
• −
• =

∇ −

• Discretization Approach

6$$ 6% 6

• =

606@A C!2;

• Geometric Multigrid Method

(= $2> E8<*8=523343$= 1'44:3:$= ?2<8312$42'324 :32$$$=$3

• =

d5 b$Ax5 Fine 42 Ac5 d5 Coarse 42 x5" x5+c5 *5fine 42

• Multigrid Transfers

Restriction

?18$$ (:432431

fine coarse adjust index to read neighbors

• utput region

coarse array i

2i 2i+1 2i$1

result

• $

Multigrid Transfers

Prolongation

8$$:3:434 (:4321

• Preconditioners
• =
• −

+ =

− +

ω

12'122 2$<

• =

FE*?

• +

+ + + =

,)E9

• +

+ =

G)??

• +

+ + + + =

G)?,)E9

• +

+ + + + + + + =

• CPU Numerical and Runtime Efficiency

+'/HHHHHH ?;)$;2@,$%0'0,

• +

+ + + =

,)E9

Gauss$Seidel Preconditioner

• =
• −

+ =

− +

ω

12'122 2$<

• #

Multi$Colored Gauss$Seidel

• G)??!E(@I

ADI$TRIDI Preconditioner

• =
• −

+ =

− +

ω

12'122 2$< G)??!)E9

• +

+ =

• SIMD Parallelism: Cyclic Reduction

/J /J!" /J0!" /J0!" /J!" /JC!0 EI *2 *4 *8 *8 *4 *2 O(N*logN)

• Memory Friendly Cyclic Reduction

B,K225 ;G0+""D

• G)?,!E(@I

ADI$TRIGS Preconditioner

• =
• −

+ =

− +

ω

12'122 2$<

• +

+ + + + =

G)?,!)E9

• Many$Core Parallelism

!:14 0!:14 $714

• $

CPU vs. GPU Numerical Efficiency

?;)$;2@,$%0'0,

• CPU vs. GPU Runtime Efficiency

0#.-+ ,G&0 + 9!.&C + G0+.+

• Multigrid on Refined Unstructured Grid

FE$gMG

242 )4$ )L4 @% ?12

Order & Storage (ELL)

• FE$gMG Results with SPAI Preconditioner

"

• #

#

Overview

Levels of Parallelism Grid Discretizations of PDEs Multigrid and Strong Smoothers Mixed Precision Iterative Refinement Layout of Multi$valued Data

• Precision Comparison

Numerical Algorithms GPU Hardware Precision %& '&$ & (" Comparison )'*' *+'*+' +% *+'*+' ,++- ''+''+ ,++. + +

• Hardware Precision

float s23e8 double s52e11 Data Error

52 bit 23 bit

"70$ +;- "7/$ +;//////// "702 +;- "7/2 +;/////////////// Roundoff Error ";+++0J+;CCC $ " "! $ " $'$ $$' ";+++0J+;CCC 2 +;CCCCCCC# "!"- 2 " $'2 $2'

• The Erratic Roundoff Error

 

!"++ !C+ ! + !.+ !#+ !-+ !+ !/+ !0+ + "+ 0+ /+ +

• +

40$'+!!M0N!"++ 40"7'"70N )2$$$<+$<O"N/!"/N0!/N/O 41 21

• Numerical Accuracy

Condition of Ax = b Discretization Error

ε ε ε

• =

− − = −

• δ

δ δ

• ⋅

= − − = −

• float s23e8 double s52e11

52 bit 23 bit

• =

∇ −

• =
• −

=

• ↓

− =

• ↑
• ⋅

• Mixed Precision Iterative Refinement

float s23e8 double s52e11

52 bit 23 bit 23 bit 52 bit 23 bit 23 bit

Iterative Refinement for Ax = b

d5 b$Ax5 Compute high 131 Ac5 d5 Solve low 1$ x5" x5+c5 Correct high 131 55" ?843431

Condition of Ax = b

ε ε ε

• =

− − = −

• ε
• ⋅

= − =

+ + +

• ⋅

• Mixed Precision Multigrid on GPU

 

, +! ,G0++! ,G0++!2 ,G0++!,

• Overview

Levels of Parallelism Grid Discretizations of PDEs Multigrid and Strong Smoothers Mixed Precision Iterative Refinement Layout of Multi$valued Data

• Multi$Valued Data

Multi$valued data is ubiquitous

"<311';4; 1288 0<2$';4; $1'1 /<1!211';4; 2'8/42

• AoS and SoA
• !

"!#$$ ! ! !

• #"!$$

$ $

• Parallel Access in AoS and SoA

$ $

• {}

{} {} {}

• Operating on Container Elements
• !
• %
• In$place update of single and indexed structs

&'()#$*#

• '%)++,-

'"!)++,-

• This is not possible with standard SoA/C++ syntax:

#'")

• Abstraction: AoS + SoA = ASA

Array of Structs (AoS)

• !

% & '() .!#$$ '"!)

Array of Structs of Arrays (ASA)

/01*01&2 3.4 5/61271 71 71 71

• 788/3.46

69992 3.4/2 % /012& '3.4/12() .!#$$ '"!) 999*788,788,

B>25; !!D

• $

$

Overview

Levels of Parallelism

11$8<?@''5'

Grid Discretizations of PDEs

)4$

Multigrid and Strong Smoothers

*4232:=

Mixed Precision Iterative Refinement

:3$1

Layout of Multi$valued Data

3$21

Questions?

Robert Strzodka Integrative Scientific Computing Max Planck Institut Informatik www.mpi$inf.mpg.de/ ~strzodka Dominik Göddeke Institute for Applied Mathematics Technical University of Dortmund www.mathematik.tu$dortmund.de/ ~goeddeke