Case Study: Gauss-Seidel on Multicores Erik Hagersten Uppsala - - PowerPoint PPT Presentation

▶
case study gauss seidel on multicores
SMART_READER_LITE
LIVE PREVIEW

Case Study: Gauss-Seidel on Multicores Erik Hagersten Uppsala - - PowerPoint PPT Presentation

Dec 15, 2023 37 likes •321 views

Case Study: Gauss-Seidel on Multicores Erik Hagersten Uppsala University Sweden Thanks: Dan Wallin(arch), Henrik Lf (sci comp) and Sverker Holmgren (sci comp) From Wallin et al, ICS 2006 Criteria for HPC Algorithms Past: Minimize

slide-1
SLIDE 1

Case Study: Gauss-Seidel on Multicores

Erik Hagersten Uppsala University Sweden

Thanks: Dan Wallin(arch), Henrik Löf (sci comp) and Sverker Holmgren (sci comp) From Wallin et al, ICS 2006

slide-2
SLIDE 2

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Criteria for HPC Algorithms

Past:

Minimize communication Maximize scalability (1000s of CPUs)

Multicores today:

Communication is “for free”

[on some multicores]

Scalability is limited to 32 threads The caches are tiny Memory bandwidth is scarse

Data locality is key!

(Both for Capacity and Capability Computing!)

slide-3
SLIDE 3

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Natural Order Gauss-Seidel

= previous = current = sweep path = data dependence

1 1 1 1,2,3,4 = iteration number

= cacheline layout

1 1

slide-4
SLIDE 4

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

1 1 1 1 1 1 1 1 1

IF (convergence_test) <done> else <iterate again>

Natural Order Gauss-Seidel

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-5
SLIDE 5

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

2 2 2 2 2

Data dependence Poor Parallelism

Natural Order Gauss-Seidel

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-6
SLIDE 6

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Red-Black Gauss-Seidel

0,5 0,5 0,5 0,5 0,5

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-7
SLIDE 7

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Red-Black Gauss-Seidel step 0,5: update the blacks

0,5 0,5 0,5 0,5 0,5

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-8
SLIDE 8

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

1

Red-Black Gauss-Seidel step 1,0 update all reds

1 1 1 1 1 1

Update all blacks <barrier> Update all reds <barrier>

great parallelism!!!

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-9
SLIDE 9

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

1

Red-Black Gauss-Seidel Parallel version

1 1 1 1 1 1

IN PARALLELL { Update all blacks <barrier> Update all reds <barrier> }

thread 0 thread 1 thread 2 thread 3 thread 4

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-10
SLIDE 10

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Any Drawbacks of the Red-Black?

Poor Cache Locality of Red-Black:

Each element will be brought into the cache

twice per iteration

Natural Order:

Each element will be brought into the cache

  • nce per iteration

You can do even better…

Natural Order with Temporal Blocking ☺

slide-11
SLIDE 11

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

G-S, temporal blocking:

several iterations per sweep

2 2 4 4 1 3

= active region

3 4

= previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

4

slide-12
SLIDE 12

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

G-S, temporal blocking:

several iterations per sweep

3 2 4 4 1 4 1 2

In this case: 4 iterations per “sweep”. (σ = 4) σ = 1,0 for natural order G-S σ = 0,5 for red-black G-S

= active region = previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout

slide-13
SLIDE 13

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

G-S 3D, σ=2

slide-14
SLIDE 14

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

G-S 3D, σ=2

slide-15
SLIDE 15

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Acumem Graph, 3D N=129

2 4 6 8 10 12 14

512 KB 1 MB 2 MB 4 MB 8 MB 16 MB 32 MB

Cache size Cache miss ratio (percent)

σ = 0,5 σ = 1 σ = 2 σ = 4 σ = 8 σ = 16

Miss ratio ~ Memory bandwidth

slide-16
SLIDE 16

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Parallel G-S, temporal blocked

3 2 4 4 1 4

thread 0 thread 1 thread 2 thread 3

1 2 3 Synchronization flags 1 = active region = previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout 1 = sync flag iteration no

slide-17
SLIDE 17

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Parallel G-S, temporal blocked

3 2 4 4 1 4

thread 0 thread 1 thread 2 thread 3

1 2 3 1 1 Synchronization flags

Wait until ”lefty” is done: Lots of communication

  • Producer/Consumer flag
  • Sharing of data values

2 = active region = previous = current = sweep path = data dependence

1,2,3,4 = iteration number

= cacheline layout 1 = sync flag iteration no

slide-18
SLIDE 18

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

t0 t1 t2 t3 Parallel G-S 3D

slide-19
SLIDE 19

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

t0 t1 t2 t3 Parallel G-S 3D

Flags

slide-20
SLIDE 20

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

t0 t1 t2 t3 Parallel G-S 3D

1

slide-21
SLIDE 21

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

t0 t1 t2 t3 Parallel G-S 3D

1

slide-22
SLIDE 22

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

t0 t1 t2 t3 Parallel G-S 3D

Stratup cost = (#threads-1)/(Nσ)

1 1 1 cacheline layout (size B bytes)

Communication:

  • one flag synchronization per N2/#threads ops
  • one communication miss per B*N/#threads bytes

2

slide-23
SLIDE 23

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

0,5 1 1,5 2 2,5 0,5 1 2 4 8 16 RBGS TBGS Execution time per step (sec)

threads=1 threads=2 threads=4 threads=8 Parallel Executiontime σ=

RedBlack

Temp.Blocked GS

Natural Order

slide-24
SLIDE 24

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Performance comparison with Red-Black σ = 0,5 N = 257, 32 threads

(Sun E15 K, US IIIcu = SMP!!)

1,0 1,5 2,0 2,5 3,0 3,5 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Threads

Performance ratio TBGS/RBGS

σ = 1 σ = 2 σ = 4 σ = 8 σ = 16

slide-25
SLIDE 25

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

0,0 0,5 1,0 1,5 2,0 2,5 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Threads

Performance ratio TBGS/RBGS

Sun 15k Simulated Multicore

Multicore Simulation σ = 1

slide-26
SLIDE 26

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

Using Gauss-Seidel Smoother in a Multigrid

G-S Important part of many real apps! EX: G-S as a Smoother in “Multigrid”

Iterative algorithm More efficient smother cuts #iterations

slide-27
SLIDE 27

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

One slide summary

Today’s algorithms assume expensive

communication

The communication cost of [some] multicores is

close to zero

Locality is becoming key to performance [again] Redesign HPC algorithms to face this fact!

(For both Capacity and Capability computing)

We show: * 3x performance gain * ~30x less bandwidth Is it time to revisit more algorithms?

slide-28
SLIDE 28

Uppsala University

Institutionen för informationsteknologi | www.it.uu.se

…8 ...

Niagara

CPU L1I L1D (wt) L2

Memory ctrl

CPU L1I L1D (wt) Xbar = 134 GB/s … 8 … L2 L2 L2

Memory ctrl Memory ctrl Memory ctrl

4 x DDR-2 = 25GB/s (!) Shared L2! 3MB Shared caches: Good or bad?