B3CC: Concurrency 08: Parallelism from Concurrency Trevor L. - - PowerPoint PPT Presentation

B3CC: Concurrency 08: Parallelism from Concurrency

Trevor L. McDonell Utrecht University, B2 2020-2021

Recap

• Concurrency: dealing with lots of things at once
• Collection of independently executing processes
• Two or more threads are making progress
• Parallelism: doing lots of things at once
• Simultaneous execution of (possibly related) computations
• At least two threads are executing simultaneously

2

Recap

• So far we have discussed concurrency as a means to write modular code

with multiple interactions

• Example: network server that interacts with multiple clients simultaneously
• Sometimes this can speed up the program by overlapping the I/O or time

spent waiting for clients to respond, but this speedup doesn’t require multiple processors to achieve

• In many cases we can use the same method to achieve real parallelism
• From now, we will talk about some of the considerations for doing this well

3

Motivations

4

The free lunch is over

• “The free lunch is over” (2005)
• Today virtually all processors include multiple cores/processing elements
• This has become the primary method for increasing performance
• This has consequences for the programmer

5

http://www.gotw.ca/publications/concurrency-ddj.htm

Why?

6

https://github.com/karlrupp/microprocessor-trend-data

epyc itanium 2 pentium 4 pentium 386

100 101 102 103 104 105 106 107 1970 1980 1990 2000 2010 2020 Transistors (thousands)

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2019 by K. Rupp

Year 48 Years of Microprocessor Trend Data

Why?

• Moore's curve (1965)
• Observation that the number of transistors in an integrated circuit doubles

roughly every two years

• In particular, to minimise the cost per transistor
• Not a law in any sense of the word (don't call it that)

7

Why?

• Dennard scaling
• As transistors get smaller, power density remains constant
• Combined with shrinking transistors, this implies performance per watt

grows at roughly the same rate as transistor density

• signal delay decreases (clock frequency increases)
• voltage and current decrease (power density remains constant)

8

Why?

9

https://github.com/karlrupp/microprocessor-trend-data

100 101 102 103 104 105 106 107 1970 1980 1990 2000 2010 2020 Frequency (MHz) Transistors (thousands) Typical Power (Watts)

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2019 by K. Rupp

Year 48 Years of Microprocessor Trend Data

epyc itanium 2 pentium 4 pentium 386

Why?

• Since ~2005 Dennard scaling breaks down
• Static power losses increased faster than the overall power supply dropped

(due to decreasing voltage & current)

• Consequence: can no longer improve performance through frequency

scaling alone

10

Why?

11

https://github.com/karlrupp/microprocessor-trend-data

100 101 102 103 104 105 106 107 1970 1980 1990 2000 2010 2020 Frequency (MHz) Single-Thread Performance (SpecINT x 103) Transistors (thousands) Typical Power (Watts)

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2019 by K. Rupp

Year 48 Years of Microprocessor Trend Data

epyc itanium 2 pentium 4 pentium 386

Why?

• Traditional approaches to increasing CPU performance:
• Frequency scaling
• Caches
• Micro-architectural improvements
• Out of order execution (increase utilisation of execution hardware)
• Branch prediction (guess the outcome of control flow)
• Speculative execution (do work before knowing if it will be needed)

12

Why?

• Frequency scaling: The Power Wall
• Power consumption of

transistors does not decrease as fast as density increases

• Performance limited by power

consumption (& dissipation)

13

Time Transistor density Transistor power Total power

Why?

• Caches: The Memory Wall
• Memory speed does not increase

as fast as computing speed

• Increasingly difficult to hide

memory latency

14

Performance Time Gap Compute Memory

Why?

• Microarchitecture improvements:

Instruction Level Parallelism Wall

• Law of diminishing returns
• Pollack rule: performance ∝

complexity2

15

Cost Serial performance

Why?

16

https://github.com/karlrupp/microprocessor-trend-data

100 101 102 103 104 105 106 107 1970 1980 1990 2000 2010 2020 Number of Logical Cores Frequency (MHz) Single-Thread Performance (SpecINT x 103) Transistors (thousands) Typical Power (Watts)

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2019 by K. Rupp

Year 48 Years of Microprocessor Trend Data

epyc itanium 2 pentium 4 pentium 386

Why?

17

https://github.com/karlrupp/microprocessor-trend-data

100 101 102 103 104 105 106 107 1970 1980 1990 2000 2010 2020 Number of Logical Cores Frequency (MHz) Single-Thread Performance (SpecINT x 103) Transistors (thousands) Typical Power (Watts)

Original data up to the year 2010 collected and plotted by M. Horowitz, F. Labonte, O. Shacham, K. Olukotun, L. Hammond, and C. Batten New plot and data collected for 2010-2019 by K. Rupp

Year 48 Years of Microprocessor Trend Data

epyc itanium 2 pentium 4 pentium 386

Aside: more cores ≠ more performance

18

https://arstechnica.com/gadgets/2020/11/a-history-of-intel-vs-amd-desktop-performance-with-cpu-charts-galore/

Considerations

19

Parallelism

• Improving application performance through parallelisation means:
• Reducing the total time to compute a single result (latency)
• Increasing the rate at which a series of results are computed (throughput)
• Reducing the power consumption of a computation

20

Problem

• To make the program run faster, we need to gain more from parallelisation

than we lose due to the overhead of adding it

• Granularity: If the tasks are too small, the overhead of managing the tasks
• utweighs any benefit you might get from running them in parallel
• Data dependencies: When one task depends on another, they must be

performed sequentially

21

Speedup

• The performance improvement, or speedup of a parallel application, is:
• Where TP is the time to execute using P threads/processors
• The efficiency of the program is:
• Here, T1 can be:
• The parallel algorithm executed on one thread: relative speedup
• An equivalent serial algorithm: absolute speedup

22

speedup = Sp = T1 TP

efficiency = Sp P = T1 PTP

Maximum speedup

• Several factors appear as overhead in parallel computations and limit the

speedup of the program

• Periods when not all processors are performing useful work
• Extra computations in the parallel version not appearing in the sequential

version (example: recompute constants locally)

• Communication time between processes

23

Amdhal

• Amdhal's law considers the execution time of a program is either:
• Wser: time spent doing (non-parallelisable) serial work
• Wpar: time spent doing parallel work
• If f is the fraction of serial work to be performed, we get the parallel speedup:

24

TP ≥ Wser + Wpar P

SP ≤ 1 f + (1 − f)/P

Amdhal

• The speedup bound is determined by the degree of sequential execution in

the program, not the number of processors

• Strong scaling (fixed sized speedup):

25

Time Serial work Parallelizable work P = 1 P = 2 P = 4 P = 8

limP →∞ SP ≤ 1/f

Amdhal

26

Gustafson-Baris

• Often the problem size can increase as the number of processes increases
• The proportion of the serial part decreases
• Weak scaling (scaled speedup):

27

Time Serial work Parallelizable work P =1 P =2 P =4 P =8

SP = 1 + (P − 1)fpar

Parallelism

A quick tour

28

What’s in a game?

• Three kinds of code:
• Gameplay simulation
• Models the state of the game world as interacting entities
• Sound, networking, user input, etc.
• Numeric computation
• Physics, collision detection, path finding, scene graph traversal, etc.
• Rendering
• Pixel & vertex attributes; runs on the GPU

29

Sekiro: Shadows Die Twice, FromSoftware

What’s in a game?

• Parallel application design
• In practice large applications consist of


a mix of concurrency and parallelism

• Parts may be run concurrently, but there are also (data) dependencies
• Usually, individual tasks are not the same size

30

The Witness, Thekla

User input Physics AI Game logic S

• u

n d Rendering Networking

What’s in a game?

• Parallel application design
• In practice large applications consist of


a mix of concurrency and parallelism

• Parts may be run concurrently, but there are also (data) dependencies
• Usually, individual tasks are not the same size

31

The Witness, Thekla

User input

Physics AI Game logic

S

• u

n d

Rendering

Networking

Task parallelism

• Task parallelism
• Problem is broken down into separate tasks
• Individual tasks are created and communicate/

synchronise with each other

• Task decomposition dictates scalability

32

Fork-Join

• Splits control flow into multiple forks which later rejoin
• Can be used to implement many other patterns

33

A C B D

data Async a do a <.- compute_A b' <.- async (compute_B a) c' <.- async (compute_C a) b <.- wait b' c <.- wait c' d <.- compute_D b c

Divide-and-conquer

• Many divide-and-conquer algorithms lend themselves to fork-join parallelism
• Sub-problems must be independent


so that they can execute in parallel

• Correct task granularity is vital
• Deep enough to expose


enough parallelism

• Not so fine-grained that


scheduling overheads
 dominate

34

divide base case combine

Load balancing

• The computation must be distributed evenly across the processors to obtain

the fastest possible execution speed

• It may be that some processors will complete their tasks before others and

become idle because the work is not evenly distributed

• The amount of work is not known prior to execution
• Differences in processor speeds (e.g. noisy system, frequency boost…)

35

Load balancing

• Static load balancing can be viewed as a scheduling or bin packing problem
• Estimate the execution time for parts of the program and their

interdependencies

• Generate a fixed number of equally sized tasks and distribute amongst the

processors in some way (e.g. round robin, recursive bisection, random…)

• Limitations:
• Accurate estimates of execution time are difficult
• Does not account for variable delays (e.g. memory access time) or

number of tasks (e.g. search problems)

36

Load balancing

• In dynamic load balancing tasks are allocated to processors during execution
• In a centralised dynamic scheme one process holds all tasks to be computed
• Worker processes request new tasks from the work-pool
• Readily applicable to divide-and-conquer problems

37

Data parallelism

• Data parallelism
• Problem is viewed as operations over parallel data
• The same operation is applied to subsets of the data
• Scales to the amount of data & number of processors

38

… n

{ { {

P1 P2 P3 …

Photo by @yukimomon

tot ziens