Functional Programming and Parallel Computing Bjrn Lisper School of - - PowerPoint PPT Presentation

Functional Programming and Parallel Computing

Björn Lisper School of Innovation, Design, and Engineering Mälardalen University bjorn.lisper@mdh.se http://www.idt.mdh.se/˜blr/

Functional Programming and Parallel Computing (revised 2013-12-03)

Parallel Processing

Multicore processors are becoming commonplace They typically consist of several processors, and a shared memory:

CPU CPU Memory CPU CPU

The different cores execute different pieces of code in parallel If several cores do useful work at the same time, the processing becomes faster

Functional Programming and Parallel Computing (revised 2013-12-03) 1

Imperative Programming and Parallel Processing

Imperative programming is inherently sequential Statements are executed one after another However, imperative programs can be made parallel through parallel processes, or threads:

CPU CPU Memory CPU CPU P4 P3 P1 P2

Functional Programming and Parallel Computing (revised 2013-12-03) 2

Side Effects and Parallel Processing

Imperative programs have side effects Variables are assigned values These variables may be accessed by several processes This introduces the risk of race conditions Since processes run asynchronously on different processors, we cannot make any assumptions about their relative speed In one situation one process may run faster, in another situation another one runs faster

Functional Programming and Parallel Computing (revised 2013-12-03) 3

A Race Condition Example

Two processes P1 and P2, which both can write a shared variable tmp:

P1: P2: . tmp = 4711 . z = tmp + 3 . . tmp = 17 . y = 2*tmp .

The intention of the programmer was that both P1 and P2 should set tmp and then use its newly set value right away. This would yield the following, intended, final contents of y and z: y = 34, z = 4714

Functional Programming and Parallel Computing (revised 2013-12-03) 4

But the programmer missed that the processes have a race condition for tmp, since they may run at different speed. Here is one possible situation:

P1: P2: tmp = 17 . . tmp = 4711 y = 2*tmp . . z = tmp + 3

Final state: y = 9422, z = 4714 Wrong value for y!

Functional Programming and Parallel Computing (revised 2013-12-03) 5

Here is another possible situation:

P1: P2: . tmp = 4711 tmp = 17 . . z = tmp + 3 y = 2*tmp .

Final state: y = 34, z = 20 Wrong value for z!

Functional Programming and Parallel Computing (revised 2013-12-03) 6

Parallel Programming is Difficult

To avoid these race conditions, the programmer must use different synchronization mechanisms to make sure processes do not interfere with each other But this is difficult! It is very easy to miss race conditions Debugging of parallel programs is also difficult! Race conditions may occur

• nly under certain conditions, that appear very seldom. It can be very hard

to reproduce bugs This is a very bad situation, since multicore processors are becoming

• commonplace. We are heading for a software crisis!

The heart of the problem is the side effects – they allow different processes to thrash each others’ data

Functional Programming and Parallel Computing (revised 2013-12-03) 7

Pure Functional Programs and Parallel Processing

Pure functional programs have no side-effects The evaluation order does not matter Thus, different parts of the same expression can always be evaluated in parallel:

+ f(x) + g(x) f(x) g(x)

Sometimes called expression parallelism

Functional Programming and Parallel Computing (revised 2013-12-03) 8

Parallelism in Collection-Oriented Primitives

Data structures like lists, arrays, sequences, are sometimes called collections Functions like map and fold are called collection-oriented Collection-oriented functions often have a lot of inherent parallelism If one can express computations with these primitives, then parallelization

• ften becomes easy

This parallelism is often called data parallelism In imperative programs these computations are often implemented by loops. Loops are sequential. A good parallelizing compiler might retrieve some of it, but there is a risk that parallelism is lost

Functional Programming and Parallel Computing (revised 2013-12-03) 9

Map on Arrays

Map is very parallel:

x1 x4 x3 x2 x5 x6 x7 x8 x9 x10 x11 x12 x1 x4 x3 x2 x5 x6 x7 x8 x9 x10 x11 x12 x14 x16 x15 x13 x14 x16 x15 x13 f f f f f x4 f x5 f x13 f x14f x15 f x16 f x11 f x10 f x9 f x12 f x8 f x7 f x6 f x1 f x2 f x3

With sufficiently many processors, map can be done in O(1) time

Functional Programming and Parallel Computing (revised 2013-12-03) 10

Parallel Fold

Fold can be parallelized if the binary function is an associative operator:

• p
• p
• p
• p
• p
• p
• p
• p
• p
• p
• p

x1 init

• p

x2 x3 x4 x5 x6 x7 x8

• p
• p
• p

x1 x7 x8 x2 x3 x4 x5 x6

Functional Programming and Parallel Computing (revised 2013-12-03) 11

If op is associative, then the expression tree can be balanced With sufficiently many processors, parallel fold can be done in O(log n) time (n = no. of elements)

Functional Programming and Parallel Computing (revised 2013-12-03) 12

Fault Tolerance Through Replicated Evaluation

If an expression has no side effects, then it can be evaluated several times without changing the result of the program

f(x) f(x) f(x) choose

This can be used to increase the fault tolerance: if one processor fails, we can use the result from another one computing the same expression

Functional Programming and Parallel Computing (revised 2013-12-03) 13

Parallelism in F#

F# has some support for parallel and concurrent processing: The System.Threading library gives threads A data type Async<’a> for asynchronous (concurrent) workflows (a kind of computation expressions) The System.Threading.Tasks library yields task parallelism Array.Parallel module provides data parallel operations on arrays More on this in Section 13 in the book

Functional Programming and Parallel Computing (revised 2013-12-03) 14

Example: Expression Parallelism using Parallel Tasks

Evaluating f x and g x in parallel when computing (f x) + (g x):

Open System.Threading.Tasks let h x = let result1 = Task.Factory.StartNew(fun () -> f x) let result2 = Task.Factory.StartNew(fun () -> g x) result1 + result2

Functional Programming and Parallel Computing (revised 2013-12-03) 15

Example: Data Parallel Search for Primes

Create a big array, then map a test for primality over it to be done in parallel Assume a predicate isPrime : int -> bool that tests whether its argument is a prime

let bigarray = [|1 .. 500000|] Array.Parallel.map isPrime bigarray

Functional Programming and Parallel Computing (revised 2013-12-03) 16

Summing Up

Freedom from side effects simplifies parallel processing a lot Collection-oriented operations are also very helpful for this It’s the design and thinking that is important – not necessarily that a functional language is used The same principles can be applied also when using conventional programming languages

Functional Programming and Parallel Computing (revised 2013-12-03) 17