Recap: Brents principle Sequential algorithms: time = work Parallel - - PowerPoint PPT Presentation

▶
recap brent s principle
SMART_READER_LITE
LIVE PREVIEW

Recap: Brents principle Sequential algorithms: time = work Parallel - - PowerPoint PPT Presentation

Dec 22, 2023 198 likes •822 views

Recap: Brents principle Sequential algorithms: time = work Parallel algorithms (PRAM): time T(n) work W(n) Brents princliple: Work matters! 1 / 61 Work efficiency Parallel alg is work-efficient if: Work is the

slide-1
SLIDE 1

1 / 61

Recap: Brent‘s principle

  • Sequential algorithms: time = work
  • Parallel algorithms (PRAM):

– time T(n) – work W(n)

  • Brent‘s princliple:
  • Work matters!
slide-2
SLIDE 2

2 / 61

Work efficiency

  • Parallel alg is work-efficient if:

– Work is the same as sequential algorithm:

  • Example:

Parallel Alg. A: Parallel Alg. B:

slide-3
SLIDE 3

3 / 61

Work efficiency

  • Parallel alg is work-efficient if:

– Work is the same as sequential algorithm:

  • Example:

Parallel Alg. A: Parallel Alg. B:

Work Efficient NOT Work Efficient

slide-4
SLIDE 4

4 / 61

Prefix Sums

  • Given A: set of n integers
  • Find B: prefix sums

3 1 1 7 2 5 9 2 4 3 3 3 4 5 12 14 19 28 30 34 37 40

A: B:

slide-5
SLIDE 5

5 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm

3 1 1 7 2 5 9 2 4 3 3

A: B:

slide-6
SLIDE 6

6 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm

3 1 1 7 2 5 9 2 4 3 3 3

A: B:

slide-7
SLIDE 7

7 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm

3 1 1 7 2 5 9 2 4 3 3 3 4

A: B:

slide-8
SLIDE 8

8 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm

3 1 1 7 2 5 9 2 4 3 3 3 4 5

A: B:

slide-9
SLIDE 9

9 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm

3 1 1 7 2 5 9 2 4 3 3

A: B:

3 4 5 12 14 19 28 30 34 37 40

slide-10
SLIDE 10

10 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm
  • Can we do this in-place?
slide-11
SLIDE 11

11 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm
  • Can we do this in-place?
slide-12
SLIDE 12

12 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm
  • What is the complexity of this algorithm?
slide-13
SLIDE 13

13 / 61

Sequential Prefix Sums

  • Trivial sequential algorithm
  • What is the complexity of this algorithm?
  • Loop runs n-1 times:

T1(n) = O(n)

slide-14
SLIDE 14

15 / 61

Parallel approach

  • Similar to summation

– Except all cells need correct values

  • “Binary tree“ approach

– Add values of increasing distance

  • Goal:

time T(n) = O(log n) work W(n) = O(n)

slide-15
SLIDE 15

16 / 61

Naive parallel algorithm

3 1 1 7 2 5 9 2 3 4 2 8 9 7 14 11

slide-16
SLIDE 16

17 / 61

Naive parallel algorithm

3 1 1 7 2 5 9 2 3 4 2 8 9 7 14 11 3 4 5 12 11 23 18 15

slide-17
SLIDE 17

18 / 61

Naive parallel algorithm

3 1 1 7 2 5 9 2 3 4 2 8 9 7 14 11 3 4 5 12 11 23 18 15 3 4 5 12 14 28 30 19

slide-18
SLIDE 18

19 / 61

Proof of correctness

  • Does this correctly solve prefix sums?

– Let‘s prove it:

  • Before step j:

– the first 2j elements are prefix sums, – each A[i] = sum of previous 2j elements

3 1 1 7 2 5 9 2 3 4 2 8 9 7 14 11 3 4 5 12 11 23 18 15

Step 1

slide-19
SLIDE 19

20 / 61

Proof of correctness

  • Claim: before each round j,
  • Proof:
  • Base case: j = 0
slide-20
SLIDE 20

21 / 61

Proof of correctness

  • Inductive step: Assume that before the jth step:
  • Step j‘ = j+1:
slide-21
SLIDE 21

22 / 61

Proof of correctness

A[i] 2j 2j A[i-2j]

slide-22
SLIDE 22

23 / 61

Proof of correctness

slide-23
SLIDE 23

24 / 61

Naive algorithm time

  • What is T(n)?

– Remember: we have unlimited processors!

slide-24
SLIDE 24

25 / 61

Naive algorithm time

  • What is T(n)?

– Inner loop is done in parallel: O(1) – Outer loop is sequential: O(log n)

  • T(n) = O(log n)
slide-25
SLIDE 25

26 / 61

Naive algorithm work

  • What is W(n)?
slide-26
SLIDE 26

27 / 61

Solving the summation

  • What is 2nd term?

k sum of internal (red) nodes =

slide-27
SLIDE 27

28 / 61

Proof of summation

  • Claim:
  • Proof by induction:
  • Base case: k = 1
slide-28
SLIDE 28

29 / 61

Proof of summation

  • Inductive Step:
  • Assume k > 1, show for k‘ = k+1
slide-29
SLIDE 29

30 / 61

Naive algorithm work

  • Is this work-efficient?
slide-30
SLIDE 30

31 / 61

Naive algorithm work

  • Is this work-efficient?
  • No. Our sequential algorithm was O(n)
slide-31
SLIDE 31

32 / 61

Ideas for a work-efficient alg.

  • What if we had half the prefix sums?
  • No help :(

3 4 5 12 3 1 1 7 2 5 9 2

A B

slide-32
SLIDE 32

33 / 61

Ideas for a work-efficient alg.

  • What about this half?
  • How would we compute the rest?

4 12 19 30 3 1 1 7 2 5 9 2

A B

slide-33
SLIDE 33

34 / 61

Ideas for a work-efficient alg.

  • B[i] = B[i-1] + A[i]

4 12 19 30 3 1 1 7 2 5 9 2

A B

3 5 14 28

slide-34
SLIDE 34

35 / 61

Work-efficient algorithm

  • Idea based on balanced binary trees:

– Depth of a tree is O(log n) – Number of nodes is O(n)

  • Compute prefix sum for every odd element
  • Use these to compute remaining
slide-35
SLIDE 35

36 / 61

Computing prefix for all 2i

  • Perform summation (reduce)
  • Save intermediate values in B

3 1 1 7 2 5 9 2 4 8 7 11

A B

slide-36
SLIDE 36

37 / 61

Computing prefix for all 2i

  • Continue until total sum is found

3 1 1 7 2 5 9 2 4 8 7 11

A B

4 12 7 18

slide-37
SLIDE 37

38 / 61

Computing prefix for all 2i

  • Continue until total sum is found

3 1 1 7 2 5 9 2 4 8 7 11

A B

4 12 7 18 4 12 7 30

slide-38
SLIDE 38

39 / 61

Remaining odd elements

  • Each prefix sum at 2i is correct!

3 1 1 7 2 5 9 2

A B

4 12 7 30

slide-39
SLIDE 39

40 / 61

Remaining odd elements

  • Each prefix sum at 2i is correct!
  • Add preceeding 2i to fix other odd B entries

3 1 1 7 2 5 9 2

A

4 12 19 30

B

4 12 7 30

slide-40
SLIDE 40

41 / 61

Defining the algorithm

  • Each odd-index prefix sum is correct
  • Empty spaces in B

– Easier to use consecutive spaces – Compress to the first n/2 positons

3 1 1 7 2 5 9 2

A B

4 12 19 30

slide-41
SLIDE 41

42 / 61

Computing prefix for all odds

3 1 1 7 2 5 9 2

A B

4 12 19 30

slide-42
SLIDE 42

43 / 61

Proof of correctness

  • Claim:
  • Proof
  • Base case:
slide-43
SLIDE 43

44 / 61

Proof of correctness

  • Inductive step:

Assume true for i > 0, show for i‘ = i + 1

slide-44
SLIDE 44

45 / 61

Even index sums

  • We compute all odd-index prefix sums
  • Simple to fill in evens:
slide-45
SLIDE 45

46 / 61

Putting it all together

Odd-index prefix sums Fill in even indices

slide-46
SLIDE 46

47 / 61

Time analysis

  • Each recursive call works on ½ as many elements

– Can be done in parallel: – Solution?

slide-47
SLIDE 47

48 / 61

Time analysis

  • Each recursive call works on ½ as many elements

– Can be done in parallel: – Solution?

slide-48
SLIDE 48

49 / 61

Work analysis

  • Have to consider work done at each level:

– Solution?

slide-49
SLIDE 49

50 / 61

Work analysis

  • Have to consider work done at each level:

– Solution? – (or using the Master Theorem)

slide-50
SLIDE 50

51 / 61

Algorithm overview

  • Parallel runtime same as our naive algorithm
  • Less work…. but is it work-efficient?
slide-51
SLIDE 51

52 / 61

Algorithm overview

  • Parallel runtime same as our naive algorithm
  • Less work…. but is it work-efficient? Yes!
slide-52
SLIDE 52

53 / 61

One more example

1

A B

1 3 2 4 1 1 3 2 6 1 8 4 2 1 2

slide-53
SLIDE 53

54 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3

A B

slide-54
SLIDE 54

55 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3 8 12 15 7

A B

slide-55
SLIDE 55

56 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3 8 12 15 7 27 15

A B

slide-56
SLIDE 56

57 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3 8 12 15 7 27 15 42

A B

slide-57
SLIDE 57

58 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3 8 12 15 7 42 15 42

A B

slide-58
SLIDE 58

59 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 4 6 2 5 7 12 3 3 15 27 42 7 42 15 42

A B

slide-59
SLIDE 59

60 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 7 13 15 20 27 39 42 3 15 27 42 7 42 15 42

A B

slide-60
SLIDE 60

61 / 61

One more example

1 1 3 2 4 1 1 3 2 6 1 8 4 2 1 2 7 13 15 20 27 39 42 3 15 27 42 7 42 15 42

A B

slide-61
SLIDE 61

62 / 61

One more example

3 4 7 9 13 14 15 18 20 26 27 35 39 41 42 2 7 13 15 20 27 39 42 3 15 27 42 7 42 15 42

A B