Toward Understanding Heterogeneity in Computing Arnold L. Rosenberg - - PowerPoint PPT Presentation

Toward Understanding Heterogeneity in Computing

Arnold L. Rosenberg Ron C. Chiang Department of Electrical and Computer Engineering Colorado State University Fort Collins, CO, USA {rsnbrg, ron.chiang@colostate.edu}

2

Motivation

• Goal

– to increase our understanding of heterogeneity in computing platforms

3

Motivation

• Goal

– to increase our understanding of heterogeneity in computing platforms

• Heterogeneous computing platforms

– different computing speeds

4

Motivation

• Goal

– to increase our understanding of heterogeneity in computing platforms

• Heterogeneous computing platforms

– different computing speeds – architecturally balanced

5

“Understanding” Heterogeneity

Suppose we have

• n+1 computers:

– the server C0 – a “cluster” C comprising n computers, C1, …, Cn

• Heterogeneity profile of C

– Ci can complete one unit of work in time – – > <

n

ρ ρ ,...,

1 i

ρ

n

ρ ρ ρ ≥ ≥ ≥ ...

2 1

6

The Cluster-Exploitation Problem (CEP)

• C0 must complete as many units of work

as possible on cluster C within a given lifespan of L time units

7

The Cluster-Exploitation Problem (CEP)

• C0 must complete as many units of work

as possible on cluster C within a given lifespan of L time units

• A worksharing protocol

– a schedule that solves the CEP

8

Fixed communication cost – setup time – latency σ

Architectural Parameters

negligible over a long lifespan λ

9

Common parameters: – transmission rate (e.g. 1 sec. / work unit) – output-to-input length ratio (= 1) For computer i, – packaging rate (e.g. 10 sec. / work unit) – unpackaging rate (e.g. 10 sec. / work unit) – workload (work units)

i

π

Architectural Parameters and Sample Values

i

π

i

w τ δ μ μ μ

10

Worksharing Protocols

1

) ( w τ π +

C0 C1 Cn

1

w

11

Worksharing Protocols

n

w ) ( τ π +

1 1

) 1 ( w ρ π +

C0 C1 Cn

n

w

12

Worksharing Protocols

1

w δ

1 1

) ( w δ τ πρ +

C0 C1 Cn

n nw

ρ π) 1 ( +

13

Worksharing Protocols

n

w δ

C0 C1 Cn

n n

w δ τ πρ ) ( +

14

The FIFO Protocol

2 2

) ( w δ τ πρ +

1 1

) 1 ( w ρ π +

3 3

) ( w δ τ πρ +

1 1

) ( w δ τ πρ +

2 2

) 1 ( w ρ π +

3 3

) 1 ( w ρ π +

C1 C2 C3

1

) (

processes processes processes results results results waits waits waits

w τ π +

2

) ( w τ π +

3

) ( w τ π +

C0 sends work to C1 sends work to C2 sends work to C3 (NOT TO SCALE)

15

The FIFO Protocol is Optimal

• Theorem [Adler-Gong-Rosenberg]

Over any sufficiently long lifespan L, for any heterogeneous cluster C — no matter what its heterogeneity profile:

– FIFO worksharing protocols provide optimal solutions to the cluster-exploitation problem – C is equally productive under every FIFO protocol, i.e., under all startup orderings

16

The Work-Production of FIFO

∑ ∏

= − =

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + + + − + − + + + + =

n i i j j i

X

1 1 1

) 1 ( ) ( 1 ) 1 ( ) ( 1 Let ρ πδ π τ π τδ τ π ρ πδ π τ π

17

The Work-Production of FIFO

L X W ⋅ + = 1 1 Then, τδ

∑ ∏

= − =

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + + + − + − + + + + =

n i i j j i

X

1 1 1

) 1 ( ) ( 1 ) 1 ( ) ( 1 Let ρ πδ π τ π τδ τ π ρ πδ π τ π

18

The Work-Production of FIFO

∑ ∏

= − =

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + + = + + = + = ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈ ≈

n i i j j j i

B A B B A X

1 1 1

1 , 1 B and A let simplify, To ρ τδ ρ ρ πδ π τ π

∑ ∏

= − =

⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ + + + + − + − + + + + =

n i i j j i

X

1 1 1

) 1 ( ) ( 1 ) 1 ( ) ( 1 Let ρ πδ π τ π τδ τ π ρ πδ π τ π

19

On Comparing Heterogeneity Profiles

• For any cluster C with heterogeneity profile

n

P ρ ρ , ... ,

1

=

20

On Comparing Heterogeneity Profiles

• For any cluster C with heterogeneity profile

n

P ρ ρ , ... ,

1

=

• C’s homogeneous-equivalent computing rate

(HECR) is

{ }

) ( ) ( max

) (

P X P X

c

≥ =

ρ ρ

ρ ρ ρ

ρ

, ... ,

) (

= P where

21

Heterogeneity Profiles

8 1 ,..., 8 6 , 8 7 , 8 8 , 8 when range a in evenly spreads which , 1 : 1 Profile = + − = n n i n

i

ρ Number of Computers 8 16 32 HECR 0.362 0.297 0.251 Recall: faster cluster has smaller HECR value

22

Heterogeneity Profiles

Number of Computers 8 16 32 HECR 0.216 0.116 0.061 8 1 ,..., 3 1 , 2 1 , 1 1 , 8 when 1 : 2 Profile = = n i

i

ρ

23

• Avg. Speed vs. Std-Dev of Speed

Randomly generate 100 profiles for each combination

Avg. Speed =0.75 Avg. Speed =0.5 Avg. Speed =0.25 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 HECR 8 computers Std-Dev=0.2 Std-Dev=0.1 Std-Dev=0.05

24

• Avg. Speed vs. Std-Dev of Speed

Std-Dev 8 computers’ HECR 0.2 0.1 0.05 0.75 0.681 0.735 0.759 0.5 0.411 0.482 0.501

• Avg. Speed

0.25 0.113 0.208 0.239 The probability that these two groups have the same mean is

10

10 2

−

×

25

• Avg. Speed vs. Std-Dev of Speed

Std-Dev 8 computers’ HECR 0.2 0.1 0.05 0.75 0.681 0.735 0.759 0.5 0.411 0.482 0.501

• Avg. Speed

0.25 0.113 0.208 0.239 Trials with 16, 32 computers show similar pattern

26

Speeding Up Clusters Optimally under FIFO Protocols

• Which one computer should you speed up,

if you can speed up only one?

27

Speeding Up Clusters Optimally under FIFO Protocols

• Which one computer should you speed up,

if you can speed up only one?

• We study two variants of this question

28

Speeding Up Clusters Optimally under FIFO Protocols

indices computer two be and let

• ...

where , ,..., profile ity heterogene have cluster let

• e,

convenienc For

2 1 1

i j i P

n n

> ≥ ≥ ≥ > =< ρ ρ ρ ρ ρ

C

29

Fixed and Proportional Speed-up

• Fixed-speedup scenario
• by a fixed amount

n j j j i i i j n j j j i i i i

P P ρ ρ φ ρ ρ ρ ρ ρ ρ ρ ρ ρ ρ ρ φ ρ ρ ρ ,..., , , ,..., , , ,..., ,..., , , ,..., , , ,...,

1 1 1 1 1 ) ( 1 1 1 1 1 ) ( + − + − + − + −

− = − =

n

ρ φ <

30

Fixed and Proportional Speed-up

• Fixed-speedup scenario (by a fixed amount )

n j j j i i i j n j j j i i i i

P P ρ ρ φ ρ ρ ρ ρ ρ ρ ρ ρ ρ ρ ρ φ ρ ρ ρ ,..., , , ,..., , , ,..., ,..., , , ,..., , , ,...,

1 1 1 1 1 ) ( 1 1 1 1 1 ) ( + − + − + − + −

− = − =

• Proportional-speedup scenario
• by a relative amount

n j j j i i i j n j j j i i i i

P P ρ ρ ψρ ρ ρ ρ ρ ρ ρ ρ ρ ρ ρ ψρ ρ ρ ,..., , , ,..., , , ,..., ,..., , , ,..., , , ,...,

1 1 1 1 1 ] [ 1 1 1 1 1 ] [ + − + − + − + −

= =

n

ρ φ <

1 < ψ

31

Proposition for Fixed-Speedup

• Under the fixed-speedup scenario, the

most advantageous single computer to speed up is C’s fastest computer

32

Terms for following figures

• Recall: work production
• Work ratio

– the ratio of work production after speedup to work production before speedup

• Speedup computer

– the single computer that is sped up

L X W ⋅ + = 1 1 τδ

33

Fixed-Speedup Scenario

0.9 1 1.1 1.2 1.3 1.4 1.5 1 2 3 4 speedup computer Work ratio

<1, 1/2, 1/3, 1/4> <1/2, 1/4, 1/6, 1/8>

16 / 1 = φ

34

Proposition for Proportional-Speedup

• If

– speeding up (faster) is better

• If

– speeding up (slower) is better

2

/B A

j i

τδ ρ ψρ <

2

/B A

j i

τδ ρ ψρ >

j

C

i

C ) and , 1 B , A : (Recall

j i

ρ ρ πδ π τ π > + + = + =

35

Proposition for Proportional-Speedup

• If

– speeding up (faster) is better

• If

– speeding up (slower) is better

5 2

10 . 1 /

−

× = < B A

j i

τδ ρ ψρ

5 2

10 . 1 /

−

× = > B A

j i

τδ ρ ψρ

j

C

i

C

Parameter Rate A 11 second / work unit B with coarse (1 sec / task) tasks 1.000011 second / work unit

) and , 1 B , A : (Recall

j i

ρ ρ πδ π τ π > + + = + =

μ

36

Proposition for Proportional-Speedup

• If

– speeding up (faster) is better

• If

– speeding up (slower) is better

5 2

10 . 1 /

−

× = < B A

j i

τδ ρ ψρ

5 2

10 . 1 /

−

× = > B A

j i

τδ ρ ψρ

j

C

i

C ) and , 1 B , A : (Recall

j i

ρ ρ πδ π τ π > + + = + =

That is, it is more advantageous to speed up the faster one unless either both computers are already “very fast” or the speedup factor is “very large.”

37

Proportional-Speedup in Action

Round 1 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

38

Proportional-Speedup in Action

Round 2 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

39

Round 3 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

Proportional-Speedup in Action

ρ

40

Proportional-Speedup in Action

Round 4 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

41

Proportional-Speedup in Action

Round 5 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

42

Proportional-Speedup in Action

Round 6 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

43

Proportional-Speedup in Action

Round 7 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

44

Proportional-Speedup in Action

Round 8 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

45

Proportional-Speedup in Action

Round 9 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

46

Proportional-Speedup in Action

Round 10 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

47

Proportional-Speedup in Action

Round 11 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

48

Proportional-Speedup in Action

Round 12 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

49

Proportional-Speedup in Action

Round 13 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

50

Proportional-Speedup in Action

Round 14 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

51

Proportional-Speedup in Action

Round 15 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

52

Proportional-Speedup in Action

Round 16 0.2 0.4 0.6 0.8 1 1.2 1 2 3 4 speedup computer

ρ

53

Proportional-Speedup in Action

• When all computers are very fast

– It is more advantageous to speed up the slower one

54

Proportional-Speedup in Action

Round 17 0.02 0.04 0.06 0.08 1 2 3 4 speedup computer

ρ

55

Proportional-Speedup in Action

Round 18 0.02 0.04 0.06 0.08 1 2 3 4 speedup computer

ρ

56

Proportional-Speedup in Action

Round 19 0.02 0.04 0.06 0.08 1 2 3 4 speedup computer

ρ

57

Proportional-Speedup in Action

Round 20 0.02 0.04 0.06 0.08 1 2 3 4 speedup computer

ρ

58

Proportional-Speedup in Action

Round 21 0.02 0.04 0.06 0.08 1 2 3 4 speedup computer

ρ

59

Summary

• Two ways to measure computing power

– the X function – the HECR value

60

Summary

• Two ways to measure computing power

– the X function – the HECR value

• Standard deviation influences work

production

61

Summary

• Two ways to measure computing power

– the X function – the HECR value

• Standard deviation influences work

production

• Speeding up a fast computer in a cluster is

almost always more advantageous than speeding up a slower one

Thank you

Questions?

63

HECR values

Number of Computers 8 16 32 Profile 1 0.362 0.297 0.251 Profile 2 0.216 0.116 0.061

n i n

i

1 : 1 Profile + − = ρ

Recall: faster cluster has smaller HECR value

i

i

1 : 2 Profile = ρ

64

• Avg. Speed vs. Std-Dev of Speed

Std-Dev 16 computers’ HECR 0.2 0.1 0.05 0.75 0.671 0.723 0.768 0.5 0.385 0.475 0.502

• Avg. Speed

0.25 0.110 0.194 0.239

65

• Avg. Speed vs. Std-Dev of Speed

Std-Dev 32 computers’ HECR 0.2 0.1 0.05 0.75 0.669 0.742 0.782 0.5 0.380 0.478 0.502

• Avg. Speed

0.25 0.115 0.197 0.239