Static Scheduling for Large-Scale Heterogeneous Platforms Yves - - PowerPoint PPT Presentation

▶
static scheduling for large scale heterogeneous platforms
SMART_READER_LITE
LIVE PREVIEW

Static Scheduling for Large-Scale Heterogeneous Platforms Yves - - PowerPoint PPT Presentation

Sep 27, 2022 121 likes •1.6k views

Static Scheduling for Large-Scale Heterogeneous Platforms Yves Robert Ecole Normale Sup erieure de Lyon joint work with Larry Carter, Henri Casanova, Jeanne Ferrante, Yang Yang Olivier Beaumont, Arnaud Legrand, Loris Marchal, Fr ed

slide-1
SLIDE 1

Static Scheduling for Large-Scale Heterogeneous Platforms

Yves Robert ´ Ecole Normale Sup´ erieure de Lyon

joint work with Larry Carter, Henri Casanova, Jeanne Ferrante, Yang Yang Olivier Beaumont, Arnaud Legrand, Loris Marchal, Fr´ ed´ eric Vivien Yves.Robert@ens-lyon.fr http://graal.ens-lyon.fr/∼yrobert

IΠ∆ΠΣ’2006

slide-2
SLIDE 2

Evolution of parallel machines

From good old parallel architectures . . .

Yves Robert Scheduling for Heterogeneous Platforms 2/ 86

slide-3
SLIDE 3

Evolution of parallel machines

. . . to heterogeneous clusters . . .

Yves Robert Scheduling for Heterogeneous Platforms 2/ 86

slide-4
SLIDE 4

Evolution of parallel machines

. . . and soon to the Holy Grid?

Yves Robert Scheduling for Heterogeneous Platforms 2/ 86

slide-5
SLIDE 5

Evolution of parallel machines

. . . and soon to the Holy Grid? Parallel algorithm design and scheduling were already difficult tasks with homogeneous machines

Yves Robert Scheduling for Heterogeneous Platforms 2/ 86

slide-6
SLIDE 6

Evolution of parallel machines

. . . and soon to the Holy Grid? Parallel algorithm design and scheduling were already difficult tasks with homogeneous machines On heterogeneous platforms, it gets worse

Yves Robert Scheduling for Heterogeneous Platforms 2/ 86

slide-7
SLIDE 7

New platforms, new problems, new solutions

Target platforms: Large-scale heterogenous platforms (networks of workstations, clusters, collections of clusters, grids, ...) New problems Heterogeneity of processors (CPU power, memory) Heterogeneity of communication links Irregularity of interconnection networks Non-dedicated platforms Need to adapt algorithms and scheduling strategies: new objective functions, new models

Yves Robert Scheduling for Heterogeneous Platforms 3/ 86

slide-8
SLIDE 8

New platforms, new problems, new solutions

Target platforms: Large-scale heterogenous platforms (networks of workstations, clusters, collections of clusters, grids, ...) New problems Heterogeneity of processors (CPU power, memory) Heterogeneity of communication links Irregularity of interconnection networks Non-dedicated platforms Need to adapt algorithms and scheduling strategies: new objective functions, new models

Yves Robert Scheduling for Heterogeneous Platforms 3/ 86

slide-9
SLIDE 9

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 4/ 86

slide-10
SLIDE 10

Background on traditional scheduling

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 5/ 86

slide-11
SLIDE 11

Background on traditional scheduling

Traditional scheduling – Framework

Application = DAG G = (T , E, w)

◮ T = set of tasks ◮ E = dependence constraints ◮ w(T) = computational cost of task T (execution time) ◮ c(T, T ′) = communication cost (data sent from T to T ′)

Platform

◮ Set of p identical processors

Schedule

◮ σ(T) = date to begin execution of task T ◮ alloc(T) = processor assigned to it Yves Robert Scheduling for Heterogeneous Platforms 6/ 86

slide-12
SLIDE 12

Background on traditional scheduling

Traditional scheduling – Constraints

w(T’)

time

w(T)

comm(T,T’)

σ(T) + w(T) σ(T)

T T’

σ(T ′)

Data dependences If (T, T ′) ∈ E then

◮ if alloc(T) = alloc(T ′) then σ(T) + w(T) ≤ σ(T ′) ◮ if alloc(T) = alloc(T ′) then σ(T) + w(T) + c(T, T ′) ≤ σ(T ′)

Resource constraints alloc(T) = alloc(T ′) ⇒ (σ(T) + w(T) ≤ σ(T ′)) or (σ(T ′) + w(T ′) ≤ σ(T))

Yves Robert Scheduling for Heterogeneous Platforms 7/ 86

slide-13
SLIDE 13

Background on traditional scheduling

Traditional scheduling – Objective functions

Makespan or total execution time MS(σ) = max

T∈T (σ(T) + w(T))

Other classical objectives:

◮ Sum of completion times ◮ With release dates: maximum flow (response time), or sum flow ◮ Fairness oriented: maximum stretch, or sum stretch Yves Robert Scheduling for Heterogeneous Platforms 8/ 86

slide-14
SLIDE 14

Background on traditional scheduling

Traditional scheduling – About the model

Simple but OK for computational resources

◮ No CPU sharing, even in models with preemption ◮ At most one task running per processor at any time-step

Very crude for network resources

◮ Unlimited number of simultaneous sends/receives per processor ◮ No contention → unbounded bandwidth on any link ◮ Fully connected interconnection graph (clique)

In fact, model assumes infinite network capacity

Yves Robert Scheduling for Heterogeneous Platforms 9/ 86

slide-15
SLIDE 15

Background on traditional scheduling

Makespan minimization

NP-hardness

◮ Pb(p) NP-complete for independent tasks and no communications

(E = ∅, p = 2 and c = 0)

◮ Pb(p) NP-complete for UET-UCT graphs (w = c = 1)

Approximation algorithms

◮ Without communications, list scheduling is a (2 − 1

p)-approximation

◮ With communications, result extends to coarse-grain graphs ◮ With communications, no λ-approximation in general Yves Robert Scheduling for Heterogeneous Platforms 10/ 86

slide-16
SLIDE 16

Background on traditional scheduling

List scheduling – Without communications

Initialization:

1 Compute priority level of all tasks 2 Priority queue = list of free tasks (tasks without predecessors)

sorted by priority While there remain tasks to execute:

1 Add new free tasks, if any, to the queue. 2 If there are q available processors and r tasks in the queue,

remove first min(q, r) tasks from the queue and execute them Priority level Use critical path: longest path from the task to an exit node Computed recursively by a bottom-up traversal of the graph

Yves Robert Scheduling for Heterogeneous Platforms 11/ 86

slide-17
SLIDE 17

Background on traditional scheduling

List scheduling – Without communications

Initialization:

1 Compute priority level of all tasks 2 Priority queue = list of free tasks (tasks without predecessors)

sorted by priority While there remain tasks to execute:

1 Add new free tasks, if any, to the queue. 2 If there are q available processors and r tasks in the queue,

remove first min(q, r) tasks from the queue and execute them Priority level Use critical path: longest path from the task to an exit node Computed recursively by a bottom-up traversal of the graph

Yves Robert Scheduling for Heterogeneous Platforms 11/ 86

slide-18
SLIDE 18

Background on traditional scheduling

List scheduling – With communications (1/2)

Priority level

◮ Use pessimistic critical path: include all edge costs in the weight ◮ Computed recursively by a bottom-up traversal of the graph

MCP Modified Critical Path

◮ Assign free task with highest priority to best processor ◮ Best processor = finishes execution first, given already taken

scheduling decisions

◮ Free tasks may not be ready for execution (communication delays) ◮ May explore inserting the task in empty slots of schedule ◮ Complexity O(|V | log |V | + (|E| + |V |)p) Yves Robert Scheduling for Heterogeneous Platforms 12/ 86

slide-19
SLIDE 19

Background on traditional scheduling

List scheduling – With communications (2/2)

EFT Earliest Finish Time

◮ Dynamically recompute priorities of free tasks ◮ Select free task that finishes execution first (on best processor),

given already taken scheduling decisions

◮ Higher complexity O(|V |3p) ◮ May miss “urgent” tasks on critical path

Other approaches

◮ Two-step: clustering + load balancing

  • DSC Dominant Sequence Clustering O((|V | + |E|) log |V |)
  • LLB List-based Load Balancing O(C log C + |V |) (C number of

clusters generated by DSC)

◮ Low-cost: FCP Fast Critical Path

  • Maintain constant-size sorted list of free tasks:
  • Best processor = first idle or the one sending last message
  • Low complexity O(|V | log p + |E|)

Yves Robert Scheduling for Heterogeneous Platforms 13/ 86

slide-20
SLIDE 20

Background on traditional scheduling

Extending the model to heterogeneous clusters

Task graph with n tasks T1, . . . , Tn. Platform with p heterogeneous processors P1, . . . , Pp. Computation costs:

  • wiq = execution time of Ti on Pq
  • wi =

Pp

q=1 wiq

p

average execution time of Ti

  • particular case: consistent tasks wiq = wi × γq

Communication costs:

  • data(i, j): data volume for edge eij : Ti → Tj
  • vqr: communication time for unit-size message from Pq to Pr

(zero if q = r)

  • com(i, j, q, r) = data(i, j) × vqr communication time from Ti

executed on Pq to Pj executed on Pr

  • comij = data(i, j) ×

P

1≤q,r≤p,q=r vqr

p(p−1)

average communication cost for edge eij : Ti → Tj

Yves Robert Scheduling for Heterogeneous Platforms 14/ 86

slide-21
SLIDE 21

Background on traditional scheduling

Rewriting constraints

Dependences For eij : Ti → Tj, q = alloc(Ti) and r = alloc(Tj): σ(Ti) + wiq + com(i, j, q, r) ≤ σ(Tj) Resources If q = alloc(Ti) = alloc(Tj), then (σ(Ti) + wiq ≤ σ(Tj)) or (σ(Tj) + wjq ≤ σ(Ti)) Makespan max

1≤i≤n

  • σ(Ti) + wi,alloc(Ti)
  • Yves Robert

Scheduling for Heterogeneous Platforms 15/ 86

slide-22
SLIDE 22

Background on traditional scheduling

HEFT: Heterogeneous Earliest Finish Time

1 Priority level: ◮ rank(Ti) = wi +

max

Tj∈Succ(Ti)(comij + rank(Tj)),

where Succ(T) is the set of successors of T

◮ Recursive computation by bottom-up traversal of the graph 2 Allocation ◮ For current task Ti, determine best processor Pq:

minimize σ(Ti) + wiq

◮ Enforce constraints related to communication costs ◮ Insertion scheduling: look for t = σ(Ti) s.t. Pq is available during

interval [t, t + wiq[

3 Complexity: same as MCP without/with insertion Yves Robert Scheduling for Heterogeneous Platforms 16/ 86

slide-23
SLIDE 23

Background on traditional scheduling

Bibliography – Traditional scheduling

Introductory book: Distributed and parallel computing, H. El-Rewini and T. G. Lewis, Manning 1997 FCP: On the complexity of list scheduling algorithms for distributed-memory systems, A. Radulescu and A.J.C. van Gemund, 13th ACM Int Conf. Supercomputing (1999), 68-75 HEFT: Performance-effective and low-complexity task scheduling for heterogeneous computing, H. Topcuoglu and S. Hariri and M.-Y. Wu, IEEE TPDS 13, 3 (2002), 260-274

Yves Robert Scheduling for Heterogeneous Platforms 17/ 86

slide-24
SLIDE 24

Background on traditional scheduling

What’s wrong? Nothing (still may need to map a DAG onto a platform!) Absurd communication model:

complicated: many parameters to instantiate while not realistic (clique + no contention)

Wrong metric: need to relax makespan minimization objective

Yves Robert Scheduling for Heterogeneous Platforms 18/ 86

slide-25
SLIDE 25

Background on traditional scheduling

What’s wrong? Nothing (still may need to map a DAG onto a platform!) Absurd communication model:

complicated: many parameters to instantiate while not realistic (clique + no contention)

Wrong metric: need to relax makespan minimization objective

Yves Robert Scheduling for Heterogeneous Platforms 18/ 86

slide-26
SLIDE 26

Background on traditional scheduling

What’s wrong? Nothing (still may need to map a DAG onto a platform!) Absurd communication model:

complicated: many parameters to instantiate while not realistic (clique + no contention)

Wrong metric: need to relax makespan minimization objective

Yves Robert Scheduling for Heterogeneous Platforms 18/ 86

slide-27
SLIDE 27

Packet routing

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 19/ 86

slide-28
SLIDE 28

Packet routing

Problem

A E G C H D F B

Routing sets of messages from sources to destinations Paths not fixed a priori Packets of same message may follow different paths

Yves Robert Scheduling for Heterogeneous Platforms 20/ 86

slide-29
SLIDE 29

Packet routing

Hypotheses

A E G C H D F B

A packet crosses an edge within one time-step At any time-step, at most one packet crosses an edge

Yves Robert Scheduling for Heterogeneous Platforms 21/ 86

slide-30
SLIDE 30

Packet routing

Hypotheses

A E G C H D F B

A packet crosses an edge within one time-step At any time-step, at most one packet crosses an edge Scheduling: for each time-step, decide which packet crosses any given edge

Yves Robert Scheduling for Heterogeneous Platforms 21/ 86

slide-31
SLIDE 31

Packet routing

Notation

k i l j nk,l nk,l

i,j

nk,l: total number of packets to be routed from k to l nk,l

i,j: total number of packets routed from k to l and crossing

edge (i, j)

Yves Robert Scheduling for Heterogeneous Platforms 22/ 86

slide-32
SLIDE 32

Packet routing

Lower bound

Congestion Ci,j of edge (i, j) = total number of packets that cross (i, j) Ci,j =

  • (k,l)|nk,l>0

nk,l

i,j

Cmax = maxi,j Ci,j Cmax lower bound on schedule makespan C∗ ≥ Cmax ⇒ “Fluidified” solution in Cmax?

Yves Robert Scheduling for Heterogeneous Platforms 23/ 86

slide-33
SLIDE 33

Packet routing

Lower bound

Congestion Ci,j of edge (i, j) = total number of packets that cross (i, j) Ci,j =

  • (k,l)|nk,l>0

nk,l

i,j

Cmax = maxi,j Ci,j Cmax lower bound on schedule makespan C∗ ≥ Cmax ⇒ “Fluidified” solution in Cmax?

Yves Robert Scheduling for Heterogeneous Platforms 23/ 86

slide-34
SLIDE 34

Packet routing

Equations (1/2)

A E G C H D F B A E G B

1 Initialization (packets leave node k):

  • j|(k,j)∈A

nk,l

k,j = nk,l

2 Reception (packets reach node l):

  • i|(i,l)∈A

nk,l

i,l = nk,l

3 Conservation law (crossing intermediate node i):

  • i|(i,j)∈A

nk,l

i,j =

  • i|(j,i)∈A

nk,l

j,i

∀(k, l), j = k, j = l

Yves Robert Scheduling for Heterogeneous Platforms 24/ 86

slide-35
SLIDE 35

Packet routing

Equations (1/2)

A E G C H D F B

G H D

1 Initialization (packets leave node k):

  • j|(k,j)∈A

nk,l

k,j = nk,l

2 Reception (packets reach node l):

  • i|(i,l)∈A

nk,l

i,l = nk,l

3 Conservation law (crossing intermediate node i):

  • i|(i,j)∈A

nk,l

i,j =

  • i|(j,i)∈A

nk,l

j,i

∀(k, l), j = k, j = l

Yves Robert Scheduling for Heterogeneous Platforms 24/ 86

slide-36
SLIDE 36

Packet routing

Equations (1/2)

G G

1 Initialization (packets leave node k):

  • j|(k,j)∈A

nk,l

k,j = nk,l

2 Reception (packets reach node l):

  • i|(i,l)∈A

nk,l

i,l = nk,l

3 Conservation law (crossing intermediate node i):

  • i|(i,j)∈A

nk,l

i,j =

  • i|(j,i)∈A

nk,l

j,i

∀(k, l), j = k, j = l

Yves Robert Scheduling for Heterogeneous Platforms 24/ 86

slide-37
SLIDE 37

Packet routing

Equations (2/2)

4 Congestion

Ci,j =

(k,l)|nk,l>0 nk,l i,j

5 Objective function

Cmax ≥ Ci,j, ∀i, j Minimize Cmax Linear program in rational numbers: polynomial-time solution. In practice use Maple or Mupad

Yves Robert Scheduling for Heterogeneous Platforms 25/ 86

slide-38
SLIDE 38

Packet routing

Equations (2/2)

4 Congestion

Ci,j =

(k,l)|nk,l>0 nk,l i,j

5 Objective function

Cmax ≥ Ci,j, ∀i, j Minimize Cmax Linear program in rational numbers: polynomial-time solution. In practice use Maple or Mupad

Yves Robert Scheduling for Heterogeneous Platforms 25/ 86

slide-39
SLIDE 39

Packet routing

Routing algorithm

1 Compute optimal solution Cmax, nk,l

i,j of previous linear program

2 Periodic schedule: ◮ Define Ω =

  • Cmax

◮ Use

Cmax

Ω

  • periods of length Ω

◮ During each period, edge (i, j) forwards (at most)

mk,l

i,j =

  • nk,l

i,jΩ

Cmax

  • packets that go from k to l

3 Clean-up: sequentially process residual packets inside network Yves Robert Scheduling for Heterogeneous Platforms 26/ 86

slide-40
SLIDE 40

Packet routing

Performance

Schedule is feasible Schedule is asymptotically optimal: Cmax ≤ C∗ ≤ Cmax + O(

  • Cmax)

Yves Robert Scheduling for Heterogeneous Platforms 27/ 86

slide-41
SLIDE 41

Packet routing

Why does it work?

Relaxation of objective function Rational number of packets in LP formulation Periods long enough so that rounding down to integer numbers has negligible impact Periods numerous enough so that loss in first and last periods has negligible impact Periodic schedule, described in compact form

Yves Robert Scheduling for Heterogeneous Platforms 28/ 86

slide-42
SLIDE 42

Packet routing

Bibliography – Packet routing

Survey of results: Introduction to parallel algorithms and architectures: arrays, trees, hypercubes, F.T. Leighton, Morgan Kaufmann (1992) NP-completeness, approximation algorithm: A constant-factor approximation algorithm for packet routing and balancing local vs. global criteria, A. Srinivasan, C.-P. Teo, SIAM

  • J. Comput. 30, 6 (2000), 2051-2068

Steady-state: Asymptotically optimal algorithms for job shop scheduling and packet routing, D. Bertsimas and D. Gamarnik, Journal of Algorithms 33, 2 (1999), 296-318

Yves Robert Scheduling for Heterogeneous Platforms 29/ 86

slide-43
SLIDE 43

Master-worker on heterogeneous platforms

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 30/ 86

slide-44
SLIDE 44

Master-worker on heterogeneous platforms

Master-worker tasking: framework

Heterogeneous resources Processors of different speeds Communication links with various bandwidths Large number of independent tasks to process Tasks are atomic Tasks have same size Single data repository One master initially holds data for all tasks Several workers arranged along a star, a tree or a general graph

Yves Robert Scheduling for Heterogeneous Platforms 31/ 86

slide-45
SLIDE 45

Master-worker on heterogeneous platforms

Application examples

Monte Carlo methods SETI@home Factoring large numbers Searching for Mersenne primes Particle detection at CERN (LHC@home) ... and many others: see BOINC at http://boinc.berkeley.edu

Yves Robert Scheduling for Heterogeneous Platforms 32/ 86

slide-46
SLIDE 46

Master-worker on heterogeneous platforms

Makespan vs. steady state

Two-different problems Makespan Maximize total number of tasks processed within a time-bound Steady state Determine periodic task allocation which maximizes total throughput

Yves Robert Scheduling for Heterogeneous Platforms 33/ 86

slide-47
SLIDE 47

Master-worker on heterogeneous platforms

Example

  • ✁✂
✄ ☎ ✁ ✄ ✂
  • ✁✂
✄ ☎ ✁ ✄ ✂ ✆ ✝ ✞ ✟ ✠ ✝ ✡ ✆ ✝ ✞ ✟ ✠ ✝ ✡ ☛ ☞ ✌ ✍ ✎ ✏ ✑ ☛ ☞ ✌ ✍ ✎ ✏ ✑ ✒ ✓ ✔ ✕ ✒ ✓ ✔ ✕ ✖ ✗ ✖ ✗ ✘ ✙ ✚ ✛✢✜ ✣ ✤ ✥ ✘ ✙ ✚ ✛ ✜ ✣ ✤ ✥ ✦ ✧ ★ ✧ ✩ ★ ✧ ✪ ★ ✩ ✫✬ ✪ ✬ ✦ ✧ ★ ✧ ✩ ★ ✧ ✪ ★ ✩ ✫✬ ✪ ✬ ✭ ✮ ✯✱✰ ✲ ✳ ✴ ✵ ✶ ✷✢✸ ✹ ✺ ✻ ✼✽ ✾✿❀ ❁ ✾ ❂ ❃ ❄ ❅ ❆❇ ❈❉ ❊ ❋ ❊
  • ❇
❉ ❊ ❍ ■ ❆❏ ❑ ▲ ▼ ◆ ❖ P ◗ ❘ ❙ ❚ ❯ ▼ ❯ ❱ ◗ ◆ ❚ ❲ ◆ ❯ ❳ ❘❨ ❳ ❖ ❱ ▼ ❯ ❳ ◆ ◗ ❖ ❳ ❚ ❩ ▼ ◆ ❩ ◗ ❨ ❬✢❭ ❯ ❳ ❯ ◗ ◗

Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-48
SLIDE 48

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

Time for computing Time for computing

  • ne task in C
  • ne task in C

Time for sending Time for sending

  • ne task from A to B
  • ne task from A to B

A is the root of the tree; A is the root of the tree; all tasks start at A all tasks start at A Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-49
SLIDE 49

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute

✠✡ ☛ ☞ ✠✡ ☛ ☞ ✌ ✌

1 1 2 2 3 3 Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-50
SLIDE 50

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute

✠✡ ☛ ☞ ✠✡ ☛ ☞ ✌ ✌

1 1 2 2 3 3 Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-51
SLIDE 51

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute

✠✡ ☛ ☞ ✠✡ ☛ ☞ ✌ ✌

1 1 2 2 3 3 Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-52
SLIDE 52

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute

✠✡ ☛ ☞ ✠✡ ☛ ☞ ✌ ✌

1 1 2 2 3 3 Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-53
SLIDE 53

Master-worker on heterogeneous platforms

Example

  • ✁
✁ ✂ ✂ ✄ ✄ ☎ ☎ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute

✠✡ ☛ ☞ ✠✡ ☛ ☞ ✌ ✌

1 1 2 2 3 3 Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-54
SLIDE 54

Master-worker on heterogeneous platforms

Example

A compute A compute A send A send B receive B receive B compute B compute C receive C receive C compute C compute C send C send D receive D receive D compute D compute 1 1 2 2 3 3 Startup Startup

Repeated Repeated pattern pattern

Clean Clean-

  • up

up

Steady Steady-

  • state: 7 tasks every 6 time units

state: 7 tasks every 6 time units

  • ✁
✁ ✂ ✂ ✄ ✄ ✄ ✄ ☎ ☎ ✆ ✆ ✝ ✝ ✞ ✞ ✟ ✟

Yves Robert Scheduling for Heterogeneous Platforms 34/ 86

slide-55
SLIDE 55

Master-worker on heterogeneous platforms

Solution for star-shaped platforms

  • ✁
✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✂ ✂ ✂ ✂ ✂ ✂ ✂ ✂ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ✄ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ☎ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✆ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✝ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✞ ✟ ✟ ✟ ✟ ✟ ✟ ✟ ✟ ✟ ✟ ✠ ✠ ✠ ✠ ✠ ✠ ✠ ✠ ✠ ✠ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ✡ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☛ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ☞ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✌ ✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍ ✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍ ✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍✎✍ ✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏ ✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏ ✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏✎✏ ✑✎✑✎✑✎✑✎✑✎✑✎✑ ✑✎✑✎✑✎✑✎✑✎✑✎✑ ✑✎✑✎✑✎✑✎✑✎✑✎✑ ✒✎✒✎✒✎✒✎✒ ✒✎✒✎✒✎✒✎✒ ✒✎✒✎✒✎✒✎✒

Communication links between master and workers have different bandwidths Workers have different computing power

Yves Robert Scheduling for Heterogeneous Platforms 35/ 86

slide-56
SLIDE 56

Master-worker on heterogeneous platforms

Rule of the game

M P1 P2 Pi Pp w1 w2 wi wp ci cp c1 c2

Master sends tasks to workers sequentially, and without preemption Full computation/communication overlap for each worker Worker Pi receives a task in ci time-units Worker Pi processes a task in wi time-units

Yves Robert Scheduling for Heterogeneous Platforms 36/ 86

slide-57
SLIDE 57

Master-worker on heterogeneous platforms

Equations

M P1 P2 Pi Pp w1 w2 wi wp ci cp c1 c2

Worker Pi executes αi tasks per time-unit Computations: αiwi ≤ 1 Communications:

i αici ≤ 1

Objective: maximize throughput ρ =

  • i

αi

Yves Robert Scheduling for Heterogeneous Platforms 36/ 86

slide-58
SLIDE 58

Master-worker on heterogeneous platforms

Solution

Faster-communicating workers first: c1 ≤ c2 ≤ . . . Make full use of first q workers, where q largest index s.t.

q

  • i=1

ci wi ≤ 1 Make partial use of next worker Pq+1 Discard other workers Bandwidth-centric strategy

  • Delegate work to the fastest communicating workers
  • It doesn’t matter if these workers are computing slowly
  • Of course, slow workers will not contribute much to the overall

throughput

Yves Robert Scheduling for Heterogeneous Platforms 37/ 86

slide-59
SLIDE 59

Master-worker on heterogeneous platforms

Example

Fully active

2 10 20 1 3 6 1 1 1

M

3

Discarded

Tasks Communication Computation 6 tasks to P1 6c1 = 6 6w1 = 18 3 tasks to P2 3c2 = 6 3w2 = 18 2 tasks to P3 2c3 = 6 2w3 = 2 11 tasks every 18 time-units (ρ = 11/18 ≈ 0.6)

Yves Robert Scheduling for Heterogeneous Platforms 38/ 86

slide-60
SLIDE 60

Master-worker on heterogeneous platforms

Example

Fully active

2 10 20 1 3 6 1 1 1

M

3

Discarded

Compare to purely greedy (demand-driven) strategy!

5 tasks every 36 time-units (ρ = 5/36 ≈ 0.14) Even if resources are cheap and abundant, resource selection is key to performance

Yves Robert Scheduling for Heterogeneous Platforms 38/ 86

slide-61
SLIDE 61

Master-worker on heterogeneous platforms

Extension to trees

1 4 3 2 1 1 1 3 1 4 2 2 2 2 2 3 3 3 3

40 39

9 5 5 5

40 39

9 5 6 6 6 6 2 6 6 6 6 5 5 9 9 5 6 6 6 6 2

10 7

5

40 67

10

1 2 1 2 1 2 1 2 1 2 1 2 1 2 1 2

1

90 53

Yves Robert Scheduling for Heterogeneous Platforms 39/ 86

slide-62
SLIDE 62

Master-worker on heterogeneous platforms

Extension to trees

Fully used node Partially used node Idle node

1 4 3 2 1 1 1 3 2 5 5 5 2 6 6 6 6 5 5 9 10

1 2 1 2

Resource selection based on local information (children)

Yves Robert Scheduling for Heterogeneous Platforms 39/ 86

slide-63
SLIDE 63

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Can we deal with return messages? In fact, can we deal with more complex applications (arbitrary collections of DAGs)?

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-64
SLIDE 64

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Yes Can we deal with return messages? In fact, can we deal with more complex applications (arbitrary collections of DAGs)?

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-65
SLIDE 65

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Yes Can we deal with return messages? In fact, can we deal with more complex applications (arbitrary collections of DAGs)?

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-66
SLIDE 66

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Yes Can we deal with return messages? Yes In fact, can we deal with more complex applications (arbitrary collections of DAGs)?

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-67
SLIDE 67

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Yes Can we deal with return messages? Yes In fact, can we deal with more complex applications (arbitrary collections of DAGs)?

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-68
SLIDE 68

Master-worker on heterogeneous platforms

Does this really work?

Can we deal with arbitrary platforms (including cycles)? Yes Can we deal with return messages? Yes In fact, can we deal with more complex applications (arbitrary collections of DAGs)? Yes, I mean, almost!

Yves Robert Scheduling for Heterogeneous Platforms 40/ 86

slide-69
SLIDE 69

Master-worker on heterogeneous platforms

LP formulation still works well . . .

Tm file emn Pj Pi Pk wi cik cji Tn

Conservation law ∀m, n

  • j

sent(Pj → Pi, emn) + executed(Pi, Tm) = executed(Pi, Tn) +

  • k

sent(Pi → Pk, emn) Computations

  • m

executed(Pi, Tm) × flops(Tm) × wi ≤ 1 Outgoing communications

  • m,n
  • j

sent(Pj → Pi, emn) × bytes(emn) × cij ≤ 1

Yves Robert Scheduling for Heterogeneous Platforms 41/ 86

slide-70
SLIDE 70

Master-worker on heterogeneous platforms

. . . but schedule reconstruction is harder

P1 → P2 P2 → P1 P1 → P3 P3 → P1 P2 → P4 P4 → P2 P3 → P4 P4 → P3 P2 → P3 P3 → P2 P4 P3 P2 P1

  • χ3

χ4

  • χ1

χ2

  • χ3

χ4

  • χ1

χ2

  • χ3

χ4

  • χ1

χ2

  • χ3

χ4

  • χ1

χ2

  • 40

80 120 160 A5 A4 A3 A2 A1

Actual (cyclic) schedule obtained in polynomial time Asymptotic optimality A couple of practical problems (large period, # buffers) No local scheduling policy

Yves Robert Scheduling for Heterogeneous Platforms 42/ 86

slide-71
SLIDE 71

Master-worker on heterogeneous platforms

The beauty of steady-state scheduling

Rationale Maximize throughput (total load executed per period) Simplicity Relaxation of makespan minimization problem Ignore initialization and clean-up phases Precise ordering/allocation of tasks/messages not needed Characterize resource activity during each time-unit:

  • which (rational) fraction of time is spent

computing for which application?

  • which (rational) fraction of time is spent receiving
  • r sending to which neighbor?

Efficiency Optimal throughput ⇒ optimal schedule (up to a constant number of tasks) Periodic schedule, described in compact form ⇒ compiling a loop instead of a DAG!

Yves Robert Scheduling for Heterogeneous Platforms 43/ 86

slide-72
SLIDE 72

Master-worker on heterogeneous platforms

The beauty of steady-state scheduling

Rationale Maximize throughput (total load executed per period) Simplicity Relaxation of makespan minimization problem Ignore initialization and clean-up phases Precise ordering/allocation of tasks/messages not needed Characterize resource activity during each time-unit:

  • which (rational) fraction of time is spent

computing for which application?

  • which (rational) fraction of time is spent receiving
  • r sending to which neighbor?

Efficiency Optimal throughput ⇒ optimal schedule (up to a constant number of tasks) Periodic schedule, described in compact form ⇒ compiling a loop instead of a DAG!

Yves Robert Scheduling for Heterogeneous Platforms 43/ 86

slide-73
SLIDE 73

Master-worker on heterogeneous platforms

Bibliography – Master-worker tasking

Steady-state scheduling: Scheduling strategies for master-worker tasking on heterogeneous processor platforms, C. Banino et al., IEEE TPDS 15, 4 (2004), 319-330 With bounded multi-port model: Distributed adaptive task allocation in heterogeneous computing environments to maximize throughput, B. Hong and V.K. Prasanna, IEEE IPDPS (2004), 52b With several applications: Centralized versus distributed schedulers for multiple bag-of-task applications, presented yesterday!

Yves Robert Scheduling for Heterogeneous Platforms 44/ 86

slide-74
SLIDE 74

Broadcast

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 45/ 86

slide-75
SLIDE 75

Broadcast

Broadcasting data

Key collective communication operation Start: one processor has the data End: all processors own a copy Vast literature about broadcast, MPI Bcast Standard approach: use a spanning tree Finding the best spanning tree: NP-Complete problem (even in the telephone model)

Yves Robert Scheduling for Heterogeneous Platforms 46/ 86

slide-76
SLIDE 76

Broadcast

Broadcasting data

Key collective communication operation Start: one processor has the data End: all processors own a copy Vast literature about broadcast, MPI Bcast Standard approach: use a spanning tree Finding the best spanning tree: NP-Complete problem (even in the telephone model)

Yves Robert Scheduling for Heterogeneous Platforms 46/ 86

slide-77
SLIDE 77

Broadcast

Broadcasting data

Key collective communication operation Start: one processor has the data End: all processors own a copy Vast literature about broadcast, MPI Bcast Standard approach: use a spanning tree Finding the best spanning tree: NP-Complete problem (even in the telephone model)

Yves Robert Scheduling for Heterogeneous Platforms 46/ 86

slide-78
SLIDE 78

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 2

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-79
SLIDE 79

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 (0) 2

Next node: minimize (Ri) + cij, Pj / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-80
SLIDE 80

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 (3) (3) 2

Next node: minimize (Ri) + cij, Pj / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-81
SLIDE 81

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 (3) (6) (6) 2

Next node: minimize (Ri) + cij, Pj / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-82
SLIDE 82

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 (3) (6) 2 (7) (7)

Next node: minimize (Ri) + cij, Pj / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-83
SLIDE 83

Broadcast

Heuristic: Earliest completing edge first (ECEF)

3 3 3 1 6 4 4 2 2 (0) (3) (6) (7) (9) (9) Broadcast finishing times (t)

Yves Robert Scheduling for Heterogeneous Platforms 47/ 86

slide-84
SLIDE 84

Broadcast

Heuristic: Look-ahead (LA)

3 3 3 1 6 4 4 2 (0) 2 (3) (1) (1)

Next node: minimize (Ri) + cij + (min cjk), Pj, Pk / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 48/ 86

slide-85
SLIDE 85

Broadcast

Heuristic: Look-ahead (LA)

3 3 3 1 6 4 4 2 2 (4) (4) (3) (2) (4) Next node: minimize (Ri) + cij + (min cjk), Pj, Pk / ∈ T

Yves Robert Scheduling for Heterogeneous Platforms 48/ 86

slide-86
SLIDE 86

Broadcast

Heuristic: Look-ahead (LA)

3 3 3 1 6 4 4 2 2 (0) (7) (7) (4) (5) (7)

Broadcast finishing times (t)

Yves Robert Scheduling for Heterogeneous Platforms 48/ 86

slide-87
SLIDE 87

Broadcast

Broadcasting longer messages

Message size goes from L to, say, 10L Communication costs scale from cij to 10cij ECEF heuristic: broadcast time becomes 90 LA heuristic: broadcast time becomes 70

Yves Robert Scheduling for Heterogeneous Platforms 49/ 86

slide-88
SLIDE 88

Broadcast

Broadcasting longer messages

Message size goes from L to, say, 10L Communication costs scale from cij to 10cij ECEF heuristic: broadcast time becomes 90 LA heuristic: broadcast time becomes 70

Yves Robert Scheduling for Heterogeneous Platforms 49/ 86

slide-89
SLIDE 89

Broadcast

Broadcasting longer messages

Message size goes from L to, say, 10L Communication costs scale from cij to 10cij ECEF heuristic: broadcast time becomes 90 LA heuristic: broadcast time becomes 70

Eh wait! What about PIPELINING?!

Yves Robert Scheduling for Heterogeneous Platforms 49/ 86

slide-90
SLIDE 90

Broadcast

Broadcasting longer messages

3 3 3 1 6 4 4 2 2

3 2 10 1

...

size = 10L

Search spanning tree . . . Objective: minimize pipelined execution time

Yves Robert Scheduling for Heterogeneous Platforms 50/ 86

slide-91
SLIDE 91

Broadcast

Broadcasting longer messages

3 3 3 1 6 4 4 2 2

3 2 10 1

...

size = 10L

Delay = inverse of throughput Node delay =

children of node comm. times

Tree delay = maximum node delay Pipelined execution time: (# edges in longest path + #packets) × tree delay Objective: minimize tree delay

Yves Robert Scheduling for Heterogeneous Platforms 51/ 86

slide-92
SLIDE 92

Broadcast

Back to the example

3 3 3 1 6 4 4 2 2 3 3 3 1 6 4 4 2 2 delay = 7 delay = 3 ECEF tree LA tree

ECEF tree turns out to have minimum delay (maximal throughput) Can we always find tree with optimal throughput?

Problem is NP-complete Still, can design simple heuristics:

SDIEF: smallest-delay-increase edge first

Yves Robert Scheduling for Heterogeneous Platforms 52/ 86

slide-93
SLIDE 93

Broadcast

Back to the example

3 3 3 1 6 4 4 2 2 3 3 3 1 6 4 4 2 2 delay = 7 delay = 3 ECEF tree LA tree

ECEF tree turns out to have minimum delay (maximal throughput) Can we always find tree with optimal throughput?

Problem is NP-complete Still, can design simple heuristics:

SDIEF: smallest-delay-increase edge first

Yves Robert Scheduling for Heterogeneous Platforms 52/ 86

slide-94
SLIDE 94

Broadcast

Back to the example

3 3 3 1 6 4 4 2 2 3 3 3 1 6 4 4 2 2 delay = 7 delay = 3 ECEF tree LA tree

ECEF tree turns out to have minimum delay (maximal throughput) Can we always find tree with optimal throughput?

Problem is NP-complete Still, can design simple heuristics:

SDIEF: smallest-delay-increase edge first

Yves Robert Scheduling for Heterogeneous Platforms 52/ 86

slide-95
SLIDE 95

Broadcast

Back to the example

3 3 3 1 6 4 4 2 2 3 3 3 1 6 4 4 2 2 delay = 7 delay = 3 ECEF tree LA tree

ECEF tree turns out to have minimum delay (maximal throughput) Can we always find tree with optimal throughput?

Problem is NP-complete Still, can design simple heuristics:

SDIEF: smallest-delay-increase edge first

Yves Robert Scheduling for Heterogeneous Platforms 52/ 86

slide-96
SLIDE 96

Broadcast

Assessing a broadcast strategy Finding optimal set of spanning trees is polynomial:

use LP formulation!

Schedule reconstruction and packet management is harder with

several trees Suggested approach:

◮ Compute optimal throughput (several trees) with LP formulation ◮ Run preferred heuristic to generate one or several ”good” spanning

trees

◮ Stop refining when performance “reasonably” close to upper bound

Will outperform MPI binomial spanning tree!

Yves Robert Scheduling for Heterogeneous Platforms 53/ 86

slide-97
SLIDE 97

Broadcast

Assessing a broadcast strategy Finding optimal set of spanning trees is polynomial:

use LP formulation!

Schedule reconstruction and packet management is harder with

several trees Suggested approach:

◮ Compute optimal throughput (several trees) with LP formulation ◮ Run preferred heuristic to generate one or several ”good” spanning

trees

◮ Stop refining when performance “reasonably” close to upper bound

Will outperform MPI binomial spanning tree!

Yves Robert Scheduling for Heterogeneous Platforms 53/ 86

slide-98
SLIDE 98

Broadcast

Bibliography – Broadcast

Complexity: On broadcasting in heterogeneous networks, S. Khuller and Y.A. Kim, 15th ACM SODA (2004), 1011–1020 Heuristics: Efficient collective communication in distributed heterogeneous systems, P.B. Bhat, C.S. Raghavendra and V.K. Prasanna, JPDC 63 (2003), 251–263 Steady-state: Pipelining broadcasts on heterogeneous platforms, O. Beaumont et al., IEEE TPDS 16, 4 (2005), 300-313

Yves Robert Scheduling for Heterogeneous Platforms 54/ 86

slide-99
SLIDE 99

Limitations Parameters

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations Parameters Communication model Topology hierarchy

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 55/ 86

slide-100
SLIDE 100

Limitations Parameters

Good news and bad news One-port model: first step towards designing realistic

scheduling heuristics

Steady-state circumvents complexity of scheduling problems

. . . while deriving efficient (often asympotically optimal) scheduling algorithms

Need to acquire a good knowledge of the platform graph Need to run extensive experiments or simulations

Yves Robert Scheduling for Heterogeneous Platforms 56/ 86

slide-101
SLIDE 101

Limitations Parameters

Good news and bad news One-port model: first step towards designing realistic

scheduling heuristics

Steady-state circumvents complexity of scheduling problems

. . . while deriving efficient (often asympotically optimal) scheduling algorithms

Need to acquire a good knowledge of the platform graph Need to run extensive experiments or simulations

Yves Robert Scheduling for Heterogeneous Platforms 56/ 86

slide-102
SLIDE 102

Limitations Parameters

Knowledge of the platform graph

For regular problems, the structure of the task graph (nodes and edges) only depends upon the application, not upon the target platform Problems arise from weights, i.e. the estimation of execution and communication times Classical answer: “use the past to predict the future” Divide scheduling into phases, during which machine and network parameters are collected (with NWS) ⇒ This information guides scheduling decisions for next phase Moving from heterogeneous clusters to computational grids causes further problems (even discovering the characteristics of the surrounding computing resources may prove a difficult task)

Yves Robert Scheduling for Heterogeneous Platforms 57/ 86

slide-103
SLIDE 103

Limitations Parameters

Knowledge of the platform graph

For regular problems, the structure of the task graph (nodes and edges) only depends upon the application, not upon the target platform Problems arise from weights, i.e. the estimation of execution and communication times Classical answer: “use the past to predict the future” Divide scheduling into phases, during which machine and network parameters are collected (with NWS) ⇒ This information guides scheduling decisions for next phase Moving from heterogeneous clusters to computational grids causes further problems (even discovering the characteristics of the surrounding computing resources may prove a difficult task)

Yves Robert Scheduling for Heterogeneous Platforms 57/ 86

slide-104
SLIDE 104

Limitations Parameters

Knowledge of the platform graph

For regular problems, the structure of the task graph (nodes and edges) only depends upon the application, not upon the target platform Problems arise from weights, i.e. the estimation of execution and communication times Classical answer: “use the past to predict the future” Divide scheduling into phases, during which machine and network parameters are collected (with NWS) ⇒ This information guides scheduling decisions for next phase Moving from heterogeneous clusters to computational grids causes further problems (even discovering the characteristics of the surrounding computing resources may prove a difficult task)

Yves Robert Scheduling for Heterogeneous Platforms 57/ 86

slide-105
SLIDE 105

Limitations Parameters

Experiments versus simulations

Real experiments difficult to drive (genuine instability of non-dedicated platforms) Simulations ensure reproducibility of measured data Key issue: run simulations against a realistic environment Trace-based simulation: record platform parameters today, and simulate the algorithms tomorrow, against recorded data Use SIMGRID, an event-driven simultation toolkit

Yves Robert Scheduling for Heterogeneous Platforms 58/ 86

slide-106
SLIDE 106

Limitations Parameters

Experiments versus simulations

Real experiments difficult to drive (genuine instability of non-dedicated platforms) Simulations ensure reproducibility of measured data Key issue: run simulations against a realistic environment Trace-based simulation: record platform parameters today, and simulate the algorithms tomorrow, against recorded data Use SIMGRID, an event-driven simultation toolkit

Yves Robert Scheduling for Heterogeneous Platforms 58/ 86

slide-107
SLIDE 107

Limitations Parameters

SIMGRID traces

server #1 client #1 client #2 client #3 server #2

router switch hub Internet CPU availability Network bandwidth Transient Failure X

See http://simgrid.gforge.inria.fr/

Yves Robert Scheduling for Heterogeneous Platforms 59/ 86

slide-108
SLIDE 108

Limitations Communication model

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations Parameters Communication model Topology hierarchy

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 60/ 86

slide-109
SLIDE 109

Limitations Communication model

Across physical links

Network = directed graph P = (V, E)

P0 P1 P3 P2

time recv 2,3 P3 T2,3(L) link e2,3 send 2,3 P2

General case: affine model (includes latencies) Common variant: sending and receiving processors busy during whole transfer

Yves Robert Scheduling for Heterogeneous Platforms 61/ 86

slide-110
SLIDE 110

Limitations Communication model

Across physical links

Network = directed graph P = (V, E)

P0 P1 P3 P2

time r2,3 r2,3 · L P3 α2,3 β2,3 · L link e2,3 s2,3 · L s2,3 P2

General case: affine model (includes latencies) Common variant: sending and receiving processors busy during whole transfer

Yves Robert Scheduling for Heterogeneous Platforms 61/ 86

slide-111
SLIDE 111

Limitations Communication model

Across physical links

Network = directed graph P = (V, E)

P0 P1 P3 P2

time recv 2,3 P3 T2,3(L) link e2,3 send 2,3 P2

General case: affine model (includes latencies) Common variant: sending and receiving processors busy during whole transfer

Yves Robert Scheduling for Heterogeneous Platforms 61/ 86

slide-112
SLIDE 112

Limitations Communication model

Multi-port

Bar-Noy, Guha, Naor, Schieber:

  • ccupation time of sender Pu independent of target Pv

time recv v Pv Tu,v(L) link eu,v send u Pu not fully multi-port model, but allows for starting a new transfer from Pu without waiting for previous one to finish

Yves Robert Scheduling for Heterogeneous Platforms 62/ 86

slide-113
SLIDE 113

Limitations Communication model

One-port

Bhat, Raghavendra and Prasanna: same parameters for sender Pu, link eu,v and receiver Pv

time ru,v · L ru,v Pv βu,v · L αu,v link eu,v su,v · L su,v Pu two flavors:

  • bidirectional: simultaneous send and receive transfers allowed
  • unidirectional: only one send or receive transfer at a given time-step

Yves Robert Scheduling for Heterogeneous Platforms 63/ 86

slide-114
SLIDE 114

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path?

Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-115
SLIDE 115

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path?

S l1 l3 l2 Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-116
SLIDE 116

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path?

S l1 l3 l2

Store & Forward : bad model for contention

Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-117
SLIDE 117

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path?

S l1 l3 l2 Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-118
SLIDE 118

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path?

pi,j s S l1 l3 l2

WormHole : computation intensive (packets), not that realistic

Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-119
SLIDE 119

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path? ∀l ∈ L,

  • r∈R s.t. l∈r

ρr ≤ cl Analytical model

Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-120
SLIDE 120

Limitations Communication model

Store & Forward, WormHole, TCP

How to model a file transfer along a path? ∀l ∈ L,

  • r∈R s.t. l∈r

ρr ≤ cl Max-Min Fairness maximize min

r∈R ρr

Proportional Fairness maximize

  • r∈R

ρr log(ρr) MCT minimization maximize min

r∈R

1 ρr TCP behavior Close to max-min. In SIMGRID: max-min + bound by 1/RTT

Yves Robert Scheduling for Heterogeneous Platforms 64/ 86

slide-121
SLIDE 121

Limitations Communication model

Bandwidth sharing

Traditional assumption: Fair Sharing Open i TCP connections, receive bw(i) bandwidth per connection bw(i) = bw(1)/i on a LAN Experimental evidence → bw(i) = bw(1) on a WAN Backbone links have so many connections that interference among a few selected connections is negligible Better model: bw(i) = bw(1) 1 + (i − 1).γ γ = 1 for a perfect LAN, γ = 0 for a perfect WAN

Yves Robert Scheduling for Heterogeneous Platforms 65/ 86

slide-122
SLIDE 122

Limitations Communication model

Bandwidth sharing

Traditional assumption: Fair Sharing Open i TCP connections, receive bw(i) bandwidth per connection bw(i) = bw(1)/i on a LAN Experimental evidence → bw(i) = bw(1) on a WAN Backbone links have so many connections that interference among a few selected connections is negligible Better model: bw(i) = bw(1) 1 + (i − 1).γ γ = 1 for a perfect LAN, γ = 0 for a perfect WAN

Yves Robert Scheduling for Heterogeneous Platforms 65/ 86

slide-123
SLIDE 123

Limitations Communication model

Bandwidth sharing

Traditional assumption: Fair Sharing Open i TCP connections, receive bw(i) bandwidth per connection bw(i) = bw(1)/i on a LAN Experimental evidence → bw(i) = bw(1) on a WAN Backbone links have so many connections that interference among a few selected connections is negligible Better model: bw(i) = bw(1) 1 + (i − 1).γ γ = 1 for a perfect LAN, γ = 0 for a perfect WAN

Yves Robert Scheduling for Heterogeneous Platforms 65/ 86

slide-124
SLIDE 124

Limitations Topology hierarchy

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations Parameters Communication model Topology hierarchy

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 66/ 86

slide-125
SLIDE 125

Limitations Topology hierarchy

Sample large-scale platform

Primergy Primergy

backbone link router front end cluster Yves Robert Scheduling for Heterogeneous Platforms 67/ 86

slide-126
SLIDE 126

Limitations Topology hierarchy

What topology?

Generated (GT-ITM, BRITE, etc.) or obtained from monitoring?

◮ Very complex (Layer 2 information) ◮ Not clear that a scheduling algorithm could exploit/know all that

information

Need a simple model that is

◮ More accurate than traditional models (e.g., LAN links,

fully-connected)

◮ Still amenable to analysis Yves Robert Scheduling for Heterogeneous Platforms 68/ 86

slide-127
SLIDE 127

Limitations Topology hierarchy

What topology?

Generated (GT-ITM, BRITE, etc.) or obtained from monitoring?

◮ Very complex (Layer 2 information) ◮ Not clear that a scheduling algorithm could exploit/know all that

information

Need a simple model that is

◮ More accurate than traditional models (e.g., LAN links,

fully-connected)

◮ Still amenable to analysis Yves Robert Scheduling for Heterogeneous Platforms 68/ 86

slide-128
SLIDE 128

Limitations Topology hierarchy

What topology? (cont’d)

Primergy Primergy

uu γ = 0.5 Unknown topology (complete graph)

Yves Robert Scheduling for Heterogeneous Platforms 69/ 86

slide-129
SLIDE 129

Limitations Topology hierarchy

What topology? (cont’d)

Primergy Primergy

backbone link router front end cluster

Hierarchy + BW sharing, but assume knowledge of Routing Backbone bandwidths CPU speeds

Yves Robert Scheduling for Heterogeneous Platforms 70/ 86

slide-130
SLIDE 130

Limitations Topology hierarchy

A first trial

sk sl gk gl b3 b1 Lk,l b2 Ck Ck

master

Ck

router

Cl

router

Cl

master

Cl

Clusters and backbone links

Yves Robert Scheduling for Heterogeneous Platforms 71/ 86

slide-131
SLIDE 131

Limitations Topology hierarchy

A first trial (cont’d)

sk sl gk gl b3 b1 Lk,l b2 Ck Ck

master

Ck

router

Cl

router

Cl

master

Cl

Clusters K clusters Ck, 1 ≤ k ≤ K Ck

master front-end processor

Ck

router router to external world

sk cumulated speed of Ck gk bandwidth of the LAN link (γ = 1) from Ck

master to Ck router

Yves Robert Scheduling for Heterogeneous Platforms 72/ 86

slide-132
SLIDE 132

Limitations Topology hierarchy

A first trial (cont’d)

sk sl gk gl b3 b1 Lk,l b2 Ck Ck

master

Ck

router

Cl

router

Cl

master

Cl

Network Set R of routers and B of backbone links li bw(li) bandwidth available for a new connection max-connect(li) max. number of connections that can be opened Fixed routing: path Lk,l of backbones from Ck

router to Cl router

Yves Robert Scheduling for Heterogeneous Platforms 72/ 86

slide-133
SLIDE 133

Limitations Topology hierarchy

Bibliography

NWS: The network weather service: a distributed resource performance forecasting service for metacomputing, R. Wolski, N.T. Spring and

  • J. Hayes, Future Generation Computer Systems 15, 10 (1999),

757-768 SIMGRID: Scheduling distributed applications: the SIMGRID simulation framework, A. Legrand, L. Marchal, and H. Casanova, 3rd IEEE CCGrid (2003), 138-145 Bandwidth sharing: Bandwidth sharing: objectives and algorithms, L. Massouli´ e and J. Roberts, IEEE/ACM Trans. Networking 10, 3 (2002), 320-328

Yves Robert Scheduling for Heterogeneous Platforms 73/ 86

slide-134
SLIDE 134

Putting all together

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 74/ 86

slide-135
SLIDE 135

Putting all together

Scheduling multiple applications

Large-scale platforms not likely to be exploited in dedicated mode/single application Investigate scenarios in which multiple applications are simultaneously executed on the platform ⇒ competition for CPU and network resources

Yves Robert Scheduling for Heterogeneous Platforms 75/ 86

slide-136
SLIDE 136

Putting all together

Target problem

1 Large complex platform: several clusters and backbone links 2 One (divisible load) application running on each cluster 3 Which fraction of the job to delegate to other clusters? 4 Applications have different communication-to-computation ratios 5 How to ensure fair scheduling and good resource utilization? Yves Robert Scheduling for Heterogeneous Platforms 76/ 86

slide-137
SLIDE 137

Putting all together

Linear program

Maximize mink

  • αk

πk

  • ,

under the constraints                                          (1a) ∀Ck,

  • l

αk,l = αk (1b) ∀Ck,

  • l

αl,k.τl ≤ sk (1c) ∀Ck,

  • l=k

αk,l.δk +

  • j=k

αj,k.δj ≤ gk (1d) ∀li,

  • li∈Lk,l

βk,l ≤ max-connect(li) (1e) ∀k, l, αk,l.δk ≤ βk,l × gk,l (1f) ∀k, l, αk,l ≥ 0 (1g) ∀k, l, βk,l ∈ N (1)

Yves Robert Scheduling for Heterogeneous Platforms 77/ 86

slide-138
SLIDE 138

Putting all together

Approach

Solution to rational linear problem as comparator/upper bound Several heuristics, greedy and LP-based Use Tiers as topology generator, and then SIMGRID

Yves Robert Scheduling for Heterogeneous Platforms 78/ 86

slide-139
SLIDE 139

Putting all together

Methodology (cont’d)

  • 1000

1000 2000 3000 4000 5000

  • 1000

1000 2000 3000 4000 5000 Vertical Distance Horizontal Distance Number of nodes: 641, Number of links:934 Original Network WAN MAN LAN 500 1000 1500 2000 2500 3000 3500 4000 500 1000 1500 2000 2500 Vertical Distance Horizontal Distance Number of nodes: 41, Number of links:45 Pruned Network

distribution K 5, 7, . . . , 90 log(bw(lk)), log(gk) normal (mean= log(2000), std=log(10)) sk uniform, 1000 — 10000 max-connect, δk, τk, πk uniform, 1 — 10 Platform parameters used in simulation

Yves Robert Scheduling for Heterogeneous Platforms 79/ 86

slide-140
SLIDE 140

Putting all together

Hints for implementation

Participants sharing resources in a Virtual Organization Centralized broker managing applications and resources Broker gathers all parameters of LP program Priority factors Various policies and refinements possible ⇒ e.g. fixed number of connections per application

Yves Robert Scheduling for Heterogeneous Platforms 80/ 86

slide-141
SLIDE 141

Putting all together

Bibliography

Tiers: Modeling Internet topology, K. Calvert, M. Doar and E.W. Zegura, IEEE Comm. Magazine 35, 6 (1997), 160-163 Scheduling multiple applications: A realistic network/application model for scheduling divisible loads

  • n large-scale platforms, L. Marchal et al., 19th IEEE IPDPS

(2005)

Yves Robert Scheduling for Heterogeneous Platforms 81/ 86

slide-142
SLIDE 142

Conclusion

Outline

1

Background on traditional scheduling

2

Packet routing

3

Master-worker on heterogeneous platforms

4

Broadcast

5

Limitations

6

Putting all together

7

Conclusion

Yves Robert Scheduling for Heterogeneous Platforms 82/ 86

slide-143
SLIDE 143

Conclusion

Key advantages of steady-state scheduling

Simplicity From local equations to global behavior Throughput characterized from activity variables Efficiency Periodic schedule, described in compact form Asymptotic optimality Adaptability Record observed performance during current period Inject information to compute schedule for next period React on the fly to resource availability variations

Yves Robert Scheduling for Heterogeneous Platforms 83/ 86

slide-144
SLIDE 144

Conclusion

Open problems

Decentralized scheduling

◮ From local strategies to provably good performance? ◮ Adapt Awerbuch-Leighton algorithm for multicommodity flows?

Concurrent scheduling

◮ Multi-criteria and fairness? ◮ Adapt economic models and buzz-words (e.g., Nash equilibrium)? Yves Robert Scheduling for Heterogeneous Platforms 84/ 86

slide-145
SLIDE 145

Conclusion

Scheduling for heterogeneous platforms

If the platform is well identified and relatively stable, try to: (i) accurately model the (expected) hierarchical structure of the platform (ii) design scheduling algorithms well-suited to this hierarchical structure If the platform is not stable enough, or if it evolves too fast, dynamic schedulers are the only option Otherwise, grab the opportunity to inject some static knowledge into dynamic schedulers:

Is this opportunity a niche? Does it encompass a wide range of applications?

Yves Robert Scheduling for Heterogeneous Platforms 85/ 86

slide-146
SLIDE 146

Conclusion

Answer to first comment

Comment Scheduling is “this thing that people in academia like to think about but that people who do real stuff sort of ignore” Answer

Thank you for your attention.

Other comments or questions?

Yves Robert Scheduling for Heterogeneous Platforms 86/ 86

slide-147
SLIDE 147

Conclusion

Answer to first comment

Comment Scheduling is “this thing that people in academia like to think about but that people who do real stuff sort of ignore” Answer

Thank you for your attention.

Other comments or questions?

Yves Robert Scheduling for Heterogeneous Platforms 86/ 86