Finding allocations for Budget-constrained PTG workflow applications - - PowerPoint PPT Presentation

Context Models Proposed solution Experimentation Conclusions and perspectives

Finding allocations for Budget-constrained PTG workflow applications

Fr´ ed´ eric Desprez1, Eddy Caron1, Adrian Mures

,an1, Fr´

ed´ eric Suter2

1Ecole Normale Sup´

erieure de Lyon, France

2IN2P3 Computing Center, CNRS, IN2P3

7th Scheduling for Large Scale Systems Workshop

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 1/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Outline

1

Context

2

Models

3

Proposed solution

4

Experimentation

5

Conclusions and perspectives

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 2/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Scientific workflow applications Montage (Astronomy) RAMSES (Astronomy) Epigenomics (Bioinformatics) Climate / ocean current / tectonic plate modeling . . . Characteristics some have sequential and parallel tasks some have non-deterministic transitions

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 3/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Our goal consider a most general model of the applications consider on-demand resources and a budget limit find a good allocation strategy Why on-demand resources? more efficient resource usage eliminate overbooking of resources

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 4/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Outline

1

Context

2

Models

3

Proposed solution

4

Experimentation

5

Conclusions and perspectives

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 5/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Application model

Non-deterministic workflows An application is a graph G = (V, E), where V = {vi|i = 1, . . . , |V |} is a set of vertexes E = {ei,j|(i, j) ∈ {1, . . . , |V |} × {1, . . . , |V |}} is a set of edges representing precedence and flow constraints Vertexes a computation [parallel, moldable] an OR-split vertex [transitions described by random variables] an OR-join vertex

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 6/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Example workflow

Figure: Example workflow

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 7/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Platform model A provider of on-demand resources from a catalog: C = {vmi = (nCPUi, costi)|i ≥ 1} nCPU represents the number of equivalent virtual CPUs cost represents a monetary cost per running hour Makespan C = maxi C(vi) is the global makespan where C(vi) is the finish time of task vi ∈ V Cost of a schedule S Cost =

∀vmi∈S⌈Tendi − Tstarti⌉ × costi

Tstarti, Tendi represent the start end end times of vmi costi is the catalog cost of virtual resource vmi

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 8/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Problem statement Given G a workflow application C a provider of resources from the catalog B a budget find a schedule S such that Cost ≤ B budget limit is not passed C (makespan) is minimized with a predefined confidence.

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 9/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Outline

1

Context

2

Models

3

Proposed solution

4

Experimentation

5

Conclusions and perspectives

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 10/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Proposed approach

1 Decompose the non-DAG workflow into DAG sub-workflows 2 Distribute the budget to the sub-workflows 3 Determine allocations by adapting an existing allocation

approach

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 11/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Step 1: Decomposing the workflow

Figure: Decomposing a nontrivial workflow

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 12/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Step 2: Allocating budget

Give each sub-workflow a ratio of the budget proportional to its work contribution. Work contribution of a sub-workflow Gi as the sum of the average execution times of its tasks average execution time computed over the catalog C task speedup model is taken into consideration multiple executions of a sub-workflow also considered

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 13/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Step 3: Determining allocations

Two algorithms based on the bi-CPA algorithm. Eager algorithm

• ne allocation for each task

good trade-off between makespan and average time-cost area fast algorithm considers allocation-time cost estimations only Deferred algorithm

• utputs multiple allocations for each task

good trade-off between makespan and average time-cost area slower algorithm

• ne allocation is chosen at scheduling time
• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 14/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Outline

1

Context

2

Models

3

Proposed solution

4

Experimentation

5

Conclusions and perspectives

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 15/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Methodology Used synthetic workflows for three types of applications

Fast Fourier Transform Strassen matrix multiplication Random workloads

Used a virtual resource catalog inspired by Amazon EC2 Used a classic list-scheduler for task mapping Measured

Cost and makespan after task mapping

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 16/21

Context Models Proposed solution Experimentation Conclusions and perspectives

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1 5 10 15 20 25 30 35 40 45 50 Relative makespan Budget limit Equality

Figure: Relative makespan (

Eager Deferred ) for all workflow applications

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 17/21

Context Models Proposed solution Experimentation Conclusions and perspectives

1 2 3 4 5 1 5 10 15 20 25 30 35 40 45 Relative cost Budget limit Equality

Figure: Relative cost ( Eager

Deferred ) for all workflow applications

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 18/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Outline

1

Context

2

Models

3

Proposed solution

4

Experimentation

5

Conclusions and perspectives

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 19/21

Context Models Proposed solution Experimentation Conclusions and perspectives

Conclusions allocations for non-DAG workflow apps that target Cloud platform proposed two algorithms – Eager and Deferred Eager is fast but cannot guarantee budget constraint after mapping Deferred is slower, but guarantees budget constraint After a certain budget they yield identical allocations Perspectives implement the two using an existing Cloud platform determine per application type which is the tipping point

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 20/21

Budget distribution algorithm

1: ω∗ ← 0 2: for all Gi = (Vi, Ei) ⊆ G do 3:

nExeci ← CDF −1(Di, Confidence)

4:

ωi ←

• vj∈Vi

  1 |C|

• vmk∈C

T(vj, vmk)   × nExeci

5:

ω∗ ← ω∗ + ωi

6: end for 7: for all Gi ⊆ G do 8:

Bi ← B × ωi

ω∗ × 1 nExeci

9: end for

Algorithm 1: Share Budget(B, G, Confidence)

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 22/21

Algorithm parameters

TA over T over

A

= 1 B′ ×

|Vi|

• j=1

(T(vj, Alloc(vj)) × cost(vj)) , TA under T under

A

= 1 B′ ×

|Vi|

• j=1

(T(vj, Alloc(vj)) × costunder(vj))

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 23/21

The eager allocation algorithm

1: for all v ∈ Vi do 2:

Alloc(v) ← {minvmi∈C CPUi}

3: end for 4: Compute B′ 5: while TCP > T over

A

∩ |Vi|

j=1 cost(vj) ≤ Bi do

6:

for all vi ∈ Critical Path do

7:

Determine Alloc′(vi) such that p′(vi) = p(vi) + 1

8:

Gain(vi) ← T(vi,Alloc(vi))

p(vi)

− T(vi,Alloc′(vi))

p′(vi)

9:

end for

10:

Select v such that Gain(v) is maximal

11:

Alloc(v) ← Alloc′(v)

12:

Update T over

A

and TCP

13: end while

Algorithm 2: Eager-allocate(Gi = (Vi, Ei), Bi)

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 24/21

Absolute makespans

00:00:00 00:10:00 00:20:00 00:30:00 00:40:00 00:50:00 01:00:00 01:10:00 01:20:00 1 5 10 15 20 25 30 35 40 45 Makespan (HH:MM:SS) Budget limit

(a) Eager for all workflow applica- tions

00:00:00 00:10:00 00:20:00 00:30:00 00:40:00 00:50:00 01:00:00 01:10:00 01:20:00 1 5 10 15 20 25 30 35 40 45 Makespan (HH:MM:SS) Budget limit

(b) Deferred for all workflow ap- plications

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 25/21

Absolute cost

10 20 30 40 50 1 5 10 15 20 25 30 35 40 45 50 Cost Budget limit Budget limit

(c) Eager for all workflow applica- tions

10 20 30 40 50 1 5 10 15 20 25 30 35 40 45 50 Cost Budget limit Budget limit

(d) Deferred for all workflow ap- plications

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 26/21

Ending with a boom!

• F. Desprez, E. Caron, A. Mures

,an, F. Suter

Budget-constrained workflow scheduling 27/21