Starting Workflow Tasks Before Theyre Ready Wladislaw Gusew, Bj - - PowerPoint PPT Presentation
Starting Workflow Tasks Before Theyre Ready Wladislaw Gusew, Bj orn Scheuermann Computer Engineering Group, Humboldt University of Berlin Agenda Introduction Execution semantics Methods and tools Simulation results
Computer Engineering Group, Humboldt University of Berlin
◮ Introduction ◮ Execution semantics ◮ Methods and tools ◮ Simulation results ◮ Experimental results ◮ Conclusion
1 / 21
2 / 21
◮ Directed Acyclic Graph
◮ Executed on distributed
◮ Aggregation and broadcast
◮ Demanding for network
3 / 21
4 / 21
4 / 21
◮ But in reality resources are limited ◮ Execute only a subset of parent tasks concurrently
◮ Congestion of network (all parent tasks have the same priority)
4 / 21
5 / 21
5 / 21
5 / 21
◮ Network congestion can slow down processing even further
◮ High delay to the start of the aggregation task ◮ Low performance and
5 / 21
6 / 21
6 / 21
6 / 21
6 / 21
6 / 21
6 / 21
6 / 21
6 / 21
7 / 21
◮ Operating system instrumentation tool ◮ Enables interception of system calls
◮ Record and evaluate logfiles with
8 / 21
◮ Operating system instrumentation tool ◮ Enables interception of system calls
◮ Record and evaluate logfiles with
0.5 1 1.5 2 2.5 3 0.5 1 1.5 2 2.5 3 Read accesses [MB] Execution progress [108 CPU cycles] Reads by mAdd in a small workflow 0.5 1 1.5 2 2.5 3 3.5 4 4.5 2 4 6 8 10 12 14 16 18 Read accesses [MB] Execution progress [108 CPU cycles] Reads by mAdd in a medium sized workflow 8 / 21
9 / 21
9 / 21
9 / 21
9 / 21
◮ Real task is transparently wrapped ◮ FUSE enables the setup of a virtual
◮ Access to input files is performed
◮ Wrapper is responsible for maintaining
10 / 21
11 / 21
◮ Java-based simulation framework for scientific workflows ◮ Simulates an execution on a Pegasus/HTCondor stack ◮ Use provided Montage workflows with 25, 50, 100, 1000 tasks ◮ Python script conducted DAG transformation of DAX files ◮ Network configured as bottleneck (by bandwidth limitation)
in distributed environments,” in eScience’12.
12 / 21
13 / 21
13 / 21
1 10 100 1000 25 50 100 1000 Total workflow runtime (log.) [s] Number of tasks Simulation results for 50 workers and max-min Normal Split 15% 19% 25% 31%
14 / 21
15 / 21
100 150 200 250 300 350 400 450 5 10 50 100 Total workflow runtime [s] Number of workers Simulation results for Montage100 and min-min Normal Split 10% 14% 26% 25%
16 / 21
17 / 21
50 100 150 200 250 300 350
M i n
i n M a x
i n R
n d
i n H E F T D H E F T R a n d
Total workflow runtime [s] Scheduling algorithm Simulation results for Montage100 on 100 workers Normal Split 25% 27% 28% 25% 17% 34%
18 / 21
◮ Small cluster of up to 10 compute nodes ◮ Intel i7 CPU@ 2.5GHz, 8GB RAM, connected to common
◮ Execute Montage 133 workflow in Pegasus/HTCondor ◮ Network bandwidth was limited on application layer to
◮ 10 repetitions, mean values with 95% confidence intervals
19 / 21
20 40 60 80 100 120 140 160 180 200 1 2 3 4 5 6 7 8 9 10 Total workflow runtime [s] Number of computing nodes Computing cluster results for 1...10 workers Original Montage133 Transformed Montage133
20 / 21
◮ Many ”legacy” workflows exist which are executed with classic
◮ Our approach is applicable to aggregation tasks that are often
◮ By using DAG transformation, no changes to task
21 / 21
◮ Many ”legacy” workflows exist which are executed with classic
◮ Our approach is applicable to aggregation tasks that are often
◮ By using DAG transformation, no changes to task
◮ Simulation and real experiment show that performance can be
◮ Potential of outperforming the original workflow grows with
21 / 21