Makeflow Work Local Condor Torque Queue W W Makefile - - PowerPoint PPT Presentation

▶
makeflow
SMART_READER_LITE
LIVE PREVIEW

Makeflow Work Local Condor Torque Queue W W Makefile - - PowerPoint PPT Presentation

May 27, 2023 120 likes •629 views

Makeflow Work Local Condor Torque Queue W W Makefile FutureGrid Private Torque W Cluster Cluster W W W Thousands of Workers in a Makeflow Personal Cloud W W W Campus Public Condor Cloud Pool

slide-1
SLIDE 1
slide-2
SLIDE 2
slide-3
SLIDE 3

Makeflow

Local Condor Torque Work Queue

slide-4
SLIDE 4

FutureGrid Torque Cluster Campus Condor Pool Public Cloud Provider Private Cluster Makefile Makeflow W W W W W W W W W W W W Thousands of Workers in a Personal Cloud

slide-5
SLIDE 5
slide-6
SLIDE 6
  • –

– –

  • –

– – –

slide-7
SLIDE 7

Sandbox for Each Task Work Queue Library Work Queue Application

C

Python Perl

$$$

Master Process submit wait

slide-8
SLIDE 8
slide-9
SLIDE 9
slide-10
SLIDE 10
slide-11
SLIDE 11

#include “work_queue.h” struct work_queue *queue; struct work_queue_task *task; queue = work_queue_create( 0 ); work_queue_specify_name( “myproject” ); task = work_queue_task_create(“sim.exe –p 50 in.dat >out.txt”); /// Missing: Specify files needed by the task. work_queue_submit( queue, task ); while(!work_queue_empty(queue)) { task = work_queue_wait( queue, 60 ); if(task) work_queue_task_delete( task ); }

slide-12
SLIDE 12

use Work_Queue; $queue = Work_Queue->new( 0 ); $queue->specify_name( “myproject” ); $task = Work_Queue::Task->new(“sim.exe –p 50 in.dat >out.txt”); ### Missing: Specify files needed by the task. $queue->submit( $task ); while(!$queue->empty()) { $task = $queue->wait( 60 ); ### Missing: Do something with the task's results }

slide-13
SLIDE 13

from work_queue import * queue = WorkQueue( port = 0 ) queue.specify_name( “myproject” ); task = Task(“sim.exe –p 50 in.dat >out.txt”) ### Missing: Specify files needed by the task. queue.submit( task ) While not queue.empty(): task = queue.wait(60)

slide-14
SLIDE 14

work_queue_task_specify_file( task,“in.dat”,”in.dat”, WORK_QUEUE_INPUT, WORK_QUEUE_NOCACHE ); work_queue_task_specify_file( task,“calib.dat”,”calib.dat”, WORK_QUEUE_INPUT, WORK_QUEUE_CACHE ); work_queue_task_specify_file( task,“out.txt”,”out.txt”, WORK_QUEUE_OUTPUT, WORK_QUEUE_NOCACHE ); work_queue_task_specify_file( task,“sim.exe”,”sim.exe”, WORK_QUEUE_INPUT, WORK_QUEUE_CACHE );

sim. exe in.dat calib.dat

  • ut.txt
slide-15
SLIDE 15

sim. exe in.dat calib.dat

  • ut.txt

$task->specify_input_file(“in.dat”); $task->specify_input_file(“calib.dat”); $task->specify_output_file(“out.txt”); $task->specify_input_file(local_name => “sim.exe”, remote_name => "sim.exe", flags => $Work_Queue::WORK_QUEUE_CACHE );

slide-16
SLIDE 16

task.specify_file( “in.dat”, ”in.dat”, WORK_QUEUE_INPUT, cache = False ) task.specify_input_file( “calib.dat” ) task.specify_output_file( “out.txt” ) task.specify_input_file( “sim.exe”, cache = True )

sim. exe in.dat calib.dat

  • ut.txt
slide-17
SLIDE 17
slide-18
SLIDE 18

Running a Work Queue Program

…

slide-19
SLIDE 19

… for Python

…

slide-20
SLIDE 20

… for Perl

…

slide-21
SLIDE 21
slide-22
SLIDE 22
slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25

Worker Catalog Work Queue

(port 9037)

slide-26
SLIDE 26

Specify Project Names in Work Queue

slide-27
SLIDE 27

Start Workers with Project Names

slide-28
SLIDE 28
slide-29
SLIDE 29

Tag a Work Queue task

  • –

–

slide-30
SLIDE 30

Cancel Work Queue task

slide-31
SLIDE 31

Retry “slow” Work Queue tasks

  • –

–

  • –

– – –

slide-32
SLIDE 32

Activating fast abort in Work Queue

#abort if task exceeds 2.5 * avg execution time

slide-33
SLIDE 33

Send intermediate buffer data as input file for Work Queue Task

slide-34
SLIDE 34

Send intermediate buffer data as input file for Work Queue Task

slide-35
SLIDE 35

Work Queue Task Structure

slide-36
SLIDE 36

Accessing Work Queue Task structure

slide-37
SLIDE 37

Work Queue Statistics

  • –

– – –

  • –
  • –
  • …
slide-38
SLIDE 38

Accessing Work Queue Statistics

slide-39
SLIDE 39

Work Queue Statistics

slide-40
SLIDE 40

Thinking Even BIGGER

slide-41
SLIDE 41

Managing Your Workforce

W W W W W W WQ Pool 100 WQ Pool 200

slide-42
SLIDE 42

Multi-Slot Workers

work_queue_worker

  • -cores 8
  • -memory 1024

1 core task 1 core task 1 core task 1 core task 1 core task

4 cores 512 MB specify_cores(4); specify_memory(512);

work_queue_worker

(implies 1 task, 1 core)

slide-43
SLIDE 43

Using Foremen

T T T T T T T T T T T T

Approx X1000 at each fanout.

$$$

work_queue_worker

  • -foreman $MASTER $PORT

$$$

California Chicago

slide-44
SLIDE 44
slide-45
SLIDE 45

Replica Exchange Work Queue

slide-46
SLIDE 46

SAND filter master SAND align master Celera

Consensu s

W W W W W W W Sequence Data

slide-47
SLIDE 47
slide-48
SLIDE 48
slide-49
SLIDE 49