Flux: Practical Job Scheduling Dong H. Ahn, Ned Bass, Al Chu, Jim - PowerPoint PPT Presentation

Flux: Practical Job Scheduling Dong H. Ahn, Ned Bass, Al Chu, Jim Garlick, Mark Grondona, Stephen Herbein , Tapasya Patki, Tom Scogland, Becky Springmeyer August 15, 2018 LLNL-PRES-757227 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE- AC52-07NA27344. Lawrence Livermore National Security, LLC

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 2 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 3 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 3 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them flickr: dannychamoro 3 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 4 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 4 LLNL-PRES-757227

What is Flux? ▪ New Resource and Job Management Software (RJMS) developed here at LLNL ▪ A way to manage remote resources and execute tasks on them 4 LLNL-PRES-757227

What about …? 5 LLNL-PRES-757227

What about …? Closed-source 6 LLNL-PRES-757227

What about …? Not designed for HPC 7 LLNL-PRES-757227

What about …? Limited Scalability, Usability, and Portability 8 LLNL-PRES-757227

Why Flux? 9 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins 9 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins ▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs 9 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins ▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs ▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler 9 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins ▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs ▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler ▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI 9 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins ▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs ▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler ▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI Flux is designed to make hard scheduling problems easy 9 LLNL-PRES-757227

Portability: Running Flux 10 LLNL-PRES-757227

Portability: Running Flux ▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else 10 LLNL-PRES-757227

Portability: Running Flux ▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else ▪ Flux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource allocation from: itself (hierarchical Flux), Slurm, Moab, PBS, LSF , etc — flux start OR srun flux start 10 LLNL-PRES-757227

Portability: Running Flux ▪ Already installed on LC systems (including Sierra) — spack install flux-sched for everywhere else ▪ Flux can run anywhere that MPI can run, (via PMI – Process Management Interface) — Inside a resource allocation from: itself (hierarchical Flux), Slurm, Moab, PBS, LSF , etc — flux start OR srun flux start ▪ Flux can run anywhere that supports TCP and you have the IP addresses — flux broker -Sboot.method=config -Sboot.config_file=boot.conf — boot.conf: session-id = "mycluster" tbon-endpoints = [ "tcp://192.168.1.1:8020", "tcp://192.168.1.2:8020", "tcp://192.168.1.3:8020"] 10 LLNL-PRES-757227

Why Flux? ▪ Extensibility — Open source — Modular design with support for user plugins ▪ Scalability — Designed from the ground up for exascale and beyond — Already tested at 1000s of nodes & millions of jobs ▪ Usability — C, Lua, and Python bindings that expose 100% of Flux’s functionality — Can be used as a single-user tool or a system scheduler ▪ Portability — Optimized for HPC and runs in Cloud and Grid settings too — Runs on any set of Linux machines: only requires a list of IP addresses or PMI 11 LLNL-PRES-757227

Usability: Submitting a Batch Job 12 LLNL-PRES-757227

Usability: Submitting a Batch Job ▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120 12 LLNL-PRES-757227

Usability: Submitting a Batch Job ▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120 ▪ Flux CLI — flux submit –N2 –n4 –t 2m sleep 120 12 LLNL-PRES-757227

Usability: Submitting a Batch Job ▪ Slurm — sbatch –N2 –n4 –t 2:00 sleep 120 ▪ Flux CLI — flux submit –N2 –n4 –t 2m sleep 120 ▪ Flux API: import json, flux jobreq = { 'nnodes' : 2, 'ntasks' : 4, 'walltime' : 120, 'cmdline' : ["sleep", "120"]} f = flux.Flux () resp = f.rpc_send ("job.submit", json.dumps(jobreq)) 12 LLNL-PRES-757227

Usability: Running an Interactive Job 13 LLNL-PRES-757227

Usability: Running an Interactive Job ▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120 13 LLNL-PRES-757227

Usability: Running an Interactive Job ▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120 ▪ Flux CLI — flux wreckrun –N2 –n4 –t 2m sleep 120 13 LLNL-PRES-757227

Usability: Running an Interactive Job ▪ Slurm — srun –N2 –n4 –t 2:00 sleep 120 ▪ Flux CLI — flux wreckrun –N2 –n4 –t 2m sleep 120 ▪ Flux API: import sys from flux import kz resp = f.rpc_send ("job.submit", json.dumps(jobreq)) kvs_dir = resp['kvs_dir'] for task_id in range(jobreq['ntasks']): kz.attach (f, "{}.{}.stdout".format(kvs_dir, task_id), sys.stdout) f.reactor_run (f.get_reactor (), 0) 13 LLNL-PRES-757227

Usability: Tracking Job Status 14 LLNL-PRES-757227

Usability: Tracking Job Status ▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID 14 LLNL-PRES-757227

Usability: Tracking Job Status ▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID ▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop 14 LLNL-PRES-757227

Usability: Tracking Job Status ▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID ▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop → quota -vf ~/quota.conf Disk quotas for herbein1: Filesystem used quota limit files /p/lscratchrza 760.3G n/a n/a 8.6M 14 LLNL-PRES-757227

Usability: Tracking Job Status ▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID ▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop Non-I/O → quota -vf ~/quota.conf I/O Disk quotas for herbein1: Runtime Stages Filesystem used quota limit files UQP Startup /p/lscratchrza 760.3G n/a n/a 8.6M Job Submission I/O File Creation File Access 14 LLNL-PRES-757227

Usability: Tracking Job Status ▪ CLI: slow, non-programmatic, inconvenient to parse — watch squeue –j JOBID — watch flux wreck ls JOBID ▪ Tracking via the filesystem — date > $JOBID.start; srun myApp; date > $JOBID.stop ▪ Push notification via Flux’s Job Status and Control (JSC): def jsc_cb (jcbstr, arg, errnum): jcb = json.loads (jcbstr) jobid = jcb['jobid'] state = jsc.job_num2state (jcb[jsc.JSC_STATE_PAIR][jsc.JSC_STATE_PAIR_NSTATE]) print "flux.jsc: job", jobid, "changed its state to ", state jsc.notify_status (f, jsc_cb, None) 14 LLNL-PRES-757227