1 CPU-Intensive Jobs Taskfarming Parallelisable, for example - PDF document

Outline Why do we need a cluster? Architecture: Machines and Properties Taskfarming Estimating Walltime Limits Common Pitfalls of Using the Cluster Joachim Wagner 2009-07-01 Why do we need a cluster? Cluster Architecture More efficient and less costly School Separate (than high-end desktop PCs) Network Networks Fileserver (100 MBit) (Gigabit) Avoid resource conflicts Logins Waiting for colleague’s job to finish Software Node 1 Trouble, e.g. disk full Medium-size jobs Job Queue Node 2 Too big for desktop PC … Too small for ICHEC maia.computi ng.dcu.ie Preparation of ICHEC runs Node N Learning Node Properties (see command pbsnodes) Selecting the Number of CPU cores (ppn) min4GB, …, min32GB: at least this much 4 or 8 CPU cores per node mem4GB, …, mem32GB: exactly this much 1, 2 or 4 GB memory per core Partitions: A single application may use more than 1 CPU Switch1/2: which fileserver network + MPI communication switch 2 groups of 8 and 4 groups of 4 (16 and 32 GB nodes) Java, C&J reranking parser, any sub-processes Proposal: run short jobs in group4b and long jobs in group4d Limit memory usage CPU type: Command ulimit -v Intel Xeon E5440 quad core, 2.83 GHz, 6 MB cache Intel Xeon E5420 quad core, 2.50 GHz, 6 MB cache Processes compete for RAM Intel Xeon 5110 dual core, 1.6 GHz, 4 MB cache Swapping of one task effects the 3 other tasks CPU Cores: If in doubt, reserve a full node Memory per core (example: mem2GBpercore and ppn = 4) Number of cores (4 or 8) ppn=4:cores4 or ppn=8:cores8 1

CPU-Intensive Jobs Taskfarming Parallelisable, for example Taskfarming Sentence by sentence processing PBS Job Executable Description Cross-validation runs (n instances) Parameter search child reading process Split into parts 1 Master n-1 Worker Run each part on a different CPU core Task file MPI or HTTP Alternatives (.tfm): Communication Task Submit large number of jobs (ppn=1) one task execution per line Taskfarming Taskfarming Options Example: Taskfarming in Action Using individual PBS jobs Can only allocate resources in multiples of 1/8 or ¼ CPU 1 000 003 006 011 Example: 3 GB task -> 4 GB job (ppn=2:cores8:mem16GB) Floods job queue CPU 2 001 005 008 009 012 MPI-based taskfarming All tasks inside one job idle 002 004 007 010 CPU 3 Example: 3 GB task -> 5 workers per 16 GB node Master blocks one CPU core HTTP/XML-RPC-based taskfarming CPU 4 Master: reads .tfm and distributes tasks Master runs on maia login node Workers can run in multiple jobs time Example: 3 GB tasks -> one job with 5 workers for 16 GB nodes and one job with 8 workers for 32 GB nodes Estimating the PBS Walltime Parameter Questions Collect durations from test run Usually high variance of execution time ? Long sentences Parameters Don’t use #packages x avg. time per package Contact: High risk (~50 %) that more time is needed Joachim Wagner Prefix jobname with, for example, “24h-” CNGL System Administrator Random sampling with observed package jwagner@computing.dcu.ie durations: /home/jwagner/tools/walltime.py (01) 700 6915 2

Installed Software PBS Job Management OpenMPI User: Job Job Submission SRILM Description MaTrEx, Moses, GIZA++ Job Queue XLE, Sicstus Johnson & Charniak’s reranking parser Job Scheduler Nodes are allocated In progress: job-exclusive for the LFG AA, incl. function labeller duration of the job (if ppn = #cores as Job Execution recommended) PBS Job Management Commands PBS Job Description Number of nodes qsub myjob.pbs #CPU cores/node submits a job PBS description: shell script with #PBS commands (ignored by shell, see next slide) Notification: end, qstat, qstat –f jobnumber begin and abort qdel jobnumber Maximum runtime pbsnodes –a list all nodes with status and properties Number of pro- cesses to start Example: Memory-Intensive Job Taskfarming Executable If Instance ID == 0 Run master code loop: Read .tfm file (arg 1) Send lines to worker Exit if no more task and all worker finished Else Run worker loop: Ask master for a task Execute task Exit if master has no more tasks 3

Example: Taskfarming PBS File Example: Taskfarming TFM File Example: Taskfarming Helper Script Example: Non-Terminating Task CPU 1 000 003 006 (does not terminate) run-package.sh idle CPU 2 001 005 008 009 011 idle 002 004 007 010 012 CPU 3 CPU 4 Master: reads .tfm and distributes tasks Killed at Walltime Limit Effect of Task Size Job will wait for last task to finish (or be killed when walltime limit is reached) What if a task crashes? Results are incomplete Next tasks is executed What if a task does not terminate? Results are incomplete Fewer CPUs available for remaining tasks Overhead of starting tasks 4