1. Download the practical work tarball wget - - PDF document

ComplexHPC Spring School’11: KOALA Amsterdam, May 10, 2011 1

ComplexHPC Spring School 2011 KOALA– Practical Work

Amsterdam, May 10, 2011

http://www.st.ewi.tudelft.nl/koala/

Key issues Application

Runners Component structure Co-allocation Staging Name (type)

PRunner

single component without executable/output hostname (sequential) pi_seq (sequential)

OMRunner

fixed/non- fixed/flexible without executable/output pi_par (MPI)

WRunner

Directed Acyclic Graph (DAG) with input/executable/output diamond (workflow) pipeline (workflow)

WRunner

conveniently parallel with input/executable/output prime search (Bag of Tasks -- BoT)

Preparing Your Environment on the DAS-4

• 1. Download the practical work tarball

wget http://bullet.st.ewi.tudelft.nl/downloads/ComplexHPC_11.tar.gz

• 2. Extract the tarball

tar xzvf ComplexHPC_11.tar.gz

• 3. Edit .bashrc

wget http://bullet.st.ewi.tudelft.nl/downloads/koala_env Append the contents of the koala_env file to the .bashrc file which is located under your home directory.

• 4. Source the updated .bashrc

source ~/.bashrc

ComplexHPC Spring School’11: KOALA Amsterdam, May 10, 2011 2

• 5. Generate public/private key pair for passwordless SSH

kssh_keygen.sh -all

• 6. Copy .bashrc file to all head-nodes

push_profile.sh

Practical Work

A: Run one sequential job on the local cluster. Key issues: pru

runner nner, fixed, without co-allocation, no staging

Application: /b

/bin/h in/hostn

• stname

ame

Commands: pr

prunne unner r -host

• st fs0

fs0.das4 das4.cs. cs.vu.n vu.nl -stdou tdout ho t host_o st_out.t ut.txt xt - c 1 / 1 /bin/ bin/host hostname name

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ B: Run one sequential job on the local cluster, and read the job’s output. Key issues: prunner, fixed, without co-allocation, no staging Application: pi

pi_seq _seq

Commands: prunner -host fs0.das4.cs.vu.nl -stdout pi_stdout.txt

• c 1 pi_seq

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ C: Run one sequential job on a remote cluster, and read the job’s output. Key issues: prunner, fixed, without co-allocation, staging executable and

• utput

Application: pi_seq Commands: prunner -host fs2.das4.science.uva.nl -stdout

pi_stdout.txt -c 1 pi_seq

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________

ComplexHPC Spring School’11: KOALA Amsterdam, May 10, 2011 3 D: Run one MPI job on the local cluster, and read the job’s output. (Make the necessary changes in the jdf file: set “resourcemanagercontact” appropriately) Key issues: omr

mrunne unner, fixed, without co-allocation, staging executable and

• utput

Application: pi

pi_par _par

Commands: om

• mrunn

runner er -f piA piApp. pp.jdf jdf

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ E: Run one MPI job on the best (Koala selected) cluster, and read the job’s output. (Make the necessary changes in the jdf file: set “resourcemanagercontact” appropriately) Key issues: omrunner, non-fixed, without co-allocation, staging output Application: pi_par Commands: omrunner -f piApp.jdf Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ F: Run a workflow on the local cluster, and read the job’s output. Key issues: wru

runner nner, staging input, staging output

Application: di

diamon amond an d and pi d pipeli peline ne

Commands:

wr wrunne unner -f wf/ wf/Diam Diamond.

• nd.xml

xml -p s p singl ingle_si e_site te -s s fs fs0.da .das4 s4.cs.vu s.vu.nl .nl wr wrunne unner -f wf/ wf/Pipe PipeLine Line.xml .xml -p p sing single_s le_site ite -s s fs0.d s0.das4 as4.cs.v cs.vu.nl .nl

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ G: Run a workflow on a single cluster (Koala selected), and read the job’s output. Key issues: wru

runner nner, staging input, staging output

Application: di

diamon amond an d and pi d pipeli peline ne

Commands: wr

wrunne unner -f wf/ wf/Diam Diamond.

• nd.xml

xml -p s p singl ingle_si e_site te wrun wrunner ner -f w f wf/Pi f/PipeLi peLine.x ne.xml ml -p si p single ngle_sit _site

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________

ComplexHPC Spring School’11: KOALA Amsterdam, May 10, 2011 4 H: Run a workflow on multiple clusters, and read the job’s output. Key issues: wrunner, staging input, staging output, with co-allocation Application: diamond and pipeline Commands: wrunner -f wf/Diamond.xml -p multi

ulti_sit _site wrunner -f wf/PipeLine.xml -p mult ulti_ i_site site

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ I: Run a BoT on multiple clusters, and read the job’s output. Key issues: wrunner, staging input, staging output, with co-allocation Application: PrimeSearch Input: range_1.in, range_2.in, and range_3.in files specify the range in which the primes will be searched. The range is of the form a-b which specifies the range [a,b). Output: primes.out in each task’s folder (e.g., ID00001/primes.out) with all the primes found in the given range. Commands: wrunner -f wf/PrimeSearch.xml -p multi_site Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ J: Free exercise: For instance, try your own application. Key issues: ___

______ ________ ________ ________ ________ ________ ________ ________ ________ ________ ________ _______ ___

Application: __

______ ________ ________ ________ ________ ________ ________ ________ ________ ________ ________ _______ ___

Commands:

__________________________________________________________________ __________________________________________________________________

Observations: _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________