1. Download the practical work tarball wget - PDF document

ComplexHPC Spring School ’11 : K OALA Amsterdam, May 10, 2011 ComplexHPC Spring School 2011 K OALA – Practical Work Amsterdam, May 10, 2011 http://www.st.ewi.tudelft.nl/koala/ Key issues Application Runners Component Co-allocation Staging Name (type) structure hostname (sequential) PRunner single component without executable/output pi_seq (sequential) fixed/non- OMRunner fixed/flexible without executable/output pi_par (MPI) Directed Acyclic input/executable/output diamond (workflow) WRunner Graph with pipeline (workflow) (DAG) prime search conveniently input/executable/output WRunner with (Bag of Tasks -- parallel BoT) Preparing Your Environment on the DAS-4 1. Download the practical work tarball wget http://bullet.st.ewi.tudelft.nl/downloads/ComplexHPC_11.tar.gz 2. Extract the tarball tar xzvf ComplexHPC_11.tar.gz 3. Edit .bashrc wget http://bullet.st.ewi.tudelft.nl/downloads/koala_env Append the contents of the koala_env file to the .bashrc file which is located under your home directory . 4. Source the updated .bashrc source ~/.bashrc 1

ComplexHPC Spring School ’11 : K OALA Amsterdam, May 10, 2011 5. Generate public/private key pair for passwordless SSH kssh_keygen.sh -all 6. Copy .bashrc file to all head-nodes push_profile.sh Practical Work A : Run one sequential job on the local cluster. Key issues : pru runner nner , fixed , without co-allocation , no staging Application : /b /bin/h in/hostn ostname ame Commands : pr prunne unner r -host ost fs0 fs0.das4 das4.cs. cs.vu.n vu.nl -stdou tdout ho t host_o st_out.t ut.txt xt - c 1 / 1 /bin/ bin/host hostname name Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ B : Run one sequential job on the local cluster, and read the job’s output . Key issues : prunner , fixed, without co-allocation, no staging Application : pi pi_seq _seq Commands : prunner -host fs0.das4.cs.vu.nl -stdout pi_stdout.txt -c 1 pi_seq Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ C : Run one sequential job on a remote cluster, and read the job’s output . Key issues : prunner , fixed, without co-allocation, staging executable and output Application : pi_seq Commands : prunner -host fs2.das4.science.uva.nl -stdout pi_stdout.txt -c 1 pi_seq Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ 2

ComplexHPC Spring School ’11 : K OALA Amsterdam, May 10, 2011 D : Run one MPI job on the local cluster, and read the job’s output . (Make the necessary changes in the jdf file: set “ resourcemanagercontact ” appropriately) Key issues : omr mrunne unner , fixed, without co-allocation, staging executable and output Application : pi pi_par _par Commands : om omrunn runner er -f piA piApp. pp.jdf jdf Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ E : Run one MPI job on the best (Koala selected) cluster, and read the job’s output . (Make the necessary changes in the jdf file : set “ resourcemanagercontact ” appropriately) Key issues : omrunner , non-fixed , without co-allocation, staging output Application : pi_par Commands : omrunner -f piApp.jdf Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ F : Run a workflow on the local cluster, and read the job’s output. Key issues : wru nner , staging input, staging output runner Application : di diamon amond an d and pi d pipeli peline ne Commands : wr wrunne unner -f wf/ wf/Diam Diamond. ond.xml xml -p s p singl ingle_si e_site te -s s fs fs0.da .das4 s4.cs.vu s.vu.nl .nl wr wrunne unner -f wf/ wf/Pipe PipeLine Line.xml .xml -p p sing single_s le_site ite -s s fs0.d s0.das4 as4.cs.v cs.vu.nl .nl Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ G : Run a workflow on a single cluster (Koala selected), and read the job’s output. Key issues : wru nner , staging input, staging output runner Application : di ne diamon amond an d and pi d pipeli peline Commands : wr wrunne unner -f wf/ wf/Diam Diamond. ond.xml xml -p s p singl ingle_si e_site te wrun wrunner ner -f w f wf/Pi f/PipeLi peLine.x ne.xml ml -p si p single ngle_sit _site Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ 3

ComplexHPC Spring School ’11 : K OALA Amsterdam, May 10, 2011 H : Run a workflow on multiple clusters , and read the job’s output. Key issues : wrunner , staging input, staging output, with co-allocation Application : diamond and pipeline Commands : wrunner -f wf/Diamond.xml -p multi ulti_sit _site site wrunner -f wf/PipeLine.xml -p mult ulti_ i_site Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ I : Run a BoT on multiple clusters, and read the job’s output. Key issues : wrunner , staging input, staging output, with co-allocation Application : PrimeSearch Input: range_1.in, range_2.in, and range_3.in files specify the range in which the primes will be searched. The range is of the form a-b which specifies the range [a,b). Output: primes.out in each task ’s folder (e.g., ID00001/primes.out) with all the primes found in the given range. Commands : wrunner -f wf/PrimeSearch.xml -p multi_site Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ J : Free exercise: For instance, try your own application. Key issues : ___ ______ ________ ________ ________ ________ ________ ________ ________ ________ ________ ________ _______ ___ Application : __ ______ ________ ________ ________ ________ ________ ________ ________ ________ ________ ________ _______ ___ Commands : __________________________________________________________________ __________________________________________________________________ Observations : _________________________________________________ _____________________________________________________________ _____________________________________________________________ _____________________________________________________________ 4