X10 Cluster SSH access X10 on your PC Eclipse for X10: x10dt - - PowerPoint PPT Presentation

X10

• Cluster
• SSH access
• X10 on your PC
• Eclipse for X10: x10dt
• From Eclipse to the cluster

Cluster

• Access via ssh through

labsrv0.math.unipd.it

• Cluster description:

http://numlab.math.unipd.it/

• labsrv0 is the submit host. The clustering

software is Torque/Maui, an evolution of PBS:

http://en.wikipedia.org/wiki/Portable_Batch_System/

Access via SSH

• Basic commands

ssh: remote terminal connection scp: file copy between hosts ssh-keygen: criptographic keys management

• Access without password, instrucion at:

http://www.linuxproblem.org/art_9.html in summary: a) local: ssh-keygen -t dsa (to generate a key couple) b) local: scp .ssh/id_dsa.pub user@remote:/tmp (public key copy) b) remote: cat /tmp/id_dsa.pub >> $HOME/.ssh/authorized_keys

X10 on your PC

• Download the archive (pre-built binaries):

http://www.x10-lang.org/

• For windows you have to install Cygwin:

http://www.cygwin.com/

• Install java: http://java.sun.com/
• Estract the archive in a convenient directory

(example: /opt/X10).

• Update PATH and JAVA_HOME enviroment

variables

Installation example

'RUN' of an example

X10 IDE (x10dt)

• Downloadable from http://www.x10-lang.org
• Extract the archive in /opt, you obtain the

directory /opt/x10dt

• To start the IDE:

koriel@righi2:~$ /opt/x10dt/x10dt <RET>

X10dt start: workspace choose

x10dt: welcome screen

x10dt: base window

x10dt: a new project

Choose 'File' menu and then 'New' and 'X10 Project (C++ back-end)'

x10dt: Hello World

x10dt: editing your code

X10dt: run your program

x10dt: remote execution

• First you need define the

remote host connection

• Click near the upper right

corner near to 'X10'

• Open the perspective

'Parallel Runtime'

x10dt: remote execution

• First you need define the

remote host connection

• Click near the upper right

corner near to 'X10'

• Open the perspective

'Parallel Runtime', the x10dt window change...

x10dt: remote execution

• In the changed x10dt

window organization click with the right button of the mouse

• n the section

'Resource manager'

• Choose

'Add resource manager'

• Choose 'Remote

Launch'

x10dt: remote execution

• In the new wizard choose

'Remote Tools' as remote service provider

• Hit 'New...' to define a new

Connection name.

• You can see the necessary

parameters on the image, here the username is 'koriel', you have to change with the your.

x10dt: start a resource manager

x10dt: running configuration

• When the resource manager is

configured and started you can define a run configuration:

• Menu` 'Run...' → 'Run

Configurations...', a new window appears.

• Click with the right mouse button on

'Parallel Application' and choose 'New'

x10dt: run: resource manager

The right side of the window is now an environment to define our

• run. You can
• Change the 'Name' with

'Remote Hello' as in the image

• Choose the previously defined

resource manager

X10dt: run: choose program

• In this window you define:

The project that produced your application, here is 'Hello'

• The remote Application

program to run, here 'remote-hello' stored in the directory '/work/koriel/X10'

• Where find the program

to run (Path to the local file), here 'Hello' stored in '/home/koriel/x10- work/Hello/bin/Hello'

• In the 'Common' tab you

can choose of display the new run configuration in the 'Run' menu` of the main window.

X10dt: remote run

You can find your run configuration in the general 'Run' menu`, choosing this item you run your 'Hello' program on the remote host and see the output on the bottom of the general window: 'Hello World from place 0'

x10dt: something of parallel

In the 'Run Configuration' setup choose the 'Environment' tab. Here you can define enviroment variables for your execution. The variable 'X10_NPLACES' defines the number of separate processes that your execution needs. In the image the variable has just been defined (using the 'New....' button) and a value of '4' was assigned. You can re-run your program and see four time the words 'Hello World from place X' with 'X' in 0,...,3.

x10dt: torque/maui at math.unipd.it

#!/bin/sh ### ### TORQUE DEFINITIONS ### #PBS -N X10-Fibonacci #PBS -r n #PBS -M righi@math.unipd.it #PBS -e localhost:${HOME}/X10/Fibonacci/Fibonacci.err #PBS -o localhost:${HOME}/X10/Fibonacci/Fibonacci.out #PBS -q cluster_short #PBS -l nodes=4:ppn=4:cluster #PBS -l mem=1g #PBS -l walltime=1:00:00 ### ### COMMANDS START ### echo Working directory is $PBS_O_WORKDIR cd $PBS_O_WORKDIR echo Running on host `hostname` echo Time is `date` echo Directory is `pwd` ### Infiniband Node Conversion echo Converting PBS_NODEFILE NEW_PBSNODEFILE=`basename $PBS_NODEFILE` /cluster/CONF/convert-ethIP-2-ibIP-X10.sh $PBS_NODEFILE $NEW_PBSNODEFILE echo MPI-Infiniband-Nodes: PBS_NODEFILE=`pwd`/${NEW_PBSNODEFILE} echo This jobs runs on: ### COMMAND export X10_NPLACES=`wc -l < $PBS_NODEFILE` export X10_HOSTLIST=`tr [:space:] , < $PBS_NODEFILE` echo "PLACES: X10_NPLACES=${X10_NPLACES}" echo "LIST: X10_HOSTLIST=${X10_HOSTLIST}" echo -n "START: "; date --rfc-3339=ns cd bin x10 Fibonacci 36 echo -n "STOP: "; date --rfc-3339=ns

Requirements for the cluster of the Department of Mathematics: You need to generate on 'labsrv0' the public/private key and then add the public part to the labsrv0 file ${HOME}/.ssh/authorized_keys Copy the file /home/koriel/known_hosts.x10 To your file ${HOME}/.ssh/known_hosts This allow every machine of the cluster to call every other machine. On the right side you see an example of a 'job' file used to submit a job to the cluster. More information on torque/maui site: http://www.adaptivecomputing.com/

x10dt: run on cluster – basics 1

• Every program in the cluster is submitted thought a

file 'NAME.job'. The command on 'labsrv0' to submit a job is: qsub NAME.job

• This file has to be present on labsrv0 disk when

we start the job. We can keep it as a file in our project, in a folder named 'PBS'.

• To run our program as a job we need transfer the

results of the compilation of our code and the job file in a convenient directory on labsrv0 and issue the command 'qsub' as described.

x10dt: run on cluster – basics 2

• We create a project named 'Fibonacci' (Java back-end)
• This back-end, on compilation, stores the results on the

folder 'bin' of the project

• We transfer all the 'bin' folder on labsrv0 on the

directory $HOME/X10/Fibonacci/bin and the job file in $HOME/X10/Fibonacci. The paths used in the job file reflects this facts

• These transfers are defined in the tab 'Syncronize' of the

'Run Configuration' and are executed before the true run

• After the transfers the 'Run Configuration' uses the 'qsub'

program with the just transferred job file to submit to the cluster our work.

x10dt: run configuration – step 1

Create a new project with 'File','New','X10 Program (Java back-end)'. Name it 'Fibonacci'. Create a new class with 'File', 'New', 'X10 Class' named 'Fibonacci'. Copy the source from the samples distributed with X10. Create a new 'Run Configuration', Choose the resource manager in tab 'Resources' and in the 'Application' tab you have to define '/export/alt/torque/bin/qsub' as 'Application program'.

x10dt: run configuration – step 3

Create a new project with 'File','New','X10 Program (Java back-end)'. Name it 'Fibonacci'. Create a new class with 'File', 'New', 'X10 Class' named 'Fibonacci'. Copy the source from the samples distributed with X10. Create a new 'Run Configuration', Choose the resource manager in tab 'Resources' and in the 'Application' tab you have to define '/export/alt/torque/bin/qsub' as 'Application program'.

x10dt: run configuration – step 3

Define the file 'fibonacci.job' as the argument to the 'qsub' program

x10dt: run configuration – step 4

In the tab 'Synchronize' of the 'Run Configuration' you defines the uploads (and eventually downloads) that have to be performed before (after) issue the 'qsub'

• command. You can transfer single files or entire folder/directories and in this case we

transfer the entire 'bin' folder of our project and the 'fibonacci.job' file

x10dt: run configuration – step 5

Now you can save ('Apply') the 'Run Configuration' and run it. On the bottom of the screen in the 'Console' tab you should observe A line like: 478411.grid0.math.unipd.it This is the output of the 'qsub' program. Now you have two choiches 1) Connect via ssh to labsrv0 and use the 'qstat' program to monitor the job 2) Define a new 'Run configuration' that runs the program /export/alt/torque/bin/qstat and – optionally – downloads the files Fibonacci.out and Fibonacci.err (defined in the 'job' file as the ouput and the error of our program) in the local 'PBS' folder to later examination.