Automatic Job Submission Simon Albright Three classes: - - PowerPoint PPT Presentation

Automatic Job Submission

Simon Albright

Three classes:

• SSHConnection
• Handles sending and receiving data, monitoring job status etc
• Remote_Optimise
• Handles itteration over variables, manipulation of .py scripts etc
• Itterable_Variable
• Contains variable to be edited as a single value or function

and sets range of values for each itteration

SSHConnection

Build on Paramiko, python module for handling SSH (text interface) and SFTP (file transfer) connections

• ssh = sp.SSHConnection('salbrigh', verb=1, workingDir = remoteDir, localDir =

localDir)

• Username = 'salbrigh'
• Verbosity = 1 (Higher levels increase output information, can be set to 0 for

*almost* no output)

• LocalDir & WorkingDir sets where all commands and files will be sent by

default, can be changed on a line by line basis

• Server, port, response to mission keys have default values but can be changed
• Password handled by “getpass”, prevents text appearing on screen when

typing and stores password in RAM.

Password method

def set_password(self): if self.verbosity > 0: print("Password for LXPLUS required") print("Password is not stored outside RAM") print("Password will not appear in history") print("Password should only come from interactive source") self.pWord = getpass.getpass()

Connection controls

start_session

• Creates initial SSH and SFTP connections
• If password is incorrect gives three attempts to try again before

ending program end_session

• Terminates SFTP and SSH connection

start_stop_connect

• Closes and then reopens SFTP and SSH connection
• Prevents session from hanging

Job submission

Small numbers of jobs ( e.g. <~ 100):

• Pass a single string as bash command with format:

'bsub -q queueName -J jobName1 < submissionScript1 ; bsub -q queueName -J jobName2 < submissionScript2 ; … Medium numbers of jobs (100 < N < 1000):

• Write a script with format:

bsub -q queueName -J jobName1 < submissionScript1 bsub -q queueName -J jobName2 < submissionScript2 ... ... … Very large numbers (N > 1000):

• As with Medium numbers but multiple scripts

JobID stored in list for identification of running/queueing and completed jobs

(Some) additional commands:

• check_jobs
• Monitoring jobs still in queue
• compress_files
• Compression of data for retrieval
• get_file
• Transfer from server
• put_file
• Transfer to server
• make_file
• Make text file from string (e.g. for submissionScript writing)
• read_file
• Return file contents as string
• Copy
• Copy files on server
• Move
• Move files on server
• Remove
• Delete file on server
• search_replace
• Search for string in file and replace