UL HPC School 2017[bis] PS1: Getting Started on the UL HPC platform - - PowerPoint PPT Presentation

▶
ul hpc school 2017 bis
SMART_READER_LITE
LIVE PREVIEW

UL HPC School 2017[bis] PS1: Getting Started on the UL HPC platform - - PowerPoint PPT Presentation

Apr 13, 2023 402 likes •1.07k views

UL HPC School 2017[bis] PS1: Getting Started on the UL HPC platform UL High Performance Computing (HPC) Team C. Parisot University of Luxembourg (UL), Luxembourg http://hpc.uni.lu C. Parisot & UL HPC Team (University of Luxembourg) UL

slide-1
SLIDE 1

UL HPC School 2017[bis]

PS1: Getting Started on the UL HPC platform

UL High Performance Computing (HPC) Team

  • C. Parisot

University of Luxembourg (UL), Luxembourg http://hpc.uni.lu

1 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-2
SLIDE 2

Latest versions available on Github: UL HPC tutorials:

https://github.com/ULHPC/tutorials

UL HPC School:

http://hpc.uni.lu/hpc-school/

PS1 tutorial sources:

https://github.com/ULHPC/tutorials/tree/devel/basic/getting_started 2 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-3
SLIDE 3

Introduction

Summary

1 Introduction 2 SSH Secure Shell 3 UL HPC Tutorial: Getting Started Step by step program of this practical session 4 Hands-On: Getting Started on ULHPC

3 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-4
SLIDE 4

Introduction

Main Objectives of this Session

Understand SSH Connect to the UL HPC Platform

֒ → SSH configuration ֒ → Generate your SSH key pair ֒ → overcome port filtering

Discovering, visualizing and reserving UL HPC resources

֒ → Working environment ֒ → Web monitoring interfaces ֒ → OAR vs. SLURM Batch Scheduler ֒ → Job management ֒ → Software / Environement Modules

4 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-5
SLIDE 5

SSH Secure Shell

Summary

1 Introduction 2 SSH Secure Shell 3 UL HPC Tutorial: Getting Started Step by step program of this practical session 4 Hands-On: Getting Started on ULHPC

5 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-6
SLIDE 6

SSH Secure Shell

SSH: Secure Shell

Ensure secure connection to remote (UL) server

֒ → establish encrypted tunnel using asymmetric keys

Public id_rsa.pub vs. Private id_rsa (without .pub) typically on a non-standard port (Ex: 8022)

limits kiddie script

Basic rule: 1 machine = 1 key pair

֒ → the private key is SECRET: never send it to anybody

Can be protected with a passphrase

6 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-7
SLIDE 7

SSH Secure Shell

SSH: Secure Shell

Ensure secure connection to remote (UL) server

֒ → establish encrypted tunnel using asymmetric keys

Public id_rsa.pub vs. Private id_rsa (without .pub) typically on a non-standard port (Ex: 8022)

limits kiddie script

Basic rule: 1 machine = 1 key pair

֒ → the private key is SECRET: never send it to anybody

Can be protected with a passphrase

SSH is used as a secure backbone channel for many tools

֒ → Remote shell i.e remote command line ֒ → File transfer: rsync, scp, sftp ֒ → versionning synchronization (svn, git), github, gitlab etc.

6 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-8
SLIDE 8

SSH Secure Shell

SSH: Secure Shell

Ensure secure connection to remote (UL) server

֒ → establish encrypted tunnel using asymmetric keys

Public id_rsa.pub vs. Private id_rsa (without .pub) typically on a non-standard port (Ex: 8022)

limits kiddie script

Basic rule: 1 machine = 1 key pair

֒ → the private key is SECRET: never send it to anybody

Can be protected with a passphrase

SSH is used as a secure backbone channel for many tools

֒ → Remote shell i.e remote command line ֒ → File transfer: rsync, scp, sftp ֒ → versionning synchronization (svn, git), github, gitlab etc.

Authentication:

֒ → password (disable if possible) ֒ → (better) public key authentication

6 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-9
SLIDE 9

SSH Secure Shell

SSH: Public Key Authentication

Client Local Machine

id_rsa.pub id_rsa known_hosts

~/.ssh/

local homedir

  • wns local private key

logs known servers

7 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-10
SLIDE 10

SSH Secure Shell

SSH: Public Key Authentication

Server Remote Machine

authorized_keys

~/.ssh/

remote homedir

knows granted

(public) key

Client Local Machine

id_rsa.pub id_rsa known_hosts

~/.ssh/

local homedir

  • wns local private key

logs known servers

7 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-11
SLIDE 11

SSH Secure Shell

SSH: Public Key Authentication

Server Remote Machine

authorized_keys

~/.ssh/

remote homedir

knows granted

(public) key

/etc/ssh/

SSH server config

ssh_host_rsa_key sshd_config ssh_host_rsa_key.pub

Client Local Machine

id_rsa.pub id_rsa known_hosts

~/.ssh/

local homedir

  • wns local private key

logs known servers

7 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-12
SLIDE 12

SSH Secure Shell

SSH: Public Key Authentication

Server Remote Machine

authorized_keys

~/.ssh/

remote homedir

knows granted

(public) key

Client Local Machine

id_rsa.pub id_rsa

~/.ssh/

local homedir

  • wns local private key

7 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-13
SLIDE 13

SSH Secure Shell

SSH: Public Key Authentication

Server Remote Machine

authorized_keys

~/.ssh/

remote homedir

knows granted

(public) key

Client Local Machine

id_rsa.pub id_rsa

~/.ssh/

local homedir

  • wns local private key
  • 1. Initiate connection
  • 2. create random

challenge, “encrypt” using public key

  • 3. solve challenge

using private key return response

  • 4. allow connection iff

response == challenge

Restrict to public key authentication: /etc/ssh/sshd_config:

PermitRootLogin no # Disable Passwords PasswordAuthentication no ChallengeResponseAuthentication no # Enable Public key auth. RSAAuthentication yes PubkeyAuthentication yes

7 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-14
SLIDE 14

SSH Secure Shell

SSH Setup on Linux / Mac OS

OpenSSH natively supported; configuration directory : ~/.ssh/

֒ → package openssh-client (Debian-like) or ssh (Redhat-like)

SSH Key Pairs (public vs private) generation: ssh-keygen

֒ → specify a strong passphrase

protect your private key from being stolen i.e. impersonation drawback: passphrase must be typed to use your key

8 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-15
SLIDE 15

SSH Secure Shell

SSH Setup on Linux / Mac OS

OpenSSH natively supported; configuration directory : ~/.ssh/

֒ → package openssh-client (Debian-like) or ssh (Redhat-like)

SSH Key Pairs (public vs private) generation: ssh-keygen

֒ → specify a strong passphrase

protect your private key from being stolen i.e. impersonation drawback: passphrase must be typed to use your key ssh-agent

8 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-16
SLIDE 16

SSH Secure Shell

SSH Setup on Linux / Mac OS

OpenSSH natively supported; configuration directory : ~/.ssh/

֒ → package openssh-client (Debian-like) or ssh (Redhat-like)

SSH Key Pairs (public vs private) generation: ssh-keygen

֒ → specify a strong passphrase

protect your private key from being stolen i.e. impersonation drawback: passphrase must be typed to use your key ssh-agent

DSA and RSA 1024 bit are deprecated now!

8 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-17
SLIDE 17

SSH Secure Shell

SSH Setup on Linux / Mac OS

OpenSSH natively supported; configuration directory : ~/.ssh/

֒ → package openssh-client (Debian-like) or ssh (Redhat-like)

SSH Key Pairs (public vs private) generation: ssh-keygen

֒ → specify a strong passphrase

protect your private key from being stolen i.e. impersonation drawback: passphrase must be typed to use your key ssh-agent

DSA and RSA 1024 bit are deprecated now!

$> ssh-keygen -t rsa -b 4096 -o -a 100

# 4096 bits RSA

(better) $> ssh-keygen -t ed25519 -o -a 100

# new sexy Ed25519

Private (identity) key

~/.ssh/id_{rsa,ed25519}

Public Key

~/.ssh/id_{rsa,ed25519}.pub

8 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-18
SLIDE 18

SSH Secure Shell

SSH Setup on Windows: the OLD way

Putty Suite, includes:

http://www.chiark.greenend.org.uk/~sgtatham/putty/ -

PuTTY, the free SSH client - Pageant, an SSH authentication agent for PuTTY tools - PLink, th PuTTy CLI - PuTTYgen, an RSA and DSA key generation utility

9 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-19
SLIDE 19

SSH Secure Shell

SSH Setup on Windows: the OLD way

Putty Suite, includes:

http://www.chiark.greenend.org.uk/~sgtatham/putty/ -

PuTTY, the free SSH client - Pageant, an SSH authentication agent for PuTTY tools - PLink, th PuTTy CLI - PuTTYgen, an RSA and DSA key generation utility

PuTTY = OpenSSH

9 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-20
SLIDE 20

SSH Secure Shell

SSH Setup on Windows: the OLD way

Putty Suite, includes:

http://www.chiark.greenend.org.uk/~sgtatham/putty/ -

PuTTY, the free SSH client - Pageant, an SSH authentication agent for PuTTY tools - PLink, th PuTTy CLI - PuTTYgen, an RSA and DSA key generation utility

PuTTY = OpenSSH

Putty keys are NOT supported by OpenSSH (yet can be exported) Binding Pageant with OpenSSH agent is NOT natively supported

֒ → Third-party tools like ssh-pageant are made for that ֒ → Combine nicely with Git bash

https://git-for-windows.github.io/

with PLink, hostnames eventually refer to PuTTY Sessions

֒ → NEVER to SSH entries in ~/.ssh/config ֒ → This usage might be hidden. . . Ex: $GIT_SSH etc.

9 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-21
SLIDE 21

SSH Secure Shell

SSH Setup on Windows: the NEW way

Use MobaXterm!

http://mobaxterm.mobatek.net/

֒ → [tabbed] Sessions management ֒ → X11 server w. enhanced X extensions ֒ → Graphical SFTP browser ֒ → SSH gateway / tunnels wizards ֒ → [remote] Text Editor ֒ → . . .

10 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-22
SLIDE 22

SSH Secure Shell

SSH Basic Usage

*.<domain>-intern.com

gitlab

bastion1.<domain>.com

11 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-23
SLIDE 23

SSH Secure Shell

SSH Basic Usage

SSH *.<domain>-intern.com

gitlab

bastion1.<domain>.com

11 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-24
SLIDE 24

SSH Secure Shell

SSH Basic Usage

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

bastion1.<domain>.com

11 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-25
SLIDE 25

SSH Secure Shell

SSH Basic Usage

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

bastion1.<domain>.com

11 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-26
SLIDE 26

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

*.<domain>-intern.com

gitlab

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-27
SLIDE 27

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-28
SLIDE 28

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

DOMAIN <domain>-intern.com NOT FOUND

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-29
SLIDE 29

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/SSH/Tunnels: Port 1080, Dynamic
  • D 1080 (SOCKS 5 Proxy)

SOCKS LISTEN: localhost:1080

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-30
SLIDE 30

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/SSH/Tunnels: Port 1080, Dynamic
  • D 1080 (SOCKS 5 Proxy)

SOCKS LISTEN: localhost:1080

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-31
SLIDE 31

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/SSH/Tunnels: Port 1080, Dynamic
  • D 1080 (SOCKS 5 Proxy)

SOCKS LISTEN: localhost:1080

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-32
SLIDE 32

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/SSH/Tunnels: Port 1080, Dynamic
  • D 1080 (SOCKS 5 Proxy)

SOCKS LISTEN: localhost:1080

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-33
SLIDE 33

SSH Secure Shell

SSH Advanced Usage: SOCKS Proxy

https://gitlab.<domain>-intern.com

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/SSH/Tunnels: Port 1080, Dynamic
  • D 1080 (SOCKS 5 Proxy)

SOCKS LISTEN: localhost:1080

bastion1.<domain>.com

12 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-34
SLIDE 34

SSH Secure Shell

SSH Advanced Usage: ProxyCommand

*.<domain>-intern.com

gitlab

bastion1.<domain>.com

13 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-35
SLIDE 35

SSH Secure Shell

SSH Advanced Usage: ProxyCommand

SSH *.<domain>-intern.com

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

bastion1.<domain>.com

13 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-36
SLIDE 36

SSH Secure Shell

SSH Advanced Usage: ProxyCommand

SSH *.<domain>-intern.com + ProxyCommand + netcat

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com Host uni_gitlab Hostname gitlab ProxyCommand ssh -q uni_bastion1 "nc %h %p" gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

Session “uni_gitlab”

  • Hostname: gitlab.<domain>-intern.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/Proxy:
  • type: local
  • Proxy hostname: bastion1.<domain>.com
  • Port: 8022
  • Username: <LOGIN>
  • Local proxy command:

plink -load “uni_bastion1” -nc %host:%port

bastion1.<domain>.com

13 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-37
SLIDE 37

SSH Secure Shell

SSH Advanced Usage: ProxyCommand

SSH *.<domain>-intern.com + ProxyCommand + netcat

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com Host uni_gitlab Hostname gitlab ProxyCommand ssh -q uni_bastion1 "nc %h %p" gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

Session “uni_gitlab”

  • Hostname: gitlab.<domain>-intern.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/Proxy:
  • type: local
  • Proxy hostname: bastion1.<domain>.com
  • Port: 8022
  • Username: <LOGIN>
  • Local proxy command:

plink -load “uni_bastion1” -nc %host:%port

bastion1.<domain>.com

13 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-38
SLIDE 38

SSH Secure Shell

SSH Advanced Usage: ProxyCommand

SSH *.<domain>-intern.com + ProxyCommand + netcat

OpenSSH ~/.ssh/config (Mac / Linux)

Host uni_* User <LOGIN> Port 8022 Host uni_bastion1 Hostname bastion1.<domain>.com Host uni_gitlab Hostname gitlab ProxyCommand ssh -q uni_bastion1 "nc %h %p" gitlab

PuTTY / PLink / Pageant (Windows)

Session “uni_bastion1”

  • Hostname: bastion1.<domain>.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>

Session “uni_gitlab”

  • Hostname: gitlab.<domain>-intern.com
  • Port: 8022
  • Connection/Data: username: <LOGIN>
  • Connection/Proxy:
  • type: local
  • Proxy hostname: bastion1.<domain>.com
  • Port: 8022
  • Username: <LOGIN>
  • Local proxy command:

plink -load “uni_bastion1” -nc %host:%port

bastion1.<domain>.com

13 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-39
SLIDE 39

SSH Secure Shell

SSH in Practice ~/.ssh/config

$> ssh [-X] [-p <port>] <login>@<hostname>

# Example: ssh -p 8022 svarrette@access-chaos.uni.lu Host <shortname> Port <port> User <login> Hostname <hostname>

~/.ssh/config:

֒ → Simpler commands ֒ → Bash completion

$> ssh cha<TAB> 14 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-40
SLIDE 40

SSH Secure Shell

SSH in Practice ~/.ssh/config

$> ssh [-X] [-p <port>] <login>@<hostname>

# Example: ssh -p 8022 svarrette@access-chaos.uni.lu Host *.ext_ul ProxyCommand ssh -q chaos-cluster \ "nc -q 0 %h %p" # UL HPC Platform -- http://hpc.uni.lu Host chaos-cluster Hostname access-chaos.uni.lu Host gaia-cluster Hostname access-gaia.uni.lu Host iris-cluster Hostname access-iris.uni.lu Host *-cluster User login #ADAPT accordingly Port 8022 ForwardAgent no Host <shortname> Port <port> User <login> Hostname <hostname>

~/.ssh/config:

֒ → Simpler commands ֒ → Bash completion

$> ssh cha<TAB> 14 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-41
SLIDE 41

SSH Secure Shell

SSH in Practice ~/.ssh/config

$> ssh [-X] [-p <port>] <login>@<hostname>

# Example: ssh -p 8022 svarrette@access-chaos.uni.lu Host *.ext_ul ProxyCommand ssh -q chaos-cluster \ "nc -q 0 %h %p" # UL HPC Platform -- http://hpc.uni.lu Host chaos-cluster Hostname access-chaos.uni.lu Host gaia-cluster Hostname access-gaia.uni.lu Host iris-cluster Hostname access-iris.uni.lu Host *-cluster User login #ADAPT accordingly Port 8022 ForwardAgent no Host <shortname> Port <port> User <login> Hostname <hostname>

~/.ssh/config:

֒ → Simpler commands ֒ → Bash completion

$> ssh cha<TAB> $> ssh chaos-cluster $> ssh work $> ssh work.ext_ul 14 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-42
SLIDE 42

SSH Secure Shell

SSH in Practice: Main CLI commands

15 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-43
SLIDE 43

SSH Secure Shell

DSH - Distributed / DancerâĂŹs Shell

http://www.netfort.gr.jp/~dancer/software/dsh.html.en

SSH wrapper that allows to run commands over multiple machines.

֒ → Linux / Mac OS only

$> { apt-get | yum | brew } install dsh

# Installation

16 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-44
SLIDE 44

SSH Secure Shell

DSH - Distributed / DancerâĂŹs Shell

http://www.netfort.gr.jp/~dancer/software/dsh.html.en

SSH wrapper that allows to run commands over multiple machines.

֒ → Linux / Mac OS only

$> { apt-get | yum | brew } install dsh

# Installation

Configuration: in ~/.dsh/

֒ → ~/.dsh/dsh.conf: main configuration file ֒ → ~/.dsh/machines.list: list of all nodes ֒ → ~/.dsh/group/: holds group definition

<name> Group definition: ~/.dsh/group/<name>:

֒ → simply list SSH shortnames (one name by line)

Bash completion file for DSH:

https://gist.github.com/920433.git 16 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-45
SLIDE 45

SSH Secure Shell

DSH configuration ~/.dsh/dsh.conf

############################################################ # ~/.dsh/dsh.conf # Configuration file for dsh (Distributed / Dancer’s Shell). # ‘man dsh.conf‘ for details ############################################################ verbose = 0 remoteshell = ssh showmachinenames = 1 # Specify 1 to make the shell wait for each individual invocation. # See -c and -w option for dsh(1) waitshell = 0 # whether to wait for execution # Number of parallel connection to create at the same time. #forklimit=8 remoteshellopt = -q

17 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-46
SLIDE 46

SSH Secure Shell

DSH Basic Usage

$> dsh [-c | -w] { -a | -g <group> | -m <hostname> } <command> Option Description

  • c

run the commands in parallel (default)

  • w

run the commands in sequential

  • a

run the command on all nodes listed in machines.list

  • g <group>

restrict the commands to the hosts group <group>

  • m <hostname>

run the command only on hostname

FAQ: sudo: sorry, you must have a tty to run sudo

֒ → requires to change the default configuration of sudo ֒ → Ex: to not requiring a tty to launch a sudo command Defaults:<login> !requiretty

18 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-47
SLIDE 47

UL HPC Tutorial: Getting Started

Summary

1 Introduction 2 SSH Secure Shell 3 UL HPC Tutorial: Getting Started Step by step program of this practical session 4 Hands-On: Getting Started on ULHPC

19 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-48
SLIDE 48

UL HPC Tutorial: Getting Started

Reference Tutorial Source

Tutorial Page:

http://ulhpc-tutorials.readthedocs.io/en/latest/basic/getting_started/ 20 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-49
SLIDE 49

UL HPC Tutorial: Getting Started

Platform overview.

Quick presentation of UL HPC platform and the new Iris cluster

֒ → as of 2017: 206.772 TFlops, 7952.4TB (shared) ֒ → For more details: http://hpc.uni.lu

[Redundant] Adminfront(s) Fast local interconnect (Infiniband EDR) 100 Gb/s [Redundant] Load balancer

Site <sitename>

10 GbE

Other Clusters network Local Institution Network

10/40 GbE QSFP+

GPFS / Lustre

Disk Enclosures

Site Shared Storage Area Puppet OAR Kadeploy supervision etc... Site router [Redundant] Site access server(s) Slurm

Site Computing Nodes

21 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-50
SLIDE 50

UL HPC Tutorial: Getting Started

First connection & SSH setup

Obj: Connecting for the 1st time & preparing your SSH environment Step 1a: Connect to UL HPC (Linux / Mac OS / Unix) Step 1b: Connect to UL HPC (Windows)

֒ → using MobaXTerm or Putty.

Step 2: Connect from one cluster to the other

֒ → Learn how to connect from one cluster to the other.

Step 2bis: Using SSH proxycommand to access the clusters

֒ → allow access from everywhere despite port filtering ֒ → use of SSH aliases to easier connection

Step 3: Transferring data files

֒ → from your laptop to the clusters ֒ → cover both Linux / Mac and Windows users

22 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-51
SLIDE 51

UL HPC Tutorial: Getting Started

First connection & SSH setup

Step 3a:Transferring data files on Linux / OS X / Unix

֒ → use command line tools (SCP, Rsync)

Step 3b: Windows / Linux / OS X / Unix GUI tools

֒ → Learn how to configure Filezilla

a graphical tool to transfer files to/from the clusters.

Step 3c: Windows [MobaXterm] file transfert

23 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-52
SLIDE 52

UL HPC Tutorial: Getting Started

Discovering & reserving HPC resources

Obj: How to reserve resources & use them to run your code on it ?

Step 1: the working environment

What software is installed on the nodes where can I put my files, my data, my results ?

֒ → How many space is available ?

Step 2: web monitoring interfaces

What is the status of the platform ? How many ressources are available and when ? Why is my job in pending state ?

24 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-53
SLIDE 53

UL HPC Tutorial: Getting Started

Discovering & reserving HPC resources

Step 3a: Reserving resources with Slurm

Now I want to run my script on the platform.

֒ → What should I do ? ֒ → How to use Slurm scheduler on iris cluster ?

Step 3b: Reserving resources with OAR

As above, yet using the OAR scheduler

֒ → available on gaia, chaos, g5k clusters ?

25 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-54
SLIDE 54

UL HPC Tutorial: Getting Started

Discovering & reserving HPC resources

Step 4: Using modules

I want to run a specific version of my software.

֒ → What software is available ? ֒ → How can I use them ?

Step 5 (advanced): Job management and Persistent Terminal Sessions using GNU Screen

Each time I close my SSH connection, my job is killed.

֒ → How can I make persistent terminal sessions ֒ → . . . to execute my code wthout disconnections.

Pre-requisite: screen configuration file ~/.screenrc Basic commands Sample Usage on the UL HPC platform: Kernel compilation

26 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-55
SLIDE 55

Hands-On: Getting Started on ULHPC

Summary

1 Introduction 2 SSH Secure Shell 3 UL HPC Tutorial: Getting Started Step by step program of this practical session 4 Hands-On: Getting Started on ULHPC

27 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-56
SLIDE 56

Hands-On: Getting Started on ULHPC

Hands-On 1: SSH Setup

http://ulhpc-tutorials.readthedocs.io/en/latest/basic/getting_started/

Your Turn!

Generating you SSH Key pair Connect to UL HPC (Linux / Mac OS / Unix / Windows)

֒ → Connect from your laptop/workstation to UL HPC access ֒ → Connect from one cluster to the other

Transferring files

28 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-57
SLIDE 57

Hands-On: Getting Started on ULHPC

Hands-on 2: First steps on UL HPC

UL HPC Environment

֒ → Operating System:

Debian 7 on gaia, chaos CentOS 7 on iris

֒ → Job Management: { oarsub | srun/sbatch } ֒ → Environment modules: modules

Not available on frontends, *Only* on compute nodes

֒ → (advanced) discovering GNU screen

Directory Max size Max #files Backup $HOME (gaia, chaos) 100 GB 1.000.000 YES $HOME (iris) 500 GB 1.000.000 YES $WORK (except iris) 3 TB NO $SCRATCH (except iris) 10 TB NO 29 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

  • Debian 8 (chaos/gaia)

CentOS 7 (iris) Infiniband [Q|E]DR Computing Nodes Computing Nodes

GPU

$SCRATCH $HOME $WORK Lustre (gaia only) SpectrumScale/GPFS

access

  • arsub [-I]

srun / sbatch ssh module avail module load … ./a.out mpirun … nvcc …

Internet

ssh rsync rsync icc …

slide-58
SLIDE 58

Hands-On: Getting Started on ULHPC

ULHPC Web monitoring interfaces

http://hpc.uni.lu/status/overview.html 30 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-59
SLIDE 59

Hands-On: Getting Started on ULHPC

ULHPC Web monitoring interfaces

http://hpc.uni.lu/{iris,gaia,chaos,g5k}/monika 30 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-60
SLIDE 60

Hands-On: Getting Started on ULHPC

ULHPC Web monitoring interfaces

http://hpc.uni.lu/{iris,gaia,chaos,g5k}/drawgantt 30 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-61
SLIDE 61

Hands-On: Getting Started on ULHPC

ULHPC Web monitoring interfaces

http://hpc.uni.lu/{iris,gaia,chaos,g5k}/ganglia 30 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-62
SLIDE 62

Hands-On: Getting Started on ULHPC

ULHPC Web monitoring interfaces

https://access-iris.uni.lu/slurm 31 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-63
SLIDE 63

Hands-On: Getting Started on ULHPC

Job management

If there are not enough resources available, use our reservations, add the parameters in red to your submission commands:

OAR (Gaia)

$> oarsub -I -t inner=4248619

SLURM (Iris)

$> srun –reservation=hpcschool –pty bash 32 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-64
SLIDE 64

Hands-On: Getting Started on ULHPC

Programming, quick start

choose a command line text editor load modules run a Matlab script run a R script use the available compilers compile and run a simple MPI program

33 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1

slide-65
SLIDE 65

Thank you for your attention...

Questions?

http://hpc.uni.lu High Performance Computing @ UL

  • Prof. Pascal Bouvry
  • Dr. Sebastien Varrette & the UL HPC Team

(V. Plugaru, S. Peter, H. Cartiaux & C. Parisot) University of Luxembourg, Belval Campus Maison du Nombre, 4th floor 2, avenue de l’Université L-4365 Esch-sur-Alzette mail: hpc@uni.lu

1

Introduction

2

SSH Secure Shell

3

UL HPC Tutorial: Getting Started Step by step program of this practical session

4

Hands-On: Getting Started on ULHPC 34 / 34

  • C. Parisot & UL HPC Team (University of Luxembourg)

UL HPC School 2017[bis]/ PS1