HyperLoom Stanislav Bhm, Vojt ch Cima https://hyperloom.eu - - PowerPoint PPT Presentation

▶

hyperloom

HyperLoom Stanislav Bhm, Vojt ch Cima https://hyperloom.eu - - PowerPoint PPT Presentation

Jan 13, 2023 218 likes •729 views

HyperLoom Stanislav Bhm, Vojt ch Cima https://hyperloom.eu It4Innovations / ADAS / DEF 10:00-11:30 Intoduction to HyperLoom Hands-on: 'Hello world' 12:30-14:00 Hands-on: Building a real pipeline Best practices

slide-1

SLIDE 1

HyperLoom

Stanislav Böhm, Vojtěch Cima https://hyperloom.eu It4Innovations / ADAS / DEF

slide-2

SLIDE 2

10:00-11:30

Intoduction to HyperLoom
Hands-on: 'Hello world'

12:30-14:00

Hands-on: Building a real pipeline
Best practices

14:30-16:00

Bring your own problems

slide-3

SLIDE 3

E CAPE

slide-4

SLIDE 4

Chemogenomics

gene GSK3B

+

ZZRDAWZXJBVQSO-UYBDAZJANA-N ZZRDINJRCNXHIQ-MCRMGHHCNA-N ZZRDSHKJUCIYQL-GPQMBLKYNA-N ZZRDKBQKAVQCMK-LILDFLRNNA-N

=

Inactive Inactive Active Inactive

slide-5

SLIDE 5 GSK3B + ZZRDAWZXJBVQSO = inactive GSK3B + ZZRDINJRCNXHIQ = inactive ... GSK3B + ZZRDSHKJUCIYQL = active GSK3B + ZZRDKBQKAVQCMK = inactive

labeled dataset

slide-6

SLIDE 6

M

modelling (machine learning) model training data (~80%)

...

slide-7

SLIDE 7

M

modelling (machine learning) model

V

validation model accuracy training data (~80%)

...

validation data (~20%)

slide-8

SLIDE 8

M V

training data (~80%)

...

TF1

validation data (~20%)

VF1

M V

training data (~80%)

TF2 VF2

validation data (~20%)

slide-9

SLIDE 9 M V M V M V M V M V

average accuracies cross-validated accuracy

slide-10

SLIDE 10 M V M V M V M V M V

cross-validated accuracy (for given parametrization)

P

modelling parameterization

slide-11

SLIDE 11

P1 P2 P3

slide-12

SLIDE 12

P1 P2 P3

pick best model best model

slide-13

SLIDE 13

best model (but only for a single target!)

slide-14

SLIDE 14

slide-15

SLIDE 15

Duration of tasks

HPC Schedulers PBS SLURM

slide-16

SLIDE 16

Duration of tasks

Workflow systems Luigi Airflow DAGMAN HPC Schedulers PBS SLURM

slide-17

SLIDE 17

Duration of tasks

Workflow systems Luigi Airflow DAGMAN HPC Schedulers PBS SLURM Task-based programming OmpSs StarPU

slide-18

SLIDE 18

Duration of tasks

Workflow systems Luigi Airflow DAGMAN HPC Schedulers PBS SLURM Task-based programming OmpSs StarPU

slide-19

SLIDE 19

Duration of tasks

Map Reduce Spark Hadoop Workflow systems Luigi Airflow DAGMAN HPC Schedulers PBS SLURM Task-based programming OmpSs StarPU

slide-20

SLIDE 20

Duration of tasks

Map Reduce Spark Hadoop Workflow systems Luigi Airflow DAGMAN HPC Schedulers PBS SLURM Task-based programming OmpSs StarPU

Dask/ Distributed

Rain Berkeley Ray

pycompss

slide-21

SLIDE 21

HyperLoom

https://code.it4i.cz/ADAS/loom

slide-22

SLIDE 22

client server worker worker worker

Architecture

pipeline

slide-23

SLIDE 23

client server worker worker worker

Architecture

server schedules tasks for execution on workers

slide-24

SLIDE 24

client server worker worker worker

Architecture

workers process tasks

slide-25

SLIDE 25

client server worker worker worker

Architecture

server returns results to client

results

slide-26

SLIDE 26

client server worker worker worker

Deployment

pipeline results

single PBS job

slide-27

SLIDE 27

HyperLoom Checklist

Task Output Inputs

Intra-node tasks

slide-28

SLIDE 28

HyperLoom Checklist

Task Output Inputs

Intra-node tasks

.py ./binary

Python code & Extern programs

slide-29

SLIDE 29

HyperLoom Checklist

Task Output Inputs

Intra-node tasks

.py ./binary

Python code & Extern programs

100k+

Many tasks

slide-30

SLIDE 30

HyperLoom Checklist

Task Output Inputs

Intra-node tasks

.py ./binary

Python code & Extern programs

100k+

Many tasks Inter-depenedant tasks

slide-31

SLIDE 31

HyperLoom Checklist

Task Output Inputs

Intra-node tasks

.py ./binary

Python code & Extern programs

100k+

Many tasks Inter-depenedant tasks Heterogenous tasks

slide-32

SLIDE 32

Example 1

import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

slide-33

SLIDE 33

Example 1

import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

const const merge

slide-34

SLIDE 34

Example 1

import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

const const merge

slide-35

SLIDE 35

Example 1

import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

const const merge

slide-36

SLIDE 36

Example 1

import loom.client as lc # Pipeline Definition task1 = lc.tasks.const("Hello ") task2 = lc.tasks.const("world!") task3 = lc.tasks.merge((task1, task2)) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task3) # Gather result result = future.gather()

const const merge

slide-37

SLIDE 37

Example 2

import loom.client as lc # Pipeline Definition task1 = lc.tasks.run("/bin/hostname") # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

slide-38

SLIDE 38

Example 3A

import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname") for i in range(100)] task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

slide-39

SLIDE 39

Example 3A

import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname") for i in range(100)] task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather()) Output: r37u12n989 r36u11n321 r36u11n320 r37u12n989 r37u12n989

slide-40

SLIDE 40

Example 3B

import loom.client as lc # Pipeline Definition tasks = [lc.tasks.run("/bin/hostname > output", shell=True,

utputs=["output"]) for i in range(100)]

task1 = lc.tasks.merge(tasks) # Create a client object client = lc.Client("localhost", 9010) # Submit tasks/pipeline future = client.submit_one(task1) # Gather result print(future.gather())

slide-41

SLIDE 41

Mapping inputs/outputs

task_load_data = ... tasks = lc.tasks.run("./my-train --train-data=train.dat --output=model.dat", inputs=[(task_load_data, "train.dat")],

utputs=["model.dat"])

slide-42

SLIDE 42

Example 4

import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

slide-43

SLIDE 43

Example 4

import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

slide-44

SLIDE 44

Example 4

import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

slide-45

SLIDE 45

Example 4

import loom.client as lc @lc.tasks.py_task() def tensorflow_task(param): import tensorflow as tf p = param.read() hello = tf.constant("{} from TensorFlow!".format(p)) sess = tf.Session() return sess.run(hello) param = lc.tasks.const("Hello") task = tensorflow_task(param) task.resource_request = lc.tasks.cpus(24) c = lc.Client("localhost", 9010) future = c.submit_one(task) result = future.gather()

slide-46

SLIDE 46

Example 5

task = ... task.resource_request = lc.tasks.cpus(24) task.checkpoint_path = "/scratch/myproject/task-1"

slide-47

SLIDE 47

Lore

Progress Node utilization Scheduling info Loom reporting

slide-48

SLIDE 48

Hands-on

user@loginX.salomon$ cp -r /scratch/temp/hyperloom-tutorial/examples/ex1 . user@loginX.salomon$ cd ex1 user@loginX.salomon$ qsub ex1.pbs

slide-49

SLIDE 49

Hands-on

user@loginX.salomon$ cp -r /scratch/temp/hyperloom-tutorial/examples/ex1 . user@loginX.salomon$ cd ex1 user@loginX.salomon$ qsub ex1.pbs user@loginX.salomon$ ml load loom user@loginX.salomon$ python3 -m pip install bokeh==0.12.6 --user user@loginX.salomon$ python3 -m loom.lore lore_logs user@loginX.salomon$ firefox output.html # Or download output.html and open it locally

slide-50

SLIDE 50

Snailwatch