Scientific Workflows Shahbaz Memon 1,2 , Dorothee Vallot 3 , Helmut - - PowerPoint PPT Presentation

▶

Mar 21, 2024 269 likes •514 views

Scientific Workflows Shahbaz Memon 1,2 , Dorothee Vallot 3 , Helmut Neukirchen 2 , Morris Riedel 1,2 1 Jlich Supercomputing Centre, Germany 2 University of Iceland, Iceland 3 Uppsala University, Sweden Outline Part 1 Automated Glacier

SLIDE 1

Scientific Workflows

Shahbaz Memon1,2, Dorothee Vallot3, Helmut Neukirchen2, Morris Riedel1,2

1Jülich Supercomputing Centre, Germany 2University of Iceland, Iceland 3Uppsala University, Sweden

SLIDE 2

Outline

Part 1 – Automated Glacier Modeling through

Scientific Workflows

Part 2 – Tutorial: Workflows

SLIDE 3

Part 1

Automated Glacier Modeling through Scientific Workflows

SLIDE 4

Motivation

Analysis pipelines – output of one as input for

next job

Iterative execution – Until certain threshold
Agglomerative and shared output data
Nested loop/control constructs (if-else, while,

for) to any depth

Easy to use GUI for scientists

SLIDE 5

Use Case: Kronebreen, Svalbard

Seasonal evolution of basal

sliding

– Inverse modelling of the surface velocity, observed for every 11 days for 3 years – Implementation of different sliding laws

SLIDE 6

Ice flow: Movement of the ice

Deformation of ice

– Fracture (crevasses) – Internal deformation or creep

Basal sliding

BED GLACIER

Velocity profile

Sliding Internal deformation

Photo: D. Vallot

24 August, 2017 Dorothée Vallot 6

Can be modelled as a continuous process

SLIDE 7

Elmer/Ice package

Elmer: Open Source Finite

Element Software for Multiphysical Problems

Elmer/ICE: Additional routines

for glaciology applications (i.e. Glen’s flow law)

Finite Element Method
Parallel processing

24 August, 2017 Dorothée Vallot 7

SLIDE 8

Calving process

Accumulation

Ablation

Calving occurs when tensile stresses are large enough to propagate fractures through the ice

24 August, 2017 Dorothée Vallot 8

?

Can be modelled as a discrete process

SLIDE 9

Helsinki Discrete Element Model (HiDEM) for ice

Numerical particle-based

model of calving (Åström et al., 2013) from 2D to 3D

Glacier divided into discrete

particles

24 August, 2017 Dorothée Vallot 9

Water depth

WATE R ICE

SLIDE 10

Coupling of Elmer/ICE and HiDEM

Input to Elmer/ICE

– Surface and bed DEM – Front position (contour)

Output of Elmer/ICE as an

input to HiDEM

– Basal friction – Surface elevation

Output to HiDEM

– Front position (contour)

SLIDE 11

Workflow Model

Main script

Shared Preprocessing ElmerSolver Generate Mesh Elmer-> Particle Particle Calving

Particle-> Elmer

Parallel Parallel Serial Serial

Loop Until Num. Observations

Serial Serial

Cluster 1 Cluster 2

SLIDE 12

Common Workflow Issues

Portability on different target resources and

batch systems

Script adoption to separate execution

environment

Manual data import and export – e.g. job outputs
Manual job dependency tracking
Boiler plate code: A mix of job, data, and

application management in one script

Single User rather than for community

SLIDE 13

UNICORE: Workflow Middleware

A set of integrated computing and storage

interfaces

– Open source – Workflow support – Standards-based, BSD licence – Seamless, secure, and intuitive – Supporting various batch systems: LoadLeveler, SLURM, Torque – Mature security for distributed research infrastructures and supercomputing deployments – Application integration – Platform independent: Implemented in JAVA and Python

SLIDE 14

UNICORE Architecture

SLIDE 15

Workflow Features

Simple graphs
Workflow Variables
Loops and Control

– For-each, while..

Condition

– Exit code, file existence, file size

Clients

– UNICORE Rich Client – Commandline Client

SLIDE 16

UNICORE Rich Client

Building, submitting and

monitoring jobs and workflows

Integrated data and storage

management

Pluggable credentials
“Simple view “ for novice

users

Based on the Eclipse

framework

SLIDE 17

Workflow Model

Main script

Shared Preprocessing ElmerSolver Generate Mesh Elmer-> Particle Particle Calving

Particle-> Elmer

Parallel Parallel Serial Serial

Loop Until Num. Observations

Serial Serial

SLIDE 18

Workflow Realization In UNICORE

Shared Preprocessing ElmerSolver Generate Mesh Elmer-> Particle Particle Calving Particle-> Elmer

Prepare shared input and output directory Generate Mesh through gmsh and ElmerGrid Elmer run and prepare output for calving Particle calving and convert output for the Elmer run

Parallel Parallel Serial Serial 2 1 3 5 Loop Until Num. Observations

Shared variables

Serial

Data Conversion Elmer -> Particle

Data Conversion Particle -> Elmer

6 Serial 1 2 3 4 5 6

SLIDE 19

Demo: UNICORE Rich Client

https://www.youtube.com/watch?v=phcV8jeBT8s

SLIDE 20

Conclusion and Outlook

Proof of concept: deployed on a single virtual

machine ( 20 cores, 20 GB RAM)

An initial effort required to develop the first

template, which can be reused by many users

Next steps

– Ready to be deployed on CSC (Finland) resources – Further evaluation and publish results

SLIDE 21

Part 2

Tutorial: UNICORE Workflows

SLIDE 22

Resources Involved

Sisu – CSC, Finland

– Cray XC40, 1688 Nodes, 3376 Procs, 64G/Node

Taito – CSC, Finland

– HP, 929 Nodes – Combination of Sandy Bridge and Haswell Nodes

JURECA – JSC, Germany

– 1872 Nodes, 128/256/512 G / Node

Cloud VM – Ubuntu 12.04 LTS – JSC Germany

SLIDE 23

Scenarios

1) Run a simple, single-site script job 2) Implement a multi-site,

sequential workflow
an iterative workflow