Homogenization of access to DCI Carlos Blanco, A. S. Cofio, V. - - PowerPoint PPT Presentation

▶
homogenization of access to dci
SMART_READER_LITE
LIVE PREVIEW

Homogenization of access to DCI Carlos Blanco, A. S. Cofio, V. - - PowerPoint PPT Presentation

Dec 15, 2022 169 likes •402 views

http://www.meteo.unican.es A multidisciplinary approach to weather & climate Santander Meteorology Group A multidisciplinary approach for weather & climate Homogenization of access to DCI Carlos Blanco, A. S. Cofio, V.

slide-1
SLIDE 1

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Homogenization of access to DCI

EGI (19-23 May 2014), Helsinki, Finland

Carlos Blanco, A. S. Cofiño, V. Fernandez-Quiruelas

Santander Meteorology Group

Department of Applied Mathematics and Computer Sciences Universidad de Cantabria, Santander, Spain

Aknowledgments: This work is funded by the Spanish PLAN NACIONAL de I+D+i 2008-2011 (Ref.# CGL2011-28864, WRF4G) and ERDF

slide-2
SLIDE 2

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Outline

  • Statement of the problem
  • DRM4G
  • How does it work ?
  • Application example: WRF4G
  • Conclusions
slide-3
SLIDE 3

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Statement of the problem

Workstation Oceano ce130.cern.ch marcream01.in2p3.fr gridgate.cs.tcd.ie CARLOS

  • Different types

PC Blizzard

Grid

Supercomputer MN3 (BSC)

DCI available

slide-4
SLIDE 4

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Statement of the problem

UC Workstation Oceano ce130.cern.ch marcream01.in2p3.fr gridgate.cs.tcd.ie CARLOS

  • Different types
  • Different institutions/organizations

PC Blizzard

Grid

PRACE Infrastructure Supercomputer MN3 (BSC)

DCI available

EGI ( VO esr )

slide-5
SLIDE 5

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Statement of the problem

UC Workstation Oceano ce130.cern.ch marcream01.in2p3.fr gridgate.cs.tcd.ie CARLOS

  • Different types
  • Different institutions/organizations
  • Different access protocols

PC Blizzard

Grid

PRACE Infrastructure Supercomputer MN3 (BSC)

DCI available

S S H SSH C R E A M

EGI ( VO esr )

slide-6
SLIDE 6

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Requirements

  • Uniform access to the available resources
  • Simple interface
  • Easy configuration
  • Easy installation (batteries included)
  • Robust and scalable
slide-7
SLIDE 7

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Outline

  • Statement of the problem
  • DRM4G
  • How does it work ?
  • Application example: WRF4G
  • Conclusions
slide-8
SLIDE 8

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Main features

  • Based on GridWay meta-scheduler
  • CLI interface
  • Tools to manage private/public keys and grid

certificates

  • Download and run
  • Requirements: python >= 2.4 and < 3.0
  • Scalable ~100,000 jobs
slide-9
SLIDE 9

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

DRM4G allows the user to merge different computing resources concurrently in a transparent way:

Local resources Remote resources (ssh) Grid infrastructures (Globus , CREAM)

  • Directly in a shell session
  • Interacting with LRMS
  • PBS
  • SGE
  • SLURM
  • LSF
  • …

DCI access

slide-10
SLIDE 10

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Outline

  • Statement of the problem
  • DRM4G
  • How does it work ?
  • Application example : WRF4G
  • Conclusions
slide-11
SLIDE 11

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Deployment

$ wget http://meteo.unican.es/drm4g-1.1.2-x86_64.tar.gz $ tar xzf drm4g-1.1.2-x86_64.tar.gz $ export PATH=$PWD/drm4g/bin:$PATH $ drm4g daemon start

Creating a DRM4G local configuration in '/home/user/.drm4g' Creating '/home/user/.drm4g/etc' directory Creating '/home/user/.drm4g/var' directory Starting DRM4G daemon .............. OK

$ drm4g daemon status

DRM4G daemon is running

$ drm4g resource edit

The resources.conf is edited and loaded dynamically

slide-12
SLIDE 12

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es resources.conf

Resource Configuration I

[DEFAULT] enable = True communicator = local frontend = localhost lrms = fork [blizzard] ncores = 1 [mn3] communicator = ssh username = user frontend = mn3.bsc.es auth = ssh-agent lrms = lsf queue = mic ncores = 400

slide-13
SLIDE 13

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es resources.conf

Resource Configuration II

[oceano] communicator = ssh username = user frontend = oceano.unican.es auth = private_key | ~/.ssh/id_rsa ncores = 60 [esr] communicator = ssh username = user frontend = ui.macc.unican.es auth = private_key | ~/.ssh/id_rsa lrms = cream vo = esr

slide-14
SLIDE 14

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Resources

$ drm4g resource esr copy-key –-identity-file id_rsa.pub

Introduce your password to log into ui.macc.uncan.es : id_rsa.pub added to authorized_keys file

$ drm4g resource esr create-proxy

Insert your GRID pass: Insert MyProxy password: Your identity: /DC=es/DC=irisgrid/O=unican/CN=josecarlos.blanco Creating proxy ........................................ Done Proxy Verify OK Your proxy is valid until: Wed May 7 10:22:42 2014 A proxy valid for 168 hours (7.0 days) for user carlos now exists on px.grid.sara.nl.

slide-15
SLIDE 15

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Resources

$ drm4g resource list

Name State

  • blizzard enabled

esr enabled

  • ceano enabled

mn3 enabled

$ drm4g resource check-frontends

  • -> Resource 'esr' ...

The front-end ui01.macc.unican.es is accessible

  • -> Resource 'mn3' ...

The front-end mn3.bsc.es is accessible

  • -> Resource 'oceano' ...

The front-end oceano.unican.es is accessible

slide-16
SLIDE 16

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

$ cat wrf4g.jt EXECUTABLE = /bin/bash ./wrfg4.sh INPUT_FILES = WRFG4-1.5.1-x86_64.tar.gz, wrfg4.sh REQUIREMENTS = ARCH = 'x86_64' $ drm4g job submit wrf4g.jt

Job id: 0

$ drm4g job status list

JID EM START END EXIT NAME HOST 0 actv 18:05:33 --:--:-- -- wrf4g.jt esr_sbgce2.in2p3.fr

$ drm4g job cancel –-jid 0

Submit, cancel and monitor jobs

slide-17
SLIDE 17

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Outline

  • Statement of the problem
  • DRM4G
  • How does it work ?
  • Application example: WRF4G
  • Conclusions
slide-18
SLIDE 18

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

WRF4G is a framework developed by the Santander Meteorology Group, provides:

  • Flexible WRF experiment design, execution and monitoring,

and ...

  • … the ability to run these experiments on different computing

resources at the same time in a transparent way (DRM4G).

WRF4G

Crédit : Laurent Fairhead (LMD/CNRS)

slide-19
SLIDE 19

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Outline

  • Statement of the problem
  • DRM4G
  • How does it work ?
  • Application example: WRF4G
  • Conclusions
slide-20
SLIDE 20

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Conclusions

As we have seem DRM4G is ….

  • Easy

–To install –To configure

  • Useful to other frameworks

–Providing a single access point –Python API

  • Scalability (~100,000 jobs)

–Stable –Reliable

slide-21
SLIDE 21

A multidisciplinary approach to weather & climate

Santander Meteorology Group

A multidisciplinary approach for weather & climate

http://www.meteo.unican.es

Thank you!

Contact: blancojc@unican.es

More info: https://www.meteo.unican.es

Aknowledgments: This work is funded by the Spanish PLAN NACIONAL de I+D+i 2008-2011 (Ref.# CGL2011-28864, WRF4G) and ERDF