Grid Runtime Environment Solutions in KnowARC Daniel Bayer Steffen - - PowerPoint PPT Presentation

▶
grid runtime environment solutions in knowarc
SMART_READER_LITE
LIVE PREVIEW

Grid Runtime Environment Solutions in KnowARC Daniel Bayer Steffen - - PowerPoint PPT Presentation

Feb 22, 2024 335 likes •511 views

Grid Runtime Environment Solutions in KnowARC Daniel Bayer Steffen Mller University of Lbeck, Institute for Neuro- and Bioinformatics EGEE07 Budapest, October 2007 Bayer, Mller (Uni Lbeck) RE Solutions in KnowARC EGEE07 1 / 14

slide-1
SLIDE 1

Grid Runtime Environment Solutions in KnowARC

Daniel Bayer Steffen Möller

University of Lübeck, Institute for Neuro- and Bioinformatics

EGEE’07

Budapest, October 2007

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 1 / 14

slide-2
SLIDE 2

Outline

Outline

Static Runtime Environments Introduction The Runtime Envrionment Registry Summary Dynamic Runtime Environments Introduction The Janitor Deployment The Catalog Outlook

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 2 / 14

slide-3
SLIDE 3

Static Runtime Environments Introduction

What are Runtime Environments?

◮ Grid jobs depend on certain programs or data being available on the

worker node.

◮ These are called Runtime Environment (RE). ◮ REs are

◮ specified in the Runtime Environment Registry. ◮ made available on the worker nodes by administrators. ◮ announced in the Grid Infosystem. ◮ used for brokering. ◮ enabled at the worker node on a per job basis. ◮ used by the job. Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 3 / 14

slide-4
SLIDE 4

Static Runtime Environments The Runtime Envrionment Registry

The Runtime Environment Registry

◮ Website at http://gridrer.csc.fi

R u n t i m e E n v i r
  • n
m e n t R e g i s t r y T h i s r e g i s t r y c
  • n
t a i n s a l i s t
  • f
G r i d R u n t i m e E n v i r
  • n
m e n t s ( R E s ) a n d s e r v e s t h e m e m b e r s
  • f
N
  • r
d u G r i d , N
  • r
d i c D a t a G r i d F a c i l i t y , S w e G r i d a n d M ✳ g r i d a n d
  • t
h e r G r i d p r
  • j
e c t s u s i n g N
  • r
d u G r i d A R C m i d d l e w a r e . F
  • r
m
  • r
e i n f
  • r
m a t i
  • n
, s e e : I n t r
  • d
u c t i
  • n
t
  • R
u n t i m e E n v i r
  • n
m e n t s a n d R e g i s t r y . H
  • w
t
  • u
s e R E ( u s e r s ) H
  • w
t
  • i
n s t a l l R E ( s i t e a d m i n s ) H
  • w
t
  • s
e t u p a n d m a i n t a i n R E ( R E m a i n t a i n e r s ) T h e c
  • n
c e p t
  • f
t h e R E R i t s e l f i s u n d e r d e v e l
  • p
m e n t . P l e a s e c
  • n
t r i b u t e t h r
  • u
g h t h e n
  • r
d u g r i d ✳ d i s c u s s @ n
  • r
d u g r i d .
  • r
g m a i l i n g l i s t
  • r
m a i l d i r e c t l y t
  • R
E R m a i n t a i n e r . L i s t
  • f
R u n t i m e E n v i r
  • n
m e n t s T h e f i r s t e n t r y f
  • r
e a c h R E i s t h e r e s e r v e d n a m e
  • f
t h e R E , a n d t h e v e r s i
  • n
n u m b e r
  • f
t h e l a t e s t s t a b l e ( p r e f e r r e d , c u r r e n t ) r e l e a s e . O t h e r a v a i l a b l e v e r s i
  • n
s a r e d
  • c
u m e n t e d
  • n
t h e R E ' s H
  • m
e p a g e . A P P S / B I O / J A S P A R ❘ C O R E ❘ 1 . D e s c r i p t i
  • n
: J A S P A R ✳ C O R E R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / J A S P A R ✳ C O R E / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 2 A P P S / B I O / L A G A N ❘ 1 . 2 D e s c r i p t i
  • n
: L A G A N R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / L A G A N / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 2 1 A P P S / B I O / T F B S ❘ . 5 . D e s c r i p t i
  • n
: T F B S R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / T F B S / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 1 6 A P P S / C A S / S I M I C S ❘ 2 . 2 . 7 ❘ 1 D e s c r i p t i
  • n
: C
  • m
p u t e r A r c h i t e c t u r e S i m u l a t
  • r
R E H
  • m
e p a g e : h t t p : / / w w w . c e . c h a l m e r s . s e / r e s e a r c h / g r
  • u
p / h p c a g / g u i d e / g r i d / S i m i c s R E . h t m l S t a t u s : F i r s t v e r s i
  • n
L a s t u p d a t e : 2 5 ✳ 5 ✳ 1 6 A P P S / C H E M / D A L T O N ❘ 1 . 2 ❘ 1 . D e s c r i p t i
  • n
: Q u a n t u m c h e m i s t r y p r
  • g
r a m R E H
  • m
e p a g e : h t t p : / / r e . s w e g r i d . s e / A P P S / C H E M / D A L T O N ✳ 1 . 2 ✳ 1 . . h t m l S t a t u s : P i l
  • t
R E s e t u p w
  • r
k i n g , a l s
  • f
  • r
p a r a l l e l r u n s u n d e r S G E . L a s t u p d a t e : 2 5 ✳ 3 ✳ 1 4 A P P S / C H E M / G A M E S S ❘ 2 4 1 1 2 2 ❘ D e s c r i p t i
  • n
: Q u a n t u m c h e m i s t r y p r
  • g
r a m f
  • r
t h e c a l c u l a t i
  • n
  • f
S C F w a v e f u n c t i
  • n
s r a n g i n g f r
  • m
R H F , R O H F , U H F , G V B , a n d M C S C F . R E H
  • m
e p a g e : h t t p : / / g r i d . u i
  • .
n
  • /
R E / G A M E S S _ R E . h t m l S t a t u s : P i l
  • t
R E s e t u p w
  • r
k i n g L a s t u p d a t e : 2 5 ✳ 1 ✳ 3 1 D e s c r i p t i
  • n
: R E H
  • m
e p a g e : h t t p : / / s t a f f . c s c . f i / a j t / r e / P
  • v
r a y R E . h t m l S t a t u s : L a s t u p d a t e :

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 4 / 14

slide-5
SLIDE 5

Static Runtime Environments The Runtime Envrionment Registry

The Runtime Environment Registry

R u n t i m e E n v i r
  • n
m e n t R e g i s t r y T h i s r e g i s t r y c
  • n
t a i n s a l i s t
  • f
G r i d R u n t i m e E n v i r
  • n
m e n t s ( R E s ) a n d s e r v e s t h e m e m b e r s
  • f
N
  • r
d u G r i d , N
  • r
d i c D a t a G r i d F a c i l i t y , S w e G r i d a n d M ✳ g r i d a n d
  • t
h e r G r i d p r
  • j
e c t s u s i n g N
  • r
d u G r i d A R C m i d d l e w a r e . F
  • r
m
  • r
e i n f
  • r
m a t i
  • n
, s e e : I n t r
  • d
u c t i
  • n
t
  • R
u n t i m e E n v i r
  • n
m e n t s a n d R e g i s t r y . H
  • w
t
  • u
s e R E ( u s e r s ) H
  • w
t
  • i
n s t a l l R E ( s i t e a d m i n s ) H
  • w
t
  • s
e t u p a n d m a i n t a i n R E ( R E m a i n t a i n e r s ) T h e c
  • n
c e p t
  • f
t h e R E R i t s e l f i s u n d e r d e v e l
  • p
m e n t . P l e a s e c
  • n
t r i b u t e t h r
  • u
g h t h e n
  • r
d u g r i d ✳ d i s c u s s @ n
  • r
d u g r i d .
  • r
g m a i l i n g l i s t
  • r
m a i l d i r e c t l y t
  • R
E R m a i n t a i n e r . L i s t
  • f
R u n t i m e E n v i r
  • n
m e n t s T h e f i r s t e n t r y f
  • r
e a c h R E i s t h e r e s e r v e d n a m e
  • f
t h e R E , a n d t h e v e r s i
  • n
n u m b e r
  • f
t h e l a t e s t s t a b l e ( p r e f e r r e d , c u r r e n t ) r e l e a s e . O t h e r a v a i l a b l e v e r s i
  • n
s a r e d
  • c
u m e n t e d
  • n
t h e R E ' s H
  • m
e p a g e . A P P S / B I O / J A S P A R ❘ C O R E ❘ 1 . D e s c r i p t i
  • n
: J A S P A R ✳ C O R E R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / J A S P A R ✳ C O R E / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 2 A P P S / B I O / L A G A N ❘ 1 . 2 D e s c r i p t i
  • n
: L A G A N R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / L A G A N / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 2 1 A P P S / B I O / T F B S ❘ . 5 . D e s c r i p t i
  • n
: T F B S R E H
  • m
e p a g e : h t t p : / / w w w . g r i d . t s l . u u . s e / R T E s / T F B S / S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✳ 9 ✳ 1 6 A P P S / C A S / S I M I C S ❘ 2 . 2 . 7 ❘ 1 D e s c r i p t i
  • n
: C
  • m
p u t e r A r c h i t e c t u r e S i m u l a t
  • r
R E H
  • m
e p a g e : h t t p : / / w w w . c e . c h a l m e r s . s e / r e s e a r c h / g r
  • u
p / h p c a g / g u i d e / g r i d / S i m i c s R E . h t m l S t a t u s : F i r s t v e r s i
  • n
L a s t u p d a t e : 2 5 ✳ 5 ✳ 1 6 A P P S / C H E M / D A L T O N ❘ 1 . 2 ❘ 1 . D e s c r i p t i
  • n
: Q u a n t u m c h e m i s t r y p r
  • g
r a m R E H
  • m
e p a g e : h t t p : / / r e . s w e g r i d . s e / A P P S / C H E M / D A L T O N ✳ 1 . 2 ✳ 1 . . h t m l S t a t u s : P i l
  • t
R E s e t u p w
  • r
k i n g , a l s
  • f
  • r
p a r a l l e l r u n s u n d e r S G E . L a s t u p d a t e : 2 5 ✳ 3 ✳ 1 4 A P P S / C H E M / G A M E S S ❘ 2 4 1 1 2 2 ❘ D e s c r i p t i
  • n
: Q u a n t u m c h e m i s t r y p r
  • g
r a m f
  • r
t h e c a l c u l a t i
  • n
  • f
S C F w a v e f u n c t i
  • n
s r a n g i n g f r
  • m
R H F , R O H F , U H F , G V B , a n d M C S C F . R E H
  • m
e p a g e : h t t p : / / g r i d . u i
  • .
n
  • /
R E / G A M E S S _ R E . h t m l S t a t u s : P i l
  • t
R E s e t u p w
  • r
k i n g L a s t u p d a t e : 2 5 ✳ 1 ✳ 3 1 D e s c r i p t i
  • n
: R E H
  • m
e p a g e : h t t p : / / s t a f f . c s c . f i / a j t / r e / P
  • v
r a y R E . h t m l S t a t u s : L a s t u p d a t e :

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 4 / 14

slide-6
SLIDE 6

Static Runtime Environments The Runtime Envrionment Registry

The Runtime Environment Homepage

N a m e : A P P S / B I O / J A S P A R ✍ C O R E ✍ 1 . D e s c r i p t i
  • n
: J A S P A R ✍ C O R E S t a t u s : A v a i l a b l e L a s t u p d a t e : 2 6 ✍ 9 ✍ 2 J A S P A R ✴ C O R E R u n t i m e E n v i r
  • n
m e n t H
  • m
e p a g e J A S P A R C O R E i s a b i
  • i
n f
  • r
m a t i c s d a t a b a s e . T h i s p a g e d e s c r i b e s t h e J A S P A R ✍ C O R E r u n t i m e e n v i r
  • n
m e n t f
  • r
g r i d s y s t e m s b a s e d
  • n
t h e N
  • r
d u g r i d A R C m i d d l e w a r e . I n t e r f a c e d e f i n i t i
  • n
O n t h e r e m
  • t
e s y s t e m J A S P A R ✍ C O R E R E i n i t i a l i z e s t h e f
  • l
l
  • w
i n g e n v i r
  • n
m e n t v a r i a b l e s : J A S P A R _ C O R E F u l l p a t h t
  • t
h e d i r e c t
  • r
y
  • f
t h e J A S P A R ✍ C O R E d a t a b a s e f i l e s V e r s i
  • n
s O l d v e r s i
  • n
s S y s t e m a d m i n i s t r a t
  • r
g u i d e I n s t a l l a t i
  • n
  • f
t h e R E A t a r b a l l c
  • n
t a i n i n g t h e J A S P A R ✍ C O R E d a t a b a s e f i l e s . V e r i f i c a t i
  • n
  • f
t h e R E A t e s t f
  • r
v e r i f y i n g t h e r u n t i m e e n v i r
  • n
m e n t a r e g i v e n b e l
  • w
: J a s p a r C
  • r
e T e s t T h e e x p e c t e d
  • u
t c
  • m
e
  • f
t h e t e s t i s i n c l u d e d i n t h e t a r b a l l . C
  • n
t a c t i n f
  • r
m a t i
  • n
Q u e s t i
  • n
s r e l a t e d t
  • t
h e J A S P A R ✍ C O R E r u n t i m e e n v i r
  • n
m e n t , s u c h a s q u e s t i
  • n
s a b
  • u
t i n s t a l l a t i
  • n
, f e a t u r e r e q u e s t s a n d i m p r
  • v
e m e n t s u g g e s t i
  • n
s , c a n b e m a i l e d t
  • R
E m a i n t a i n e r m a t t i a s . e l l e r t @ t s l . u u . s e .

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 5 / 14

slide-7
SLIDE 7

Static Runtime Environments Summary

Summary

◮ traditional static REs are well suited to provide a few, stable and

well-known runtime environments

◮ but they are not suited for a huge amount of fast changing REs as

necessary for e. g. bioinformatics ⇒ Dynamic Runtime Environments

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 6 / 14

slide-8
SLIDE 8

Dynamic Runtime Environments Introduction

What are Dynamic Runtime Environments

◮ Dynamic Runtime Environments are REs prepared to be installed

automatically and dynamically. automated once triggered, the installation is done without human supervision dynamic installation can be triggered by requirements of incoming jobs

◮ Are implemented with help of a new Service, the Janitor

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 7 / 14

slide-9
SLIDE 9

Dynamic Runtime Environments The Janitor

The Janitor

◮ is informed by the Grid Manager about requirements of jobs and its

state ⇒ triggering dynamic installation

◮ knows which REs are available and passes this information to the Grid

Information System

◮ removes dynamically installed DREs which are unused for a long time

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 8 / 14

slide-10
SLIDE 10

Dynamic Runtime Environments The Janitor

Interactions of the Janitor with the middleware

Job Grid Manager Grid Information System Filesystem LRMS helper LRMS

  • 1. submit

searches for REs

  • 2. calles

LRMS helper

  • 3. submit

Figure: Interaction of the Janitor with the ARC 0.6 middleware

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 9 / 14

slide-11
SLIDE 11

Dynamic Runtime Environments The Janitor

Interactions of the Janitor with the middleware

Job Grid Manager Grid Information System Janitor LRMS helper LRMS

  • 1. submit

requests list

  • f REs
  • 2. registers Job
  • 3. deploy REs
  • 7. unregisters Job
  • 4. calles

LRMS helper

  • 5. retrieves

RE info

  • 6. submit

Figure: Interaction of the Janitor with the ARC 0.6 middleware

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 9 / 14

slide-12
SLIDE 12

Dynamic Runtime Environments Deployment

Deployment

◮ currently only one supported package format, TAR packages,

containing:

◮ the software to deploy and ◮ metadata to control the deployment and usage of the software

◮ other kind of packages are possible, but currently not implemented ◮ constraints set by the site admin specify the eligibility of packages ◮ a shared directory is necessary

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 10 / 14

slide-13
SLIDE 13

Dynamic Runtime Environments The Catalog

The Catalog

◮ describes all REs:

◮ name ◮ version (part of the name) ◮ RE homepage ◮ last change ◮ set of tags

◮ for DRE also describes how to deploy ◮ currently the Catalog is a RDF file, it will be a service in the future

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 11 / 14

slide-14
SLIDE 14

Dynamic Runtime Environments The Catalog

The RDF Schema

kb#1 WEKA APPS/BIO/WEKA kb#MetaPackage

kb#description kb#name ns#type

kb#2 type::app kb#Tag

kb#tag ns#name ns#type

kb#3 http://.../weka-3.4.8.tar.gz kb#TarPackage

kb#instance kb#url ns#type

kb#4 http://.../jre_1.5-1.tar.gz

kb#depends kb#url ns#type

kb#5 Debian Sid debian::sid kb#BaseSystem

kb#basesystem kb#basesystem kb#short_description kb#name ns#type

Figure: A part of the RDF graph describing the WEKA runtime environment.

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 12 / 14

slide-15
SLIDE 15

Outlook

Acceptance in the Community

◮ Current Installations:

◮ Lübeck ◮ Košice ◮ Kopenhavn

◮ Upcoming Installations:

◮ Helsinki and ◮ the NDGF Biogrid Community Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 13 / 14

slide-16
SLIDE 16

Outlook

Ongoing Development

◮ use of virtualisation, which will

◮ provide for every job a clean environment, ◮ allow to easily draw software packages from Linux distributions

◮ the RER will be implemented with help of the Catalog ◮ the Catalog will become a service ◮ expose DRE states in the Information System

Bayer, Möller (Uni Lübeck) RE Solutions in KnowARC EGEE’07 14 / 14