Uniform access to heterogeneous Uniform access to heterogeneous - - PowerPoint PPT Presentation

▶
uniform access to heterogeneous uniform access to
SMART_READER_LITE
LIVE PREVIEW

Uniform access to heterogeneous Uniform access to heterogeneous - - PowerPoint PPT Presentation

Oct 05, 2023 265 likes •586 views

Uniform access to heterogeneous Uniform access to heterogeneous grid infrastructures with grid infrastructures with ISGC 2008, Taiwan Sylvain Reynaud Plan Plan Introduction Infrastructure heterogeneity Interoperability solutions

slide-1
SLIDE 1

Uniform access to heterogeneous Uniform access to heterogeneous grid infrastructures with grid infrastructures with

ISGC 2008, Taiwan Sylvain Reynaud

slide-2
SLIDE 2

Uniform access to heterogeneous grid infrastructures with JSAGA 2

Plan Plan

Introduction

– Interoperability solutions – What is JSAGA ?

Middleware heterogeneity

– Security – Data management – Execution management

Infrastructure heterogeneity

– Security – Data management – Execution management

Transport data to/from

worker nodes

Conclusion

slide-3
SLIDE 3

Uniform access to heterogeneous grid infrastructures with JSAGA 3

Introduction Introduction

Interoperability solutions Interoperability solutions

The medium term solution (GIN)

– Forward jobs to other grid infrastructures

  • Adapt existing components
  • Or use CE as a gateway

The long term solution (OMII-Europe)

– Adapt components to set common interfaces

  • Standard (Open Grid Forum)
  • Use common mechanism (e.g. for security)

Client-side adaptors and translators solution

– Meta-schedulers: GridWay, GridSAM… – Client API: GAT, CoG-Kit… …and

EGEE user

regional grids regional grids cluster cluster

slide-4
SLIDE 4

Uniform access to heterogeneous grid infrastructures with JSAGA 4

Introduction Introduction

What is JSAGA ?... A simple example What is JSAGA ?... A simple example

job desc. gLite plug-ins Globus plug-ins

JSAGA JSAGA

WMS WMS

input data

SRM SRM

GridFTP GridFTP WS WS-

  • GRAM

GRAM LCG LCG-

  • CE

CE LCG LCG-

  • CE

CE WS WS-

  • GRAM

GRAM firewall

RUGBI EGEE

job

RSL

staging graph

JDL

delegate selection & files staging

job

slide-5
SLIDE 5

Uniform access to heterogeneous grid infrastructures with JSAGA 5

Plan Plan

Introduction

– Interoperability solutions – What is JSAGA ?

Middleware heterogeneity

– Security – Data management – Execution management

Infrastructure heterogeneity

– Security – Data management – Execution management

Transport data to/from

worker nodes

Conclusion

slide-6
SLIDE 6

Uniform access to heterogeneous grid infrastructures with JSAGA 6

Hide middleware heterogeneity Hide middleware heterogeneity

SAGA as the uniform interface SAGA as the uniform interface

  • Simple API for Grid Application

–

  • bject-oriented API

– to enable end-users to … develop easily grid applications

  • << functional packages for fundamental

programming capabilities >>

  • Non-standard API

– service-oriented API – to enable engine to provide SAGA capabilities as efficiently as possible

  • With low-level or optimized plug-in methods
  • Plug-ins expose supported methods & techno
  • Plug-ins throw homogeneous exceptions

jsaga-context-init jsaga-context-info jsaga-context-destroy jsaga-cat jsaga-cp jsaga-ls jsaga-logical jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-test jsaga-job-run jsaga-job-submit jsaga-job-monitor jsaga-job-getoutput protocol

Data Service Data Service SAGA API SAGA API Job File Context

control monitor

Job Service Job Service COMMANDS COMMANDS JSAGA ENGINE JSAGA ENGINE

security

Security Mgr Security Mgr

slide-7
SLIDE 7

Uniform access to heterogeneous grid infrastructures with JSAGA 7

Hide middleware heterogeneity Hide middleware heterogeneity

SAGA as the uniform interface SAGA as the uniform interface

Implementation focuses on Implementation focuses on… …

Modularity and efficiency

– see next slides…

Software quality

– plug-ins are validated with a reusable test-suite

Lightness

– minimize external dependencies

  • resolve libraries version conflicts
  • regenerate stubs
  • remove useless

dependencies

– install only what you need…

protocol

Data Service Data Service

control monitor

Job Service Job Service JSAGA ENGINE JSAGA ENGINE

security

Security Mgr Security Mgr

slide-8
SLIDE 8

Uniform access to heterogeneous grid infrastructures with JSAGA 8

Hide middleware heterogeneity Hide middleware heterogeneity

Security Security

Security Mgr Security Mgr

Security Mgr Security Mgr Data Service Data Service Job Service Job Service

slide-9
SLIDE 9

Uniform access to heterogeneous grid infrastructures with JSAGA 9

/etc/grid-security/certificates /tmp/x509up_u_$UID ~/.globus/user*.pem

Hide middleware heterogeneity Hide middleware heterogeneity

Security Security Data and execution plug-ins

– declare

  • supported security plug-ins

Security plug-ins

– declare

  • attributes usage rules
  • attributes default values

– engine set the context instance to first matching usage rule

X509 PKI InMemCred Globus

  • G. Legacy
  • G. RFC820

MyProxy VOMS Login / pwd jsaga-context-init jsaga-context-info jsaga-context-destroy planned construction done

proxy proxy path CA cert key p12 proxy pass phrase

slide-10
SLIDE 10

Uniform access to heterogeneous grid infrastructures with JSAGA 10

Hide middleware heterogeneity Hide middleware heterogeneity

Data management Data management

Data Service Data Service

Data Service Data Service Security Mgr Security Mgr Job Service Job Service

slide-11
SLIDE 11

Uniform access to heterogeneous grid infrastructures with JSAGA 11

Hide middleware heterogeneity Hide middleware heterogeneity

Data management Data management The engine enables caching

– opened connections – information about entries – content of entries through

The engine chooses the most

appropriate plug-in methods to efficiently provide the requested functionality

– e.g. third-party transfer

catalog Logical files lfn rns P h y s i c a l f i l e s planned construction done cache srb gsiftp v1 gsiftp v2 dCache gsiftp

Data Service Data Service

http https mail file ftp sftp rbyteio tar zip cache gsiftp

slide-12
SLIDE 12

Uniform access to heterogeneous grid infrastructures with JSAGA 12

Data Service Data Service

Hide middleware heterogeneity Hide middleware heterogeneity

Data management Data management

SAGA srm

Data Service Data Service

planned construction done catalog Logical files lfn rns P h y s i c a l f i l e s srb

The engine enables caching

– opened connections – information about entries – content of entries through

The engine chooses the most

appropriate plug-in methods to efficiently provide the requested functionality

– e.g. third-party transfer

cache http https mail file ftp sftp rbyteio tar zip cache gsiftp srm

slide-13
SLIDE 13

Uniform access to heterogeneous grid infrastructures with JSAGA 13

Job Service Job Service

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management Data Service Data Service Security Mgr Security Mgr Job Service Job Service

Job control Job control Job monitoring Job monitoring

slide-14
SLIDE 14

Uniform access to heterogeneous grid infrastructures with JSAGA 14

Job control Job control Job monitoring Job monitoring

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management

SAGA user interface: getStatus / waitFor

Monitoring

Plug-in interfaces: querying / listening individual job / list of jobs / filtered jobs

Monitoring and job control are

independent

– e.g. monitor with gatekeeper, gLite- LB or gsiftp LIST command

Monitoring strategy of the engine

depends on

– monitoring capabilities of the plug-in – monitoring activities of the user

  • current, on going and past activities

gatekeeper planned construction done gLite-WMS wsgram unicore6 cream gridvm query individual job listen individual job query filtered jobs PBS ssh local remote gsiftp-ls gatekeeper gLite-LB wsgram unicore6 cream gridvm … getStatus waitFor

slide-15
SLIDE 15

Uniform access to heterogeneous grid infrastructures with JSAGA 15

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management

jsaga-context-init jsaga-context-info jsaga-context-destroy jsaga-cat jsaga-cp jsaga-ls jsaga-logical jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-test jsaga-job-run jsaga-job-submit jsaga-job-monitor jsaga-job-getoutput jsaga-jobcollection-run jsaga-jobcollection-submit jsaga-jobcollection-monitor jsaga-jobcollection-getoutput language expression

Collection Mgr Collection Mgr

protocol

Data Service Data Service SAGA API SAGA API Job File Context Job Collection SAGA SAGA-

  • like +

like +

jobs file

control monitor

Job Service Job Service COMMANDS COMMANDS JSAGA ENGINE JSAGA ENGINE

security

Security Mgr Security Mgr

slide-16
SLIDE 16

Uniform access to heterogeneous grid infrastructures with JSAGA 16

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management

Collection Mgr Collection Mgr

Collection Mgr Collection Mgr Data Service Data Service Security Mgr Security Mgr Job Service Job Service

slide-17
SLIDE 17

Uniform access to heterogeneous grid infrastructures with JSAGA 17

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management JSDL + extensions

– Job collection

  • Parametric job
  • Termination rules

– Elapsed time – % jobs per status

– Individual jobs

  • Job identification
  • Application

– POSIX application – Script application

  • Resources requirements

– Candidate tags

  • Data staging

– Alternatives (e.g. local/remote) – Pre/post processing

  • Retry rules

– Elapsed time – Status

  • Selected resources

– Identifier – Information

  • Middleware specific extension

SAGA JDL RSL-2 RSL-4 planned construction done JSDL+ext.

slide-18
SLIDE 18

Uniform access to heterogeneous grid infrastructures with JSAGA 18

Hide middleware heterogeneity Hide middleware heterogeneity

Execution management Execution management basic

– Get parametric job index

default

– Get system properties (-Dxxx) – Evaluate math expressions – Manipulate strings – Format dates and floats

JEP

– Java Math Expression Parser

– https://sourceforge.net/projects/jep/ Bean Shell

– Lightweight Scripting for Java

– http://www.beanshell.org/

basic default NFunk BeanShell planned construction done

… <POSIXApplication> <Executable>myexec</Executable> <Argument>-from @{date('dd/MM/yyyy', now+24*INDEX)}</Argument> </POSIXApplication> …

slide-19
SLIDE 19

Uniform access to heterogeneous grid infrastructures with JSAGA 19

Plan Plan

Introduction

– Interoperability solutions – What is JSAGA ?

Middleware heterogeneity

– Security – Data management – Execution management

Infrastructure heterogeneity

– Security – Data management – Execution management

Transport data to/from

worker nodes

Conclusion

slide-20
SLIDE 20

Uniform access to heterogeneous grid infrastructures with JSAGA 20

Hide infrastructure heterogeneity Hide infrastructure heterogeneity

Security Security Hide middleware heterogeneity

– Security context types

  • e.g. X509, UserPass, VOMS

Hide infrastructure heterogeneity

– Context instances (e.g. VOMS)

  • Different certificates & CA files

– e.g. EGEE, a regional grid

World World Grid Grid

E G E E E G E E l

  • c

a l h

  • s

t l

  • c

a l h

  • s

t

VOMS VOMS VOMS VOMS Globus Globus

Regional grid Regional grid

C C C C

  • I

N 2 P 3 I N 2 P 3

slide-21
SLIDE 21

Uniform access to heterogeneous grid infrastructures with JSAGA 21

Hide infrastructure heterogeneity Hide infrastructure heterogeneity

Data management Data management Hide middleware heterogeneity

– Data management protocols

  • e.g. srm, srb, http, tar, gsiftp

Hide infrastructure heterogeneity

– Protocol parameters (e.g. gsiftp)

  • Different versions

– e.g. v1, v2, dCache impl., .NET

  • Different attributes

– e.g. NumStreams, BufferSize

World World Grid Grid

E G E E E G E E l

  • c

a l h

  • s

t l

  • c

a l h

  • s

t gsiftp:// (v1)

VOMS VOMS VOMS VOMS Globus Globus

Regional grid Regional grid

C C C C

  • I

N 2 P 3 I N 2 P 3

gsiftp:// (v2)

slide-22
SLIDE 22

Uniform access to heterogeneous grid infrastructures with JSAGA 22

Hide infrastructure heterogeneity Hide infrastructure heterogeneity

Execution management Execution management Hide middleware heterogeneity

– Job management technologies

  • e.g. CREAM, WMS, SSH, GK

Hide infrastructure heterogeneity

– Computing Elements (e.g. GK)

  • Different grid or site policies

– e.g. network filtering, shared FS

  • Different environment variables

– e.g. $VO_?_SW_DIR, /usr/local

  • Different attributes

– e.g. monitor service URL, shell path on cygwin, default SE URL

  • Commands available on worker

– e.g. globus-url-copy, srmcp, Scp, wget, tar

World World Grid Grid

E G E E E G E E Regional grid Regional grid l

  • c

a l h

  • s

t l

  • c

a l h

  • s

t

C C C C

  • I

N 2 P 3 I N 2 P 3

http:// tar:// srb:// gsiftp:// (v1) srm:// lfn:// gatekeeper gatekeeper gatekeeper gatekeeper wsgram wsgram WMS WMS gatekeeper gatekeeper

VOMS VOMS VOMS VOMS Globus Globus

gsiftp:// (v2)

slide-23
SLIDE 23

Uniform access to heterogeneous grid infrastructures with JSAGA 23

targeted grid or site

– domain name in URL – prefix of host name in URL

user

– grid name in URL fragment – initialized context instances

protocol / execution service

– supported context types

security context

– installed plug-ins plug-ins

JSAGA JSAGA

url://

execution execution

Hide infrastructure heterogeneity Hide infrastructure heterogeneity

Selected security context depends on Selected security context depends on… …

World World Grid Grid

E G E E E G E E Regional grid Regional grid l

  • c

a l h

  • s

t l

  • c

a l h

  • s

t

C C C C

  • I

N 2 P 3 I N 2 P 3

http:// tar:// srb:// gsiftp:// (v1) srm:// lfn:// gatekeeper gatekeeper gatekeeper gatekeeper wsgram wsgram WMS WMS gatekeeper gatekeeper

VOMS VOMS VOMS VOMS Globus Globus

gsiftp:// (v2)

slide-24
SLIDE 24

Uniform access to heterogeneous grid infrastructures with JSAGA 24

Hide infrastructure heterogeneity Hide infrastructure heterogeneity

Selected security context depends on Selected security context depends on… …

plug-ins

JSAGA JSAGA

targeted grid or site

– domain name in URL – prefix of host name in URL

user

– grid name in URL fragment – initialized context instances

protocol / execution service

– supported context types

security context

– installed plug-ins

url://

World World Grid Grid

EGEE EGEE R e g i
  • n
a l g r i d R e g i
  • n
a l g r i d localhost localhost CC CC-
  • IN2P3
IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gsiftp:// (v2) gatekeeper gatekeeper gatekeeper gatekeeper wsgram wsgram WMS WMS gatekeeper gatekeeper VOMS VOMS VOMS VOMS Globus Globus

execution execution

slide-25
SLIDE 25

Uniform access to heterogeneous grid infrastructures with JSAGA 25

Transport data to/from workers Transport data to/from workers

Transfer path depends on Transfer path depends on… …

plug-ins

JSAGA JSAGA

grid or site

– network filtering policy – commands available on workers – services available from workers (close Storage Element, shared FS) – supported context instances

data to stage execution service

– protocols supported for staging

transfer protocol

– access mode (RO, WO, RW) – third-party transfer – supported data protection level

security context

– delegation capability – shared by several jobs – installed on some worker nodes – file size – required data protection level

job job desc desc. .

World World Grid Grid

EGEE EGEE R e g i
  • n
a l g r i d R e g i
  • n
a l g r i d localhost localhost CC CC-
  • IN2P3
IN2P3 http:// tar:// srb:// gsiftp:// srm:// lfn:// gsiftp:// (v2) gatekeeper gatekeeper gatekeeper gatekeeper wsgram wsgram WMS WMS gatekeeper gatekeeper VOMS VOMS VOMS VOMS Globus Globus
slide-26
SLIDE 26

Uniform access to heterogeneous grid infrastructures with JSAGA 26

E E1

1

R R1

1

C C C' C'C'' C'' common E E E E

src src

executable

D D1

1

input data result std-error

RUGBI RUGBI

RUGBI RUGBI RUGBI EGEE CA

SMTP SMTP GSIFTP GSIFTP GSIFTP GSIFTP HTTP HTTP GSIFTP GSIFTP

X509

HTTPS HTTPS

Transport data to/from workers Transport data to/from workers

Transfer path depends on Transfer path depends on… …

grid or site

– network filtering policy – commands available on workers – services available from workers (close Storage Element, shared FS) – supported context instances

data to stage

– shared by several jobs – installed on some worker nodes – file size – required data protection level

job job job

C" C" E E

src src

D D1

1

E E C C R R1

1

E E1

1

C' C'

TAR TAR TAR TAR

execution service

– protocols supported for staging

transfer protocol

– access mode (RO, WO, RW) – third-party transfer – supported data protection level

security context

– delegation capability

slide-27
SLIDE 27

Uniform access to heterogeneous grid infrastructures with JSAGA 27

Transport data to/from workers Transport data to/from workers

An example with 3 jobs (x2) An example with 3 jobs (x2)

glite-job-list-match myJobs.jdl | ????? > EGEE-rsc.xml jsaga-jobcollection-run myJobs.jdl EGEE-rsc.xml

EGEE EGEE

????? > OpenPlast-rsc.xml jsaga-jobcollection-run myJobs.jdl OpenPlast-rsc.xml

Regional grid Regional grid

slide-28
SLIDE 28

Uniform access to heterogeneous grid infrastructures with JSAGA 28 cat << EOF > resources.xml <Selection xmlns="http://www.in2p3.fr/jsdl-extension"> <ResourceRequest name="MYJOB_@{PARAM}"> `xpath resources-OpenPlast.xml //SelectedGrid` `xpath resources-EGEE.xml //SelectedGrid` </ResourceRequest> </Selection> EOF jsaga-jobcollection-run myJobs.jdl resources.xml

Transport data to/from workers Transport data to/from workers

An example with 6 jobs An example with 6 jobs Regional grid Regional grid EGEE EGEE

slide-29
SLIDE 29

Uniform access to heterogeneous grid infrastructures with JSAGA 29

Plan Plan

Introduction

– Interoperability solutions

Middleware heterogeneity

– Security – Data management – Execution management

Infrastructure heterogeneity

– Security – Data management – Execution management

Transport data to/from

worker nodes

Conclusion

slide-30
SLIDE 30

Uniform access to heterogeneous grid infrastructures with JSAGA 30

Conclusion Conclusion

Summary of features Summary of features Hide middleware heterogeneity

– uniform interface

Hide infrastructure heterogeneity

– select security contexts, protocol versions, environment variables – transport data to/from workers

…through standard interfaces

– SAGA – JSDL

…with efficiency

– choose the most efficient strategy for job monitoring, job in/out staging, file operations – enable caching connections, files information and content

…with flexibility

– extensible (plug-ins) – customizable (configuration) – use infrastructures as they are – middleware specific features

Describe your job once, submit it worldwide ! Describe your job once, submit it worldwide !

http://grid.in2p3.fr/jsaga/

slide-31
SLIDE 31

Uniform access to heterogeneous grid infrastructures with JSAGA 31

Conclusion Conclusion

Related projects Related projects

Many plug-ins contributed by JSAGA is being integrated into…

– Elis@

  • a web portal for submitting jobs to industrial and

research grid infrastructures

– JJS (Java Job Submission)

  • a tool for submitting job collections to EGEE
  • optimized for short-life jobs (resource selection

based on QoS observed while submitting jobs)

– JUX (Java Universal eXplorer)

  • a multi-protocol file browser