Introduction to gLite Middleware Malik Ehsanullah - - PowerPoint PPT Presentation

1

Introduction to gLite Middleware

Malik Ehsanullah

(ehsan@barc.gov.in)

BARC – Mumbai

2

Introduction

• The Grid relies on advanced software, called

middleware, which interfaces between resources and the applications

• gLite 3.1 based on Scientific Linux 4
• gLite 3.2 based on Scientific Linux 5

3

What gLite does?

• What gLite does not do

– Somehow my application just run faster – My application can run as long as it needs – Users can access to any resource – Users can rely of a huge amount of software, libraries

• What gLite can do

– Provides sharing of resources (CPUs, Storage, Sensors …) – Allows the creation of virtual organizations (People, Groups) – Applications will run faster only if properly developed and best if thought for Grid environments (Trivial parallelization, MPIs) – Provides access to computational/storage/other resources accordingly to defined: Policies and Access rights

4

gLite Grid

gLite Grid System aims to: Integrate Virtualize Manage

RESOURCEs and SERVICEs across different Vos The gLite middleware is the set of software packages able to do this www.glite.org

5

gLite evolution

GTK2 Web services based gLite-2 gLite-1 LCG-2 LCG-1 gLite-3 GTK4

gLite - Lightweight Middleware for Grid Computing

www.glite.org

6

Grid Scenario

• Usage scenario

– Many users of different

• rganizations

geographically ditributed (Virtual Oranizations VOs) requesting high computational and storage capacities, collaborating each other – Many computational resources (computing power and storage) belongs to different institutions but transparently accessible

? ?

VO VO VO

University of Catania Italian CNR Italian institute

• f Particle

Physics

Garr-B

Overview

• The user joins to a VO

7

INTERNET

Overview

• The user joins to a VO
• Each VO shares grid resources to
• ther VOs accordingly to several

policies.

8

INTERNET

Overview

• The user joins to a VO
• Each VO shares grid resources to
• ther Vos accordingly to several

policies.

• The Grid middleware allow to

use and share:

– “Computing Elements (CE)” – “Storage Element (SE)”

9

INTERNET

Overview

• The users join to a VO
• Each VO shares grid resources to
• ther Vos accordingly to several

policies.

• The Grid middleware allow to

use and share:

– “Computing Elements (CE)” – “Storage Element (SE)”

• Plus Additional services to

empower the capabilities of the Grid

10

INTERNET

Overview

• The users join to a VO
• Each VO shares grid resources to
• ther Vos accordingly to several

policies.

• The Grid middleware allow to

use and share:

– “Computing Elements (CE)” – “Storage Element (SE)”

• Plus Additional servicees to

empower the capabilities of the Grid

• Result:

COLLABORATION

11

INTERNET

12

gLite Main components services

• UI: User Interface
• WMS: Workload management system
• LB: Logging and bookkeeping service
• VOMS: Virtual Organization Management service
• BDII: Information system
• CE: Computing element (LCG/gLite)
• WN: Worker nodes
• SE: Storage element
• LFC: File catalog
• MyProxy: User Credential Storage

13

Components

• The User Interface (UI) is the user entry point normally considered as the user
• workstation. It is normally considered as a WMS component.
• The Workload Management System (WMS) is a set of services having the

responsibility to find the best available computing element where to submit user’s job in a transparent fashion

• The Logging and bookkeeping service (LB), keep track of user job execution in

terms of statuses: Ready, Scheduled, Waiting, Running, Done

• The Computing element (CE) is the computational resource, the entry point to

a cluster or PCs handled by a job queue management system; in particular: TORQUE, PBS, LSF, CONDOR

• The Worker Nodes are the machines where jobs are really executed and

managed by the CE’ queue management system

14

Components

• The Information System and Monitoring maintain data related to available

grid resources and their health status.

• The Virtual Organization Management service (VOMS) , is the way gLite

improves the management of authentication and authorization to the Grid

• resources. The VOMS allows to their own members to define different access

rights to VO’ resources

• The Storage element (SE) and the File catalogue (LFC), allow to manage Grid

files and offer a mechanism to locate them easily for users and jobs.

15

Job life cycle

16

UI JDL Logging & Book-keeping Resource Broker Job Submission Service Storage Element Computing Element

Job Status

LFC Catalog

DataSets info

Author. &Authen.

Job Submit Event Job Query Input “sandbox” RSL Output “sandbox” Job Status Publish Expanded JDL

Job Workflow in gLite

17

Job Workflow in gLite

UI JDL Resource Broker Job Submission Service Storage Element

Job Status

LFC Catalog

DataSets info Job Submit Event Job Query Input “sandbox” RSL Output “sandbox” Job Status Publish Expanded JDL

18

gLite services

• gLite services can be grouped in 5 main high level set of services

Grid Access Security Information system & Monitoring Job Workload Management System Data Management

19

gLite – Grid access

Two possibilities: APIs or CLI

Built on top of them there exist GridPortals and GUIs

20

gLite Security

• User authentication is based on X.509 (PKI)‏

 Authorized Certification Authorities (CA) can generate user and service

certificates who identify univocally people or Grid services in the whole Grid

 Each Grid service may support or not certificates coming from different

CAs

 To reduce the vulnerabilities the identification of users in to the grid is

done through the use of proxy certificates. Proxies are signed ‘copies’ of the original user certificate, having a limited lifetime.

• The use of Proxy certificates allows the following:

 Delegation: Any grid service can operates on the user behalf making

signed copies of the original proxy. (Single Sign On)

 Add additional info (Add VO specific information provided by VOMS)‏  Store a long term proxy on a secure server (MyProxy)  Renewal (A Proxy close to the expiration time can be automatically

renewed)‏

21

gLite Security: AutH/AhtZ

• Authentication

– The user receive a certificate from a CA (PKI third party) – He connects to the UI via SSH – He Creates the proxy (single sign on) – All grid services will use this proxy to identify the user.

• Authorization

– The user has to subscribe to a VO (VOMS) – The VO establishes the user rights – In any Grid service it will be verified if the user belongs to the VO and assigns the proper access rights to the user

– A special configuration file named the “gridmapfile”, maintains the correspondency between grid users and resource users (unix pool accounts)

22

VOMS

• Virtual Organization Membership Service

– Manages many Virtual Organizations (VOs). – Multiple user roles can be defined inside each VO

• Extends the X509 schema
• Extensions are Digitally Signed

– Service maintenance provided by a web front-end – Support MyProxy (stored proxies) – Allow the access rights by VO or by Role – Each Grid site associates to each VO member or role

• Allows to implement fine grained security policies to grid

resources

23

VOMS

Authentication Request Auth DB

C=IT/O=INFN /L=CNAF /CN=Pinco Palla /CN=proxy

VOMS AC

VOMS AC

24

Joining a Virtual Organisation

• Users (and machines) are identified by

certificates.

• Steps

– User obtains certificate from Certification Authority – User registers at the VO

• usually via a web form

– VO manager authorizes the user

• VO DB updated

– User information is replicated onto VO resources within 24 hours

CA

Obtaining certificate: Annually

VOMS database Grid sites

VO Membership Service Replicating VOMS DB

• nce a day

User’s identity in the Grid = Subject of certificate: /C=IN/O=DAE/OU=BARC/CN=mvineet

VO manager

Joining VO: Once VO Membership Service

VOMS database

25

MyProxy

• MyProxy

– Stores a long term proxy certificates to allow the automatic proxy renewal mechanism – Allow to execute jobs requesting a computation time larger that the normal proxy lifetime (normally 12 hrs)

• The WMS is the responsible for the proxy renewal
• Users should not use long lived proxy directly

– Allow the user to access grid resources without carrying

• ut the public and private keys.
• Proxy Delegation

26

Information System and Monitoring

GIIS INFN sez. CT GIIS Merida (gilda) GRISes GRISes Other GIIS (gilda) GRISes BDII (gilda)

Globus MDS

Berkeley Database Information Index (BDII) The information hierarchically stored via tree modeling (The LDAP implementation of GLUE) GRIS Stores information at resource level Site BDII Stores information at site level BDII Stores information at VO level

VO Level Site Level Resource Level

27

Workload Management

WMS set of middleware components responsible of distribution and management of

jobs across Grid resources.

Two core components of WMS WM: accepts and satisfy requests for job management.

(Matchmaking) is the process of assigning the best available resource.

Logging & Bookeeping : keeps track of job execution in term of events:

(Submitted, Running, Done,...)

28

Computing Element

Service that represents the computing resource that is responsible to manage the queue

• f jobs to execute
• The CE may be used by a Generic Client: an end-user interacting

directly with the Computing Element,

• r
• by the Workload Manager, which submits a given job to an appropriate CE found by the

matchmaking process. Two job submission models : PUSH (Eager Scheduling) (jobs pushed to CE), PULL

(Lazy Scheduling) (jobs coming from WMS when CE has free slots)

27

Computing Element: Architecture

• A CE refer to a set of computational

resources (cluster, computing farm, etc.): – CE Aceptance (CEA): generic interface to cluster. Includes the functionality of a site Gatekeeper – LRMS (batch system): Condor, OpenPBS, Torque/Maui, LSF – The cluster itself: Worker Nodes (WNs) – CE Monitor (CEMon): deals with notifications about CE status, requests jobs to WMS (pull mode) For job submission, CE is able to work in pull or in push mode

30

Storage Element

SE Services are at least:

Storage back-end (Drivers and Hardware) Storage Resource Manager (SRM) Interface (Interface to manage the specific storage solution: dpm, rfio, …) Transfer service (Protocols: GridFTP(gsiftp), glubus-url-copy, …) Native POSIX like file I/O API (GFAL)

31

LFC File Catalog

LFN (Logical file name) GUID (Grid unique identifier) SimLinks SURL (Site URL) TURL (Transfer URL)

32

Grid Services and their interactions

• Grid Access

– User Interface – Info system – Security

• MyProxy (Normal, Long

term)

• VOMS
• Job submission

– WMS – Computing Element – Worker Node

• Data management

– Catalogs – Storage elements

BDII

33

Questions …