Development and Operation of the D-Grid Infrastructure Thomas - - PowerPoint PPT Presentation

▶
development and operation of the d grid infrastructure
SMART_READER_LITE
LIVE PREVIEW

Development and Operation of the D-Grid Infrastructure Thomas - - PowerPoint PPT Presentation

Jan 02, 2023 158 likes •337 views

Development and Operation of the D-Grid Infrastructure Thomas Fieseler (t.fieseler@fz-juelich.de) Wolfgang Grich (w.guerich@fz-juelich.de) Forschungszentrum Jlich (FZJ), Germany Jlich Supercomputing Centre (JSC) International Symposium

slide-1
SLIDE 1

Development and Operation of the D-Grid Infrastructure

International Symposium on Grid Computing 2008 07-11 April 2008, Academia Sinica, Taipei, Taiwan Thomas Fieseler (t.fieseler@fz-juelich.de) Wolfgang Gürich (w.guerich@fz-juelich.de) Forschungszentrum Jülich (FZJ), Germany Jülich Supercomputing Centre (JSC)

slide-2
SLIDE 2

D-Grid / integration project DGI

  • D-Grid – projects for a german national grid initiative funded

by BMBF since 2005

  • General grid environment for e-Science and business
  • Community grid projects + integration project
  • Integration of developments from other projects (EGEE,

UNICORE, Globus)

  • Integration of developments from communities
  • Provision of software needed by communities
  • Scalable, extensible, generic grid platform
  • Installation of a reliable and sustainable infrastructure

 Core D-Grid

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 2
slide-3
SLIDE 3

Community grids / integration project

Community Grids Integration Project

Generic Grid Middleware and Grid Services

Grid specific Developments

High Energy Physics

CG Middle- ware

Grid specific Developments

Astrophysics

CG Middle- ware

Grid specific Developments

Medicine and Lifesciences

CG Middle- ware

Grid specific Developments

Climate Research

CG Middle- ware

Grid specific Developments

Engineering

CG Middle- ware

…

  • Initially 7 community projects (D-Grid 1, 2005-2008)
  • 15 new community projects (D-Grid 2, 2007-2010)
  • Possibly more new community projects (D-Grid 3)
  • Integration project DGI-1 (2005-2007) / DGI-2 (2008-2010)

– Installation, operation, and development of the grid infrastructure

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 3
slide-4
SLIDE 4

Community grids / integration project

…

Community Grids Integration Project

Generic Grid Middleware and Grid Services

Grid specific Developments

High Energy Physics

CG Middle- ware

Grid specific Developments

Astrophysics

CG Middle- ware

Grid specific Developments

Medicine and Lifesciences

CG Middle- ware

Grid specific Developments

Climate Research

CG Middle- ware

Grid specific Developments

Engineering

CG Middle- ware

  • Initially 7 community projects (D-Grid 1, 2005-2008)
  • 15 new community projects (D-Grid 2, 2007-2010)
  • Possibly more new community projects (D-Grid 3)
  • Integration project DGI-1 (2005-2007) / DGI-2 (2008-2010)

– Installation, operation, and development of the grid infrastructure

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 3

total funding of approx. 100 mill. EUR by BMBF

slide-5
SLIDE 5

Partners and hardware

  • Initially 10 partners, own compute and storage resources

– extremely heterogeneous (from cluster to supercomputer) – installation of middleware stack and integration complicated – experience with different middleware on different platforms

  • Additional funding of investments by BMBF

– 2006: compute and storage resources at 25 partner sites, 2200 CPU-Cores, 800 TB Disk, 1.4 PB Tape – 2007: further investments of about 20 mill. EUR – hardware less heterogeneous – new concept to operate several middleware systems on one resource – new development of a Grid Resource Registry Service (GRRS)

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 4
slide-6
SLIDE 6

Grid middleware

  • Reference installation
  • Whole middleware stack
  • n single resource

– Globus – LCG / gLite – UNICORE – OGSA-DAI – dCache

  • Example for resource

providers

  • Maintained and updated
  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 5
slide-7
SLIDE 7

Central services for the D-Grid infrastructure

  • Point of information http://www.d-grid.de
  • Public Key Infrastructure (2 CAs, RAs all over the country)
  • User and VO management
  • Resource management
  • Monitoring status of resources and accounting
  • User support / trouble ticket system http://dgus.d-grid.de
  • Provision of grid middleware for resources and clients
  • Training courses
  • Central D-Grid middleware services

– Globus: MDS4 index, WebMDS, MyProxy – LCG / gLite: BDII, R-GMA, broker, file catalog, VOMS, central user interface – UNICORE: gateway

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 6
slide-8
SLIDE 8

Central services for the D-Grid infrastructure

  • Point of information http://www.d-grid.de
  • Public Key Infrastructure (2 CAs, RAs all over the country)
  • User and VO management
  • Resource management
  • Monitoring status of resources and accounting
  • User support / trouble ticket system http://dgus.d-grid.de
  • Provision of grid middleware for resources and clients
  • Training courses
  • Central D-Grid middleware services

– Globus: MDS4 index, WebMDS, MyProxy – LCG / gLite: BDII, R-GMA, broker, file catalog, VOMS, central user interface – UNICORE: gateway

increase availability of services by redundant infrastructure components

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 6
slide-9
SLIDE 9

Central services for the D-Grid infrastructure

  • Point of information http://www.d-grid.de
  • Public Key Infrastructure (2 CAs, RAs all over the country)
  • User and VO management
  • Resource management
  • Monitoring status of resources and accounting
  • User support / trouble ticket system http://dgus.d-grid.de
  • Provision of grid middleware for resources and clients
  • Training courses
  • Central D-Grid middleware services

– Globus: MDS4 index, WebMDS, MyProxy – LCG / gLite: BDII, R-GMA, broker, file catalog, VOMS, central user interface – UNICORE: gateway

increase availability of services by redundant infrastructure components

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 6
slide-10
SLIDE 10

Integration of users and resources

… … … …

GRRS

Resources

VOMRS

VO 1

User information

Users VO 2 VO 3 Registration Mapping Registration Access Resource DB dgridmap server

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 7
slide-11
SLIDE 11

Integration of users and resources (2)

  • Registration of resources at Grid Resource Registry

Service (GRRS)

– resource database – https://dispatch.fz-juelich.de:8814/D-Grid-Resource-List

  • Registration of users at Virtual Organization Management

Registration Service (VOMRS) for their community / VO

– member database – currently 14 VOs – https://dispatch.fz-juelich.de:8814/D-Grid-VO-Member

  • Support of groups (sub-VOs)
  • dgridmap service: server in GRRS, client on resource

– user mapping for each middleware system

  • Globus, LCG/gLite, UNICORE, OGSA-DAI, dCache
  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 8
slide-12
SLIDE 12

Monitoring / Accounting

  • Globus: MDS4 / Web-MDS for Globus resources

– geo-data / resource status – hierarchical structure site / resource

  • LCG / gLite: site functional tests
  • UNICORE: client monitoring window

– new developments with UNICORE 6

  • Separate D-Grid project for monitoring D-MON

– development of a unified monitoring framework – collect data from Globus, gLite, UNICORE – integration of D-MON results into the D-Grid infrastructure

  • Accounting system under development

– based on the Distributed Grid Accounting System (DGAS) – collect data from Globus, gLite, UNICORE

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 9
slide-13
SLIDE 13

Policy relations between users and resource providers (1)

VO resource provider rules for VOs VO AUP rules for providers VO member resource AUP sample resource AUP sample VO AUP

accept release accept accept examine provide accept release release release

D-Grid

Requirement of AUPs for providers and VOs

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 10
slide-14
SLIDE 14

Policy relations between users and resource providers (2)

GRRS

Provider

VOMRS

VO

Registration Mapping Registration Access

P V Provider AUP VO AUP P&V

User information Agreement

Requirement of a more flexible mapping resource - VO

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 11
slide-15
SLIDE 15

Further requirements / open topics

  • Providing new services required by business partners
  • Creation of a legal entity: D-Grid Entwicklungs- und

Betriebsgesellschaft mbH (~ limited liability corporation)

  • Automated creation of virtual organizations

– support of VOs with shorter life time

  • Short Lived Certificates (SLC) issued by a Short Lived

Credential Service (SLCS)

  • Quality of Service (QoS) / Service Level Agreements (SLA)
  • Integration of scheduling / brokering services
  • Common production environment

– compilers, MPI etc., application software from communities by means of modules and images

  • Further development of operational concept
  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 12
slide-16
SLIDE 16

Summary / Outlook

  • Infrastructure for a rapidly growing grid
  • Support for communities with different background and

requirements

– scientific communities – business partners

  • Concept for easy integration of resources
  • Support of 3 middleware systems for compute resources

and 2 middleware systems for storage resources

  • Common production environment
  • Common monitoring and accounting infrastructure
  • D-Grid is involved in the European Grid Initiative (EGI)
  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 13
slide-17
SLIDE 17

Partners of DGI-2

Univer sit ät Dor t mund

  • T. Fieseler, W. Gürich, FZJ/JSC D-Grid Infrastructure ISGC2008 14