German e-Science Initiative 2003 Agenda 2006: The German e-Science - - PowerPoint PPT Presentation

▶
german e science initiative
SMART_READER_LITE
LIVE PREVIEW

German e-Science Initiative 2003 Agenda 2006: The German e-Science - - PowerPoint PPT Presentation

Jan 10, 2024 263 likes •617 views

GridKa Karlsruhe, September 11 15, 2006 Building the German D-Grid *) Wolfgang Gentzsch D-Grid, RENCI, Duke, OGF, PCAST *) funded by the German Ministry for Education and Research 1 September, 2006 Wolfgang Gentzsch, D-Grid German

slide-1
SLIDE 1

1 Wolfgang Gentzsch, D-Grid September, 2006

GridKa Karlsruhe, September 11 – 15, 2006

Building the German D-Grid *)

Wolfgang Gentzsch

D-Grid, RENCI, Duke, OGF, PCAST

*) funded by the German Ministry for Education and Research

slide-2
SLIDE 2

2 Wolfgang Gentzsch, D-Grid September, 2006

German e-Science Initiative

2003 ‘Agenda 2006‘:

“ The German e-Science Initiative is a strategic task ! ”

Build Sustainable Grid Infrastructure in Germany

Combine existing grid activities and infrastructure Middleware components driven by Community Grids

Develop e-science services for research community

Science Service Grid

Important: Sustainability

Continuing production grid after the end of the funding period

Integration of new grid communities (2. generation) Business models for grid services

slide-3
SLIDE 3

3 Wolfgang Gentzsch, D-Grid September, 2006

  • The 90s: Several Government funded projects for the area of

“distributed computing”, like e.g. Unicore,…

  • 2000++:

Many German research centers are part of European grid projects, like DataGrid, GridLab, EuroGrid, CrossGrid, EGEE, GridCoord, CoreGrid, UniGrids, NextGrid,…

  • 01/2003: Scientists started D-Grid working groups => White Paper for Govt
  • 03/2004: German Govt (BMBF) announced e-Science Initiative
  • 08/2004: Call for Grid proposals
  • 01/2005: “Pre-Project”: Finalists gather in 4-month D-Grid workshops => final review
  • 09/2005: Start of the D-Grid projects, 20 Mio Euro funding
  • 06/2006: Deadline D-Grid-2 Call, 20 Mio Euro funding
  • 01/2007: Start D-Grid-2

History of D-Grid Initiative

slide-4
SLIDE 4

4 Wolfgang Gentzsch, D-Grid September, 2006

To minimize overlapping of tasks and parallel activities: −Monitoring (von Voigt, Gentzsch) −Portale, User Interfaces (Wehrens, Harms) −Scheduling, Workflow (Schwiegelshohn, Gentzsch) −Storage Management, Archivierung (Ludwig, Reuter) −Metadaten, Datenmanagement (Kasemann, Reuter) −Lizenzmanagement, Accounting, Virtuelle Organisationen (Mickel, Harms) −Security, Privacy (Sax, Gentzsch)

=> Final Review => 6 Projects

D-Grid Pre-Project Workshops

February – May 2006

slide-5
SLIDE 5

5

Wolfgang Gentzsch, D-Grid September, 2006

Study on Major Grid Projects

  • UK-e-Science: Tony Hey, Steven Newhouse,

Carole Goble, Malcolm Atkinson, John Darlington, Trevor Cooper Chadwick, Monica Schraefel, Luc Moreau, Paul Watson, Aaron Turner

  • TeraGrid: Charlie Catlett, Dane Skow
  • ChinaGrid: Hai Jin
  • Naregi: Kazushige Saga, Satoshi Matsuoka,

Kenichi Miura

  • EGEE: Bob Jones, Dieter Kranzlmueller
  • RENCI: Dan Reed and Alan Blatecky
slide-6
SLIDE 6

6 Wolfgang Gentzsch, D-Grid September, 2006

Community Community Grids Grids

Generic Grid Middleware and Grid Services

Information and Knowledge Management

Grid specific Developments

Application CG Middle- ware

Grid specific Development

Application CG Middle- ware

D-Grid Structure

Courtesy Dr. Krahl PT/ BMBF

Integration Project Integration Project

slide-7
SLIDE 7

7 Wolfgang Gentzsch, D-Grid September, 2006

D-Grid Structure

Generic Grid Middleware and Grid Services

Integration Project

Astro-Grid C3-Grid HEP-Grid IN-Grid MediGrid ONTOVERSE WIKINGER WIN-EM Textgrid D-Grid Knowledge Management

. . .

Im Wissensnetz

slide-8
SLIDE 8

8 Wolfgang Gentzsch, D-Grid September, 2006

D-Grid Structure

Generic Grid Middleware and Grid Services

Integration Project

Astro-Grid C3-Grid HEP-Grid IN-Grid MediGrid ONTOVERSE WIKINGER WIN-EM Textgrid D-Grid Knowledge Management

. . .

Im Wissensnetz

D-Grid-2 Projects

slide-9
SLIDE 9

9 Wolfgang Gentzsch, D-Grid September, 2006

German Core Grid

Visions

  • establishment and operation of the Core D-Grid infrastructure

by the D-Grid integration project (DGI)

  • evaluation of interoperability and performance of resources

The Core D-Grid comprises:

  • acquisition, implementation and provision of middleware

(Globus, gLite, UNICORE)

  • attachment of storage robots to the storage element of the D-

Grid software (dCache, SRM/SRB)

  • work for the admission of users to the resources (AA)
  • peration of a virtual Grid Operating Center (GOC)
slide-10
SLIDE 10

10 Wolfgang Gentzsch, D-Grid September, 2006

Nutzer Application Development and User Access GAT API

Data/ Software

Resources in D-Grid High-level Grid Services Basic Grid Services

Distributed Data Archive

User

Network Infrastructur

LCG/gLite Globus 4.0.1 Accounting Billing User/VO-Mngt

Scheduling Workflow Management

Data management Security Plug-In UNICORE

Distributed Compute Resources

GridSphere Monitoring

D-Grid Middleware

slide-11
SLIDE 11

Wolfgang Gentzsch, D-Grid September, 2006

DGI Infrastructure Project

WP 1: D-Grid basic software components, large storage, data interfaces, virtual organizations, management WP 2: Develop, operate and support robust core grid infrastructure, resource description, monitoring, accounting, and billing WP 3: Network (transport protocols, VPN) Security (AAI, CAs, Firewalls) WP 4: Business platform and sustainability, project management, communication and coordination

Scalable, extensible, generic grid platform for future Longterm, sustainable grid operation, SLAs based

slide-12
SLIDE 12

12 Wolfgang Gentzsch, D-Grid September, 2006

Monitoring, Accounting, Billing

Reliable monitoring systems will be developed, error tolerant and scalable, for resource and job information Accounting will answer conceptual, technical and legal questions.

  • Determination, definition, delivery, allocation of calculation units
  • customized presentation of resulting fees

A comprehensive billing solution for the mutual provision and usage of grid resources will be established and a transparent and comprehensible billing framework for grid-computing is going to be developed.

slide-13
SLIDE 13

13 Wolfgang Gentzsch, D-Grid September, 2006

GOC, German Core Grid sites

PC² RRZN TUD RZG LRZ RWTH FZJ FZK FHG/ ITWM Uni-KA Site Resource Amount

FZJ/ZAM IBM Supercomputer with 8,5 TFlops STK data robot system with 2,8 PByte 32 CPUs 300 TByte FZK/IWR 8 nodes Opteron 2x2.2 GHz 8 processors of a system NEC SX-5 1 p630 with 4 processors 1 SX-6i to do tests 2 nodes Opteron 2x2.2 GHz to do tests 100% 50% 50% 50% 50% LRZ SGI high performance system with 20 TFlop/s Intel IA32 and IA 64 Cluster, IBM p690, SunFire 80 5% 5% 5% MPI/RZG IBM supercomputer with 4,5 TFlops, PC cluster with 2 TFlops Data robot system with 8 PByte 32 CPUs 400 TByte PC² Cluster of 400 Xeon 64 Bit processors, high performance visualization and FPGAs 10% RWTH/RZ 2 SunFire 6900 with 24 UltraSPARC IV each 100% TU- Dresden/ZIH SGI O2K(56 proc)/O3K(192 proc.) : T3E (64 proc): PC cluster with 30 processors, end off 2005: new system with 1000 proc. 10% 20% 20% 2% Uni-H/RRZN PC-Cluster mit 64 CPUs assoc. Uni-KA PC-Pool assoc. FHG/ITWM assoc.

slide-14
SLIDE 14

14 Wolfgang Gentzsch, D-Grid September, 2006

  • Each resource with full middleware stack (GT4, gLite, UNICORE).
  • Storage systems accessible through DGI tools

(dCache, SRB, OGSA-DAI)

  • All systems should run the existing middleware stack, to reduce

migration overhead

  • Resources should be available for ALL communities

(DGrid1 and DGrid2)

  • Sustainability at least until the end of resource life cycle
  • Different resources from different grid communities in one location

have to be consolidated

Requirements for Core Grid Resources

5.3 Mio Euro Investment in D-Grid in October 2006

slide-15
SLIDE 15

15 Wolfgang Gentzsch, D-Grid September, 2006

  • 8 DGI Centers to build the resource and service backbone for

all communities

  • These centers are responsible for a reliable and sustainable

grid operation

  • These backbone resources are available for all communities

exclusively for grid activities

  • In addition, each community is building a community resource

infrastructure with similar quality and services

  • FZK provides supervision, coordination, monitoring as part of

this 3-tier architecture

Core D-Grid Infrastructure

slide-16
SLIDE 16

16

  • T. Fieseler, FZJ/ZAM, 10.07.2006

Joint EGEE / D-Grid Meeting 16

Installation of a Support Infrastructure

Community A Consulting Center Community A Consulting Center Community A Consulting Center Community Consulting Center

Dienste- Anbieter Dienste- Anbieter Dienste- Anbieter Service provider User support Operation mngt.

DGUS

D-Grid User Support

from both FZK and FZJ

Entwickler J Entwickler J Developer Deployment-Team

Communities DGI

common trouble ticket system or compatible systems

slide-17
SLIDE 17

17

  • T. Fieseler, FZJ/ZAM, 10.07.2006

Joint EGEE / D-Grid Meeting 17

D-Grid User Support

slide-18
SLIDE 18

18 Wolfgang Gentzsch, D-Grid September, 2006

DGI Services, Available Dec 2006

  • Sustainable grid operation environment with set of core

D-Grid middleware services for all grid communities

  • Central registration and information management for all

(including new) resources

  • Packaged middleware components for gLite, Globus and

Unicore and for data management systems SRB, dCache and OGSA-DAI

  • D-Grid support infrastructure will support new communities

/ VOs with installation and integration of new grid resources into D-Grid Help-Desk, Monitoring System and central Information Portal

slide-19
SLIDE 19

19 Wolfgang Gentzsch, D-Grid September, 2006

DGI Services, Dec 2006, cont.

  • Tools for managing VOs based on VOMS and Shibboleth
  • Test implementation for Monitoring & Accounting for Grid

resources, and first concept for billing system

  • Network and security support for Communities (firewalls

in grids, alternative network protocols,...)

  • DGI operates „Registration Authorities“, for application of

internationally accepted Grid certificates of DFN & GridKa

  • Partners support new D-Grid members with building

their own „Registration Authorities“

slide-20
SLIDE 20

20 Wolfgang Gentzsch, D-Grid September, 2006

  • DGI will offer resources to other Communities, with access

via gLite, Globus Toolkit 4, and UNICORE

  • Portal-Framework Gridsphere can be used by future users

as a graphical user interface

  • For administration and management of large scientific

datasets, DGI will offer dCache for testing

  • New users can use the D-Grid resources of the core grid

infrastructure upon request

DGI Services, Dec 2006, cont.

slide-21
SLIDE 21

21 Wolfgang Gentzsch, D-Grid September, 2006

AstroGrid

slide-22
SLIDE 22

22 Wolfgang Gentzsch, D-Grid September, 2006

Climate research moves towards new levels of complexity:

Stepping from Climate (=Atmosphere+Ocean) to Earth System Modelling Earth system model wishlist: Higher spatial and temporal resolution Quality: Improved subsystem models Atmospheric chemistry (ozone, sulfates,..) Bio-geochemistry (Carbon cycle, ecosystem dynamics,..)

Increased Computational demand factor: O(1000 -10000)

C3 Grid: Collaborative Climate Community

Data and Processing Grid

slide-23
SLIDE 23

23 Wolfgang Gentzsch, D-Grid September, 2006

HEP-Grid: p-p collisions at LHC at CERN

(from 2007 on)

Crossing rate 40 MHz Event Rates: ~ 109 Hz Max LV1 Trigger 100 kHz Event size ~ 1 Mbyte Readout network 1 Terabit/ s Filter Farm ~ 107 Si2K Trigger levels 2 Online rejection 99.9997% (100 Hz from 50 MHz) System dead time ~ % Event Selection: ~1/1013 Crossing rate 40 MHz Event Rates: ~ 109 Hz Max LV1 Trigger 100 kHz Event size ~ 1 Mbyte Readout network 1 Terabit/ s Filter Farm ~ 107 Si2K Trigger levels 2 Online rejection 99.9997% (100 Hz from 50 MHz) System dead time ~ % Event Selection: ~1/1013

Event rate

“Discovery” rate

Luminosity Low 2x1033 cm-2 s-1 High 1034 cm-2 s-1 Data analysis: ~ 1PB/ year Level 1 Trigger Rate to tape

Courtesy David Stickland

slide-24
SLIDE 24

24 Wolfgang Gentzsch, D-Grid September, 2006

Gridspezifische Entwicklungen

Integration project

Cooperation and business models

InGrid: Virtual Prototyping & Modeling in Industry

Molding Metal Form ing Fluid Processes Groundw ater Transportation

Knowledge-based support for engineering-specific decision support Support for engineering- specific Workflows Distributed simulations-based product & process

  • ptimization

Methods and m odels for solving engineering problem s in Grids

Fluid-Structur/ Magneto-Hydro- dynam ic I nteraction

Security and trust models Grid-specific developm ents

AP 2 AP 3 AP 4

slide-25
SLIDE 25

25 Wolfgang Gentzsch, D-Grid September, 2006

Raw Data

Metadata

Molecule

Homogenization Target data Metadata

Population

Metadata

Patient

Metadata

Illness

Metadata

Organ/Tissue

Metadata

Cell

Search, Find, Select

Access Control

Correlate, Process, Analyze Resulting Data Presentation Final Result

MediGrid: Mapping of Characteristics, Features, Raw Data, etc

slide-26
SLIDE 26

26 Wolfgang Gentzsch, D-Grid September, 2006

TextGrid

Modular Platform for distributed cooperative scientific text processing for the humanities

Workbench for Publishing, Text Processing, Text Retrieval,

Linking, and Workflow

Focus of TextGrid is on:

  • Quantity: Full text instead excerpts, text and pictures
  • Speed of reprography: stability of text, volatility of medium
  • Precision: Maximum requirements on correctness
  • Availability: international open standards
slide-27
SLIDE 27

27 Wolfgang Gentzsch, D-Grid September, 2006

D-Grid 2 Call April 2006

‘Horizontal’ Service Grids: professional Service Providers for heterogeneous user groups in research and industry ‘Vertical’ Community Service Grids using existing D-Grid infrastructure and services, supported by Service Providers D-Grid extensions, based on a D-Grid 1 gap analysis

  • Tools for operating a professional grid service
  • Adding business layer on top of D-Grid infrastructure
  • Pilot service phase with Service Providers and ‘customers’

!! Reliable grid services require sustainable grid infrastructure !!

slide-28
SLIDE 28

28 Wolfgang Gentzsch, D-Grid September, 2006

slide-29
SLIDE 29

29 Wolfgang Gentzsch, D-Grid September, 2006

D-Grid and its European Relations

Courtesy Dieter Kranzlmueller

slide-30
SLIDE 30

30 Wolfgang Gentzsch, D-Grid September, 2006

  • Difficult to differentiate reality from hype
  • Sensitive data, sensitive applications (medical patient records)
  • Different organizations have different ROI
  • Accounting, who pays for what (sharing!)
  • Security policies: consistent and enforced across the grid !
  • Lack of standards prevent interoperability of components
  • Current IT culture is not predisposed to sharing resources
  • Not all applications are grid-ready or grid-enabled
  • Open source is not equal open source (read the little print)
  • SLAs based on open source (liability?)
  • “Static” licensing model don’t embrace grid
  • Protection of intellectual property
  • Legal issues (FDA, HIPAA, multi-country grids)

Challenges for Research and Industry

slide-31
SLIDE 31

31 Wolfgang Gentzsch, D-Grid September, 2006

  • Difficult to differentiate reality from hype
  • Sensitive data, sensitive applications (medical patient records)
  • Different organizations have different ROI
  • Accounting, who pays for what (sharing!)
  • Security policies: consistent and enforced across the grid !
  • Lack of standards prevent interoperability of components
  • Current IT culture is not predisposed to sharing resources
  • Not all applications are grid-ready or grid-enabled
  • Open source is not equal open source (read the small print)
  • SLAs based on open source (liability?)
  • “Static” licensing model don’t embrace grid
  • Protection of intellectual property
  • Legal issues (FDA, HIPAA, multi-country grids)

Challenges for Research and Industry

slide-32
SLIDE 32

32 Wolfgang Gentzsch, D-Grid September, 2006

Lessons Learned and Recommendations

– Continuity: Grid infrastructure should be modified and improved in large cycles

  • nly: applications depend on infrastructure !

– Sustainability: Funding should be available after end of project, to guarantee services, support and continuous improvement. – Interoperability: Use open-source software and standards especially in the infrastructure and application middleware layer. – Collaboration: between infrastructure developers and the applications, to best utilize grid services and to avoid application silos. – User-Friendliness: for easy adoption for new communities. Infrastructure group should offer installation, operation and support services. – Grid Services: Centers of Excellence should specialize on specific services, e.g. integration of new communities, grid operation, utility services, training, support, etc. – Participation of Industry: has to be industry-driven. Push from outside, even with govmnt funding, is not promising. Success comes only from real needs e.g. through existing collaborations between research and industry.

slide-33
SLIDE 33

33 Wolfgang Gentzsch, D-Grid September, 2006

Summary D-Grid: Towards a Sustainable Infrastructure for Science and Industry

Govt is changing policies for resource acquisition (HBFG ! ) to enable a service model 2nd Call: Focus on Service Provisioning for Sciences & Industry Strong collaboration with: Globus Project, EGEE, Deisa, CrossGrid, CoreGrid, GridCoord, GRIP, UniGrids, NextGrid, … Application and user-driven, not infrastructure-driven Focus on implementation and production, not grid research, in a multi-technology environment (Globus, Unicore, gLite, etc) D-Grid is the Core of the German e-Science Initiative

slide-34
SLIDE 34

34 Wolfgang Gentzsch, D-Grid September, 2006

The Innovation Engine

wgentzsch@d-grid.de wgentzsch@renci.org

Thank You !

Slides are available