IBERGRID towards the EOSC Presented by: Joao Pina (LIP-Lisbon), on - - PowerPoint PPT Presentation

▶
ibergrid towards the eosc
SMART_READER_LITE
LIVE PREVIEW

IBERGRID towards the EOSC Presented by: Joao Pina (LIP-Lisbon), on - - PowerPoint PPT Presentation

Jan 13, 2023 27 likes •290 views

IBERGRID towards the EOSC Presented by: Joao Pina (LIP-Lisbon), on behalf of IBERGRID Distributed computing infrastructure Ministrio da Cincia, Tecnologia e Ensino Superior 2 Iberian Distributed Computing Infrastructure Distributed

slide-1
SLIDE 1

IBERGRID towards the EOSC

Presented by: Joao Pina (LIP-Lisbon),

  • n behalf of IBERGRID
slide-2
SLIDE 2

Distributed computing infrastructure

Iberian Distributed Computing Infrastructure

Ministério da Ciência, Tecnologia e Ensino Superior

2

slide-3
SLIDE 3

Distributed computing infrastructure

Iberian Distributed Computing Infrastructure

a) Federates infrastructures from Iberian research and academic

  • rganizations (PT + ES) mainly focused;
  • Grid Computing
  • Cloud Computing
  • Data Processing.

b) Enables a joint participation in the European Grid Initiative (EGI) and

  • ther initiatives (EOSC, ESFRIs, WLCG, etc);
  • Provides the regional operations coordination for the computing and data

processing activities of several user communities;

c) Forum for common R&D activities and sharing of knowledge.

  • Participation in European and cross-border projects;
  • Conferences and events.

3

slide-4
SLIDE 4

EOSC Framework

Iberian Distributed Computing Infrastructure

slide-5
SLIDE 5

Vision towards EOSC implementation

Expanding the Capacity and Capabilities of EOSC by leveraging the experience, effort and resources of national publicly-funded digital infrastructures in a coherent way, therefore acting also as an incentive for national resource providers.

  • European Framework: project EOSC-synergy
  • Led by IBERGRID
  • Funded in infraeosc5-b,
  • Starts September 1st 2019 (30 months)
  • 16 partners

Iberian Distributed Computing Infrastructure

slide-6
SLIDE 6

Iberian Distributed Computing Infrastructure

slide-7
SLIDE 7

CAPACITY EXPANSION BUILDING CAPABILITIES SKILLS DEVELOPMENT FOSTERING SERVICE INTEGRATION

  • Deployment of an EOSC-enabled digital

infrastructure with a critical mass

  • Designing a rational strategy to deploy

FAIR National Data Repositories

  • Enable auto-training in EOSC
  • Explore and implement a MooC solution

following a Marketplace approach

  • Integration of production-level National

Thematic Services in EOSC

  • Thematic service certification based on SQAaaS.
  • Definition of an EOSC software quality baseline
  • Service Quality Assurance as a Service: SQAaaS
  • FAIRness checking “as a Service”

Iberian Distributed Computing Infrastructure

slide-8
SLIDE 8

Integration of Capacity in EOSC

IBERGRID towards the EOSC

8 Iberian Distributed Computing Infrastructure

slide-9
SLIDE 9

Know-how in federating infrastructures:

2001 2004 2010 2017 Building Capabilities via User Community Engagement

Iberian Distributed Computing Infrastructure

slide-10
SLIDE 10

IBERGRID contribution to Federation work

https://ibergrid.eu/federating-clouds/

  • Accounting Portal:

○ http://accounting.egi.eu

  • Cloud Accounting software – CASO

○ https://github.com/IFCA/caso

  • Processes and Policies: SQA Coordination

○ Software Quality Assurance Processes and Policies

  • Author.& Authent. OpenStack-VOMS

○ https://github.com/IFCA/voms-auth-system-openstack

10 Iberian Distributed Computing Infrastructure

slide-11
SLIDE 11

Lessons – Grid achievements

❑

Technical

▪

Standards effort is very important

▪

Interoperability (not perfect or complete, but very valuable)

▪

Sophisticated data an resource management

▪

Worldwide authentication for scientific grids

▪

Common usage and security policies

▪

Many developments for privacy and security

❑

European Policy and coordination structure

▪

Creation of National Grid Initiatives supported by the governments

▪

Creation of the European Grid Initiative – EGI in 2009

▪

Model for Long Term sustainability

Iberian Distributed Computing Infrastructure

slide-12
SLIDE 12

Grey Zone

Integration at the AAI level (the Virtual Organization model worked, but it has limitations) Automatization of Resource provisioning (it always worked manually: VO configuration) In a nutshell

  • Policy Level: Actions to be supported by technical developments, notably in the front
  • f Security (data privacy policies) and access policies
  • Technical Integration Level: work to be done/extended, building on the previous

experience.

2001 2004 2010 2017

Building on what we have

EGI Federated Model Open Science EOSC Model

Integration at the security level Technical and Policy - wise Monitoring

Making the leap to EOSC

Iberian Distributed Computing Infrastructure

slide-13
SLIDE 13

SOFTWARE AND SERVICE DEVELOPMENT FOR EOSC

IBERGRID towards the EOSC

13 Iberian Distributed Computing Infrastructure

slide-14
SLIDE 14

Iberian Grid Initiative

2002-2005 11 countries / 21 partners (5 Iberian) CSIC coordinated 2005-2007 7 countries / 12 partners (5 Iberian)

2010-2013

EGI.eu is funded in Feb. 2009. Software Quality task EGI-wide

2014-2017

Software activity history as Iberian Federation

2009… 2005-2010 “Southwest Federation” 50 partners/8-10 Iberian 2006-2010 Many… @ Jesús Marco & Jorge Gomes

14

2018 - 2020 @ Isabel Campos & Jorge Gomes

slide-15
SLIDE 15

Popular IBERGRID developments:

  • CONTAINER SERVICES

○ UDocker (LIP)

  • COMPUTING SERVICES

○ OpenStack Preemptible Instances - OPIE (CSIC)

  • PaaS SERVICES

○ Infrastructure Manager - IM (UPVLC)

  • THEMATIC SERVICES

○ LIFEWATCH (Lifewatch ERIC) ○ OPENCOASTS (LNEC)

15 Iberian Distributed Computing Infrastructure

slide-16
SLIDE 16

The INDIGO-DC Software Stack Catalogue is a catalogue of open source software components that are follow the Architecture defined by the INDIGO-DataCloud project, funded by the European Union under the Horizon 2020 Framework Program with Grant Agreement 653549. This Collaboration Agreement targets to sustain and further develop the INDIGO–DataCloud architecture, the original INDIGO-DataCloud Software Catalogue and as well as the „INDIGO brand” beyond the lifetime of the INDIGO-DataCloud Project, through a not-for-profit, liability-free mutual….

INDIGO-DC Software Collaboration Agreement

Ibergrid teams are involved in the development of software to support innovative services for researchers in the European Open Science Cloud framework. CSIC, LIP and UPVLC are signatories

  • f the INDIGO-Datacloud Software

Collaboration Agreement, a key Technology provider for the EOSC ecosystem.

INDIGO-Datacloud paper (Open Access): https://link.springer.com/sharelink/10.1007/ s10723-018-9453-3

16 Iberian Distributed Computing Infrastructure

slide-17
SLIDE 17

Software Quality Management

See: http://digital.csic.es/handle/10261/160086

17 Iberian Distributed Computing Infrastructure

slide-18
SLIDE 18

Software Quality Assurance in

Consolidation of an EOSC Software Maturity Baseline ฀ Consolidation of best practices that seek software quality Implementation of an EOSC Service Integration Platform: SQAaaS

Iberian Distributed Computing Infrastructure

slide-19
SLIDE 19

USER COMMUNITIES

IBERGRID towards the EOSC

19 Iberian Distributed Computing Infrastructure

slide-20
SLIDE 20

User Communities: Examples

Iberian Distributed Computing Infrastructure 20

Community Computing resources Type of resources WLCG 7 600M CPU*hours Grid Lifewatch 16M CPU*hours Cloud INSTRUCT On demand Cloud WeNMR and MoBrain 3M CPU*hours Grid (cooperation in integration and packaging) AUGER 12M CPU*hours Grid OPENCoastS 0.5M (recent) Cloud and Grid (EOSC Thematic service) Genetics 90.5M CPU*hours Grid

slide-21
SLIDE 21

User Communities: Examples

Iberian Distributed Computing Infrastructure 21

slide-22
SLIDE 22

฀ udocker ฀ event-driven computing

Two examples of user support

Iberian Distributed Computing Infrastructure

slide-23
SLIDE 23

udocker: running containers everywhere

Iberian Distributed Computing Infrastructure

Technologies:

  • Docker containers
  • PTRACE,

SECCOMP

  • Library interception
  • OpenContainers
  • etc
  • INDIGO, EOSC, DEEP HybridDataCloud
  • Execute Docker in batch systems, HPC and HTC transparently
  • no docker, no installation, no privileges, download and run
  • Open source available in: https://github.com/indigo-dc/udocker

Usage:

  • Many communities
  • DEEP HybridDataCloud
  • INDIGO-DataCloud
  • EOSC
slide-24
SLIDE 24
  • OSCAR: open-source serverless computing platform for data-processing applications that can be

deployed on multi-Clouds.

  • https://github.com/grycap/oscar
  • Creates highly-parallel event-driven file-processing serverless applications:
  • Executes customized runtime environments provided by Docker containers.
  • Runs on an elastic Kubernetes cluster (dynamic provision of nodes).
  • Users upload files to a storage back-end triggering the parallel invocation of a function that runs a

user-defined script in charge of processing each file.

  • Integrated with EGI Applications on Demand, EGI Federated Cloud and EGI DatHub.

OSCAR: supporting event-driven computing. (biodiversity in plant recognition)

Iberian Distributed Computing Infrastructure

slide-25
SLIDE 25

OSCAR: Open Source based

Iberian Distributed Computing Infrastructure

slide-26
SLIDE 26

Co-located with EOSC-synergy KoM When: 23rd-26th September 2019 Where: Santiago de Compostela Opportunities for co-location: drop us an e-mail. More details in: http://www.ibergrid.eu/

(*) IBERGRID 2007 – Santiago de Compostela First Ibergrid Conference

Returning to the origins*: IBERGRID 2019

Iberian Distributed Computing Infrastructure

http://old.ibergrid.eu/2007