INDIGO-DataCloud EGI Fed Cloud contribution Giacinto Donvito (INFN) - - PowerPoint PPT Presentation

▶
indigo datacloud egi fed cloud contribution
SMART_READER_LITE
LIVE PREVIEW

INDIGO-DataCloud EGI Fed Cloud contribution Giacinto Donvito (INFN) - - PowerPoint PPT Presentation

May 01, 2023 366 likes •715 views

INDIGO-DataCloud EGI Fed Cloud contribution Giacinto Donvito (INFN) INDIGO-DataCloud Technical Director RIA-653549 EGI Fed Cloud F2F Meeting Outline General approach of the INDIGO Platform The Platform in the proposal The overall

slide-1
SLIDE 1

INDIGO-DataCloud EGI Fed Cloud contribution

RIA-653549

Giacinto Donvito (INFN) INDIGO-DataCloud Technical Director EGI Fed Cloud F2F Meeting

slide-2
SLIDE 2

Outline

  • General approach of the INDIGO Platform
  • The Platform in the proposal
  • The overall view of the INDIGO-Platform
  • Standards, protocols, and implementations.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 2

slide-3
SLIDE 3

Implementation approach

  • Rely on standards
  • µService approach
  • Modularity
  • Pick the services you really need for your use-case
  • And build your own platform based on your needs
  • Each layer has clear interfaces and could be exploited directly by the end users
  • The authentication/Authorization is based on the concept of “Delegation”
  • Each service could decide autonomously about the authorization
  • Each service is requested exploiting the real end-user credential

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 3

slide-4
SLIDE 4

Implementation approach

  • Automation based on orchestrating resources
  • This is done at different level (IaaS/PaaS/SaaS)
  • Open:
  • Not only OpenSource,
  • But the possibility to plug-in any supported services/protocols/resources in order to

build the needed infrastructure

  • Both private and public cloud resources could be part of the same INDIGO instance
  • You can build your own (private) infrastructure or provide a multi-tenant

solution for your users.

  • Depending on your goal

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 4

slide-5
SLIDE 5

The high level view of the Architecture

JSAGA/JSAGA Adaptors Future Gateway Engine Future Gateway REST API Other Science Gateways

Mobile Apps Open Mobile Toolkit

Ophidpia plugin LONI plugin Taverna, Kepler plugin

Admin Portlets User Portlets Data Analitics Workflow Portlets SG Mon GUI Clients

Future Gateway Portal Workflows Mobile clients

Support services

WP6 Services

Kubernetes Cluster IAM Service PaaS Orchestrator QoS/SLA CloudProvider Ranker Monitoring Infrastructure Manager

TOSCA TOSCA

WP5 Services

Onedata Dynafed FTS Data Services

REST/CDMI/Wedbav/posix/Gridftp OIDC

Accounting

Non-INDIGO IaaS

Native IaaS API

Heat/IM

TOSCA

WP4 Services

Mesos Cluster Mesos Cluster

  • Aut. Scaling

Service

Storage Service

S3/CDMI/Posix/Webdav GridFTP

Smart Scheduling Spot Istances Native Docker QoS Support Identity Armonization Local Repository

This is the INDIGO-DataCloud General Architecture*

*: see details in http://arxiv.org/abs/1603.09536 or in https://www.indigo-datacloud.eu/documents-deliverables

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 5

slide-6
SLIDE 6

Some possible implementation scenario:

  • 1. Enhanced Resource Virtualization -> Computing
  • 2. Enhanced Resource Virtualization -> Storage
  • 3. A complete federated storage access solution
  • 4. Interactive usage of a Docker container with ssh
  • 5. A web portal that uses a batch system to run applications
  • 6. Virtual infrastructures for executing scientific applications

and services exploiting Mesos

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 6

slide-7
SLIDE 7

Enhanced Resource Virtualization -> Computing (OpenNebula)

OpenNebula

OneDock Orchestrator+TOSCA Support (IM) OCCI Support

  • 1. IM: Provides

a) Advanced IaaS Orchestrator capabilities b) TOSCA Support

  • 2. OCCI:

a) Enhanced Network capabilities b) Docker support

  • 3. OneDock:

a) Support for native Docker (on bare- metal)

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 7

slide-8
SLIDE 8

Enhanced Resource Virtualization -> Computing (OpenStack)

OpenStack

NovaDocker Orchestrator+TOSCA Support (HEAT) OCCI Support

  • 1. TOSCA on HEAT
  • 2. OCCI:

a) Enhanced Network capabilities b) Docker support

  • 3. Nova Docker:

a) Support for native Docker (on bare- metal)

  • 4. Synergy:

a) Fair-share on cloud resource usage

  • 5. Spot-istances

Synergy Spot Istances

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 8

slide-9
SLIDE 9

Regarding IaaS product status for Mitaka

(INDIGO and CMD)

  • OOI -> works in Mitaka
  • Syncrepos -> works in Mitaka
  • Cloud-info-provider -> works in Mitaka
  • IAM integration -> works in Mitaka
  • TOSCA translation -> works in Mitaka
  • Nova-docker -> code works, lacks packaging. à Will be ready soon
  • OPIE -> work in progress for porting on Mitaka
  • Synergy -> will finish Liberty support first, will provide an estimate

during december.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 9

slide-10
SLIDE 10

Enhanced Resource Virtualization -> Storage (QoS)

CEPH

CDMI

1. CDMI service provides the capability to manage the QoS of storage 2. Independent from the technology used 3. CDMI is not used to access files at the site level

a) The files still could be accessed/stored using the original protocols

a) WebDAV b) Posix c) S3 d) GridFTP

POSIX dCache

CDMI CDMI

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 10

slide-11
SLIDE 11

Interactive usage of a Docker container with ssh - Overview

3 Future Gateway API Server Orchestrator OneDock nova-docker WP6 WP5 WP4 TOSCA Documents and Dockerfiles per Use Case Other PaaS Core Services Cloud Site Docker Container Public IP SSHd INDIGO-DataCloud Docker Hub OrganizaLon Provider

Champion + JRA

User 1.a.1) build, push 1.a.2) Dockerfile (commit) 1.b) Automated Build 3) Deploy TOSCA 2) Stage Data 5) Mount 4)Access App IM

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 11

slide-12
SLIDE 12

Interactive usage of a Docker container with ssh - INDIGO Services

1. TOSCA Template to describe the user service 2. Future Gayeway to “configure and submit” TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker + SLAM/QoS:

a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+swrequirements

4. Infrastructure Manager at the PaaS level in case the IaaS do not supports native TOSCA enabled orchestrator 5. IaaS Orchestrator (Heat/IM) supporting TOSCA 6. OneDock or Nova Docker to run Docker on bare metal at IaaS level

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 12

slide-13
SLIDE 13

Future Gateway API Server WP6 WP5 Front-End Public IP Provider User 2) Deploy TOSCA with Vanilla VM / Container 1) Stage Data 5) Mount 6)Access Web Portal Galaxy WN WN WN … Virtual Elastic LRMS Cluster Orchestrator IM OpenNebula WP4 Other PaaS Core Services Cloud Sites OpenStack Heat Clues IM

TOSCA Documents and Dockerfiles per Use Case INDIGO-DataCloud Docker Hub Organization

Champion + JRA

1.a.1) build, push 1.a.2) Dockerfile (commit) 1.b) Automated Build

A web portal that uses a batch system to run applications - Overview

OneZone TOSCA TOSCA

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 13

slide-14
SLIDE 14

Mesos PaaS solution exploiting INDIGO platform

Future Gateway API Server WP6 WP5 Mesos Masters

Public IP

Provider User 2) Deploy TOSCA with Vanilla VM / Container 1) Stage Data 5) Mount 6) Access Mesos Services Chronos/ Marathon 4) Install / Configure Workers … Virtual Elastic Mesos Cluster Orchestrator IM OpenNebula WP4 Other PaaS Core Services Cloud Site OpenStack Heat Clues IM

TOSCA Documents and Dockerfiles per Use Case INDIGO-DataCloud Docker Hub Organization

Champion + JRA

1.a.1) build, push 1.a.2) Dockerfile (commit) 1.b) Automated Build

Workers

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 14

slide-15
SLIDE 15

A dynamic cluster to run applications – INDIGO Services

1. TOSCA Template to describe the user service 2. Future Gayeway to “configure and submit” TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker+ SLAM/QoS:

a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+sw requirements e) That hosts the required data

4. Infrastructure Manager at the PaaS level in case the IaaS do not supports native TOSCA enabled orchestrator 5. IaaS Orchestrator (Heat/IM) supporting TOSCA 6. Onedata for shared and distributed data access 7. Clues for driving the automatic resource provisioning based on the usage

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 15

slide-16
SLIDE 16

DATA IN MULTI-CLOUD ENVIRONMENTS

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 16

slide-17
SLIDE 17

DIFFERENT TYPES OF STORAGES VIRTUALIZED

S3 POSIX Ceph OpenStack Swift

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 17

slide-18
SLIDE 18

Deployment plan on EGI Fed Cloud

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

18

slide-19
SLIDE 19

Deployment at NCG-INGRID-PT

  • Not yet back in EGI Fed Cloud as they prefer to implement AAI based on OpenID

and not on X509

  • Supporting already Lifewatch and EMSODEV
  • INDIGO Services already in production:
  • INDIGO IAM integration in keystone à DONE
  • nova-docker à DONE
  • OOI à DONE
  • Syncrepos à DONE
  • Cloud-InfoProviderà DONE
  • INDIGO Services under test, will be in production soon:
  • Tosca in Heat, IM, Onedata, CDMI - QoS - backend storage ceph, Accouting probes
  • INDIGO Services will be provided depending on the users requirements:
  • Synergy, OPIE, Kubernetes, IAM, Mesos/Marathon/Chronos.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 19

slide-20
SLIDE 20

Deployment at CSIC

  • INDIGO Services already in production:
  • INDIGO IAM integration in keystone à DONE
  • nova-docker à DONE
  • OOI à DONE
  • Syncrepos à DONE
  • uDocker à DONE
  • Accouting probes à DONE
  • INDIGO Services under test, will be in production soon:
  • Tosca in Heat, Onedata, cloud-info-provider, OPIE

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 20

slide-21
SLIDE 21

Deployment at INFN-PADOVA-STACK

  • Cloud framework: OpenStack/Liberty
  • Capacity: 144 physical CPUs, 282 GB RAM, 10.7 TB ephemeral disk, 2.2 TB block storage
  • Communities supported: Atlas, Cms, Lhcb, Structural Biology (WeNMR/West-Life project),

EMSO

  • INDIGO components deployed in production environment:
  • OOI
  • Synergy
  • Other componentesplanned to be deployed soon in production and future plans:
  • openid-keystone(end of December)
  • OneData (end of December, requested by Structural Biology community)
  • Work in progress with CMS team to implement "CMS analysis cluster on-demand" use-case

(OpenStackHEAT deployed so far for this)

  • Upgradingthe site to Openstack/Mitakaversion as soon as a Mitaka compliantSynergy

version available(timescalenot defined yet).

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 21

slide-22
SLIDE 22

Deployment at ReCaS-Bari

  • Cloud framework: OpenStack/Juno
  • Capacity: 1100 physical CPUs, 5.5 TB RAM, 160 TB block storage
  • IaaS INDIGO components deployed in production:
  • OOI
  • IaaS INDIGO components under test in pre-production:
  • OpenID-keystone, Nova-docker, TOSCA Heat, Reposync, OPIE, QoS Storage for

Swift/Posix and CEPH, Synergy, udocker

  • PaaS INDIGO components under test in pre-production:
  • PaaS Orchestrator + Other µService needed
  • Onedata
  • Kubernetes, Mesos/Marathon/Chronos

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 22

slide-23
SLIDE 23

Cloud Accesibility and Usability

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

23

slide-24
SLIDE 24

AAI Features

  • INDIGO provides an advanced set of AAI features that includes:
  • User authentication (supporting SAML, OIDC, X.509)
  • Identity harmonization (link heterogeneous AuthN mechanisms

to a single VO identity)

  • Management of VO membership (i.e., groups and other

attributes)

  • Management of registration and enrolment flows
  • Provisioning of VO structure and membership information to

services

  • Management, distribution and enforcement of authorization

policies

  • A Token Translation Service (TTS), creating credentials for

services that do not natively support OpenID Connect. Services that do not support OpenID Connect are for example ssh, X509 based services S3 storage, OpenNebula.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 24

slide-25
SLIDE 25

IAM: more details

  • Provided a central, integrated solution that
  • supports external authentication mechanisms (Google, SAML) as well as local username/password
  • provides an OIDC/OAuth2 authorisation server that allows to dynamically register clients, manage

user consent and revoke issued tokens if needed

  • integrates a registration service and management dashboard to manage the organisation/VO

structure and membership

  • provides SCIM v2.0 compliant provisioning interfaces that relying services can use to get

information about an organisation members/groups/attributes

  • provides a OAuth token exchange implementation that supports chained delegation for services
  • can be easily deployed as a Docker container

25 7-8/11/2015 INDIGO-DataCloud -- WP5 -- PaaS Layer

slide-26
SLIDE 26

udocker: capabilities

  • It is a tool to execute content of docker containers in user space when docker is not available
  • enables download of docker containers from dockerhub
  • enables execution of docker containers by non-privileged users
  • It is executed under the regular user id (no root privileges needed anymore).
  • privileges are not used in any step not for running not for installing
  • It can be used to execute the content of docker containers in Linux batch systems and

interactive clusters managed by others

  • Acts as a wrapper around other tools to mimic docker capabilities
  • More info and downloads at:
  • https://www.gitbook.com/book/indigo-dc/udocker/details
  • https://indigo-dc.gitbooks.io/udocker/content/doc/user_manual.html

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 26

slide-27
SLIDE 27

udocker: basic description

  • Everything is stored in the $HOME or some other directory belonging to the user (tunable parameter).
  • Container layersare download to the above specified directory
  • Directory trees can be created/extracted from these container layers
  • Uses the ptrace mechanism to change pathnames and execute transparently inside a directory tree
  • No impact on read/write or execution, only impact on system calls using pathnames (open, chdir, etc)

NO IMPACT ON THE PERFORMANCE OF THE CODES

  • Does not require installation of additional softwarein the host system: udockeris a python script

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 27

slide-28
SLIDE 28

Scenarios

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 28

slide-29
SLIDE 29

TOSCA Template

PaaS Orchestrator

JSAGA/JSAGA Adaptors Future Gateway Engine Future Gateway REST API

Other Science Gateways Ophidia Kepler plugin Data Analitics Workflow Portlets GUI Clients Future Gateway Portal Workflows App Portlets Mobile Apps OMT

High-level Architecture

IAM Data Services

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 29

slide-30
SLIDE 30

User interfaces, toolkits and services

FutureGateway Liferay IAM JSAGA RM Ophidia Kepler Orchent, TTSc OMT Portlets

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 30

slide-31
SLIDE 31

Conclusions

  • INDIGO has a modular approach in building its Architecture
  • This provides to the end users and to the resource/e-

infrastructure providers, the capability to build their platforms including the needed “building blocks”

  • Or including new services developed by INDIGO in their own, already

available, platforms

  • INDIGO provide an high level of legacy compatibility:
  • This allow also user community to easily plug their already available

applications/services within the INDIGO Platform

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 31

slide-32
SLIDE 32

Backup slides

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

32

slide-33
SLIDE 33

INDIGO FAQ

  • How do INDIGO achieve resource redundancy and high availability?
  • This is achieved at multiple levels:
  • at the data level, redundancy can be implemented exploiting the capability of INDIGO's Onedata of

replicating data across different data centers.

  • at the site level, it is possible to ask for copies of data to be for example on both disk and tape using the

INDIGO QoS storage features.

  • for services, the INDIGO architecture uses Mesos and Marathon to provide automatic service high-

availability and load balancing. This automation is easily obtainable for stateless services; for stateful services this is application-dependent but it can normally be integrated into Mesos through, for example, a custom framework (examples of which are provided by INDIGO).

  • How do INDIGO achieve resource scalability?
  • First of all, we can distinguish between vertical (scale up) and horizontal (scale out) scalability.

INDIGO provides both:

  • Mesos and Marathon handle vertical scalability by deploying Docker containers with an increasing

amount of resources.

  • The INDIGO PaaS Orchestrator handles horizontal scalability through requests made at the IaaS level to

add resources when needed.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 33

slide-34
SLIDE 34

INDIGO FAQ

  • How do INDIGO achieve resource scalability?
  • The INDIGO software does this in a smart way, i.e. for example it does not look at

CPU load only:

  • In the case of a dynamically instantiated LRMS, it checks the status of jobs and queues and

accordingly adds or remove computing nodes.

  • In the case of a Mesos cluster, in case there are applications to start and there no free

resources, INDIGO starts up more nodes. This happens within the limits of the submitted TOSCA

  • templates. In other words, any given user stays within the limits of the TOSCA template he has

submitted; this is true also for what regards accounting purposes.

  • How do you know when and where resources are available?
  • We have extended the Information System available in the European Grid

Infrastructure (EGI) to inform the INDIGO PaaS orchestrator about the available IaaS infrastructures and about the services they provide. It is therefore possible for the INDIGO orchestrator to optimally choose a certain IaaS infrastructure given, for example, the location of a certain dataset.

Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting 34