INDIGO-DataCloud EGI Fed Cloud contribution Giacinto Donvito (INFN) - PowerPoint PPT Presentation

INDIGO-DataCloud EGI Fed Cloud contribution Giacinto Donvito (INFN) INDIGO-DataCloud Technical Director RIA-653549 EGI Fed Cloud F2F Meeting

Outline • General approach of the INDIGO Platform • The Platform in the proposal • The overall view of the INDIGO-Platform • Standards, protocols, and implementations. 2 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Implementation approach • Rely on standards • µService approach • Modularity • Pick the services you really need for your use-case • And build your own platform based on your needs • Each layer has clear interfaces and could be exploited directly by the end users • The authentication/Authorization is based on the concept of “ Delegation ” • Each service could decide autonomously about the authorization • Each service is requested exploiting the real end-user credential 3 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Implementation approach • Automation based on orchestrating resources • This is done at different level ( IaaS/PaaS/SaaS ) • Open: • Not only OpenSource , • But the possibility to plug-in any supported services/protocols/resources in order to build the needed infrastructure • Both private and public cloud resources could be part of the same INDIGO instance • You can build your own (private) infrastructure or provide a multi-tenant solution for your users. • Depending on your goal 4 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

WP6 Services The high level view of Admin User Ophidpia Mobile Apps plugin Portlets Portlets Other GUI the Architecture Science SG Mon LONI plugin Clients Workflow Data Gateways Open Mobile Taverna, Analitics Portlets Toolkit Support Kepler plugin services Future Gateway Portal Mobile clients Workflows Future Gateway REST API Future Gateway Engine JSAGA/JSAGA Adaptors REST/CDMI/Wedbav/posix/Gridftp OIDC TOSCA Kubernetes Cluster Dynafed Onedata PaaS Data Services FTS Orchestrator QoS/SLA This is the INDIGO-DataCloud Accounting IAM Service Monitoring TOSCA WP5 General Architecture* CloudProvider Services Infrastructure Ranker Manager TOSCA S3/CDMI/Posix/Webdav GridFTP Mesos Mesos Aut. Scaling Cluster Cluster Service Storage Native IaaS API Service QoS Support Heat/IM Non-INDIGO Identity Smart IaaS Spot Istances Armonization *: see details in http://arxiv.org/abs/1603.09536 or in Scheduling https://www.indigo-datacloud.eu/documents-deliverables Native WP4 Services Docker Local Repository 5 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Some possible implementation scenario: 1. Enhanced Resource Virtualization -> Computing 2. Enhanced Resource Virtualization -> Storage 3. A complete federated storage access solution 4. Interactive usage of a Docker container with ssh 5. A web portal that uses a batch system to run applications 6. Virtual infrastructures for executing scientific applications and services exploiting Mesos 6 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Enhanced Resource Virtualization -> Computing (OpenNebula) 1. IM : Provides Orchestrator+TOSCA OCCI a) Advanced IaaS Orchestrator Support (IM) capabilities Support b) TOSCA Support OneDock 2. OCCI : a) Enhanced Network capabilities OpenNebula b) Docker support 3. OneDock : a) Support for native Docker (on bare- metal) 7 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Enhanced Resource Virtualization -> Computing (OpenStack) 1. TOSCA on HEAT Orchestrator+TOSCA OCCI 2. OCCI : Support (HEAT) Support a) Enhanced Network capabilities b) Docker support NovaDocker Synergy 3. Nova Docker : a) Support for native Docker (on bare- OpenStack metal) 4. Synergy : a) Fair-share on cloud resource usage Spot Istances 5. Spot-istances 8 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Regarding IaaS product status for Mitaka (INDIGO and CMD) • OOI -> works in Mitaka • Syncrepos -> works in Mitaka • Cloud-info-provider -> works in Mitaka • IAM integration -> works in Mitaka • TOSCA translation -> works in Mitaka • Nova-docker -> code works, lacks packaging. à Will be ready soon • OPIE -> work in progress for porting on Mitaka • Synergy -> will finish Liberty support first, will provide an estimate during december. 9 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Enhanced Resource Virtualization -> Storage (QoS) 1. CDMI service provides the capability to manage the QoS of storage 2. Independent from the technology CDMI CDMI CDMI used 3. CDMI is not used to access files at the CEPH POSIX dCache site level a) The files still could be accessed/stored using the original protocols a) WebDAV b) Posix c) S3 d) GridFTP 10 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Interactive usage of a Docker container with ssh - Overview WP6 3) Deploy Future Gateway User TOSCA API Server WP5 2) Stage TOSCA Documents and Data Dockerfiles per Use Case 4)Access Other PaaS 1.a.2) Orchestrator 1.b) Core Services Dockerfile Automated (commit) Build Provider IM 5) Mount Champion Cloud + JRA 1.a.1) INDIGO-DataCloud Site WP4 App build, Docker Hub OrganizaLon SSHd Docker push OneDock nova-docker Container Public IP 3 11 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Interactive usage of a Docker container with ssh - INDIGO Services 1. TOSCA Template to describe the user service 2. Future Gayeway to “configure and submit” TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker + SLAM/QoS : a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+swrequirements 4. Infrastructure Manager at the PaaS level in case the IaaS do not supports native TOSCA enabled orchestrator 5. IaaS Orchestrator ( Heat/IM ) supporting TOSCA 6. OneDock or Nova Docker to run Docker on bare metal at IaaS level 12 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

A web portal that uses a batch system to run applications - Overview WP6 2) Deploy TOSCA with Future Gateway User Vanilla VM / Container API Server WP5 TOSCA Documents and Dockerfiles per Use Case 1.a.2) 1.b) 1) Stage Data 6)Access Dockerfile Automated Other PaaS (commit) Web Portal Orchestrator Build Core Services Champion + JRA IM OneZone 1.a.1) INDIGO-DataCloud build, Docker Hub Organization push WP4 TOSCA Cloud TOSCA Clues Virtual Elastic LRMS Cluster Sites IM Heat 5) Mount Galaxy … WN Front-End WN WN Public IP Provider OpenNebula OpenStack 13 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Mesos PaaS solution exploiting INDIGO platform WP6 2) Deploy TOSCA with Future Gateway User Vanilla VM / Container API Server WP5 TOSCA Documents and 1) Stage 6) Access Dockerfiles per Use Case 1.a.2) Data 1.b) Mesos Dockerfile Automated Other PaaS (commit) Services Orchestrator Build Core Services Champion Provider + JRA IM 1.a.1) INDIGO-DataCloud build, Docker Hub Organization push 5) Mount WP4 Cloud Public IP 4) Install / Virtual Elastic Mesos Site IM Chronos/ Clues Configure Heat Cluster Marathon Mesos … Workers Workers Masters OpenNebula OpenStack 14 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

A dynamic cluster to run applications – INDIGO Services 1. TOSCA Template to describe the user service 2. Future Gayeway to “configure and submit” TOSCA Template in an easy way 3. Orchestrator + PaaS Core services + CloudProviderRanker+ SLAM/QoS : a) To find the available IaaS b) That are correctly working c) That has SLA with the given user d) And supports the hw+sw requirements e) That hosts the required data 4. Infrastructure Manager at the PaaS level in case the IaaS do not supports native TOSCA enabled orchestrator 5. IaaS Orchestrator (Heat/IM) supporting TOSCA 6. Onedata for shared and distributed data access 7. Clues for driving the automatic resource provisioning based on the usage 15 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

DATA IN MULTI-CLOUD ENVIRONMENTS 16 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

DIFFERENT TYPES OF STORAGES VIRTUALIZED S3 POSIX Ceph OpenStack Swift 17 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting

Deployment plan on EGI Fed Cloud Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed 18 Cloud F2F Meeting

Deployment at NCG-INGRID-PT • Not yet back in EGI Fed Cloud as they prefer to implement AAI based on OpenID and not on X509 • Supporting already Lifewatch and EMSODEV • INDIGO Services already in production : • INDIGO IAM integration in keystone à DONE • nova-docker à DONE • OOI à DONE • Syncrepos à DONE • Cloud-InfoProvider à DONE • INDIGO Services under test, will be in production soon : • Tosca in Heat, IM, Onedata, CDMI - QoS - backend storage ceph, Accouting probes • INDIGO Services will be provided depending on the users requirements: • Synergy, OPIE, Kubernetes, IAM, Mesos/Marathon/Chronos. 19 Giacinto DONVITO -- INDIGO Technical Director -- EGI Fed Cloud F2F Meeting