OpenStack on the EGI Federated Cloud Enol En l Fe Fernndez - - PowerPoint PPT Presentation

▶
openstack on the egi federated cloud
SMART_READER_LITE
LIVE PREVIEW

OpenStack on the EGI Federated Cloud Enol En l Fe Fernndez - - PowerPoint PPT Presentation

May 16, 2023 120 likes •437 views

OpenStack on the EGI Federated Cloud Enol En l Fe Fernndez Cloud ud Archi hitect EGI Founda undation eno nol.ferna nande ndez@egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme www.egi.eu www of the

slide-1
SLIDE 1

EGI-Engage is co-funded by the Horizon 2020 Framework Programme

  • f the European Union under grant number 654142

www www.egi.eu

Cloud ud Archi hitect – EGI Founda undation eno nol.ferna nande ndez@egi.eu

OpenStack on the EGI Federated Cloud

En Enol l Fe Fernández

slide-2
SLIDE 2

2 12/ 12/15/ 15/16 16

EG EGI: Advanced Computing for Re Research

slide-3
SLIDE 3

3 12/ 12/15/ 15/16 16

http://go.egi.eu/ServiceCatalogue

OpenStack Summit Barcelona

slide-4
SLIDE 4

4 12/ 12/15/ 15/16 16 OpenStack Summit Barcelona

slide-5
SLIDE 5

5 12/ 12/15/ 15/16 16

EGI Federation, , 2016 QR3

The he largest di distribut buted d comput pute e-Infra worldw dwide de

23 Cloud providers, +300 data centres +250 000 instantiated VMs/year 1.7 Million jobs/day 2.6 Billion CPU hours/year +26% >48 000 users, +25%

OpenStack Summit Barcelona

slide-6
SLIDE 6

6 12/ 12/15/ 15/16 16

In Interna natio ional nal Par artne nership hips

Africa and Arabia Council for Scientific and Industrial Research, South Africa India Centre for Development of Advanced Comp. China Inst. Of HEP Chinese Academy

  • f Sciences

Latin America Universidade Federal do Rio de Janeiro Ukraine Ukrainian National Grid USA Canada Asia Pacific Region Academia Sinica at Taiwan

slide-7
SLIDE 7

7 12/ 12/15/ 15/16 16

Se Serving ing resear arche hers and and inno innovators

ESFRIs, FET flagships

Size of individual groups

Multinational communities ‘Long tail’

WLCG CTA ELIXIR EPOS EISCAT_3D BBMRI CLARIN LOFAR EMSO ELI LifeWatch ICOS EMSO CORBEL ENVRIplus … VRE projects WeNMR DRIHM VERCE MuG AgINFRA CMMST LSGC SuperSites Exploitation Environmental sci. neuGRID … PeachNote CEBA Galaxy eLab Semiconductor design Main-belt comets Quantum pysics studies Virtual imaging (LS) Bovine tuberculosis spread Convergent evol. in genomes Geography evolution Seafloor seismic waves 3D liver maps with MRI Metabolic rate modelling Genome alignment Tapeworms infection on fish …

Industry, SMEs

Agroknow CloudEO CloudSME Ecohydros gnubila Sinergise SixSq TEISS Terradue Ubercloud …

OpenStack Summit Barcelona

slide-8
SLIDE 8

8 12/ 12/15/ 15/16 16

EG EGI Federated Cloud

  • EGI Federated Cloud is a

collaboration of communities developing, innovating,

  • perating and using cloud

federations for research and education.

  • 23 providers from 14 NGIs

– 16 OpenStack – 6 OpenNebula – 1 Synnefo

  • ~7K CPU cores
slide-9
SLIDE 9

9 12/ 12/15/ 15/16 16

Cl Cloud ud Realms ms

EGI Core Infrastructure Platform

AAI, Service Registry, Accounting, Monitoring, Federated Service Management

Collaboration Platform

VM image catalogue, Helpdesk EGI endorsed images

Cloud Realm Cloud Realm Community Platform Community Platform

Cloud Realm subset of cloud providers exposing homogeneous cloud management interfaces and capabilities which use the the services of the EGI Core Infrastructure Platform for creating a federation Community Platforms provide community-specific data, tools and applications and can be supported by one or more realms.

Cloud Realm Cloud Realm

OpenStack Summit Barcelona

slide-10
SLIDE 10

10 10 12/ 12/15/ 15/16 16

OpenStack OpenNebula OpenStack OpenNebula OpenStack Synnefo

Harmonised

  • peration

Service registry Information system Virtual Machine marketplace Usage accounting Access control

Cl Cloud ud Fede deration

Uniform user interfaces

OpenStack realm Open Standards realm

slide-11
SLIDE 11

11 11 12/ 12/15/ 15/16 16

EG EGI AAI

  • Users identified with X.509 certificates (IGTF Federation) with VOMS

extensions

  • VOMS (Virtual Organization Management System) provides attributes on

membership to VOs, groups and roles on the VO

  • Not user-friendly, problematic in web-based GUIs
  • Now in transition to new EGI AAI (EGI CheckIn)
  • Federated identity standards (SAML, OpenID Connect)
  • Allows users to authenticate with their institutional accounts
  • Integration with Attribute Authorities beyond VOMS

OpenStack Summit Barcelona

slide-12
SLIDE 12

12 12 12/ 12/15/ 15/16 16

Ke Keystone-VO VOMS

  • WSGI filter for Keystone V2 API
  • Extracts information from VOMS proxies to perform

AuthN/AuthZ

  • Can manage federation users
  • Add users to Keystone
  • Add roles to users in tenants
  • Mapping VOMS → Keystone defined on file

https://github.com/IFCA/Keystone-VOMS

OpenStack Summit Barcelona

slide-13
SLIDE 13

13 13 12/ 12/15/ 15/16 16

Apache HTTPD

EG EGI Che CheckIn In + + Ke Keystone

Horizon Keystone mod_shib EGI CheckIn User IdP

Enter credentials SAML assertion SAML assertion with claims Token Token

IdP

Attribute Authority

EGI UID First name, last name email affiliation Mandatory Attributes

OpenStack Summit Barcelona

slide-14
SLIDE 14

14 14 12/ 12/15/ 15/16 16

Se Servic ice Regis gistry

  • All Resource Centers must

register their services at the EGI central catalog: https://goc.egi.eu

  • Static information about

services endpoints

– org.openstack.nova and

  • rg.openstack.swift service

types

  • Web frontend and API

access

slide-15
SLIDE 15

15 15 12/ 12/15/ 15/16 16

In Informa matio ion n Dis Discovery

  • Real-time information provided by BDII

– Hierarchical information discovery system based on LDAP – Using standard Glue Schema 2

  • Resource Centers publish actual capabilities

– Available images & flavors – Supported user groups (VOs) – Available resources

  • Cloud-bdii-provider

– Gathers information from OS services using public APIs and puts it into Glue Schema – https://github.com/EGI-FCTF/cloud-bdii-provider

OpenStack Summit Barcelona

slide-16
SLIDE 16

16 16 12/ 12/15/ 15/16 16

Acc Accounting

  • Collect, aggregate and display

usage information across the whole federation.

  • OGF Usage Record extended for

Cloud

  • cASO produces accounting

records using nova (and

  • ptionally ceilometer) APIs
  • https://github.com/IFCA/caso
slide-17
SLIDE 17

17 17 12/ 12/15/ 15/16 16

Mon Monitor

  • ring
  • Health monitoring of

services

  • Automatic discovery of

services using GOCDB

  • A/R metrics for SLA/OLAs
  • Powered by EGI ARGO
  • https://argoeu.github.io/
slide-18
SLIDE 18

18 18 12/ 12/15/ 15/16 16

VM VM Image Marketplace: Ap AppDB

  • Open Library of Virtual Appliances
  • Use on clouds or for personal download
  • Re-use, share, associate

contextualization

  • EGI endorsed VM images, securely

configured and tested

  • Community curated sets of images
  • Automatic distribution of sets to cloud

providers

  • https://github.com/alvarolopez/atrope
slide-19
SLIDE 19

19 19 12/ 12/15/ 15/16 16

VM VM Image Marketplace: Ap AppDB

slide-20
SLIDE 20

20 20 12/ 12/15/ 15/16 16

Ope OpenSt nStack OCCI OCCI interface (ooi

  • oi)
  • OC

OCCI (Open Cloud Computing Interface, OGF)

– RESTFul protocol and API focusing on cloud interoperability – Primarily for IaaS (manage VMs and Block Storage), extensible to other areas

  • ooi (OpenStack OCCI interface)

– Completely written from scratch OCCI implementation – Uses only public OpenStack APIs – Support for VM, volumes and network operations – Can be installed along an existing nova-api endpoint or as a separate WSGI application

https://launchpad.net/ooi

OpenStack Summit Barcelona

slide-21
SLIDE 21

21 21 12/ 12/15/ 15/16 16

  • oi
  • oi in

in the glo global al OCCI I pic icture

  • oi

OpenStack Summit Barcelona

slide-22
SLIDE 22

22 22 12/ 12/15/ 15/16 16

EG EGI – Ope OpenSt nStack integr gration n (I) (I)

Apache+SSL+ mod_wsgi

Keystone Keystone- VOMS nova-api

  • oi

glance ceilometer atrope cASO + SSM OpenStack resource center EGI accounting repository cloud-bdii- provider AppDB (vmcaster)

Extract usage information Extract information Register/update images Publish accounting records Publish information Subscribe to image lists

EGI BDII

VOMS x509 proxy Get access token IaaS Operations OCCI clients OpenStack clients

EGI monitoring

slide-23
SLIDE 23

23 23 12/ 12/15/ 15/16 16

EG EGI – Ope OpenSt nStack integr gration n (II) (II)

Apache+SSL+ mod_wsgi

Keystone

OS- FEDERATION

nova-api

  • oi

glance ceilometer atrope cASO + SSM OpenStack resource center EGI accounting repository cloud-bdii- provider AppDB (vmcaster)

Extract usage information Extract information Register/update images Publish accounting records Publish information Subscribe to image lists

EGI BDII

Get access token IaaS Operations OCCI clients OpenStack clients

EGI monitoring EGI CheckIn

slide-24
SLIDE 24

24 24 12/ 12/15/ 15/16 16

Ope OpenSt nStack Fe FedCloud Ap Appliance ce

  • A single VM with all the components using the public

OpenStack interfaces

– Accounting, Information discovery, VMI replication – Packaged as Docker containers, available at docker hub https://hub.docker.com/u/egifedcloud/

  • Documentation:

– https://wiki.egi.eu/wiki/MAN10#Integration_with_EGI_FedCloud_Appl iance

  • Appliance at AppDB:

– https://appdb.egi.eu/store/vappliance/fedcloud.integration.appliance.

  • penstack

OpenStack Summit Barcelona

slide-25
SLIDE 25

25 25 12/ 12/15/ 15/16 16

Cl Cloud R Realms ms: b build c customi mized f federati tions

Service Cloud Realm Integration EGI Technology EGI Federated Service Management mandatory

  • EGI Service Registry

mandatory GOCDB EGI AAI compliance mandatory Keystone-VOMS EGI Accounting mandatory, depends on EGI AAI compliance cASO EGI Monitoring mandatory, done externally. Monitoring of IaaS interfaces (OCCI, OpenStack, CDMI) requires EGI AAI compliance EGI ARGO EGI Information Discovery

  • ptional

cloud-bdii-provider VMI replication

  • ptional

atrope VMI catalogue

  • ptional

AppDB IaaS standard interface

  • ptional, all providers of the realm must provide homogeneous

interface

  • oi

EGI helpdesk

  • ptional

GGUS

OpenStack Summit Barcelona

slide-26
SLIDE 26

26 26 12/ 12/15/ 15/16 16

Co Communiti ties s usi sing Cl Cloud Co Compute (I)

The EXTraS project is harvesting 13 years of data collected on-board the ESA’s X-ray space

  • bservatory XMM-Newton.

The project is using Cloud Compute to implement four lines of analysis with ad-hoc software pipelines The DRIHM project is prototyping an e- infrastructure to simulate extreme hydro- meteorological events such as ash flooding. The National Bioinformatics Infrastructure of Sweden uses Cloud Compute to provide bioinformatics tools to their researchers, including high- profile tools to predict 3D protein structures, for

  • example. So far, more than

6,700 unique users in 73 countries have made the most of these resources

OpenStack Summit Barcelona

https://www.egi.eu/use-cases/

slide-27
SLIDE 27

27 27 12/ 12/15/ 15/16 16

Co Communiti ties s usi sing Cl Cloud Co Compute (II)

Konrad Förstner and team used EGI Federacted cloud to analyse the RNAs produced by the Salmonella and humans at the same time, in the same experiments to discover that a piece of Salmonella RNA called PinT is heavily involved in what happens right after the infection. Results published in Nature. VERCE platform is a science gateway, developed jointly by seismologists and IT experts, as a tool to create accurate earthquake simulations. VERCE helped seismologists to make sense of the August 2016 Amatrice earthquake using the resources of EGI Cloud Peachnote is a music score search engine and analysis

  • platform. The system is the first
  • f its kind and can be thought as

an analog of Google Books Ngram Viewer and Google Books search for music scores. Peachnote is visited by tens of thousands of users every day from all over the world.

OpenStack Summit Barcelona

Pictures from wikimedia commons

https://www.egi.eu/use-cases/

slide-28
SLIDE 28

28 28 12/ 12/15/ 15/16 16

Wh What’s c com

  • ming
  • Improve user experience

– Certificate-less access with new EGI AAI – AppDB extension to provide VM management

  • OCCI 1.2

– Improved version of the standard, better networking support

  • Go beyond IaaS

– Exploit results from INDIGO-Project to offer PaaS to our users

  • Get more involved in OpenStack

– Scientific WG, identity federation, …

OpenStack Summit Barcelona

slide-29
SLIDE 29

29 29 12/ 12/15/ 15/16 16

So Some me referenc nces

  • EGI: http://www.egi.eu
  • Federated Cloud at EGI wiki:

https://wiki.egi.eu/wiki/EGI_Federated_Cloud

– Installation manual: https://wiki.egi.eu/wiki/MAN10

  • EGI Federated Cloud list: fedcloud-tf@mailman.egi.eu

OpenStack Summit Barcelona

slide-30
SLIDE 30

Tha Thank nk you u for your ur attention. n.

Qu Questions?

This work by Parties of the EGI-Engage Consortium is licensed under a Creative Commons Attribution 4.0 International License.

www www.egi.eu