FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 - - PowerPoint PPT Presentation

fedcloud f2f
SMART_READER_LITE
LIVE PREVIEW

FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 - - PowerPoint PPT Presentation

FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142 Agenda for F2F EGI-ENGAGE EGI FedCloud status overview (10) Resource Providers and operations


slide-1
SLIDE 1

EGI-Engage is co-funded by the Horizon 2020 Framework Programme

  • f the European Union under grant number 654142

www.egi.eu

FedCloud F2F

slide-2
SLIDE 2

2

EGI-ENGAGE

Agenda for F2F

  • EGI FedCloud status overview (10’)
  • Resource Providers and operations views

and feedback (30’)

– How can we attract more resources?

  • Roadmap discussion (1h)

– State of the Scenarios – Next steps

  • AOB (20’)
slide-3
SLIDE 3

3

EGI-ENGAGE

EGI FedCloud Status

10/11/15

slide-4
SLIDE 4

4

EGI-ENGAGE

The infrastructure

slide-5
SLIDE 5

5

EGI-ENGAGE

Nov 2014 – May 2015

  • 21 sites
  • 28.3K VMs*
  • 335K CPU hours*
  • 20 VOs

May 2015 – Nov 2015

  • 20 sites (SZTAKI gone)
  • 29.4K VMs*
  • 257K CPU hours*
  • 24 VOs
  • CC are starting to

apporach fedcloud: LifeWatch, MoBrain, BBMRI, ELIXIR…

Infrastructure numbers

* Accoun(ng numbers removing LHC and local VOs

Missing clear informa(on from long- running VMs, these may have grown in the last period

10/11/15

slide-6
SLIDE 6

6

EGI-ENGAGE

Sites joining

  • New sites actively joining in:

– BITP (UA) – EBI (UK)

  • Long-standing integrations

– KISTI (KR)

  • Got stuck with network configuration, no progress since August

– CSC (FI)

  • Proof of concept working, still not integrated into production.

– RAL (UK)

  • Under risk assessment (network separation)
slide-7
SLIDE 7

7

EGI-ENGAGE

Documentation

  • Wiki completely reviewed
  • Technology Section

– Services and tools that build the federation

  • Site-related documentation

– Complete installation manuals for sites

  • OpenNebula and OpenStacl

– Several HOWTOS for configuration of specific functionality – Being tested by new sites now

  • User/Developers related-documentation

– OCCI-client FAQ reviewed – API, SDKs, CLIs page

slide-8
SLIDE 8

8

EGI-ENGAGE

EGI images

  • Allow users to get started quickly

– Cloud-init configured (no user/root passwords)

  • Automatically built with packer

– Easy to get updated versions

  • Set of basic OS images

– Ubuntu 12, Ubuntu 14, Centos 6 – Centos 7 ready, not deployed due to problems with older versions

  • f Xen
  • FedCloud clients VM

– rOCCI – Preconfigured fedcloud.egi.eu + training.egi.eu VOMS

  • Training images

– MoinMoin Wiki – Fractal Application

slide-9
SLIDE 9

9

EGI-ENGAGE

Docker

  • Demand for Docker support increasing (PanCancer,

HBP, READemption, )

  • Docker is supported on FedCloud

– Just install it on a VM – A Ubuntu 14.04 + Docker image now available!

  • Docker container with voms + OCCI clients installed

– Run the EGI FedCloud clients without messing with your machine configuration – Will use it in training tomorrow

  • Next step:

– Documentation – Clusters?

slide-10
SLIDE 10

10

EGI-ENGAGE

Training

  • Training infrastructure available

– CESNET, BIFI, CIEMAT, CATANIA, UKIM

  • Use of Per User Sub Proxies

– No need for certificates for the trainees – Supported in OpenStack and OpenNebula*

  • Training modules

– Introduction with OCCI CLI, already used in three events – For this week:

  • Preparation of VMIs, VMDIRAC, COMPSs, D4Science

– Coming:

  • Docker

*not complete user separa(on

slide-11
SLIDE 11

11

EGI-ENGAGE

Expanding the Federation

  • IaaS interfaces

– OCCI as main & preferred API – Openstack API now also supported – Swift API coming soon – PROC19 being tested on:

  • OpenNebula + rOCCI
  • OpenStack + OCCI-OS
  • OpenStack Nova
  • FedCloud as a technology provider for building

federations

– See wiki, technology section.

10/11/15

slide-12
SLIDE 12

12

EGI-ENGAGE

Last news from the FedCloud Use Cases

  • BILS – Swedish ELIXIR Node:

– SLA to be signed by the end of Nov 2016 – CPU cores: 324, RAM: 648 GB, Temp Storage: 1425 GB, Permanent Storage: 5400 GB

  • Chipster (NGS) – CSC Finnish ELIXIR NODE:

– In production with VO chipster.csc.fi – CSC/EGI Training events:

  • NGS event in Thessaloniki (Oct 2015)
  • 2 tutorials on Wednesday: CHIPSTER deployment (11:00),

Usage for NGS analysis (15:30)

  • Planning a tutorial in March 2016 in Helsinki (collaboration with

ELIXIR's EXCELERATE project)

FedCloud F2F @ Bari

slide-13
SLIDE 13

13

EGI-ENGAGE

Last news from the FedCloud Use Cases

  • EMSO/EMSODev:

– First test running in the FedCloud:

  • Setup an HADOOP cluster
  • Scipion/INSTRUCT:

– It is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy – MoBrain CC – First testing deployments in the Fedcloud ready

slide-14
SLIDE 14

14

EGI-ENGAGE

Main issues (personal view)

  • VMI replication to sites is not reliable

– Lack of monitoring

  • OCCI support in OpenStack

– OCCI-OS not maintained, ooi not released – FZJ 0% A/R because of this!

  • Need to find balance between security and usability

– We need to properly define who and what can be done and enforce it

  • Lack of tool ecosystem (e.g. support from libraries like

jclouds, lacking support of orchestrators, vagrant, no GUIs, ...)

10/11/15

slide-15
SLIDE 15

15

EGI-ENGAGE

Resource Providers

10/11/15

slide-16
SLIDE 16

16

EGI-ENGAGE

Sites joining

  • New sites joining in:

– BITP (UA) – EBI (UK)

  • Long-standing integrations

– KISTI (KR)

  • Got stuck with network configuration, no progress since August

– CSC (FI)

  • Proof of concept working, still not integrated into production.

– RAL (UK)

  • Under risk assessment (network separation)

Why integra(on takes so long?

  • Documenta(on?
  • SoKware Quality?
  • Security?
slide-17
SLIDE 17

17

EGI-ENGAGE

We need feedback!

  • What are the biggest issues as a resource

provider in FedCloud?

  • How can we get more providers joining?

10/11/15

slide-18
SLIDE 18

18

EGI-ENGAGE

Roadmap discussion

10/11/15

slide-19
SLIDE 19

19

EGI-ENGAGE

Scenarios

  • VM Image Management
  • VM endorsement
  • Brokering
  • Security
  • Intra Cloud Networking
  • VM Management
  • Data Management
  • Information Discovery
  • Accounting
  • Monitoring
  • Federated AAI

10/11/15

slide-20
SLIDE 20

20

EGI-ENGAGE

VM Management

  • PUSP support
  • OCCI extensions and move to OCCI 1.2

– Compute -- resize – Compute – save

  • Deployment of OOI for OpenStack
  • Native API support
  • Improvements/integration for client tools

10/11/15

slide-21
SLIDE 21

21

EGI-ENGAGE

Information Discovery

  • VO specific info

– Who can prepare the schema?

  • Available resources

– Should this be published?

  • Alternatives to BDII?
  • Leadership of this scenario?

10/11/15

slide-22
SLIDE 22

22

EGI-ENGAGE

Accounting

  • Storage, IP, any other resource accounting?
  • How to make users aware of their usage?

10/11/15

slide-23
SLIDE 23

23

EGI-ENGAGE

Monitoring

  • Rework OCCI probe to use an AppDB

VMI

– Test all of the EGI VMIs?

  • VMCatcher probe
  • Block Storage probe
  • Swift probes

10/11/15

slide-24
SLIDE 24

24

EGI-ENGAGE

AAI

  • Need integration with broader EGI AAI

plans

– Technology? – Roadmap?

  • Who can act as bridge?

10/11/15

slide-25
SLIDE 25

25

EGI-ENGAGE

VMI management

  • Single VMI per VA version
  • Reliable image distribution

– Disk consumption – Documentation – Allow external implementations (HEPiX list format specification)

  • Banning VA versions

– independent of endorsement?

10/11/15

slide-26
SLIDE 26

26

EGI-ENGAGE

VMI endorsement

  • Endorsing images not that easy to implement

– Signing? Who? When? How? un-endorsement?

  • Need to find balance between security and

usability

– Automatic checks?

  • This does not solve that the may go

(insecurely) crazy during execution!

10/11/15

slide-27
SLIDE 27

27

EGI-ENGAGE

Networking

  • Is OCCI network enough?

– How to deal with Neutron? – Firewalls?

  • How to deal with multiple sites?

– VPNs – DNS as a Service

10/11/15

slide-28
SLIDE 28

28

EGI-ENGAGE

Brokering, user interfaces

  • How can users profit from the federation?
  • GUIs?

– AppDB as VM Management tool

  • No integration with existing cloud ecosystem
  • Leadership?

10/11/15

slide-29
SLIDE 29

29

EGI-ENGAGE

Security

  • Security Threat Risk Assessment with Cloud

focus

– Need participation from a few others in the EGI Federated Cloud

  • Security Requirements related to the EGI Fed

cloud.

– VA and VM drafted, needs iteration, probably also software requirements extracting

  • Probably also we should write down the

EGI Federated Cloud Security Model

  • Security Monitoring?

10/11/15

slide-30
SLIDE 30

30

EGI-ENGAGE

Data Management

10/11/15

"The Elephant in the Room Banksy-Barely legal-2006" by Bit Boy - Flickr: The Elephant in the

  • Room. Licensed under CC BY 2.0 via Commons - hWps://commons.wikimedia.org/
slide-31
SLIDE 31

31

EGI-ENGAGE

AOB

10/11/15