FedCloud F2F www.egi.eu EGI-Engage is co-funded by the Horizon 2020 - - PowerPoint PPT Presentation

EGI-Engage is co-funded by the Horizon 2020 Framework Programme

• f the European Union under grant number 654142

www.egi.eu

FedCloud F2F

2

EGI-ENGAGE

Agenda for F2F

• EGI FedCloud status overview (10’)
• Resource Providers and operations views

and feedback (30’)

– How can we attract more resources?

• Roadmap discussion (1h)

– State of the Scenarios – Next steps

• AOB (20’)

3

EGI-ENGAGE

EGI FedCloud Status

10/11/15

4

EGI-ENGAGE

The infrastructure

5

EGI-ENGAGE

Nov 2014 – May 2015

• 21 sites
• 28.3K VMs*
• 335K CPU hours*
• 20 VOs

May 2015 – Nov 2015

• 20 sites (SZTAKI gone)
• 29.4K VMs*
• 257K CPU hours*
• 24 VOs
• CC are starting to

apporach fedcloud: LifeWatch, MoBrain, BBMRI, ELIXIR…

Infrastructure numbers

* Accoun(ng numbers removing LHC and local VOs

Missing clear informa(on from long- running VMs, these may have grown in the last period

10/11/15

6

EGI-ENGAGE

Sites joining

• New sites actively joining in:

– BITP (UA) – EBI (UK)

• Long-standing integrations

– KISTI (KR)

• Got stuck with network configuration, no progress since August

– CSC (FI)

• Proof of concept working, still not integrated into production.

– RAL (UK)

• Under risk assessment (network separation)

7

EGI-ENGAGE

Documentation

• Wiki completely reviewed
• Technology Section

– Services and tools that build the federation

• Site-related documentation

– Complete installation manuals for sites

• OpenNebula and OpenStacl

– Several HOWTOS for configuration of specific functionality – Being tested by new sites now

• User/Developers related-documentation

– OCCI-client FAQ reviewed – API, SDKs, CLIs page

8

EGI-ENGAGE

EGI images

• Allow users to get started quickly

– Cloud-init configured (no user/root passwords)

• Automatically built with packer

– Easy to get updated versions

• Set of basic OS images

– Ubuntu 12, Ubuntu 14, Centos 6 – Centos 7 ready, not deployed due to problems with older versions

• f Xen
• FedCloud clients VM

– rOCCI – Preconfigured fedcloud.egi.eu + training.egi.eu VOMS

• Training images

– MoinMoin Wiki – Fractal Application

9

EGI-ENGAGE

Docker

• Demand for Docker support increasing (PanCancer,

HBP, READemption, )

• Docker is supported on FedCloud

– Just install it on a VM – A Ubuntu 14.04 + Docker image now available!

• Docker container with voms + OCCI clients installed

– Run the EGI FedCloud clients without messing with your machine configuration – Will use it in training tomorrow

• Next step:

– Documentation – Clusters?

10

EGI-ENGAGE

Training

• Training infrastructure available

– CESNET, BIFI, CIEMAT, CATANIA, UKIM

• Use of Per User Sub Proxies

– No need for certificates for the trainees – Supported in OpenStack and OpenNebula*

• Training modules

– Introduction with OCCI CLI, already used in three events – For this week:

• Preparation of VMIs, VMDIRAC, COMPSs, D4Science

– Coming:

• Docker

*not complete user separa(on

11

EGI-ENGAGE

Expanding the Federation

• IaaS interfaces

– OCCI as main & preferred API – Openstack API now also supported – Swift API coming soon – PROC19 being tested on:

• OpenNebula + rOCCI
• OpenStack + OCCI-OS
• OpenStack Nova
• FedCloud as a technology provider for building

federations

– See wiki, technology section.

10/11/15

12

EGI-ENGAGE

Last news from the FedCloud Use Cases

• BILS – Swedish ELIXIR Node:

– SLA to be signed by the end of Nov 2016 – CPU cores: 324, RAM: 648 GB, Temp Storage: 1425 GB, Permanent Storage: 5400 GB

• Chipster (NGS) – CSC Finnish ELIXIR NODE:

– In production with VO chipster.csc.fi – CSC/EGI Training events:

• NGS event in Thessaloniki (Oct 2015)
• 2 tutorials on Wednesday: CHIPSTER deployment (11:00),

Usage for NGS analysis (15:30)

• Planning a tutorial in March 2016 in Helsinki (collaboration with

ELIXIR's EXCELERATE project)

FedCloud F2F @ Bari

13

EGI-ENGAGE

Last news from the FedCloud Use Cases

• EMSO/EMSODev:

– First test running in the FedCloud:

• Setup an HADOOP cluster
• Scipion/INSTRUCT:

– It is an image processing framework to obtain 3D models of macromolecular complexes using Electron Microscopy – MoBrain CC – First testing deployments in the Fedcloud ready

14

EGI-ENGAGE

Main issues (personal view)

• VMI replication to sites is not reliable

– Lack of monitoring

• OCCI support in OpenStack

– OCCI-OS not maintained, ooi not released – FZJ 0% A/R because of this!

• Need to find balance between security and usability

– We need to properly define who and what can be done and enforce it

• Lack of tool ecosystem (e.g. support from libraries like

jclouds, lacking support of orchestrators, vagrant, no GUIs, ...)

10/11/15

15

EGI-ENGAGE

Resource Providers

10/11/15

16

EGI-ENGAGE

Sites joining

• New sites joining in:

– BITP (UA) – EBI (UK)

• Long-standing integrations

– KISTI (KR)

• Got stuck with network configuration, no progress since August

– CSC (FI)

• Proof of concept working, still not integrated into production.

– RAL (UK)

• Under risk assessment (network separation)

Why integra(on takes so long?

• Documenta(on?
• SoKware Quality?
• Security?
• …

17

EGI-ENGAGE

We need feedback!

• What are the biggest issues as a resource

provider in FedCloud?

• How can we get more providers joining?

10/11/15

18

EGI-ENGAGE

Roadmap discussion

10/11/15

19

EGI-ENGAGE

Scenarios

• VM Image Management
• VM endorsement
• Brokering
• Security
• Intra Cloud Networking
• VM Management
• Data Management
• Information Discovery
• Accounting
• Monitoring
• Federated AAI

10/11/15

20

EGI-ENGAGE

VM Management

• PUSP support
• OCCI extensions and move to OCCI 1.2

– Compute -- resize – Compute – save

• Deployment of OOI for OpenStack
• Native API support
• Improvements/integration for client tools

10/11/15

21

EGI-ENGAGE

Information Discovery

• VO specific info

– Who can prepare the schema?

• Available resources

– Should this be published?

• Alternatives to BDII?
• Leadership of this scenario?

10/11/15

22

EGI-ENGAGE

Accounting

• Storage, IP, any other resource accounting?
• How to make users aware of their usage?

10/11/15

23

EGI-ENGAGE

Monitoring

• Rework OCCI probe to use an AppDB

VMI

– Test all of the EGI VMIs?

• VMCatcher probe
• Block Storage probe
• Swift probes

10/11/15

24

EGI-ENGAGE

AAI

• Need integration with broader EGI AAI

plans

– Technology? – Roadmap?

• Who can act as bridge?

10/11/15

25

EGI-ENGAGE

VMI management

• Single VMI per VA version
• Reliable image distribution

– Disk consumption – Documentation – Allow external implementations (HEPiX list format specification)

• Banning VA versions

– independent of endorsement?

10/11/15

26

EGI-ENGAGE

VMI endorsement

• Endorsing images not that easy to implement

– Signing? Who? When? How? un-endorsement?

• Need to find balance between security and

usability

– Automatic checks?

• This does not solve that the may go

(insecurely) crazy during execution!

10/11/15

27

EGI-ENGAGE

Networking

• Is OCCI network enough?

– How to deal with Neutron? – Firewalls?

• How to deal with multiple sites?

– VPNs – DNS as a Service

10/11/15

28

EGI-ENGAGE

Brokering, user interfaces

• How can users profit from the federation?
• GUIs?

– AppDB as VM Management tool

• No integration with existing cloud ecosystem
• Leadership?

10/11/15

29

EGI-ENGAGE

Security

• Security Threat Risk Assessment with Cloud

focus

– Need participation from a few others in the EGI Federated Cloud

• Security Requirements related to the EGI Fed

cloud.

– VA and VM drafted, needs iteration, probably also software requirements extracting

• Probably also we should write down the

EGI Federated Cloud Security Model

• Security Monitoring?

10/11/15

30

EGI-ENGAGE

Data Management

10/11/15

"The Elephant in the Room Banksy-Barely legal-2006" by Bit Boy - Flickr: The Elephant in the

• Room. Licensed under CC BY 2.0 via Commons - hWps://commons.wikimedia.org/

31

EGI-ENGAGE

AOB

10/11/15