Ellip: a collaborative workplace for EO Open Science Herv Caumont - - PowerPoint PPT Presentation

Ellip: a collaborative workplace for EO Open Science

Lisbon, October 9th, 2018

Hervé Caumont

Platform Operations, Terradue

Copernicus Sentinels as the nextgen Earth Observation satellites Typical EO data volumes 500MB (S-2) < a single product < 8GB (S-1)

~5000 ~400

Why Terradue ?

▪ Terradue is an ESA spin-off (2006) ▪ From day one providing support to application builders in earth sciences, to use satellite EO data as information source ▪ Early days Grid Computing. Nowadays, Cloud PaaS, complemented with APIs for Cloud bursting, aimed at the transfer in production of released applications ▪ Business model: Platform-centered, a collaborative workplace for value adders to interact & co-create

Now opening new Ellip solutions to beta testers, who will help us improve our Platform services

Why value co-creation ?

▪ There is still need for significant capacity building within the earth sciences community in exploiting EO data ▪ We believe the value co-creation model is the way forward to unleash innovation ▪ Since 2014 we’re evolving Terradue Cloud Platform, user-driven, via ESA & EU funding ▪ A steady and careful process, as well as a learning process

Now opening new Ellip solutions to beta testers, who will help us improve our Platform services

Funding & User communities

Hosted Communities

↗

H2020 Copernicus App Lab (2016-2018)

↗

H2020 INTAROS (2016-2021) ESA Exploitation Platforms (2014 onwards)

70+

EC FP7 MELODIES (2014-2016)

14

EC FP7 SenSyF (2013-2015)

8

EC FP7 GEOWOW (2011-2013)

5

Processing Campaigns (2013-onwards)

35+ ▪ Climate Change ▪ Carbon market ▪ Oceanography ▪ Climatology ▪ Biodiversity ▪ Volcanology ▪ Ship logistics ▪ Water quality ▪ Desertification ▪ Seismic analysis ▪ Urban accounting ▪ Land management ▪ Disaster Risk Reduction

H2020 Co-ReSyF (2016-2018)

15+ 10+

H2020 NextGEOSS (2016-2020)

↗

H2020 ECOPOTENTIAL (2015-2019)

5+ ↗

H2020 EVER-EST (2015-2018)

4+ 4+ 3+

▪ EO data is a unique source of global measurements over decades

▪ When calibrated and combined with other sources, it empowers validation & interpolation models

▪ Collaboration process: a feedback loop of providers & consumers

▪ More providers of pre-processed data layers ▪ New demands for structured information

The growth model is pulled by the communities

Earth Observation Platform

Producers Consumers

Partnerships

We are always seeking for new Partnerships to expand our Platform services to users : ▪ Data providers ▪ e-Infra / IaaS providers ▪ Technology providers ▪ Scientific Toolbox/Algorithm/Model providers A particular focus from community requirements is to boost the joint use of EO & in-situ data

EGI.eu & Terradue Partnership

M2M communications realised initially via OCCI and then via native APIs for

cloud computing, ensuring integration, portability, and interoperability.

Use of standard interfaces

We can rely on well-defined operational processes (e.g. FitSM), having the EGI Foundation acting as single point of contact of the EGI Federated Cloud.

Reliable operations

We use Cloud resources within the European boundaries. In fact, it is mandatory for certain processing services due to their organisations’ policies.

European cloud resources

The partnership with EGI Foundation enhances the sustainability of the involved projects, via SLAs / OLAs helping to sustain operational lifetimes.

Project(s) sustainability

• Oct. 2018

https://www.egi.eu/about/newsletters/egi-terradue-a-long-lasting-partnership/

• Nov. 2015

Integration roadmap

2016/09 - 1st MoU

2015 2 1 6 2 1 7 2018

Feasibility Setup Dev A c c e p t a n c e P i l

• t

s (cntd) M

• n

i t

• r

(cntd)

M

• n

i t

• r

Pilots A c c e p t a n c e

(cntd)

2018/09 - 2nd MoU

Other partnership examples

Science providers:

• research and commercial

(GEP, HEP, NextGEOSS) Data providers:

• EPOS RI on Geohazards TEP

for data uploads and processing

• Open Data policies in the EO

data world

• (cf. RDA interview)

Technology providers:

• OpenNebula (since before ESA

TEPs early stage, circa 2012)

• OGC standards (Terradue as

standard editor) Cloud IaaS providers:

• EGI.eu (NextGEOSS)
• EOSC.eu (EOSC-hub)
• DIAS (AppLab, GEP)

https://www.rd-alliance.org/interview-terradue

Opportunities to partner

• Since Terradue’s inception 12 years ago, our

company DNA pushes us to innovate

• We believe technology serves new ways to

design products, that people can use to better work collaboratively

• Platform Operations goals

○ Manage well defined usage scenarii as ‘products’ ○ Pilot a unified foundation for software solutions ○ Innovate smoother, end-to-end business processes matching customer needs ○ Guarantee the ability to share data among applications ○ ...

Why do Platform Operations exist ?

• Platform Operations require a thoughtful mix of

stability & innovation

• We’re embarking on a new journey to learn how

to tune this mix, based on: ○ The market needs ○ Our own skills and capacities to deliver software of high quality ○ The cost / benefit of the solutions that we decide to market ○ The market recognition of what we do

How to run Platform Operations ?

Stability Timely Innovation

How to work with Ellip ?

Federated IdPs & Terradue SSO

Possibility to keep your preferred Identity Provider OpenID Connect preferred Under work: User Managed Access (UMA) profile of OAuth 2.0

• Business model for

sustainability of hosted processing operations

• Public funding,

commercial revenues, and costs-sharing

• Cloud providers,
• Processors developers,
• Applications

Integrators

• Scientific users

Subscription Plans

EXPLORER

Low cost

• App. development

For Data science users

SCALER

Low cost Cluster processing For Producer users

PREMIUM PARTNER

Higher cost Innovation & platform evolutions For Integrator users 4 Partner Programs

◆ Technology Alliance ◆ Virtual Archive ◆ Data Science ◆ Cloud Appliance

Ellip Dashboard

• Ellip Notebooks

○ “Create interactive laboratory notebooks”

• Ellip Workflows

○ “Design scalable processing chains”

• Ellip Launchpads

○ “Select a data processor and deploy it”

• Ellip Infohubs

○ “Connect a WPS to a custom Geobrowser”

Ellip Solutions

Vestibulum congue Vestibulum congue Vestibulum congue Vestibulum congue Ellip Infohubs Ellip Notebooks

Ellip Launchpads

Ellip Workflows

Ellip Solutions in a nutshell

I want to build and maintain a processing service Interactively prototype Integrate for scalability Run a scale Monitor and collaborate

V e s t i b u l u m c

• n

g u e V e s t i b u l u m c

• n

g u e V e s t i b u l u m c

• n

g u e V e s t i b u l u m c

• n

g u e

Ellip Exchange

E l l i p I n f

• h

u b s E l l i p N

• t

e b

• k

s E l l i p L a u n c h p a d s E l l i p W

• r

k f l

• w

s

The Ellip Exchange as the core enabler

• f Ellip as a

‘collaborative workplace’

. Application code . Packaged application . Tutorial . Running WPS at a selected Cloud Provider . New information layers . Reproducible experiment as code . Interactive data visualisations . OWS Context template . ad-hoc Community Portal

So, what does a platform mean nowadays?

The Ellip Solutions are built for earth sciences practitioners: ▪ to share about their findings, streamline their creation of new data products and make these accessible, interoperable, ▪ and document their developments on the PaaS as reproducible experiments In line with the FAIR guiding principles to make data findable, accessible, interoperable and reusable

Build Integrate & Test Package Dockerize

Code Commit Scan Repository Production Environments

Continuous Integration & Deployment

Code Upload Application Release Integration Environment

Hybrid Cloud

Openstack API - powered by jclouds

Poznań Supercomputing and Networking Center

EC2 - powered by jclouds

Amazon Web Services

OCCI, native cloud APIs

EGI Federated Cloud

Opennebula registered partner

Terradue

Openstack API - powered by libcloud

IPT.PL

AWS

EGI

IPT

PSNC

T2

www.terradue.com

Scheduled Data Processing

29

24 processing nodes provided by EGI.eu (NextGEOSS partner) Growing season 2016-2017

Biopar products

Terradue Cloud Platform

Biopar Application Burst to EGI.eu

www.terradue.com

Scheduled Data Processing

30

It’s a data driven systematic processing. The service will follow a ramp-up period starting from September 2016 until end 2017:

■

EU Tectonic area ■ World tectonic area (40%) ■ World tectonic area (70%) It currently processes 150+ Sentinel-1 SLC pairs per day.

DLR InSAR Browse Medium Resolution Service

Supported by BELNET-BEGRID (Belgium)

www.terradue.com

User-Driven Data Processing

31

This service provides the coherence and intensity changes for Sentinel-1 TOPSAR IW data pairs performed through SNAP. SNAP is a common architecture for all Sentinel Toolboxes, which is ideal for Earth Observation processing and analysis.

ESA SNAP Sentinel-1 COherence and INtensity (COIN)

Supported by GOEGRID-GWGD (Germany)

H2020 project BETTER

H2020 project BETTER

All of it: an Open Cloud strategy

■ Open APIs

Embrace Cloud bursting APIs that can be easily plugged into the Platform’s codebase, so to expand the Platform offering with Providers offering complementary strategic advantages for different user communities.

■ Developers community

Support and nurture Cloud communities that collaborate on evolving open source technologies, including at the level of the Platform engineering team, when it comes to deliver modular extensions.

■ Self-service provisioning and management of resources

The Platform’s end-users are able to self-provision their required ICT resources and to work autonomously.

■ Users rights to move data as needed

By supporting distributed instances of its EO Data management layer, the Platform delivers the required level of data locality to ensure high performance processing with

• ptimized costs.

All of it: an Open Cloud strategy

▪ The Ellip subscribers remain the owners of their created assets, and decide how to share these on the platform, for other users to find and reuse these assets. ▪ They are also empowered to decide on the target production environment of their packaged applications (no lock in with Cloud provider)

Producers Consumers

That’s it!

▪ Last message: we’re always looking for new collaborators ▪ Join our team in Rome La Città Eterna

15 members, 5 EU nationalities

▪ We are welcoming experts in EO applications and cloud technologies ▪ Exciting new challenges rolling out Ellip for Earth Science

Looking forward hearing from you!

https://www.terradue.com

Hervé Caumont, Platform Operations herve.caumont@terradue.com