Ellip: a collaborative workplace for EO Open Science Hervé Caumont Platform Operations, Terradue Lisbon, October 9 th , 2018
Copernicus Sentinels as the nextgen Earth Observation satellites Typical EO data volumes 500MB (S-2) < a single product < 8GB (S-1)
~5000 ~400
Why Terradue ? ▪ Terradue is an ESA spin-off (2006) ▪ From day one providing support to application builders in earth sciences, to use satellite EO data as information source ▪ Early days Grid Computing. Nowadays, Cloud PaaS, complemented with APIs for Cloud bursting, aimed at the transfer in production of released applications ▪ Business model: Platform-centered, a collaborative workplace for value adders to interact & co-create Now opening new Ellip solutions to beta testers, who will help us improve our Platform services
Why value co-creation ? ▪ There is still need for significant capacity building within the earth sciences community in exploiting EO data ▪ We believe the value co-creation model is the way forward to unleash innovation ▪ Since 2014 we’re evolving Terradue Cloud Platform, user-driven, via ESA & EU funding ▪ A steady and careful process, as well as a learning process Now opening new Ellip solutions to beta testers, who will help us improve our Platform services
Funding & User communities
Hosted Communities ▪ Climate Change ESA Exploitation Platforms ▪ Carbon market 70+ (2014 onwards) H2020 ECOPOTENTIAL ▪ Oceanography ↗ 5+ (2015-2019) ▪ Climatology EC FP7 MELODIES 14 (2014-2016) H2020 EVER-EST ▪ Biodiversity ↗ 4+ (2015-2018) ▪ Volcanology EC FP7 SenSyF 8 H2020 Co-ReSyF ▪ Ship logistics (2013-2015) 15+ (2016-2018) ▪ Water quality H2020 Copernicus App Lab ↗ EC FP7 GEOWOW ▪ Desertification 3+ 5 (2016-2018) (2011-2013) ▪ Seismic analysis H2020 NextGEOSS 10+ ▪ Urban accounting (2016-2020) Processing Campaigns 35+ ▪ Land management H2020 INTAROS (2013-onwards) ↗ 4+ (2016-2021) ▪ Disaster Risk Reduction
Earth Observation Platform ▪ EO data is a unique source of global measurements over decades Consumers ▪ When calibrated and combined with other sources, it empowers validation & interpolation models ▪ Collaboration process : a feedback loop of providers & consumers ▪ More providers of pre-processed data layers ▪ New demands for structured information The growth model is pulled by the communities Producers
Partnerships We are always seeking for new Partnerships to expand our Platform services to users : ▪ Data providers ▪ e-Infra / IaaS providers ▪ Technology providers ▪ Scientific Toolbox/Algorithm/Model providers A particular focus from community requirements is to boost the joint use of EO & in-situ data
EGI.eu & Terradue Partnership Oct. 2018 Project(s) sustainability The partnership with EGI Foundation enhances the sustainability of the involved projects, via SLAs / OLAs helping to sustain operational lifetimes. Reliable operations We can rely on well-defined operational processes (e.g. FitSM) , having the EGI Foundation acting as single point of contact of the EGI Federated Cloud. European cloud resources We use Cloud resources within the European boundaries. In fact, it is mandatory for certain processing services due to their organisations’ policies. https://www.egi.eu/about/newsletters/egi-terradue-a-long-lasting-partnership/ Use of standard interfaces Nov. 2015 M2M communications realised initially via OCCI and then via native APIs for cloud computing , ensuring integration, portability, and interoperability.
2015 Feasibility Setup 2 0 1 6 Dev A c c e p 2016/09 - 1st MoU t a n c e Pilots M o n i t Integration roadmap o r 2 0 1 7 A c c e p t a n c e (cntd) P i l o t s (cntd) M o n i t o r (cntd) 2018/09 - 2nd MoU 2018
Other partnership examples Science providers: research and commercial ● Technology providers: (GEP, HEP, NextGEOSS) OpenNebula (since before ESA ● TEPs early stage, circa 2012) Data providers: OGC standards (Terradue as ● standard editor) EPOS RI on Geohazards TEP ● for data uploads and processing Cloud IaaS providers: Open Data policies in the EO ● EGI.eu (NextGEOSS) ● data world EOSC.eu (EOSC-hub) ● (cf. RDA interview) ● DIAS (AppLab, GEP) ● https://www.rd-alliance.org/interview-terradue
Opportunities to partner
Why do Platform Operations exist ? ● Since Terradue’s inception 12 years ago, our company DNA pushes us to innovate ● We believe technology serves new ways to design products, that people can use to better work collaboratively Platform Operations goals ● Manage well defined usage scenarii as ‘products’ ○ Pilot a unified foundation for software solutions ○ Innovate smoother, end-to-end business processes matching ○ customer needs Guarantee the ability to share data among applications ○ ... ○
How to run Platform Operations ? ● Platform Operations require a thoughtful mix of stability & innovation ● We’re embarking on a new journey to learn how Stability to tune this mix , based on: ○ The market needs Timely ○ Our own skills and capacities to deliver software of high quality Innovation ○ The cost / benefit of the solutions that we decide to market ○ The market recognition of what we do
How to work with Ellip ?
Federated IdPs & Terradue SSO Possibility to keep your preferred Identity Provider OpenID Connect preferred Under work: User Managed Access (UMA) profile of OAuth 2.0
Subscription Plans PREMIUM • Business model for PARTNER sustainability of hosted processing operations Higher cost EXPLORER SCALER Innovation • Public funding, & platform evolutions commercial revenues, For and costs-sharing Integrator users Low cost Low cost Cloud providers, • App. development Cluster processing 4 Partner Programs Processors developers, • ◆ Technology Alliance Applications • For For ◆ Virtual Archive Integrators Data science users Producer users ◆ Data Science Scientific users ◆ Cloud Appliance •
Ellip Dashboard
Ellip Solutions ● Ellip Notebooks ○ “Create interactive laboratory notebooks” ● Ellip Workflows ○ “Design scalable processing chains” Vestibulum Vestibulum Ellip Workflows Ellip Infohubs congue congue ● Ellip Launchpads ○ “Select a data processor and deploy it” Vestibulum congue Vestibulum Notebooks Launchpads Ellip Ellip congue ● Ellip Infohubs ○ “Connect a WPS to a custom Geobrowser”
Ellip Solutions in a nutshell I want to build and maintain a processing service Interactively prototype Monitor and collaborate Run a scale Integrate for scalability
. Application code . Packaged application . Tutorial The Ellip E s e V l l b u e i Exchange as g p s u n t i h o b W c u o l m o u f the core enabler m n r u k I l u c f p b o l o i n i t of Ellip as a l s g w l e E u V e s ‘ collaborative Ellip . Running WPS workplace ’ Exchange at a selected Cloud Provider E . New V l V l i e e p s s s t t d information p i i N b b a u i u l o l l p l u u layers E t m m h . OWS Context template e c b c c n o o o . ad-hoc Community Portal n n u o g g a u u k e e L s . Reproducible experiment as code . Interactive data visualisations
So, what does a platform mean nowadays? The Ellip Solutions are built for earth sciences practitioners: ▪ to share about their findings , streamline their creation of new data products and make these accessible, interoperable, ▪ and document their developments on the PaaS as reproducible experiments In line with the FAIR guiding principles to make data findable, accessible, interoperable and reusable
Continuous Integration & Deployment Dockerize Integrate & Test Build Package Code Application Upload Release Scan Repository Code Commit Integration Production Environment Environment s
Hybrid Cloud Amazon Web Services EC2 - powered by jclouds AWS Poznań Supercomputing PSNC and Networking Center Openstack API - powered by jclouds T2 EGI Federated Cloud EGI OCCI, native cloud APIs Terradue IPT.PL IPT Opennebula registered partner Openstack API - powered by libcloud
Scheduled Data Processing Growing season 2016-2017 Terradue Cloud Platform Biopar Application Burst to EGI.eu 24 processing nodes provided by EGI.eu (NextGEOSS partner) Biopar products 29 www.terradue.com
Scheduled Data Processing Supported by BELNET-BEGRID (Belgium) DLR InSAR Browse Medium Resolution Service It’s a data driven systematic processing. The service will follow a ramp-up period starting from September 2016 until end 2017: EU Tectonic area ■ World tectonic area (40%) ■ World tectonic area (70%) ■ It currently processes 150+ Sentinel-1 SLC pairs per day. 30 www.terradue.com
User-Driven Data Processing Supported by GOEGRID-GWGD (Germany) ESA SNAP Sentinel-1 COherence and INtensity (COIN) This service provides the coherence and intensity changes for Sentinel-1 TOPSAR IW data pairs performed through SNAP. SNAP is a common architecture for all Sentinel Toolboxes, which is ideal for Earth Observation processing and analysis. 31 www.terradue.com
Recommend
More recommend
