O2A Observations to Archive Data Flow Framework Arndt Steinhage - - PowerPoint PPT Presentation

▶
o2a observations to archive
SMART_READER_LITE
LIVE PREVIEW

O2A Observations to Archive Data Flow Framework Arndt Steinhage - - PowerPoint PPT Presentation

Oct 02, 2022 460 likes •687 views

O2A Observations to Archive Data Flow Framework Arndt Steinhage Alfred Wegener Institute - Computing Center presented by Angela Schfer D4IR 2017 30.11.2017 Use Case : Arctic long-term observatory FRAM Ice tethered platform networks

slide-1
SLIDE 1

O2A – Observations to Archive

Data Flow Framework

Arndt Steinhage

Alfred Wegener Institute - Computing Center presented by Angela Schäfer D4IR 2017 – 30.11.2017

slide-2
SLIDE 2

Use Case: Arctic long-term observatory FRAM

Ice tethered platform networks … radiation, snow height, ice thickness, temperature, salinity, oxygen, chlorophyll a …

Medieningenieure Bremen / Sabine Lüdeling

slide-3
SLIDE 3

Use Case: FRAM

Medieningenieure Bremen / Sabine Lüdeling

Water column … fluorescence, nutrients, salinity, temperature, conductivity, acoustic Doppler current profiler, water and phytoplankton samples, …

Medieningenieure Bremen / Sabine Lüdeling

slide-4
SLIDE 4

Use Case: FRAM

Medieningenieure Bremen / Sabine Lüdeling

Ocean floor … photo, video, benthic flux, physico-chemical, ...

Medieningenieure Bremen / Sabine Lüdeling

slide-5
SLIDE 5

Data Flow Framework

slide-6
SLIDE 6

Data Flow Framework

slide-7
SLIDE 7

Objectives

  • Generic infrastructure for data flows
  • Sustainability and up-to-date services
  • Interoperability and standards
  • e.g. Open Geospatial Consortium
  • Seamless integration with our infrastructure
  • Web GIS
  • Workspace/Sandbox
  • Web Portals
  • Data Archive / Publishing
slide-8
SLIDE 8

Architecture

Metadata + Inventory AWI Data Pool multiple Storage

ETLService

DSHIP Data Archive

  • Data

Acquisition

ETLService ETLService ETLService

Offline

Data Archives

  • Data

Acquisition NRT Data Archive

  • Data

Acquisition DMS Data Archive

  • External Systems

Data Acquisition

Dashboard.awi.de Data.awi.de-Portal Data Services Sensor.awi.de

Sensor ML Device Device Device Device Device Device Device Device e.g. ships e.g. aircraft decentral not automated Rest Service API Rest Service API

Research Applications

slide-9
SLIDE 9

Challenges

  • Heterogeneity of scientific needs and workflows
  • Vast Number of different instruments, data

sources and formats

  • Multitude of Standards
  • Integration with existing solutions, e.g. for the

data flow, but also administrative information

  • Limited additional Effort acceptable by science
  • Limited Bandwidth ship to shore
slide-10
SLIDE 10

Use Case: MOSAiC

Multidisciplinary drifting Observatory for the Study of Arctic Climate, the first year-round expedition into the central Arctic to explore the Arctic climate system during 2019 to 2020

Based on year round operation of RV Polarstern, drifting with the sea ice across Arctic

slide-11
SLIDE 11

O2A: MOSAiC

MOSAiC Raw Data Polarstern Data Storage

Onboard Data Transfer Ship-to-shore Data Transfer Polarstern Satellite Link for Data Monitoring and Remote Service

  • nly 2 x 100MB/day ?

“direct” satellite links to partner sites

slide-12
SLIDE 12

O2A: MOSAiC

Polarstern Data

MOSAiC Comprehensive

  • nshore

Data Collection PANGAEA

slide-13
SLIDE 13

O2A components

  • SENSOR.awi.de

ready

  • DASHBOARD.awi.de

ready

  • MAPS.awi.de

ready

  • PANGAEA.de

always

  • DATA.awi.de

prototype internal only

slide-14
SLIDE 14

SENSOR.awi.de - Description

  • Platform and device

descriptions for provenance information and reduced data integration effort

  • Versioning and

citability

  • Interoperability and

standards

  • ~1200 descriptions

available and counting

slide-15
SLIDE 15

DASHBOARD.awi.de

  • User-customizable,

flexible dashboards for data monitoring

  • Automatic data

streaming of near-real time and delayed-mode data

  • Based on sensor

descriptions and configurations

slide-16
SLIDE 16

NRT Data Presentation – Decision Support

slide-17
SLIDE 17

DATA.awi.de (release summer ‘18)

slide-18
SLIDE 18

Portal – data combined

slide-19
SLIDE 19

Data Flow Framework

slide-20
SLIDE 20

Current work

  • Developing a science community workspace for

data sharing and data analyses within the Helmholtz Data Federation (HDF)

  • State-of-the-art storage, AWI-part distributed

between Bremerhaven and Potsdam

  • User-friendly, high bandwidth compute/analysis

solutions with virtual machines and containers

  • Hadoop big data analysis based on Hortonworks

data flow and data platform

  • Raster data management and analysis with

rasdaman (hypercube data analytics)

slide-21
SLIDE 21

Thank you very much for your attention!

Team: Roland Koppe, Peter Gerchow, Ana Macario, Antonie Haas, Christian Schäfer-Neth, Hans Pfeiffenberger, presented by Angela.Schaefer@awi.de Alfred Wegener Institute, Helmholtz Centre for Polar and Marine Research Bremerhaven, Germany

We are looking for IT-staff!