GFZ German Research Centre for Geosciences, Centre for Geoinformation - - PDF document

1

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

TEODOOR, a blueprint for distributed terrestrial

• bservation data

infrastructures

Ralf Kunkel1, Jürgen Sorg1, Martin Abbrent2, Erik Borg3, Rainer Gasche4, Olaf Kolditz2, Frank Neidl4, Eckard Priesack5, Vivien Stender6

1: Research Centre Juelich, IBG-3, Juelich, Germany (r.kunkel@fz-juelich.de) 2: Helmholtz Centre for Environmental Research, Leipzig, Germany 3: German Aerospace Center, Earth Observation Center, Neustrelitz, Germany 4: Karlsruhe Institute of Technology, Garmisch-Partenkirchen, Germany 5: German Center for Environmental Health, BIOP, Oberschleißheim, Germany 6: GFZ German Research Centre for Geosciences, Centre for Geoinformation Technology, Potsdam

EGU General Assembly 2017, Vienna, Austria, 23 - 28 April 2017

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

The TERENO network

Ø Regional different effects of global climate change on terrestrial systems Ø Global change affects all terrestrial compartments (water, soil, vegetation, atmosphere) Ø Most existing observation networks focus on specific compartments and/or scientific questions

Ø TERENO:

§ Long-term observations (> 15 years) of hydrological and ecological parameters

• n different scales

§ Investigation of interaction between the different compartments § Bridging the gap between measurement, modelling and management § Currently 4 observatories, each

• perated by one individual Helmholtz

Center § Project duration: 2008 until >2023

2

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Multi-scale and multi-compartment monitoring concept of TERENO

SEITE 3

0.00E+00 5.00E+07 1.00E+08 1.50E+08 2.00E+08 2.50E+08 3.00E+08 3.50E+08 Lysimeter data Meteorological data Soil data Surface water data Other data DLR FZJ GFZ HMGU KIT UFZ

Ø 1065 stations Ø 3 weather radar devices Ø 129 lysimeters Ø 400 file metadata sets

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

TEODOOR Distributed Spatial Data Infrastructure

3

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Interconnecting observatories

Ø Common data policy

§ Quality management § Time-limits for data delivery § Retention times for data publication § Accessibility of data

Ø Syntactical interoperability by consequent usage of standardized (OGC) web services and interfaces Ø Semantical interoperability:

§ Common metadata profile § Common sensorML profiles § Common thesauri § Standardisation (parameters, units, ...)

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

100,000,000 200,000,000 300,000,000 400,000,000 500,000,000 600,000,000 700,000,000 800,000,000 900,000,000 Eifel-Rur SoilCan SN Wüstebach SN Rollesbroich Sampledata

data values

checked data unchecked data raw data

95 % 15 % 42 % 47 % 100 %

TERENO Quality management policy

Ø Establishing workflows for data collection, quality assessment and publication Ø Nomination of responsible people Ø Prohibition to circulate unevaluated data § Technical inspection (mandatory): Identification and tagging of obviously wrong data values § Validity checks (optional): Checking the continuity of time series and the definite conclusion that the observed data are representing the measured property Ø Common system to assign § Quality flags (good, suspicious, bad data) § Processing status (unevaluated, quality checked,…) Ø Automatic publication of quality assessed data

4

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Data services connected to the DDP

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

TEODOOR: The TERENO Data Portal

http.//www.tereno.net

Ø Central portal for information exchange, data search and data access Ø Querying the TERENO Metadata Catalogues Ø Connected local data infrastructures from FZJ, DLR, GFZ, HMGU, KIT, UFZ Ø Custom multi-condition queries Ø Predefinded queries to data Ø Free access to data from more than 900 sites

5

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Client applications using standardized web services

Animation of weather radar data using raster SOS Visualization and download of time series data Animation of automatically interpolated soil moisture data using raster SOS

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Interconnecting infrastructures using OGC interfaces

Ø No central database Ø Not portal dependent Ø Interoperability through OGC services

6

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Naming issues

Ø Currently, naming of sites is ambiguous.

§ Different sites have identical names. § Sites are renamed. § Metadata that allow unique identification are often missing. § Institutions have their own naming protocols, no assurance that names are unique on a global scale.

Ø Access to information about observation sites

§ Need to ensure proper evaluation and facilitate interpretation

• f data.

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

DEOS - A Centralized Approach

provides a resolvable, persistent, interoperable link Ø resolvable – standard identifier syntax + network resolution mechanism (Handle System) Ø persistent – through:

§ technical infrastructure (registry database, proxy support, etc.) § social infrastructure (obligations by Registration Agencies)

Ø interoperable - through a data model (semantic interoperability)

7

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

DEOS - A Centralized Approach

Ø Registration service currently hosted at FZJ: https://deos-id.org/deos/ Ø Structure: TERENO.ER012345 Ø Generated by DEOS or by users Ø Does not replace personal

• r institutional names

Ø Building an inventory of

• bservation facilities

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

8

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

TEODOOR download statistics

• 100,000,000

200,000,000 300,000,000 400,000,000 500,000,000 600,000,000 700,000,000 800,000,000 FZJ->FZJ GFZ->GFZ UFZ->UFZ KIT-KIT HMGU->HMGU FZJ->other GFZ->other UFZ->other KIT->other HMGU->other 733,108,533 1,199,549 6,240

• 352,476,008

4,209,845 140,448 471,644 96

Number of data values provided by TEODOOR 08/13 - 07/16

• 50,000,000

100,000,000 150,000,000 GFZ Universität Potsdam CUAHSI TU Wien Universität Würzburg ETH Zürich Universität für Bodenkultur Wien HFT StuKgart Bundeswehr Universität Wien LUP Umwelt University of Bristol Universität Augsburg Universität Hohenheim FZJ Universität Hamburg TU DelQ University of Wisconsin-Madison ZALF UFZ University of Salerno, Italy University Gent RWTH Aachen University of Michigan University Cologne Universität Marburg Universität Trier unknown Universität Bonn

94 96 3,166 4,322 19,921 23,648 80,463 83,900 231,109 345,953 574,464 770,434 807,460 1,360,894 1,399,876 1,632,624 1,838,359 1,922,383 2,455,584 2,904,488 3,037,599 3,080,396 3,285,312 5,617,575 29,815,719 38,372,614 40,749,743 42,994,213 173,666,408

Number of data values provided to external ins4tu4ons by TEODOOR 08/13 - 07/16

Ø Number of downloads: 3548 Ø Number of data series: 140,696 Ø Number of data values (est): 1,091,393,139 Ø Mean monthly dowloads: 25,000,000

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Persistent data identifiers

Ø Unique, digital identifier, allowing persistent citation of publications and data

§ Eases access to research data § Increases visibility of data

Ø Identifier refers to “landing page” containing:

§ Metadata (for station or data set) § Individual data sets § Licensing information (e.g. data policy)

Ø Landing page (and data, in general) hosted by issuing institution (here: GFZ) Ø Internal agreement to be able to link to data from the TERENO portal Ø Currently, 24 data sets/stations were identified through persistent identifiers (see https://search.datacite.org/works? query=TERENO) Ø Drawbacks

§ File based approach § “Snapshot creation” of data from data bases required § Dynamical referencing to data in planning § Manual process § Hosting the same metadata on two systems in parallel (GFZ, TEODOOR)

http://doi.org/10.5880/TERENO.256 http://doi.org/10.5880/TERENO. 2016.001

9

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Examples

Ø Graswang § TERENO: „Graswang“ (http://tereno.imk-ifu.kit.edu/Graswang/) § ICOS: „DE-GWG“ (https://meta.icos-cp.eu/edit/stationentry/) § Fluxnet: „DW-GWG“ (https://fluxnet.ornl.gov/site/4147) Ø Bad Lauchstädt § TERENO: „Lysimeterstation Bad Lauchstädt“ (multiple entries, e.g. http://teodoor.icg.kfa-juelich.de/ibg3searchportal2/dispatch?searchparams=freetext- lauch&metadata.detail.view.id=urn:ogc:object:feature:Sensor:UFZ:970) § LTER: „TERENO - Bad Lauchstaedt“ https://data.lter-europe.net/deims/site/lter_eu_de_019 Ø FZJ Climate Tower § TERENO: RU_K_001 ( http://teodoor.icg.kfa-juelich.de/ibg3searchportal2/dispatch?searchparams=freetext- RU_K&metadata.detail.view.id=RU_K_001) § ICOS: JUE (https://meta.icos-cp.eu/ontologies/stationentry/AS/N2)

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

Local Infrastructure „Eifel-Rur“

Ø Static data (usually file based)

§ Descriptive data (reports) § Geodata § Other static data (statistics, …)

Ø Time series data

§ Runoff, water quality, soil, climate

§ 589 stations 10‘-60‘, offline)

§ Eddy-Covariance

§ 7 stations (20 Hz-10‘)

§ Weather radar

§ 2 radar devices (5-10‘)

§ Lysimeters (SoilCan)

§ 36 lysimeters (1’-15‘)

§ Regular sampling campaign data

50,000,000 100,000,000 150,000,000 200,000,000 250,000,000 300,000,000 350,000,000 2008 2009 2010 2011 2012 2013 2014 2015 2016

data values per year

checked data unchecked data raw data

10

Institute for Bio- and Geosciences - Agrosphere (IBG-3)

• 1. Data Importing & Processing
• 2. Storage
• 5. Administration
• 3. Standardized

Access 4. Publication

Developed based on open source software and open standards.

Time series Management System