GFZ German Research Centre for Geosciences, Centre for Geoinformation - PDF document

TEODOOR, a blueprint for distributed terrestrial observation data infrastructures Ralf Kunkel 1 , Jürgen Sorg 1 , Martin Abbrent 2 , Erik Borg 3 , Rainer Gasche 4 , Olaf Kolditz 2 , Frank Neidl 4 , Eckard Priesack 5 , Vivien Stender 6 1: Research Centre Juelich, IBG-3, Juelich, Germany (r.kunkel@fz-juelich.de) GFZ German Research Centre for Geosciences, Centre for Geoinformation Technology, Potsdam 2: Helmholtz Centre for Environmental Research, Leipzig, Germany 3: German Aerospace Center, Earth Observation Center, Neustrelitz, Germany 4: Karlsruhe Institute of Technology, Garmisch-Partenkirchen, Germany 5: German Center for Environmental Health, BIOP, Oberschleißheim, Germany 6: EGU General Assembly 2017, Vienna, Austria, 23 - 28 April 2017 Institute for Bio- and Geosciences - Agrosphere (IBG-3) The TERENO network Ø Regional different effects of global climate change on terrestrial systems Ø Global change affects all terrestrial compartments (water, soil, vegetation, atmosphere) Ø Most existing observation networks focus on specific compartments and/or scientific questions Ø TERENO: § Long-term observations (> 15 years) of hydrological and ecological parameters on different scales § Investigation of interaction between the different compartments § Bridging the gap between measurement, modelling and management § Currently 4 observatories, each operated by one individual Helmholtz Center § Project duration: 2008 until >2023 Institute for Bio- and Geosciences - Agrosphere (IBG-3) 1

Multi-scale and multi-compartment monitoring concept of TERENO DLR FZJ GFZ HMGU KIT UFZ Ø 1065 stations Other data Ø 3 weather radar devices Ø 129 lysimeters Surface water data Ø 400 file metadata sets Soil data Meteorological data Lysimeter data 0.00E+00 5.00E+07 1.00E+08 1.50E+08 2.00E+08 2.50E+08 3.00E+08 3.50E+08 Institute for Bio- and Geosciences - Agrosphere (IBG-3) SEITE 3 TEODOOR Distributed Spatial Data Infrastructure Institute for Bio- and Geosciences - Agrosphere (IBG-3) 2

Interconnecting observatories Ø Common data policy § Quality management § Time-limits for data delivery § Retention times for data publication § Accessibility of data Ø Syntactical interoperability by consequent usage of standardized (OGC) web services and interfaces Ø Semantical interoperability: § Common metadata profile § Common sensorML profiles § Common thesauri § Standardisation (parameters, units, ...) Institute for Bio- and Geosciences - Agrosphere (IBG-3) TERENO Quality management policy Ø Establishing workflows for data collection, quality assessment and publication Ø Nomination of responsible people Ø Prohibition to circulate unevaluated data § Technical inspection (mandatory): Identification and tagging of obviously wrong data values § Validity checks (optional): Checking the continuity of time series and the definite conclusion that the observed data are representing the measured property Ø Common system to assign § Quality flags (good, suspicious, bad data) 42 % 900,000,000 checked data § Processing status (unevaluated, quality unchecked data 800,000,000 checked, … ) raw data 700,000,000 data values 600,000,000 Ø Automatic publication of quality 15 % 47 % 500,000,000 assessed data 400,000,000 95 % 300,000,000 200,000,000 100 % 100,000,000 0 Institute for Bio- and Geosciences - Agrosphere (IBG-3) Eifel-Rur SoilCan SN Wüstebach SN Rollesbroich Sampledata 3

Data services connected to the DDP Institute for Bio- and Geosciences - Agrosphere (IBG-3) TEODOOR: The TERENO Data Portal http.//www.tereno.net Ø Central portal for information exchange, data search and data access Ø Querying the TERENO Metadata Catalogues Ø Connected local data infrastructures from FZJ, DLR, GFZ, HMGU, KIT, UFZ Ø Custom multi-condition queries Ø Predefinded queries to data Ø Free access to data from more than 900 sites Institute for Bio- and Geosciences - Agrosphere (IBG-3) 4

Client applications using standardized web services Visualization and download of time series data Animation of weather radar data using raster SOS Animation of automatically interpolated soil moisture data using raster SOS Institute for Bio- and Geosciences - Agrosphere (IBG-3) Interconnecting infrastructures using OGC interfaces Ø No central database Ø Not portal dependent Ø Interoperability through OGC services Institute for Bio- and Geosciences - Agrosphere (IBG-3) 5

Naming issues Ø Currently, naming of sites is ambiguous. § Different sites have identical names. § Sites are renamed. § Metadata that allow unique identification are often missing. § Institutions have their own naming protocols, no assurance that names are unique on a global scale. Ø Access to information about observation sites § Need to ensure proper evaluation and facilitate interpretation of data. Institute for Bio- and Geosciences - Agrosphere (IBG-3) DEOS - A Centralized Approach provides a resolvable, persistent, interoperable link Ø resolvable – standard identifier syntax + network resolution mechanism (Handle System) Ø persistent – through: § technical infrastructure (registry database, proxy support, etc.) § social infrastructure (obligations by Registration Agencies) Ø interoperable - through a data model (semantic interoperability) Institute for Bio- and Geosciences - Agrosphere (IBG-3) 6

DEOS - A Centralized Approach Ø Registration service currently hosted at FZJ: https://deos-id.org/deos/ Ø Structure: TERENO.ER012345 Ø Generated by DEOS or by users Ø Does not replace personal or institutional names Ø Building an inventory of observation facilities Institute for Bio- and Geosciences - Agrosphere (IBG-3) Institute for Bio- and Geosciences - Agrosphere (IBG-3) 7

TEODOOR download statistics Number of data values provided by TEODOOR 08/13 - 07/16 733,108,533 800,000,000 Number of data values provided to external ins4tu4ons by 700,000,000 TEODOOR 08/13 - 07/16 352,476,008 600,000,000 173,666,408 Universität Bonn 500,000,000 unknown 42,994,213 Universität Trier 40,749,743 400,000,000 Universität Marburg 38,372,614 300,000,000 University Cologne 29,815,719 1,199,549 4,209,845 University of Michigan 5,617,575 200,000,000 140,448 471,644 6,240 RWTH Aachen 3,285,312 96 100,000,000 - - 3,080,396 University Gent University of Salerno, Italy 3,037,599 - UFZ 2,904,488 FZJ->FZJ GFZ->GFZ UFZ->UFZ KIT-KIT HMGU->HMGU FZJ->other GFZ->other UFZ->other KIT->other HMGU->other ZALF 2,455,584 University of Wisconsin-Madison 1,922,383 1,838,359 TU DelQ Universität Hamburg 1,632,624 FZJ 1,399,876 Universität Hohenheim 1,360,894 Ø Number of downloads: 3548 Universität Augsburg 807,460 University of Bristol 770,434 LUP Umwelt 574,464 Universität Wien 345,953 Ø Number of data series: 140,696 Bundeswehr 231,109 HFT StuKgart 83,900 Universität für Bodenkultur Wien 80,463 ETH Zürich 23,648 Ø Number of data values (est): 1,091,393,139 19,921 Universität Würzburg TU Wien 4,322 CUAHSI 3,166 Ø Mean monthly dowloads: 25,000,000 Universität Potsdam 96 GFZ 94 - 50,000,000 100,000,000 150,000,000 Institute for Bio- and Geosciences - Agrosphere (IBG-3) Persistent data identifiers http://doi.org/10.5880/TERENO.256 Ø Unique, digital identifier, allowing persistent citation of publications and data § Eases access to research data § Increases visibility of data Ø Identifier refers to “landing page” containing: § Metadata (for station or data set) § Individual data sets § Licensing information (e.g. data policy) http://doi.org/10.5880/TERENO. 2016.001 Ø Landing page (and data, in general) hosted by issuing institution (here: GFZ) Ø Internal agreement to be able to link to data from the TERENO portal Ø Currently, 24 data sets/stations were identified through persistent identifiers (see https://search.datacite.org/works? query=TERENO) Ø Drawbacks § File based approach § “Snapshot creation” of data from data bases required § Dynamical referencing to data in planning § Manual process § Hosting the same metadata on two systems in parallel (GFZ, TEODOOR) Institute for Bio- and Geosciences - Agrosphere (IBG-3) 8