eReporting and INSPIRE Eionet NRC Environmental Information Systems meeting 6 – 7 December 2016 Anders Foureaux Process Manager Data feeds – Data hosting Swedish EPA, Research and Assessment Department Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 1
Background > 200 consultants for collecting data 290 municipalities 21 counties Most of them shall deliver data to 10 - 15 Datahosts. Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 2
Data collection is mainly financed by the Swedish EPA and The Swedish Agency for Marine and Water Management Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 3
So, what ´ s the problem? On the provider side: • A very time consuming manual QA-process • The datahost are often forced to fix what the submitter are paid for to fix. Not fair! On the consumer side: • It takes too long between the collection of data until anyone can use it • It is hard for the end user to find relevant quality information at the same place were the data is stored or made available. Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 4
And what are the objectives? 1. Speed up the data flow from collection to end user. 2. Check the data quality against agreed quality standards and quality measures. 3. Create metadata "on-the fly" for performed quality control. 4. Gather all the requirements for data quality into a Data Product Specification. 5. Save time so that the contracted experts can use their time to more advanced analyzes of data instead of correcting incomplete data files. 6. To the extent possible, use the same OGC- and ISO standards for Environmental Data, as the INSPIRE community uses for Geodata. Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 5
The pie chart is based on time analysis 130 different data flows in 2010 Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 6
Data Product Specification ISO 19131 DPS 1) ” Core model ” of O&M + 2) ”Data theme specific ” model of O&M Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 7
Cloudbased Validation Service Data- Data- Codelists specification model Data Data host Specificaton Translation Validation compliance Metadata Datafile in xls is transformed into xml according to O&M 8 Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 8
Step one We have a very simple website for a consultant to upload data (FME at the backend) • Tell who you are • Your e-mail adress • The datahost´s e-mail adress • Select the appropriate media type • Submit the data file Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 9
Step two • Validation of data file starts • Checks the values • Checks the parameter codes • Against REGISTRY at JRC • Against a clone of Data Dictionary at Swedish EPA • Against the Station Registry • According to our Data Product Specificaton Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 10
Step tree • The submitter recieves an e-mail. • The file did not pass the validation! • Only the submitter gets this. • Link to report in HTML • Link to zip-file with report in HTML + xml (ISO 19139). Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 11
Step four The HTML report tells what´s wrong. • "MediaMonitored" Sediment is not in the code list http://inspire.ec.europa.eu/codelist/MediaValue/ • "PROVTAG_MET" på rad 22. Value GEMINIX is not in the code list Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 12
Step five No uppercase ”s” Name of the • Check the code list at JRC sampler is not GEMINX • Check the code list at Swedish EPA Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 13
Step six • Fix the errors • Submit a new data file Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 14
Step seven • After validation, check your mail box • OK! • Download! Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 15
Step eight The Datahost and the submitter now gets the same files 1. The original xls-file that was uploaded 2. The transformed data file into xml and O&M 3. The specification compliance- report 4. Quality information in xml according to ISO 19139 The Datahost can now start the process for importing new data into the database and publish it. The submitter is finished with his/her work Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 16
Cloudbased Validation Service Data- Data- Codelists specification model Data Data host Specificaton Translation Validation compliance Metadata Datafile in xls is transformed into xml according to O&M 17 Naturvårdsverket | Swedish Environmental Protection Agency Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 17
Near Real Time data EEA / EU Air Quality Via middleware/ or other server SOS Datahost Swedish EPA (Sensor DB Observation Service) Validation service Other consumers Direct from logger at Instead of the website for uploading data MontitoringStation (page 9), now the machines are doing the job at municipality level (FME at the backend) Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 18
Realtime Data – Data specification and dataformat for ”sensors/ machines ” Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 19
Realtime Data – Sensor Observation Service (SOS) Standardized web service that meets the INSPIRE Directive requirements for making data available Data are available in different formats (JSON, XML, SOAP, etc.) for the development of client applications Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 20
Realtime Data – Charts at Swedish EPA • Developed with www.highcharts.com open software • API "Greenbox" developed to read & translate data from the SOS service for use in Highcharts • Work is underway to expand with data from several municipalities Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 21
Realtime Data – other consumers Länkar: http://www.eionet.europa.eu/aqportal/products/viewers http://waqi.info/ Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 22 www.plumelabs.com
Observations & Measurements ISO 19156 • An open standard from OGC • Descssions must be made • We try to find a ” core ” • Then add specific extentions for each ”Data theme specific ” type of data Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 23
Observations & Measurements ISO 19156 Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 24
”Data theme specific ” model ” Core model ” Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 25
Metals and hazardous substances in sediments GEMINI sampler EKMAN sampler Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 26
Supporting services like codelists, registries • Registry at JRC • A clone of Registry at National level (?) • Data Dictionary at EEA • A clone of Data Dictionary at National level • A Station Registry for monitoring stations. A WFS in the cloud (OpenGeo Suite + Azure) Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 27
INSPIRE REGISTRY An exellent stack of software based on open source! Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 28
We are now testing DD from EEA / GitHub as a Swedish ” clone ” ( open source) Also a very nice stack of software based on open source Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 29
More than 3000 parameters in our ” clone ” Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 30
Registry of Monitoring Stations A Web Service (WFS) in ”the Cloud” OpenGeo Suite + Azure Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 31
Applied standards (some of them) • ISO 19156 - Observations and measurements • ISO 19157 - Data quality • ISO 19139 - Metadata - XML schema implementation • ISO 19131 - Data product specifications Naturvårdsverket | Swedish Environmental Protection Agency 2016-12-07 32
Recommend
More recommend
