eReporting and INSPIRE Eionet NRC Environmental Information Systems - - PowerPoint PPT Presentation

▶
ereporting and inspire
SMART_READER_LITE
LIVE PREVIEW

eReporting and INSPIRE Eionet NRC Environmental Information Systems - - PowerPoint PPT Presentation

May 06, 2023 201 likes •549 views

eReporting and INSPIRE Eionet NRC Environmental Information Systems meeting 6 7 December 2016 Anders Foureaux Process Manager Data feeds Data hosting Swedish EPA, Research and Assessment Department Naturvrdsverket | Swedish

slide-1
SLIDE 1

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 1

Anders Foureaux

Process Manager Data feeds – Data hosting Swedish EPA,

Research and Assessment Department

eReporting and INSPIRE

Eionet NRC Environmental Information Systems meeting 6 – 7 December 2016

slide-2
SLIDE 2

Background

> 200 consultants for collecting data 290 municipalities 21 counties Most of them shall deliver data to 10 - 15 Datahosts.

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 2

slide-3
SLIDE 3

Data collection is mainly financed by the Swedish EPA and The Swedish Agency for Marine and Water Management

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 3

slide-4
SLIDE 4

So, what´s the problem?

On the provider side:

  • A very time consuming manual QA-process
  • The datahost are often forced to fix what the

submitter are paid for to fix. Not fair! On the consumer side:

  • It takes too long between the collection of data

until anyone can use it

  • It is hard for the end user to find relevant quality

information at the same place were the data is stored or made available.

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 4

slide-5
SLIDE 5

And what are the objectives?

1. Speed up the data flow from collection to end user. 2. Check the data quality against agreed quality standards and quality measures. 3. Create metadata "on-the fly" for performed quality control. 4. Gather all the requirements for data quality into a Data Product Specification. 5. Save time so that the contracted experts can use their time to more advanced analyzes of data instead of correcting incomplete data files. 6. To the extent possible, use the same OGC- and ISO standards for Environmental Data, as the INSPIRE community uses for Geodata.

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 5

slide-6
SLIDE 6

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 6

The pie chart is based on time analysis 130 different data flows in 2010

slide-7
SLIDE 7

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 7

Data Product Specification ISO 19131 1) ”Core model” of O&M + 2) ”Data theme specific” model of O&M DPS

slide-8
SLIDE 8

8 8

Data- specification Data- model Codelists

Cloudbased Validation Service

Data Specificaton compliance Metadata

Translation Validation

Data host

Datafile in xls is transformed into xml according to O&M

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency

slide-9
SLIDE 9

Step one

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 9

  • Tell who you are
  • Your e-mail

adress

  • The datahost´s

e-mail adress

  • Select the

appropriate media type

  • Submit the data

file

We have a very simple website for a consultant to upload data (FME at the backend)

slide-10
SLIDE 10

Step two

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 10

  • Validation of data file starts
  • Checks the values
  • Checks the parameter

codes

  • Against REGISTRY at JRC
  • Against a clone of Data

Dictionary at Swedish EPA

  • Against the Station Registry
  • According to our Data

Product Specificaton

slide-11
SLIDE 11

Step tree

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 11

  • The submitter recieves an

e-mail.

  • The file did not pass the

validation!

  • Only the submitter gets this.
  • Link to report in HTML
  • Link to zip-file with report in

HTML + xml (ISO 19139).

slide-12
SLIDE 12

Step four

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 12

The HTML report tells what´s wrong.

  • "MediaMonitored" Sediment is not in the code list

http://inspire.ec.europa.eu/codelist/MediaValue/

  • "PROVTAG_MET" på rad 22. Value GEMINIX is not in the

code list

slide-13
SLIDE 13

Step five

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 13

  • Check the code list at JRC
  • Check the code list at

Swedish EPA No uppercase ”s” Name of the sampler is not GEMINX

slide-14
SLIDE 14

Step six

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 14

  • Fix the errors
  • Submit a new data file
slide-15
SLIDE 15

Step seven

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 15

  • After validation, check

your mail box

  • OK!
  • Download!
slide-16
SLIDE 16

Step eight

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 16

The Datahost and the submitter now gets the same files 1. The original xls-file that was uploaded 2. The transformed data file into xml and O&M 3. The specification compliance- report 4. Quality information in xml according to ISO 19139 The Datahost can now start the process for importing new data into the database and publish it. The submitter is finished with his/her work

slide-17
SLIDE 17

17 Naturvårdsverket | Swedish Environmental Protection Agency 17

Data- specification Data- model Codelists

Cloudbased Validation Service

Data Specificaton compliance Metadata

Translation Validation

Data host

Datafile in xls is transformed into xml according to O&M

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency

slide-18
SLIDE 18

Near Real Time data Air Quality

Direct from logger at MontitoringStation at municipality level Via middleware/ or

  • ther server

Datahost DB EEA / EU Swedish EPA Other consumers SOS

(Sensor Observation Service)

Validation service

Instead of the website for uploading data (page 9), now the machines are doing the job (FME at the backend)

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 18

slide-19
SLIDE 19

Realtime Data – Data specification and dataformat for ”sensors/ machines”

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 19

slide-20
SLIDE 20

Realtime Data – Sensor Observation Service (SOS)

  • Standardized web service that meets the INSPIRE Directive

requirements for making data available

  • Data are available in different formats (JSON, XML, SOAP,

etc.) for the development of client applications

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 20

slide-21
SLIDE 21

Realtime Data – Charts at Swedish EPA

  • Developed with www.highcharts.com open software
  • API "Greenbox" developed to read & translate data from the

SOS service for use in Highcharts

  • Work is underway to expand with data from several

municipalities

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 21

slide-22
SLIDE 22

Realtime Data –

  • ther consumers

Länkar: http://www.eionet.europa.eu/aqportal/products/viewers http://waqi.info/ www.plumelabs.com

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 22

slide-23
SLIDE 23

Observations & Measurements ISO 19156

  • An open standard from OGC
  • Descssions must be made
  • We try to find a ”core”
  • Then add specific extentions for each ”Data

theme specific” type of data

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 23

slide-24
SLIDE 24

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 24

Observations & Measurements ISO 19156

slide-25
SLIDE 25

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 25

”Data theme specific” model ”Core model”

slide-26
SLIDE 26

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 26

Metals and hazardous substances in sediments

GEMINI sampler EKMAN sampler

slide-27
SLIDE 27

Supporting services like codelists, registries

  • Registry at JRC
  • A clone of Registry at National level (?)
  • Data Dictionary at EEA
  • A clone of Data Dictionary at National level
  • A Station Registry for monitoring stations.

A WFS in the cloud (OpenGeo Suite + Azure)

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 27

slide-28
SLIDE 28

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 28

An exellent stack of software based on open source!

INSPIRE REGISTRY

slide-29
SLIDE 29

We are now testing DD from EEA / GitHub as a Swedish ”clone” (open source)

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 29

Also a very nice stack of software based on open source

slide-30
SLIDE 30

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 30

More than 3000 parameters in our ”clone”

slide-31
SLIDE 31

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 31

Registry of Monitoring Stations A Web Service (WFS) in ”the Cloud” OpenGeo Suite + Azure

slide-32
SLIDE 32

Applied standards (some of them)

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 32

  • ISO 19156 - Observations and measurements
  • ISO 19157 - Data quality
  • ISO 19139 - Metadata - XML schema implementation
  • ISO 19131 - Data product specifications
slide-33
SLIDE 33

Applied standards (all of them ?)

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 33

SIS/TK 489 N247 Geodata- Nationell metadataprofil – DPS och vägledning version 3.1.1 SIS-TR 14:2012 Geografisk information – Metadata på svenska SIS-TR 40:2012 Geografisk information – Tekniskt ramverk – Handbok för Dataproduktspecifikation SS-EN ISO 19107:2005 Geografisk information – Modell för att beskriva rumsliga aspekter SS-EN ISO 19108:2005 Geografisk information – Modell för att beskriva tidsaspekter SS-EN ISO 19109:2006 Geografisk information – Regler för applikationsschema SS-EN ISO 19110:2006 Geografisk information - Struktur för katalogisering av

  • bjekttyper

SS-EN ISO 19115:2014 Geografisk information – Metadata – Del 1:Grunder SS-EN ISO 19131:2008 Geographic information – Specifikation av datamängder SS-EN ISO/TS 19139:2007 Geografisk information – Metadata – implementering med XML-schema SS-EN ISO 19156:2013 Geografisk information - Observationer och mätningar SS-EN ISO 19157:2013 Geografisk information – Datakvalitet

slide-34
SLIDE 34

Anders.foureaux@naturvardsverket.se

2016-12-07 Naturvårdsverket | Swedish Environmental Protection Agency 34

Thanks for your attention!