Dynamic analysis, reporting and visualization of data catalogs Use - - PowerPoint PPT Presentation

▶
dynamic analysis reporting and visualization of data
SMART_READER_LITE
LIVE PREVIEW

Dynamic analysis, reporting and visualization of data catalogs Use - - PowerPoint PPT Presentation

Jul 27, 2023 254 likes •681 views

Dynamic analysis, reporting and visualization of data catalogs Use cases INSPIRE Dashboard for all Member States and EEA MedSea project at Ifremer INSPIRE Directive in Europe INSPIRE is based on the infrastructures for spatial information

slide-1
SLIDE 1

Dynamic analysis, reporting and visualization of data catalogs

Use cases INSPIRE Dashboard for all Member States and EEA MedSea project at Ifremer

slide-2
SLIDE 2

INSPIRE Directive in Europe

INSPIRE is based on the infrastructures for spatial information established and operated by the 28 Member States of the European Union Yearly report made to the Commission.

> INSPIRE MIWP-16 group goal is “to support MS in making annual report to INSPIRE” > INSPIRE dashboard prototype

slide-3
SLIDE 3

Another need - MedSea checkpoint

slide-4
SLIDE 4

Scenario

#1 Collect #2 Analyze #3 Visualize & DSS

slide-5
SLIDE 5

Collecting

INSPIRE Monitoring Metadata record MS CSW MS CSW MS CSW

XML XML

slide-6
SLIDE 6

Analyzing

slide-7
SLIDE 7

Reporting

slide-8
SLIDE 8

Which tool for that ?

We need

  • a search engine
  • a simple (& fast) harvester
  • analysis framework
  • reporting
slide-9
SLIDE 9

Which tool for that ?

We need

  • a search engine
  • a simple & fast harvester
  • analysis framework
  • dashboards

banana/silk project

slide-10
SLIDE 10

daobs - data analyzer and observer

  • pensource
  • n github
slide-11
SLIDE 11

Collecting

slide-12
SLIDE 12

Collecting through CSW

MS CSW ? GetRecords with no hits to get number of records Compute number of pages p1 p2 p3 Index GetRecords

slide-13
SLIDE 13

MS CSW

Harvesting fast

Parallel processing configurable

MS CSW Index MS CSW

slide-14
SLIDE 14

Harvester module

Support CSW protocol Tested on 30 CSW endpoints from different MS harvesting more than 200K records Quite fast Based on Camel routing engine

slide-15
SLIDE 15

Analyzing

slide-16
SLIDE 16

Analysis - 3 types

indexing background tasks reports

slide-17
SLIDE 17

Analyzing / Indexing

slide-18
SLIDE 18

First pass analysis

Made when indexing records

  • should be quick to not slow down imports
  • based on Solr/Lucene analysis features
  • basic multilingual support
slide-19
SLIDE 19

Generic indexing with dynamic fields

  • eg. categorize keywords
slide-20
SLIDE 20

Keep word filter

ISO19139 metadata record Index

view download discovery transformation invoke

  • ther

OGC:WMS view

KeepWordFilter

view

slide-21
SLIDE 21

Synonym analysis

ISO19139 metadata record Index Synonyms xpath = index field

Orthoimagery > Annex II Ortofotosnímky

inspireTheme=Ortofotosnímky inspireTheme_syn=Orthoimagery inspireAnnex=II

ManagedSynonymFilterFactory

slide-22
SLIDE 22

Analyzing / Background tasks

slide-23
SLIDE 23

INSPIRE validation task

task

(1) Get documents to validate (2) Validate (3) Save validation report and completeness indicator

slide-24
SLIDE 24

Indexing related documents

  • 4. Search on

metadata and related documents references a Catalog of metadata record Document

  • html page
  • pdf
  • spreadsheet
  • dbf (in a shp)
  • ...

extract content 3.Analyze and index 1.Get document URLs

slide-25
SLIDE 25

Indexing Geographic Features

WFS services references a 2.GetFeature 3.Analyze and index 1.Get service URLs Catalog of metadata record

  • 4. Search on

metadata and features

slide-26
SLIDE 26

Tasks chaining

JMS Messaging bus - topic & queue

slide-27
SLIDE 27

Analyzing / Reports

slide-28
SLIDE 28

Configure reports

Indicators Indicators Variables Variables Variable Indicator Report configuration file

= query = mathematical expression based on variables

slide-29
SLIDE 29

Configure INSPIRE report

INSPIRE monitoring reporting

http://inspire.ec.europa.eu/index.cfm/pageid/5022

MDv11 +(resourceType:dataset resourceType:series) +inspireAnnex:i MDi11 MDv11 / DSv_Num1 MDi1 (MDv11 + MDv12 + MDv13 + MDv14) / (DSv_Num + SDSv_Num)

DSv_Num1 DSv_Num DSv_Num DSv_Num DSv_Num

slide-30
SLIDE 30

Report configuration file INSPIRE Monitoring INSPIRE Monitoring

slide-31
SLIDE 31

Analysis & data visualization

slide-32
SLIDE 32

Based on banana/silk project

A dashboard web creation tool on top of Solr

slide-33
SLIDE 33

A dashboard is a set of rows

slide-34
SLIDE 34

Different widgets to build dashboard

slide-35
SLIDE 35

TagClouds Graphs

slide-36
SLIDE 36

Heatmaps

slide-37
SLIDE 37

Maps

slide-38
SLIDE 38

Dynamic filtering

Click on items to apply filters

slide-39
SLIDE 39

Dynamic filtering

slide-40
SLIDE 40

Examples

http://apps.titellus.net/dashboard/ https://inspire-dashboard.eea.europa.eu/dashboard http://sextant.ifremer.fr/dashboard/

slide-41
SLIDE 41

Technical overview

Messaging - ActiveMQ - JMS Camel tasks

Can run in a webapp or as CLI

Camel harvesters

Can run in a webapp or as CLI

Solr index

  • data
  • dashboard

Report module Banana

(AngularJS app)

Main app

(AngularJS app)

Spring Security Spring MVC

slide-42
SLIDE 42

Roadmap & future

  • Reporting

○ history of records ○ scheduled analysis to follow the trends

  • Indexing

○ geographic features for improving search

  • Dashboard: migration to Silk (next banana

version)