Heaven, a framework for a systematic comparative research approach - - PowerPoint PPT Presentation

▶
heaven a framework for a systematic comparative research
SMART_READER_LITE
LIVE PREVIEW

Heaven, a framework for a systematic comparative research approach - - PowerPoint PPT Presentation

Feb 23, 2024 119 likes •450 views

Heaven, a framework for a systematic comparative research approach of RSP Engines Stream Reasoning Workshop 2016 R. Tommasini,E. Della Valle, M. Balduini, D. DellAglio 1 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini -

slide-1
SLIDE 1

Stream Reasoning Workshop 2016

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Heaven, a framework for a systematic comparative research approach of RSP Engines

1
  • R. Tommasini,E. Della Valle, M. Balduini, D. Dell’Aglio
slide-2
SLIDE 2 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

SRW 16

What is the goal of a benchmark?

2
slide-3
SLIDE 3

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB) The goal of a domain specific benchmark 
 is to foster technological progress 
 by guaranteeing a fair assessment.

  • Jim Gray, The Benchmark Handbook
for Database and Transaction Systems, 1993 
 3
slide-4
SLIDE 4

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

RSP Engines Vs Benchmarking

4 time (2008) (2010) (2011) (2015) Citybench (2016) YABench (2012) LSBench SRBench CSRBench EP-SPARQL C-SPARQL Sparkwave MorphStream CQELS SKB INSTANS Qualitative (2013)
slide-5
SLIDE 5

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

A Well-Known Hypothesis

The incremental maintenance of the materialisation 
 is faster then full re-materialisation 

  • f the ontological entailment

when content changes are 
 small enough (e.g. about 10%). 5
slide-6
SLIDE 6

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Uncomfortable Truths in RSP Benchmarking

6 Memory (mb) Latency (ms) 6 4 0.10 0.20 0.05 Window Cardinality (# Triples ) 1000 100 10 1 Naive Incremental
slide-7
SLIDE 7

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Uncomfortable Truths in RSP Benchmarking

7 2 5 1 Memory (mb) Latency (ms) 6 4 Abox Cardinality (# Triples ) 1000 100 10 1 Naive Incremental
slide-8
SLIDE 8

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Observations

  • RSP Engines show heterogenous operational semantics that called for
  • Comparative Research (Benchmarks).
  • Unifying formal models (RSP-QL).
  • The community is doing a lot of effort to develop benchmarking
solutions.
  • Each of the proposed benchmarks highlights new aspects regarding RSP
performance. 8
slide-9
SLIDE 9

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Adoption of Comparative Research

  • It is natively case driven:
  • It considers cases as a combination of known properties
  • It defines analysis guidelines through baselines
  • It is extensively used to analyse complex systems
  • It provides layered frameworks to
  • systematically examine cases
  • identify similarities/differences enabling us to catch more insights.
9
slide-10
SLIDE 10

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

In practice

  • Qualitatively, is there a solution that always outperforms the others?
  • If no dominant solution can be found, when does a solution work better
than another one?
  • Quantitatively, is there a solution that distinguishes itself from the
  • thers?
  • Why does a solution performs better than another solution under a
certain experimental condition? 10
slide-11
SLIDE 11 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

SRW 16

Do we need comparative research?

11

let’s take a look to what other fields do

slide-12
SLIDE 12 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Linked Data

Benchmarks 12
  • Systems standard semantics
(SPARQL 1.0/1.1) 🙃
  • Several Benchmarks 🙃
  • No benchmark principles ☹
  • Benchmark Requirements 🙃
  • Clear Metrics 🙃
  • Baselines 🙃
  • Standard by Adoption ☹
  • LDBC and Hobbit 😑
slide-13
SLIDE 13 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

OWL Reasoning

Benchmarks
  • Systems standard semantics

(OWL Profiles) 🙃
  • Few Benchmarks (growing) 🙃
  • No benchmark principles ☹
  • Benchmark Requirements 😑
  • Clear Metrics 🙃
  • Baselines 🙃
  • Standard by Adoption ☹
  • ORE 😑
13
slide-14
SLIDE 14 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Databases

Benchmarks 14
  • (Several) System standard
semantics (SQL dialects) 🙄
  • Domain Specific Benchmarks 🙃
  • Jim Gray’s benchmark principles 🙃
  • Benchmark Requirements* 🙃
  • Clear Metrics 🙃
  • Baselines 🙃
  • Standard by TPC 🙃
slide-15
SLIDE 15 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

SRW 16

What about RSP?

15
slide-16
SLIDE 16 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

RSP

Benchmarks
  • System standard semantics 

(But unified) 🙃
  • Several Benchmarks 🙃
  • Seven Commandments for RSP
benchmarking 🙃
  • No benchmark requirements ☹
  • Baselines by adoption 😑
  • No clear metrics ☹
  • No standard ☹
16
slide-17
SLIDE 17

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Claims

  • Comparative research is not systematic for RSP:
  • it needs requirements/methodology
  • it needs fixed metrics
  • it needs fixed baselines
17
slide-18
SLIDE 18 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

SRW 16

Can we enable a systematic comparative research approach for RSP Engines?

18
slide-19
SLIDE 19

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Heaven

A set of requirements to satisfy. An architecture for an RSP engine Test Stand. Two baseline RSP engine architectures A proof-of-concept implementation (open source) 19
slide-20
SLIDE 20 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Comparability

Requirements

[R1] RSP engine agnostic [R2] Independent from the measured key performance indicators [R3] Identify baseline RSP engines 20
slide-21
SLIDE 21 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Reproducibility

Requirements

[R4] Data independent [R5] Query independent 21
slide-22
SLIDE 22 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Repeatability

Requirements

[R6] Minimise the experimental error 22
slide-23
SLIDE 23

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB) RSP Experiment is a function

RSP Experiment

23

Exp(____________)

E T Q D K , , , ,

[R1] [R4] [R5] [R2]

R

eport

slide-24
SLIDE 24 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Test Stands

Black box evaluation of complex system that controls the experimental error [R6] 24
slide-25
SLIDE 25

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB) 25 RSPEngine < ,Q> Input
  • utput
Start Stop MB MB Interface Interface

T

Streamer D Receiver ResultCollector K
slide-26
SLIDE 26 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

RSP Baselines

The minimal meaningful approaches to realise an RSP engine [R3] Pipeline of DSMS and a reasoner; Support reasoning under the ρDF entailment regime; Data can flows from the DSMS to the reasoner via snapshots or differences. They exploit absolute time, i.e. their internal clock can be externally controlled. 26
slide-27
SLIDE 27

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Conclusion

RSP needs comparative analysis [Claim]. Heaven enables a systematic execution of experiments, required by comparative investigations. [Promise] 27
slide-28
SLIDE 28

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Promise

  • Qualitatively, is there a solution that always outperforms the others?
  • If no dominant solution can be found, when does a solution work better
than another one?
  • Quantitatively, is there a solution that distinguishes itself from the
  • thers?
  • Why does a solution performs better than another solution under a
certain experimental condition? 28
slide-29
SLIDE 29

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Example of Intra Experiment Comparison

29
slide-30
SLIDE 30

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB) 30

Example of Inter Experiment Analysis

slide-31
SLIDE 31

SRW16

Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Future Works

Systematic analysis of existing solutions A web-based environment where a users can: choose one of existing benchmarks (datasets, queries) design of experiments; consult and compare the results against the baselines or existing integrated RSP engines. 31
slide-32
SLIDE 32 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)

Questions?

Email: riccardo.tommasini@polimi.it
 Twitter: @rictomm Github: riccardotommasini Web: streamreasoning.org 32