Heaven, a framework for a systematic comparative research approach of RSP Engines Stream Reasoning Workshop 2016 R. Tommasini,E. Della Valle, M. Balduini, D. Dell’Aglio 1 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
SRW 16 What is the goal of a benchmark? 2 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
SRW16 The goal of a domain specific benchmark is to foster technological progress by guaranteeing a fair assessment . - Jim Gray, The Benchmark Handbook for Database and Transaction Systems, 1993 3 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
RSP Engines Vs Benchmarking Qualitative SRW16 SKB Citybench LSBench C-SPARQL EP-SPARQL YABench SRBench Morph Stream Sparkwave CSRBench CQELS INSTANS (2008) (2010) (2011) (2012) (2013) (2015) (2016) time 4 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
A Well-Known Hypothesis SRW16 The incremental maintenance of the materialisation is faster then full re-materialisation of the ontological entailment when content changes are small enough (e.g. about 10%). 5 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Uncomfortable Truths in RSP Benchmarking 1000 SRW16 6 100 Memory (mb) 10 4 1 0.05 0.10 0.20 Naive Latency (ms) Window Cardinality Incremental (# Triples ) 6 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Uncomfortable Truths in RSP Benchmarking SRW16 1000 6 100 Memory (mb) 10 4 1 1 2 5 Latency (ms) Abox Cardinality Naive (# Triples ) Incremental 7 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Observations SRW16 •RSP Engines show heterogenous operational semantics that called for •Comparative Research (Benchmarks). •Unifying formal models (RSP-QL). •The community is doing a lot of effort to develop benchmarking solutions. •Each of the proposed benchmarks highlights new aspects regarding RSP performance. 8 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Adoption of Comparative Research SRW16 •It is natively case driven: •It considers cases as a combination of known properties •It defines analysis guidelines through baselines •It is extensively used to analyse complex systems •It provides layered frameworks to •systematically examine cases •identify similarities/differences enabling us to catch more insights. 9 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
In practice SRW16 •Qualitatively, is there a solution that always outperforms the others? •If no dominant solution can be found, when does a solution work better than another one? •Quantitatively, is there a solution that distinguishes itself from the others? •Why does a solution performs better than another solution under a certain experimental condition? 10 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
SRW 16 Do we need comparative research? let’s take a look to what other fields do 11 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
•Systems standard semantics (SPARQL 1.0/1.1) 🙃 Linked •Several Benchmarks 🙃 •No benchmark principles ☹ Data •Benchmark Requirements 🙃 •Clear Metrics 🙃 Benchmarks •Baselines 🙃 •Standard by Adoption ☹ •LDBC and Hobbit 😑 12 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
•Systems standard semantics (OWL Profiles) 🙃 •Few Benchmarks (growing) 🙃 OWL •No benchmark principles ☹ Reasoning •Benchmark Requirements 😑 •Clear Metrics 🙃 Benchmarks •Baselines 🙃 •Standard by Adoption ☹ • ORE 😑 13 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
•(Several) System standard semantics (SQL dialects) 🙄 •Domain Specific Benchmarks 🙃 Databases •Jim Gray’s benchmark principles 🙃 •Benchmark Requirements* 🙃 Benchmarks •Clear Metrics 🙃 •Baselines 🙃 •Standard by TPC 🙃 14 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
SRW 16 What about RSP? 15 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
• System standard semantics (But unified) 🙃 • Several Benchmarks 🙃 • Seven Commandments for RSP RSP benchmarking 🙃 Benchmarks • No benchmark requirements ☹ • Baselines by adoption 😑 • No clear metrics ☹ • No standard ☹ 16 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Claims SRW16 •Comparative research is not systematic for RSP: •it needs requirements/methodology •it needs fixed metrics •it needs fixed baselines 17 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
SRW 16 Can we enable a systematic comparative research approach for RSP Engines? 18 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Heaven SRW16 A set of requirements to satisfy. An architecture for an RSP engine Test Stand. Two baseline RSP engine architectures A proof-of-concept implementation (open source) 19 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
[R1] RSP engine agnostic Comparability [R2] Independent from the measured key performance indicators Requirements [R3] Identify baseline RSP engines 20 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Reproducibility [R4] Data independent Requirements [R5] Query independent 21 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Repeatability [R6] Minimise the experimental error Requirements 22 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
RSP Experiment SRW16 RSP Experiment is a function Exp(____________) E T D Q , , , , K [R1] [R4] [R5] [R2] R eport 23 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Test Stands Black box evaluation of complex system that controls the experimental error [R6] 24 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
MB MB SRW16 Interface Interface Streamer Input output Receiver RSPEngine D < , Q > T Start Stop ResultCollector K 25 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Pipeline of DSMS and a reasoner; Support reasoning under the RSP Baselines ρ DF entailment regime; Data can flows from the DSMS to the reasoner via snapshots or differences. The minimal meaningful approaches to realise an They exploit absolute time, RSP engine [R3] i.e. their internal clock can be externally controlled. 26 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Conclusion SRW16 RSP needs comparative analysis [Claim]. Heaven enables a systematic execution of experiments , required by comparative investigations. [Promise] 27 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Promise SRW16 •Qualitatively, is there a solution that always outperforms the others? •If no dominant solution can be found, when does a solution work better than another one? •Quantitatively, is there a solution that distinguishes itself from the others? •Why does a solution performs better than another solution under a certain experimental condition? 28 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Example of Intra Experiment Comparison SRW16 29 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Example of Inter Experiment Analysis SRW16 30 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Future Works SRW16 Systematic analysis of existing solutions A web-based environment where a users can: choose one of existing benchmarks (datasets, queries) design of experiments; consult and compare the results against the baselines or existing integrated RSP engines. 31 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Questions? Email : riccardo.tommasini@polimi.it Twitter : @rictomm Github : riccardotommasini Web : streamreasoning.org 32 Stream Reasoning Workshop, Berlin, 2016 - Riccardo Tommasini - Politecnico di Milano (DEIB)
Recommend
More recommend
