QueryArrow: Semantically Unified Query and Update of Heterogeneous - - PowerPoint PPT Presentation

QueryArrow: Semantically Unified Query and Update of Heterogeneous Data Stores

Hao Xu xuhao@renci.org Ben Keller Antoine de Torcy Jason Coposky

University of North Carolina at Chapel Hill

June 14, 2017

Xu (UNC CH) QueryArrow June 14, 2017 1 / 15

Acknowledgement

This research is paritally supported by The National Science Foundation under Grant Number OCI 0940841 ”DataNet Federation Consortium” (Any opinions, findings, and conclusions or recommendations expressed in this material are those

• f the authors and do not necessarily reflect the views of the National

Science Foundation) and iRODS Consortium

Xu (UNC CH) QueryArrow June 14, 2017 2 / 15

Resources

QueryArrow: https://github.com/xu-hao/QueryArrow fs branch QueryArrow Database Plugin: http://github.com/xu-hao/temporary-irods-mod qa3 branch

Xu (UNC CH) QueryArrow June 14, 2017 3 / 15

Motivting Applications

Big Metadata Challenge: Aggregation: integrating metadata from multiple data stores Policies: for example, metadata access control Unify query and update Semantically specification Heterogeneity: NoSQL, Indexing Improve Testing, Modularity, Robustness

Xu (UNC CH) QueryArrow June 14, 2017 4 / 15

QueryArrow Elements

QueryArrow is a semantically unified query and update system for heterogeneous metadata store QueryArrow Service (QAS): Register Databases and Execution of QAL QueryArrow Language (QAL): Configuration, QL/DML QueryArrow Plugins (QAP): Mappings between QAL and Databases iRODS QueryArrow Database Plugin

Xu (UNC CH) QueryArrow June 14, 2017 5 / 15

QueryArrow Architecture

iRODS Core QueryArrow Database Plugin QAS Postgres QAP Postgres SQL Neo4j QAP Neo4j Cypher ElasticSearch QAP ElasticSearch EQL QAL Update API GenQuery Query irodsAgent QueryArrow

Xu (UNC CH) QueryArrow June 14, 2017 6 / 15

QAPs

Name Description Sum QAP aggregation Translation QAP policy support Cache QAP caching Mutable Map QAP in-memory mutable map Immutable Map QAP in-memory immutable map ElasticSearch QAP interfacing with ElasticSearch Neo4j QAP interfacing with Neo4j PostgreSQL QAP interfacing with Postgres SQLite3 QAP interfacing with SQLite3 CockroachDB QAP interfacing with CockroachDB FileSystem QAP file system

Xu (UNC CH) QueryArrow June 14, 2017 7 / 15

QAP Composition

Higher-order plugin Cache QAP Translation QAP Sum QAP QAP 1 Data store 1 QAP 2 Data store 2 QAP 3 Data store 3

Xu (UNC CH) QueryArrow June 14, 2017 8 / 15

QAL Syntax

t ∶∶= p ∣ v terms a ∶∶= P(t1,...,tn) atom c ∶∶= a ∣ insert a ∣ delete a ∣ g c ∣ b ∣ 1 ∣ 0 ∣ c ⊕ c ∣ c ⊗ c command pt ∶∶= input? output? key? ty parameter type R ∶∶= rewrite a c ∣ rewrite insert a c ∣ rewrite delete a c ∣ predicate P(pt1,...,ptn) ∣ import ... ∣ export ... configuration

Xu (UNC CH) QueryArrow June 14, 2017 9 / 15

QAL Semantics

”store-and-heap” Monad https://github.com/xu-hao/CertifiedQueryArrow

Xu (UNC CH) QueryArrow June 14, 2017 10 / 15

QAL Translation

P A B C . . . . . . . . . Q A B C . . . . . . . . . :P :Q P.C ”type” : ”P” ”A” : . . . ”B” : . . . ”C” : . . . ”type” : ”Q” ”A” : . . . ”B” : . . . ”C” : . . .

Xu (UNC CH) QueryArrow June 14, 2017 11 / 15

QueryArrow Service

Local Unix domain socket TCP HTTP

Xu (UNC CH) QueryArrow June 14, 2017 12 / 15

QueryArrow Database Plugin

Plugs into iRODS, requires no changes to iRODS core except for bug fixes Talks to QueryArrow Service Implements 84 db plugin functions Backward compatibility with GenQuery Tested on 4.2.x:

4.2.0: PostgreSQL QAP passed python tests suite except 2 tests using ssl plugin on Ubuntu 16.04/Centos 7 4.2.1: PostgreSQL QAP passed python tests suite except 2 tests using ssl plugin on Ubuntu 16.04/Centos 7 is running

Xu (UNC CH) QueryArrow June 14, 2017 13 / 15

Install Script

Run iRODS install script through QueryArrow

Xu (UNC CH) QueryArrow June 14, 2017 14 / 15

Demo

Only Postgresql Combined Postgresql and Neo4j

Xu (UNC CH) QueryArrow June 14, 2017 15 / 15