Incremental Graph Queries for Cypher Gbor Szrnyas, Jzsef Marton - - PowerPoint PPT Presentation

▶
incremental graph queries for cypher
SMART_READER_LITE
LIVE PREVIEW

Incremental Graph Queries for Cypher Gbor Szrnyas, Jzsef Marton - - PowerPoint PPT Presentation

Mar 13, 2024 143 likes •1.05k views

Incremental Graph Queries for Cypher Gbor Szrnyas, Jzsef Marton Budapest University of Technology and Economics McGill University, Montral Budapest University of Technology and Economics Department of Measurement and Information Systems

slide-1
SLIDE 1

Budapest University of Technology and Economics Department of Measurement and Information Systems Department of Telecommunications and Media Informatics

Budapest University of Technology and Economics McGill University, Montréal

Incremental Graph Queries for Cypher

Gábor Szárnyas, József Marton

slide-2
SLIDE 2

Live railway model

slide-3
SLIDE 3

Live railway model

slide-4
SLIDE 4

Live railway model

slide-5
SLIDE 5

Live railway model

slide-6
SLIDE 6

Live railway model

Proximity detection

slide-7
SLIDE 7

Live railway model

Proximity detection

slide-8
SLIDE 8

Live railway model

Proximity detection

slide-9
SLIDE 9

Live railway model

Trailing the switch Proximity detection

slide-10
SLIDE 10

Live railway model

slide-11
SLIDE 11

Live railway model

slide-12
SLIDE 12

Live railway model

c d e g f div 2 a b 1

slide-13
SLIDE 13

Live railway model

c d e g f div 2

NEXT NEXT STRAIGHT TOP ON

a b 1

NEXT ON NEXT

slide-14
SLIDE 14

Proximity detection

Proximity detection

≤ 𝟑 segments

slide-15
SLIDE 15

Proximity detection

seg1

NEXT: 1..2

t1

ON

Proximity detection

seg2

t2

ON

≤ 𝟑 segments

slide-16
SLIDE 16

Proximity detection

seg1

NEXT: 1..2

t1

ON

MATCH (t1:Train)-[:ON]->(seg1:Segment)

  • [:NEXT*1..2]->(seg2:Segment)

<-[:ON]-(t2:Train) RETURN t1, t2, seg1, seg2

Proximity detection

seg2

t2

ON

≤ 𝟑 segments

slide-17
SLIDE 17

Proximity detection

seg1

NEXT: 1..2

t1

ON

MATCH (t1:Train)-[:ON]->(seg1:Segment)

  • [:NEXT*1..2]->(seg2:Segment)

<-[:ON]-(t2:Train) RETURN t1, t2, seg1, seg2

Proximity detection

seg2

t2

ON

≤ 𝟑 segments

slide-18
SLIDE 18

Trailing the switch

slide-19
SLIDE 19

Trailing the switch

seg div t

STRAIGHT ON

slide-20
SLIDE 20

Trailing the switch

seg div t

STRAIGHT ON

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

slide-21
SLIDE 21

Trailing the switch

seg div t

STRAIGHT ON

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

slide-22
SLIDE 22

Trailing the switch

seg div t

STRAIGHT ON

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

Evaluate continuously

slide-23
SLIDE 23

Incremental queries

slide-24
SLIDE 24

Incremental queries

  • Register a set of standing queries
  • Continuously evaluate queries on changes
slide-25
SLIDE 25

Incremental queries

  • Register a set of standing queries
  • Continuously evaluate queries on changes
  • The Rete algorithm (1974)
  • Originally for rule-based expert systems
  • Indexes the graph and caches interim query results
slide-26
SLIDE 26

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

div

STRAIGHT

Trailing the switch

ON

slide-27
SLIDE 27

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

slide-28
SLIDE 28

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

slide-29
SLIDE 29

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

a 1

ON

e 2

ON

slide-30
SLIDE 30

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

a 1

ON

e 2

ON

slide-31
SLIDE 31

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

slide-32
SLIDE 32

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e div

STRAIGHT

slide-33
SLIDE 33

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e div

STRAIGHT

e div

STRAIGHT

slide-34
SLIDE 34

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

slide-35
SLIDE 35

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e div

STRAIGHT

e 2

ON

slide-36
SLIDE 36

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e div

STRAIGHT

e 2

ON

e div 2

STRAIGHT ON

slide-37
SLIDE 37

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

e div

STRAIGHT

2

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e div

STRAIGHT

e 2

ON

e div 2

STRAIGHT ON

slide-38
SLIDE 38

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

div

STRAIGHT ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e 2 div

STRAIGHT ON

e 2

slide-39
SLIDE 39

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

div

STRAIGHT ON

e div

STRAIGHT

2

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

e 2 div

STRAIGHT ON

e 2

slide-40
SLIDE 40

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

e div

STRAIGHT

2

ON

e div

STRAIGHT

2

ON

c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

div 2

slide-41
SLIDE 41

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

e div

STRAIGHT

2

ON

e div

STRAIGHT

2

ON

div 2 c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

div 2

slide-42
SLIDE 42

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

e div

STRAIGHT

2

ON

e div

STRAIGHT

2

ON

div 2 c d e g f div 2

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON ON

div

STRAIGHT

Trailing the switch

ON

div 2

slide-43
SLIDE 43

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

e 2

ON

a 1

ON

e div

STRAIGHT

2

ON

e div

STRAIGHT

2

ON

div 2 c e g f div

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON

div

STRAIGHT

Trailing the switch

ON

div

ON

2 d

slide-44
SLIDE 44

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

d 2

ON

a 1

ON

e div

STRAIGHT

2

ON

e div

STRAIGHT

2

ON

div 2 c e g f div

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON

div

STRAIGHT

Trailing the switch

ON

div

ON

2 d

slide-45
SLIDE 45

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

d 2

ON

a 1

ON

e div

STRAIGHT

2

ON

div 2 c e g f div

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON

div

STRAIGHT

Trailing the switch

ON

div

ON

2 d

slide-46
SLIDE 46

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

d 2

ON

a 1

ON

div 2 c e g f div

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON

div

STRAIGHT

Trailing the switch

ON

div

ON

2 d

slide-47
SLIDE 47

πt.number, sw σsw.position = ′diverging′

⋈

STRAIGHT ON

e div

STRAIGHT

d 2

ON

a 1

ON

c e g f div

NEXT NEXT STRAIGHT TOP

a b 1

NEXT NEXT ON

div

STRAIGHT

Trailing the switch

ON

div

ON

2 d

slide-48
SLIDE 48

Batch vs. incremental queries

  • Batch queries

(pull / request-driven):

  • 1. Client selects a query
  • 2. Results are calculated
  • Query results
  • btained on demand
slide-49
SLIDE 49

Batch vs. incremental queries

  • Batch queries

(pull / request-driven):

  • 1. Client selects a query
  • 2. Results are calculated
  • Query results
  • btained on demand
  • Incremental queries

(push / event-driven):

  • 1. Client registers queries
  • 2. Graph is changed
  • 3. Results are maintained
  • 4. Goto 2
  • Query results are

always available

slide-50
SLIDE 50

Incremental query engines

  • CLIPS

C structures NASA

  • Drools

POJO Red Hat

  • VIATRA

EMF BME / IncQuery Labs.

slide-51
SLIDE 51

Incremental query engines

  • CLIPS

C structures NASA

  • Drools

POJO Red Hat

  • VIATRA

EMF BME / IncQuery Labs.

  • INSTANS

RDF Aalto University

  • i3QL

POJO TU Darmstadt

  • IncQuery-D

RDF BME

slide-52
SLIDE 52

Incremental query engines

  • CLIPS

C structures NASA

  • Drools

POJO Red Hat

  • VIATRA

EMF BME / IncQuery Labs.

  • INSTANS

RDF Aalto University

  • i3QL

POJO TU Darmstadt

  • IncQuery-D

RDF BME

  • No implementations for property graphs yet
slide-53
SLIDE 53

ingraph

  • An incremental, in-memory graph query engine
slide-54
SLIDE 54

ingraph

  • An incremental, in-memory graph query engine

ingraph client

slide-55
SLIDE 55

ingraph

  • An incremental, in-memory graph query engine

ingraph client

register queries

slide-56
SLIDE 56

ingraph

  • An incremental, in-memory graph query engine

ingraph client

register queries update graph

slide-57
SLIDE 57

ingraph

  • An incremental, in-memory graph query engine

ingraph client

register queries query results update graph

slide-58
SLIDE 58

ingraph

  • An incremental, in-memory graph query engine

ingraph client

register queries query results change notifications update graph

slide-59
SLIDE 59

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

  • penCypher

query

slide-60
SLIDE 60

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

  • penCypher

query Query syntax tree

slide-61
SLIDE 61

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

Query parser

  • penCypher

query Query syntax tree

slide-62
SLIDE 62

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

Query parser

  • penCypher

query Relational algebra model Query syntax tree

slide-63
SLIDE 63

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

Relational algebra builder Query parser

  • penCypher

query Relational algebra model Query syntax tree

slide-64
SLIDE 64

MATCH (t:Train)-[:ON]->(seg:Segment) <-[:STRAIGHT]-(sw:Switch) WHERE sw.position = 'diverging' RETURN t.number, sw

Relational algebra builder Query parser

  • penCypher

query Relational algebra model Query syntax tree

slide-65
SLIDE 65

Relational algebra model Rete network Rete network model

slide-66
SLIDE 66

Relational algebra model Rete network Rete network model Transformer and optimizer VIATRA

slide-67
SLIDE 67

Relational algebra model Rete network Rete network model Transformer and optimizer VIATRA

slide-68
SLIDE 68

Relational algebra model Rete network Rete network model Transformer and optimizer Query deployer VIATRA

slide-69
SLIDE 69

FORMALIZATION OF OPENCYPHER

slide-70
SLIDE 70

Relational algebra

  • Standard relational algebra
  • 𝜌, 𝜏
  • ∪,∩,∖
  • ×, ⋈
  • Common extensions
  • 𝜀 – duplicate elimination
  • 𝛿 – grouping
  • 𝜐 – sorting
slide-71
SLIDE 71

Graph-specific operators

Jürgen Hölsch, Michael Grossniklaus: An Algebra and Equivalences to Transform Graph Patterns in Neo4j, GraphQ 2016, EDBT, http://ceur-ws.org/Vol-1558/paper24.pdf

GetVertices: returns a graph relation containing all vertices of the underlying graph G Expand: return the neighbors of a given node

slide-72
SLIDE 72

Additional extensions

GetEdges: returns a graph relation containing all edges of the underlying graph G

Gábor Szárnyas, József Marton:

  • penCypher specification, Technical report

http://docs.inf.mit.bme.hu/ingraph/pub/

  • pencypher-report.pdf
slide-73
SLIDE 73

While pattern matching, Neo4j makes sure to not include matches where the same graph relationship is found multiple times in a single pattern.

Uniqueness of edges

All-different operator

slide-74
SLIDE 74

EXAMPLE QUERIES

slide-75
SLIDE 75

Uniqueness of edges

slide-76
SLIDE 76

Get neighbours

slide-77
SLIDE 77

Filter out based on node prop name

slide-78
SLIDE 78

Use multiple MATCH clauses to do a Cartesian product

slide-79
SLIDE 79

Two subgraphs

slide-80
SLIDE 80

SwitchMonitored

slide-81
SLIDE 81
slide-82
SLIDE 82

INCREMENTAL GRAPH QUERIES WITH OPENCYPHER

slide-83
SLIDE 83
  • penCypher constructs
  • Standard constructs
  • pattern matching
  • filtering
  • lists, maps
  • data manipulation
  • variable length paths
slide-84
SLIDE 84
  • penCypher constructs
  • Standard constructs
  • pattern matching
  • filtering
  • lists, maps
  • data manipulation
  • variable length paths
  • Legacy constructs
  • indexing, constraints
  • regular expressions
  • some list functions,

including reduce

  • most predicate functions
  • shortest path functions
  • CASE expressions
  • id()
slide-85
SLIDE 85
  • penCypher constructs
  • Standard constructs
  • pattern matching
  • filtering
  • lists, maps
  • data manipulation
  • variable length paths
  • Legacy constructs
  • indexing, constraints
  • regular expressions
  • some list functions,

including reduce

  • most predicate functions
  • shortest path functions
  • CASE expressions
  • id()

Difficult to handle incrementally

slide-86
SLIDE 86

Challenges for incremental openCypher

  • Lists
  • ['a', 1, 2, true]
  • ['a', [1, [2]], true]
  • UNWIND
  • Efficient aggregation
  • min(), max()
  • collect()
  • Bag semantics, ORDER BY, SKIP and LIMIT
  • Idea: collect(x ORDER BY x.name)
slide-87
SLIDE 87

Incremental queries – use cases

Standing queries on large & quickly changing graph

  • Runtime monitoring (train example)
  • Model validation: The Train Benchmark
  • Static analysis of JavaScript source code
  • Fraud detection
  • IT infrastructure monitoring
slide-88
SLIDE 88

Future work

Path operator

Giacomo Bergami, Matteo Magnani, and Danilo Montesi, A join operator for property graphs, GraphQ 2016, EDBT, http://jackbergus.alwaysdata.net/paper_graph_graphq2 017.pdf

Unwind operator: 𝜕

Elena Botoeva et al.: A Formal Presentation of MongoDB, https://arxiv.org/abs/1603.09291

slide-89
SLIDE 89

Open-Source Projects

Incremental Graph Engine:

https://github.com/ftsrg/ingraph

Train Benchmark:

https://github.com/ftsrg/trainbenchmark

BME-MODES3:

https://github.com/ftsrg/bme-modes3

Available under EPL v1.0.