[PPT] - Batch & Stream Graph Processing with Apache Flink Vasia PowerPoint Presentation

SLIDE 1

Batch & Stream Graph Processing with Apache Flink

Vasia Kalavri vasia@apache.org @vkalavri

SLIDE 2

SLIDE 3

Outline

Distributed Graph Processing
Gelly: Batch Graph Processing with Flink
Gelly-Stream: Continuous Graph Processing with Flink

SLIDE 4

WHEN DO YOU NEED DISTRIBUTED GRAPH PROCESSING?

SLIDE 5

MY GRAPH IS SO BIG, IT DOESN’T FIT IN A SINGLE MACHINE

Big Data Ninja

MISCONCEPTION #1

SLIDE 6

A SOCIAL NETWORK

SLIDE 7

YOUR INPUT DATASET SIZE IS _OFTEN_ IRRELEVANT

SLIDE 8

INTERMEDIATE DATA: THE OFTEN

▸ Naive Who(m) to Follow:

▸ compute a friends-of-friends

list per user

▸ exclude existing friends ▸ rank by common

connections

SLIDE 9

DISTRIBUTED PROCESSING IS ALWAYS FASTER THAN SINGLE-NODE

Data Science Rockstar

MISCONCEPTION #2

SLIDE 10

SLIDE 11

GRAPHS DON’T APPEAR OUT OF THIN AIR

Expectation…

SLIDE 12

GRAPHS DON’T APPEAR OUT OF THIN AIR

Reality!

SLIDE 13

HOW DO WE EXPRESS A DISTRIBUTED GRAPH ANALYSIS TASK?

SLIDE 14

GRAPH APPLICATIONS ARE DIVERSE

▸ Iterative value propagation

▸ PageRank, Connected Components, Label Propagation

▸ Traversals and path exploration

▸ Shortest paths, centrality measures

▸ Ego-network analysis

▸ Personalized recommendations

▸ Pattern mining

▸ Finding frequent subgraphs

SLIDE 15

LINEAR ALGEBRA

Adjacency Matrix

Partition by rows, columns, blocks
Efficient representation of non-zero elements
Algorithms expressed as vector-matrix multiplications

1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

1 5 4 3 2

SLIDE 16

BREADTH-FIRST SEARCH

1 5 4 3 2

1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

SLIDE 17

BREADTH-FIRST SEARCH

1 5 4 3 2

1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

SLIDE 18

BREADTH-FIRST SEARCH

1 5 4 3 2

1

X =

1 1 1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

SLIDE 19

BREADTH-FIRST SEARCH

1 5 4 3 2

1

X =

1 1 1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

SLIDE 20

BREADTH-FIRST SEARCH

1 5 4 3 2

1 1

X =

1 1 1 1 2 3 4 5 1 1 1 2 1 1 3 4 1 1 1 5 1

SLIDE 21

RECENT DISTRIBUTED GRAPH PROCESSING HISTORY

2004

MapReduce Pegasus

2009

Pregel

2010

Signal-Collect PowerGraph

2012 Iterative value propagation

Giraph++

2013 Graph Traversals

NScale

2014 Ego-network analysis

Arabesque

2015 Pattern Matching

Tinkerpop

SLIDE 22

PREGEL: THINK LIKE A VERTEX

1 5 4 3 2 1 3, 4 2 1, 4 5 3

. . .

SLIDE 23

PREGEL: SUPERSTEPS

(Vi+1, outbox) <— compute(Vi, inbox)

1

3,

2

1,

5

3

. .

1

3,

2

1,

5

3

. .

Superstep i Superstep i+1

SLIDE 24

PAGERANK: THE WORD COUNT OF GRAPH PROCESSING

VertexID Out-degree Transition Probability

1 2 1/2 2 2 1/2 3

4

3 1/3 5 1 1

1 5 4 3 2

SLIDE 25

PAGERANK: THE WORD COUNT OF GRAPH PROCESSING

VertexID Out-degree Transition Probability

1 2 1/2 2 2 1/2 3

4

3 1/3 5 1 1

1 5 4 3 2

PR(3) = 0.5*PR(1) + 0.33*PR(4) + PR(5)

SLIDE 26

PREGEL EXAMPLE: PAGERANK

void compute(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum) for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for

sum up received messages

update vertex rank

distribute rank to neighbors

SLIDE 27

SIGNAL-COLLECT

utbox <— signal(Vi)

1

3,

2

1,

5

3

. .

1

3,

2

1,

5

3

. .

Superstep i

Vi+1 <— collect(inbox)

1

3,

2

1,

5

3

. .

Signal Collect Superstep i+1

SLIDE 28

SIGNAL-COLLECT EXAMPLE: PAGERANK

void signal(): for (edge <- getOutEdges()) do sendMessageTo( edge.target(), getValue()/numEdges) end for void collect(messages): sum = 0.0 for (m <- messages) do sum = sum + m end for setValue(0.15/numVertices() + 0.85*sum)

sum up received messages update vertex rank distribute rank to neighbors

SLIDE 29

GATHER-SUM-APPLY (POWERGRAPH)

1

. . . . . .

Gather Sum

1 2 5

. . .

Apply

3 1 5 5 3 1

. . .

Gather

3 1 5 5 3

Superstep i Superstep i+1

SLIDE 30

GSA EXAMPLE: PAGERANK

double gather(source, edge, target): return target.value() / target.numEdges() double sum(rank1, rank2): return rank1 + rank2 double apply(sum, currentRank): return 0.15 + 0.85*sum

compute partial rank combine partial ranks update rank

SLIDE 31

PROBLEMS WITH VERTEX-CENTRIC MODELS

▸ Excessive communication ▸ Worker load imbalance ▸ Global Synchronization ▸ High memory requirements

▸ inbox /outbox can grow too large ▸ overhead for low-degree vertices in GSA

SLIDE 32

Vertex-Centric Connected Components

Propagate the minimum

value through the graph

In each superstep, the value

propagates one hop

Requires diameter + 1

supersets to converge

SLIDE 33

THINK LIKE A (SUB)GRAPH

1 5 4 3 2

compute() on the entire partition
Information flows freely inside

each partition

Network communication between

partitions, not vertices

1 5 4 3 2

SLIDE 34

Subgraph-Centric Connected Components

In each superstep, the value

propagates throughout each subgraph

Communication between

partitions only

Requires less (possibly)

supersteps to converge

SLIDE 35

RECENT DISTRIBUTED GRAPH PROCESSING HISTORY

2004

MapReduce Pegasus

2009

Pregel

2010

Signal-Collect PowerGraph

2012 Iterative value propagation

Giraph++

2013 Graph Traversals

NScale

2014 Ego-network analysis

Arabesque

2015 Pattern Matching

Tinkerpop

SLIDE 36

CAN WE HAVE IT ALL?

▸ Data pipeline integration: built on top of an

efficient distributed processing engine

▸ Graph ETL: high-level API with abstractions and

methods to transform graphs

▸ Familiar programming model: support popular

programming abstractions

SLIDE 37

Gelly

the Apache Flink Graph API

SLIDE 38

Flink Stack

Gelly Table ML SAMOA DataSet (Java/Scala) DataStream (Java/Scala)

Hadoop M/R

Local Remote Yarn Embedded Dataflow

Dataflow (WiP)

Table Cascading Streaming dataflow runtime CEP

SLIDE 39

Why Graph Processing with Apache Flink?

Native Iteration Operators
DataSet Optimizations
Ecosystem Integration

SLIDE 40

Flink Iteration Operators

Input

Iterative Update Function

Result Replace Workset

Iterative Update Function

Result Solution Set

State

SLIDE 41

Optimization

the runtime is aware of the iterative execution
no scheduling overhead between iterations
caching and state maintenance are handled automatically

Push work  “out of the loop” Maintain state as index Cache Loop-invariant Data

SLIDE 42

Beyond Iterations

Performance & Scalability
Memory management
Efficient serialization framework
Operations on binary data
Automatic Optimizations
Choose best execution strategy
Cache invariant data

SLIDE 43

Meet Gelly

Java & Scala Graph APIs on top of Flink
graph transformations and utilities
iterative graph processing
library of graph algorithms
Can be seamlessly mixed with the DataSet Flink

API to easily implement applications that use both record-based and graph-based analysis

SLIDE 44

Hello, Gelly!

ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<Edge<Long, NullValue>> edges = getEdgesDataSet(env); Graph<Long, Long, NullValue> graph = Graph.fromDataSet(edges, env); DataSet<Vertex<Long, Long>> verticesWithMinIds = graph.run( new ConnectedComponents(maxIterations)); val env = ExecutionEnvironment.getExecutionEnvironment val edges: DataSet[Edge[Long, NullValue]] = getEdgesDataSet(env) val graph = Graph.fromDataSet(edges, env) val components = graph.run(new ConnectedComponents(maxIterations))

Java Scala

SLIDE 45

Graph Methods

Graph Properties

getVertexIds getEdgeIds numberOfVertices numberOfEdges getDegrees ...

Transformations

map, filter, join subgraph, union, difference reverse, undirected getTriplets

Mutations

add vertex/edge remove vertex/edge

SLIDE 46

Example: mapVertices

// increment each vertex value by one  val graph = Graph.fromDataSet(...)    // increment each vertex value by one  val updatedGraph = graph.mapVertices(v => v.getValue + 1)

4 2 8 5 5 3 1 7 4 5

SLIDE 47

Example: subGraph

val graph: Graph[Long, Long, Long] = ...    // keep only vertices with positive values  // and only edges with negative values  val subGraph = graph.subgraph( vertex => vertex.getValue > 0, edge => edge.getValue < 0 )

SLIDE 48

Neighborhood Methods

Apply a reduce function to the 1st-hop

neighborhood of each vertex in parallel

graph.reduceOnNeighbors( new MinValue, EdgeDirection.OUT)

SLIDE 49

Iterative Graph Processing

Gelly offers iterative graph processing abstractions
n top of Flink’s Delta iterations
vertex-centric
scatter-gather
gather-sum-apply
partition-centric*

SLIDE 50

Vertex-Centric SSSP

final class SSSPComputeFunction extends ComputeFunction {

verride def compute(vertex: Vertex, messages: MessageIterator) = {

var minDistance = if (vertex.getId == srcId) 0 else Double.MaxValue while (messages.hasNext) { val msg = messages.next if (msg < minDistance) minDistance = msg } if (vertex.getValue > minDistance) { setNewVertexValue(minDistance) for (edge: Edge <- getEdges) sendMessageTo(edge.getTarget, vertex.getValue + edge.getValue) }

SLIDE 51

Library of Algorithms

PageRank*
Single Source Shortest Paths*
Label Propagation
Weakly Connected Components*
Community Detection
Triangle Count & Enumeration
Clustering Coefficient
Jaccard & Adamic-Adar Similarity
Graph Summarization
val ranks = inputGraph.run(new PageRank(0.85, 20))
*: both scatter-gather and GSA implementations

SLIDE 52

Gelly-Stream

single-pass stream graph processing with Flink

SLIDE 53

Real Graphs are dynamic

Graphs are created from events happening in real-time

SLIDE 54

SLIDE 55

How we’ve done graph processing so far

1. Load: read the graph

from disk and partition it in memory

SLIDE 56

2. Compute: read and

mutate the graph state

How we’ve done graph processing so far

1. Load: read the graph

from disk and partition it in memory

SLIDE 57

3. Store: write the final

graph state back to disk

How we’ve done graph processing so far

2. Compute: read and

mutate the graph state

1. Load: read the graph

from disk and partition it in memory

SLIDE 58

What’s wrong with this model?

It is slow
wait until the computation is over before you see

any result

pre-processing and partitioning
It is expensive
lots of memory and CPU required in order to

scale

It requires re-computation for graph changes
no efficient way to deal with updates

SLIDE 59

Graph Streaming Challenges

Maintain the

dynamic graph structure

Provide up-to-date

results with low latency

Compute on fresh

state only

SLIDE 60

Single-Pass Graph Streaming

Each event is an edge addition
Maintain only a graph summary
Recent events are grouped in graph

windows

SLIDE 61

SLIDE 62

Graph Summaries

spanners for distance estimation
sparsifiers for cut estimation
sketches for homomorphic properties

graph summary algorithm algorithm

~

R1 R2

SLIDE 63

1 4 3 2 5

i=0

Batch Connected Components

6 7 8

SLIDE 64

1 4 3 2 5 6 7 8

i=0

Batch Connected Components

1 4 3 4 5 2 3 5 2 4 7 8 6 7 6 8

SLIDE 65

1 2 1 2 2

i=1

Batch Connected Components

6 6 6

SLIDE 66

1 2 1 1 2 6 6 6

i=1

Batch Connected Components

2 1 2 2 1 1 2 1 2 7 6 6 6

SLIDE 67

1 1 1 1 1

i=2

Batch Connected Components

6 6 6

SLIDE 68

5 4 7 6 8 6 4 2 3 1 5 2

Stream Connected Components

Graph Summary: Disjoint Set (Union-Find)

Only store component IDs

and vertex IDs

SLIDE 69

5 4 7 6 8 6 4 2 4 3 3 1 5 2 1 3

Cid = 1

SLIDE 70

5 4 7 6 8 6 4 2 4 3 8 7 3 1 5 2 1 3

Cid = 1

2 5

Cid = 2

SLIDE 71

5 4 7 6 8 6 4 2 4 3 8 7 4 1 3 1 5 2 1 3

Cid = 1

2 5

Cid = 2

4

SLIDE 72

5 4 7 6 8 6 4 2 4 3 8 7 4 1 3 1 5 2 1 3

Cid = 1

2 5

Cid = 2

4 6 7

Cid = 6

SLIDE 73

5 4 7 6 8 6 4 2 4 3 8 7 4 1 3 1 5 2 1 3

Cid = 1

2 5

Cid = 2

4 6 7

Cid = 6

8

SLIDE 74

5 4 7 6 8 6 4 2 4 3 8 7 4 1 3 1 5 2 1 3

Cid = 1

2 5

Cid = 2

4 6 7

Cid = 6

8

SLIDE 75

5 4 7 6 8 6 4 2 4 3 8 7 4 1 5 2 6 7

Cid = 6

8 1 3

Cid = 1

2 5

Cid = 2

4

SLIDE 76

5 4 7 6 8 6 4 2 4 3 8 7 4 1 5 2 1 3

Cid = 1

2 5 4 6 7

Cid = 6

8

SLIDE 77

Distributed Stream Connected Components

SLIDE 78