The case against specialized graph engines Jing Fan, - PowerPoint PPT Presentation

The ¡case ¡against ¡specialized ¡ graph ¡engines ¡ Jing Fan, Adalbert Gerald Soosai Raj, and Jignesh M. Patel University of Wisconsin – Madison 01/06/2015 University of Wisconsin–Madison 1

Motivation ¡ • Graph analytics is now common • Response = new specialized graph engines The house always wins 01/06/2015 University of Wisconsin–Madison 2

Motivation ¡ • Graph analytics is now common • Response = new specialized graph engines graphs, 18% Stanford GPS others, 82% 01/06/2015 University of Wisconsin–Madison 3

Motivation ¡ • Graph analytics is now common • Response = new specialized graph engines Stanford GPS Question: Is graph processing that different from other types of data processing? Our Answer: No. Can be subsumed by “traditional” relational processing 01/06/2015 University of Wisconsin–Madison 4

What ¡is ¡appealing ¡about ¡these ¡new ¡engines? ¡ Easy to write Higher Vertex- graph programmer centric API programs productivity 01/06/2015 University of Wisconsin–Madison 5

Graph ¡API: ¡Giraph ¡ Vertex Centric: do { e 2 e 3 e 1 foreach vertex in the graph { V i receive_messages(); mutate_vertex_value(); New Val i Val i if (send_to_neighbors()) { send_messages_to_neighbors(); e 5 e 4 } } } until (has_converged() || reached_limit()) 01/06/2015 University of Wisconsin–Madison 6

Example: ¡Shortest ¡path ¡ Computation & Communication Pattern B 2 1 A D V B 3 2 \ C ∞ Input Graph V D V A ∞ 0 Iteration 1 V C ∞ 01/06/2015 University of Wisconsin–Madison 7

Example: ¡Shortest ¡path ¡ Computation & Communication Pattern B 2 1 A D V B 3 2 \ C 1 Input Graph ( D , 3 ) V D V D V A ∞ 3 ∞ 0 Iteration 2 V C 2 01/06/2015 University of Wisconsin–Madison 8

Example: ¡Shortest ¡path ¡ Computation & Communication Pattern B 2 1 A D V B 3 2 \ C 1 Input Graph V D V A 3 0 Iteration 3 V C 2 01/06/2015 University of Wisconsin–Madison 9

GraphLab ¡ 1. Gather values (from neighbors) e 2 e 3 e 1 2. Apply updates to local state Gather 3. Scatter signals to your neighbors V i New Val i Val i signal signal 01/06/2015 University of Wisconsin–Madison 10

What ¡is ¡appealing ¡about ¡these ¡new ¡engines? ¡ Easy to write Higher Vertex- graph programmer centric API programs productivity 01/06/2015 University of Wisconsin–Madison 11

But ¡… ¡ ¡ • Can we build a similar vertex-centric simple API? • … and then map it to SQL, with good performance The GRAIL API Advantages: • Already have SQL in the enterprise stack • Huge advantage to “one size fits many” • O(N 2 ) headaches when maintaining N specialized systems • Economies of scale 01/06/2015 University of Wisconsin–Madison 12

¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡Example: ¡Shortest ¡path ¡ B 2 1 A D 3 2 \ C Input Graph id val id val id val id val A 0 A 0 A 0 A ∞ next next next Vertex B 1 B ∞ B 1 B ∞ C 2 C 2 C ∞ C ∞ D 3 D ∞ D ∞ D ∞ message message message id val src dest val id val id val B 1 D 3 A B 1 Edge D 5 A C 2 C 2 B D 2 Iteration 3 Iteration 1 Iteration 2 C D 3 01/06/2015 University of Wisconsin–Madison 13

1 DECLARE @flag int; The Grail API 2 SET @flag = 1; 3 SELECT vertex.id, 2147483647 AS val Initialize 4 INTO next 5 FROM vertex; 1 VertexValType: INT 6 CREATE TABLE message( Initialize the message 2 MessageValType: INT 7 id int, 8 val int table 9 ); 3 InitiateVal : INT_MAX 10 INSERT INTO message values(1,0); 11 WHILE (@flag != 0) 4 InitialMessage : (1, 0) T-SQL Code 12 BEGIN 13 SELECT message.id AS id, MIN(message.val) AS val Aggregate the messages 14 INTO cur 5 CombineMessage: MIN(message) 15 FROM message 16 GROUP BY message.id; 6 UpdateAndSend: update=cur.val<getVal() 17 DROP TABLE message; 7 if (update) { 18 SELECT cur.id AS id, cur.val AS val 19 INTO update 20 FROM cur, next 8 setVal(cur.val) 21 WHERE cur.id = next.id AND cur.val < next.val; Update and generate 22 UPDATE next 9 send(out, cur.val+1) 23 SET next.val = update.val messages for the next 24 FROM update, next 25 WHERE next.id = update.id; 10 } iteration 26 SELECT edge.dest AS id, update.val + 1 AS val 11 End: NO_MESSAGE 27 INTO message 28 FROM update, edge 29 WHERE edge.src = update.id; 30 DROP TABLE cur; 31 DROP TABLE update; Stop when no new msgs. 32 SELECT @flag = COUNT(*) FROM message; 33 END 01/06/2015 University of Wisconsin–Madison 14

1 DECLARE @flag int; 2 SET @flag = 1; 3 SELECT vertex.id, 2147483647 AS val Initialize 4 INTO next 5 FROM vertex; 6 CREATE TABLE message( 7 id int, Initialize the message table 8 val int 9 ); 10 INSERT INTO message values(1,0); 11 WHILE (@flag != 0) T-SQL Code 12 BEGIN 13 SELECT message.id AS id, MIN(message.val) AS val Aggregate the messages 14 INTO cur 15 FROM message 16 GROUP BY message.id; 17 DROP TABLE message; Create an update table and 18 SELECT cur.id AS id, cur.val AS val 19 INTO update only consider updated vertices 20 FROM cur, next 21 WHERE cur.id = next.id AND cur.val < next.val; 22 UPDATE next Update the next table 23 SET next.val = update.val 24 FROM update, next 25 WHERE next.id = update.id; 26 SELECT edge.dest AS id, update.val + 1 AS val Generate the message table 27 INTO message 28 FROM update, edge for the next iteration 29 WHERE edge.src = update.id; 30 DROP TABLE cur; 31 DROP TABLE update; Stop when there are no new messages 32 SELECT @flag = COUNT(*) FROM message; 33 END 01/06/2015 University of Wisconsin–Madison 15

Aggregate function Scalar computation Scalar computation Join attributes (can be a UDAF) (can be a UDF) (can be a UDF) control the direction For single source shortest path min sum identity Outgoing edges 01/06/2015 University of Wisconsin–Madison 16

Grail: ¡Implementation ¡and ¡Evaluation ¡ Test Machine (single node) Query Results • Dual 1.8GHz Xeon E2450L • 96GB of main memory API Compare with Translator • Giraph (v.1.1.0) Grail • GraphLab (v 2.2): sync and async Dataset #nodes #edges size Optimizer web-google (GO) 9K 5M 71MB SQL Queries com-Orkut (OR) 3M 117M 1.6GB Twitter-10 (TW) 41.6M 1.5B 24GB uk-2007-05 (UK) 100M 3.3B 56GB SQL Server Queries Single source shortest-path • Page Rank • Weakly connect components • 01/06/2015 University of Wisconsin–Madison 17

Results: ¡Single ¡Source ¡Shortest ¡Path ¡ 1827 800 718 Execution time in seconds Grail 700 601 Giraph 600 GraphLab (sync) 500 GraphLab (async) 400 315 300 200 67.5 88 85 62 100 12.4 12.7 DNF DNF DNF DNF DNF 31 0 GO (71MB) OR (1.6GB) TW (24GB) UK (56GB) Grail is slower than GraphLab for the smallest datasets, … but catches up as the dataset size grows, … and can handle the largest datasets, while the other systems fail 01/06/2015 University of Wisconsin–Madison 18

Summary: ¡Graph ¡Analytics ¡on ¡RDBMS ¡ Simple API (Grail) addresses the programmer productivity issue Produces far more robust and deployable solutions than specialized graph engines Interesting physical schema design and optimization issues 01/06/2015 University of Wisconsin–Madison 19

Why ¡does ¡Mike ¡not ¡have ¡GraphDB ¡Inc.? ¡ The ¡general ¡case ¡against ¡GraphDB ¡Inc. ¡ 01/06/2015 University of Wisconsin–Madison 20

Thanks! David DeWitt Jae Young Do Alan Halverson Ian Rae 21