One Trillion Edges: Graph Processing at Facebook-Scale GraphHPC - PowerPoint PPT Presentation

One Trillion Edges: Graph Processing at Facebook-Scale GraphHPC 2015, Moscow Avery Ching Sergey Edunov Maja Kabiljo Facebook Facebook Facebook Dionysios Logothetis Sambavi Muthukrishnan Facebook Facebook

Social Graph

Social Graph Example Question: Are Jay and Sambavi friends?

Ranking Features 7.6 9.3 6.4 8.2

Ranking Features Recommendations 7.6 9.3 6.4 8.2

Ranking Features Recommendations Data Partitioning 7.6 9.3 6.4 8.2

Benchmark Graphs Benchmark to Social Graphs Vertices Clueweb 09 Edges Twitter research Friendster Yahoo! web 0 1750 3500 5250 7000

Benchmark Graphs Benchmark to Social Graphs Vertices Clueweb 09 Edges Twitter research Friendster 70x larger than benchmarks! Yahoo! web 2015 Twitter Approx. 2015 Facebook Approx. 0 125000 1750 250000 3500 375000 5250 50000 7000

Requirements

Requirements Efficient iterative computing model • • Easy to program and debug graph-based API • Scale to real world Facebook graph sizes (1B+ nodes and hundreds of billions of edges) • Easily interoperable with existing data (Hive) • Run multiple jobs in a multi-tenant environment

Apache Giraph Maximum Vertex Example • Highly scalable graph processing engine loosely based on Pregel Combiners are used to aggregate message values • • Aggregators are global data generated on every superset

Apache Giraph Maximum Vertex Example Processor 1 5 Processor 2 1 2 Time • Highly scalable graph processing engine loosely based on Pregel Combiners are used to aggregate message values • • Aggregators are global data generated on every superset

Apache Giraph Maximum Vertex Example Processor 1 5 5 Processor 2 1 1 5 2 2 Time • Highly scalable graph processing engine loosely based on Pregel Combiners are used to aggregate message values • • Aggregators are global data generated on every superset

Apache Giraph Maximum Vertex Example Processor 1 5 5 5 Processor 2 1 5 1 5 2 2 2 5 Time • Highly scalable graph processing engine loosely based on Pregel Combiners are used to aggregate message values • • Aggregators are global data generated on every superset

Apache Giraph Maximum Vertex Example Processor 1 5 5 5 5 Processor 2 1 5 5 1 5 2 5 2 2 5 Time • Highly scalable graph processing engine loosely based on Pregel Combiners are used to aggregate message values • • Aggregators are global data generated on every superset

Pipelines Data Pipelines Framework Core Applications Analytics Execution Framework MapReduce (Scheduler) Storage HDFS

Architecture

Architecture Loading the graph Input Format Split 0 Worker 0 Load/Send Graph Split 1 Master Split 2 Worker 1 Load/Send Graph Split 3

Architecture Loading the graph Compute / Iterate Input Format In-memory graph Split 0 Part 0 Worker 0 Worker 0 Load/Send Compute/ Send Graph Messages Split 1 Part 1 Master Master Split 2 Part 2 Worker 1 Worker 1 Load/Send Compute/ Send Graph Messages Split 3 Part 3 Send stats/iterate!

Architecture Loading the graph Compute / Iterate Storing the graph Input Format In-memory graph Output Format Split 0 Part 0 Part 0 Part 0 Worker 0 Worker 0 Worker 0 Load/Send Compute/ Send Graph Messages Part 1 Split 1 Part 1 Part 1 Master Master Split 2 Part 2 Part 2 Part 2 Worker 1 Worker 1 Worker 0 Load/Send Compute/ Send Graph Messages Split 3 Part 3 Part 3 Part 3 Send stats/iterate!

Parallelization Model Compute / Iterate In-memory graph Part 0 Worker 0 Compute/ Send Messages Part 1 Master Part 2 Worker 1 Compute/ Send Messages Part 3 Send stats/iterate!

Parallelization Model Compute / Iterate Worker parallelization: Rely on scheduling In-memory graph framework for parallelization (e.g. Part 0 Worker 0 Compute/ Send more mappers) Messages Part 1 Master Part 2 Worker 1 Compute/ Send Messages Pros: Simple Part 3 Send stats/iterate!

Parallelization Model Pros: Fewer Compute / Iterate connections, better Worker memory usage (e.g. parallelization: shared message Rely on scheduling In-memory graph buffering) framework for parallelization (e.g. Part 0 Worker 0 Compute/ Send more mappers) Messages Part 1 Master Multithreading Part 2 Worker 1 parallelization: 
 Compute/ Send Messages Pros: Simple Multicore machines Part 3 leverage up to (partitions / worker) Send stats/iterate! threads


 
 
 
 
 
 Efficient Java Object Support Edges >> vertices (> 2 orders) • /** 
 * Interface for data structures that store out-edges for a vertex. 
 * 
 • Allow custom out edge * @param < I > Vertex id 
 * @param < E > Edge value 
 implementations */ 
 public interface OutEdges<I extends WritableComparable, E extends Writable> extends Iterable<Edge<I, E>>, Writable { 
 Example: Java primitive arrays, • void initialize(Iterable<Edge<I, E>> edges); 
 void initialize(int capacity); 
 FastUtil libraries void initialize(); 
 void add(Edge<I, E> edge); 
 • Serialize messages into large void remove(I targetVertexId); 
 byte arrays int size(); 
 }

Page Rank Map Reduce (Hadoop)

Page Rank Map Reduce (Hadoop)

Page Rank Map Reduce (Hadoop) Giraph public class PageRankComputation extends BasicComputation<LongWritable, DoubleWritable, FloatWritable, DoubleWritable> { public void compute( Vertex<LongWritable, DoubleWritable, FloatWritable> vertex, Iterable<DoubleWritable> messages) { // Calculate new page rank value if (getSuperstep() >= 1) { double sum = 0; for (DoubleWritable message : messages) { sum += message.get(); } vertex.getValue().set(0.15d / getTotalNumVertices() +0.85d * sum); } // Send page rank value to neighbors if (getSuperstep() < 30) { sendMsgToAllEdges(new DoubleWritable(getVertexValue().get() / getNumOutEdges())); } else { voteToHalt(); } } }

Page Rank Map Reduce (Hadoop) Giraph public class PageRankComputation extends BasicComputation<LongWritable, DoubleWritable, FloatWritable, DoubleWritable> { public void compute( Vertex<LongWritable, DoubleWritable, FloatWritable> vertex, Iterable<DoubleWritable> messages) { // Calculate new page rank value if (getSuperstep() >= 1) { double sum = 0; for (DoubleWritable message : messages) { sum += message.get(); } vertex.getValue().set(0.15d / getTotalNumVertices() +0.85d * sum); } // Send page rank value to neighbors if (getSuperstep() < 30) { sendMsgToAllEdges(new DoubleWritable(getVertexValue().get() / getNumOutEdges())); } else { voteToHalt(); } } }

Page Rank Map Reduce (Hadoop) Giraph public class PageRankComputation extends BasicComputation<LongWritable, DoubleWritable, FloatWritable, DoubleWritable> { public void compute( Vertex<LongWritable, DoubleWritable, FloatWritable> vertex, Iterable<DoubleWritable> messages) { // Calculate new page rank value if (getSuperstep() >= 1) { double sum = 0; for (DoubleWritable message : messages) { sum += message.get(); } vertex.getValue().set(0.15d / getTotalNumVertices() +0.85d * sum); } // Send page rank value to neighbors if (getSuperstep() < 30) { sendMsgToAllEdges(new DoubleWritable(getVertexValue().get() / getNumOutEdges())); } else { voteToHalt(); } } }

Pregel Extensions

Pregel Model Limitations

Pregel Model Limitations Difficult to construct “multi-stage” graph applications •

Pregel Model Limitations Difficult to construct “multi-stage” graph applications • • Hard to reuse code

Pregel Model Limitations Difficult to construct “multi-stage” graph applications • • Hard to reuse code

Extensions

Extensions • Make Computation first class object

Extensions • Make Computation first class object • Define computation on a master, worker, and a vertex

Extensions • Make Computation first class object • Define computation on a master, worker, and a vertex Master computation is executed centrally to set the • computation, combiner for the workers

Extensions • Make Computation first class object • Define computation on a master, worker, and a vertex Master computation is executed centrally to set the • computation, combiner for the workers • All computations are now composable and reusable

First Class Computation

First Class Computation class Vertex { public: virtual void Compute(MessageIterator* msgs) = 0; … };

First Class Computation class Vertex { public: … }; public interface Computation { void compute(Vertex<I, V, E> vertex, Iterable<M1> messages); … }