Mizan: A System for Dynamic Load Balancing in Large-scale Graph Processing Zuhair Khayyat 1 Karim Awara 1 Amani Alonazi 1 Hani Jamjoom 2 Dan Williams 2 Panos Kalnis 1 1 King Abdullah University of Science and Technology Thuwal, Saudi Arabia 2 IBM Watson Research Center Yorktown Heights, NY 1/34
Importance of Graphs Graphs abstract application-specific algorithms into generic problems represented as interactions using vertices and edges Max flow in road network Diameter of WWW Ranking in social networks Simulating protein interactions Algorithms vary in their computational requirements 2/34
How to Scale Graph Processing Pregel was introduced by Google as a scalable abstraction for graph processing Overcomes the limitations of processing graphs on MapReduce Based on vertex-centric computation Utilizes bulk synchronous parallel (BSP) System Programming Data Computations Abstraction Exchange Key based map() MapReduce grouping Disk based combine() reduce() compute() Message Pregel combine() passing In memory aggregate() 3/34
Pregel’s BSP Graph Processing Superstep 1 Superstep 2 Superstep 3 Worker 1 Worker 1 Worker 1 Worker 2 Worker 2 Worker 2 Worker 3 Worker 3 Worker 3 BSP Barrier Balanced computation and communication is fundamental to Pregel’s efficiency 4/34
Optimizing Graph Processing Existing work focus on optimizing for graph structure (static optimization): Optimize graph partitioning: Simple graph partitioning schemes (e.g., hash or range): Giraph User-defined partitioning function: Pregel Sophisticated partitioning techniques (e.g., min-cuts): GraphLab, PowerGraph, GoldenOrb and Surfer Optimize graph data access: Distributed data stores and graph indexing: GoldenOrb, Hama and Trinity 5/34
Optimizing Graph Processing Existing work focus on optimizing for graph structure (static optimization): Optimize graph partitioning: Simple graph partitioning schemes (e.g., hash or range): Giraph User-defined partitioning function: Pregel Sophisticated partitioning techniques (e.g., min-cuts): GraphLab, PowerGraph, GoldenOrb and Surfer Optimize graph data access: Distributed data stores and graph indexing: GoldenOrb, Hama and Trinity What about algorithm behavior? Pregel provides coarse-grained load balancing, is it enough? 5/34
Types of Algorithms Algorithms behaves differently, we classify algorithms in two categories depending on their behavior Algorithm Type In/Out Vertices Examples Messages State PageRank, Stationary Predictable Fixed Diameter estimation, WCC Distributed minimum spanning tree, Non-stationary Variable Variable Belief propagation, Graph queries, Ad propagation 6/34
Types of Algorithms – First Superstep V2 V8 V2 V8 V4 V1 V6 V4 V1 V6 V7 V7 V3 V5 V3 V5 PageRank DMST 7/34
Types of Algorithms – Superstep k V2 V8 V2 V8 V4 V1 V6 V4 V1 V6 V7 V3 V5 V7 V3 V5 PageRank DMST 8/34
Types of Algorithms – Superstep k + m V2 V8 V2 V8 V4 V1 V6 V4 V1 V6 V7 V3 V5 V7 V3 V5 PageRank DMST 9/34
What Causes Computation Imbalance in Non-stationary Algorithms? Superstep 1 Superstep 2 Superstep 3 Worker 1 Worker 1 Worker 1 Worker 2 Worker 2 Worker 2 Worker 3 Worker 3 Worker 3 BSP Barrier -Vertex response time -Time to receive in messages -Time to send out messages 10/34
Why to Optimize for Algorithm Behavior? Difference between stationary and non-stationary algorithms 1000 In Messages (Millions) 100 10 1 0.1 0.01 0.001 0 10 20 30 40 50 60 SuperSteps PageRank - T otal DMST - T otal PageRank - Max/W DMST - Max/W 11/34
What is Mizan? BSP-based graph processing framework Uses runtime fine-grained vertex migrations to balance computation and communication Follows the Pregel programming model Open source, written in C++ 12/34
PageRank’s compute() in Mizan void compute(messageIterator * messages, userVertexObject * data, messageManager * comm) { double currVal = data->getVertexValue().getValue(); double newVal = 0; double c = 0.85; if (data->getCurrentSS() > 1) { while (messages->hasNext()) { newVal = newVal + messages->getNext().getValue(); } newVal = newVal * c + (1.0 - c) / ((double) vertexTotal); data->setVertexValue(mDouble(newVal)); } else {newVal = currVal;} if (data->getCurrentSS() <= maxSuperStep) { mDouble outVal(newVal / ((double) data->getOutEdgeCount())); for (int i = 0; i < data->getOutEdgeCount(); i++) { comm->sendMessage(data->getOutEdgeID(i), outVal); } } else {data->voteToHalt();} } 13/34
Migration Planning Objectives Decentralized Simple Transparent No need to change Pregel’s API Does not assume any a priori knowledge to graph structure or algorithm 14/34
Mizan’s Migration Barrier Mizan performs both planning and migrations after all workers reach the BSP barrier Superstep 2 Superstep 1 Superstep 3 Worker 1 Worker 1 Worker 1 Worker 2 Worker 2 Worker 2 Worker 3 Worker 3 Worker 3 Migration Planner Migration Barrier BSP Barrier 15/34
Vertex Response Time Remote Outgoing Messages V2 V1 V6 V4 V3 V5 Mizan Mizan Worker 1 Worker 2 Local Incoming Messages Remote Incoming Messages Mizan’s Migration Planning Steps 1 Identify the source of imbalance: By comparing the worker’s execution time against a normal distribution and flagging outliers 16/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance: By comparing the worker’s execution time against a normal distribution and flagging outliers Mizan monitors for each vertex: Remote outgoing messages All incoming messages Response time High level summaries are broadcast to each worker Vertex Response Time Remote Outgoing Messages V2 V1 V6 V4 V3 V5 Mizan Mizan Worker 1 Worker 2 Local Incoming Messages Remote Incoming Messages 16/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance 2 Select the migration objective: Mizan finds the strongest cause of workload imbalance by comparing statistics for outgoing messages and incoming messages of all workers with the worker’s execution time 17/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance 2 Select the migration objective: Mizan finds the strongest cause of workload imbalance by comparing statistics for outgoing messages and incoming messages of all workers with the worker’s execution time The migration objective is either: Optimize for outgoing messages, or Optimize for incoming messages, or Optimize for response time 17/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance 2 Select the migration objective 3 Pair over-utilized workers with under-utilized ones: All workers create and execute the migration plan in parallel without centralized coordination Each worker is paired with one other worker at most W7 W2 W1 W5 W8 W4 W0 W6 W9 W3 1 0 2 3 4 5 6 7 8 18/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance 2 Select the migration objective 3 Pair over-utilized workers with under-utilized ones 4 Select vertices to migrate: Depending on the migration objective 19/34
Mizan’s Migration Planning Steps 1 Identify the source of imbalance 2 Select the migration objective 3 Pair over-utilized workers with under-utilized ones 4 Select vertices to migrate 5 Migrate vertices: How to migrate vertices freely across workers while maintaining vertex ownership and fast updates? How to minimize migration costs for large vertices? 20/34
Vertex Ownership Mizan uses distributed hash table (DHT) to implement a distributed lookup service: V can execute at any worker V ’s home worker ID = ( hash ( ID ) mod N ) Workers ask the home worker of V for its current location The home worker is notified with the new location as V migrates Where is V5 & V8 V1 V2 V8 V4 V6 V7 V8 Compute V5 V1 V3 V9 V3 V2 V5 V9 V1:W1 V4:W2 V7:W3 V2:W1 V5:W2 V8:W3 V3:W1 V6:W2 V9:W3 DHT Worker 3 Worker 1 Worker 2 21/34
Migrating Vertices with Large Message Size Introduce delayed migration for very large vertices: At SS t : only ownership of vertex v is moved to worker new Superstep t+2 Superstep t Superstep t+1 Worker_old Worker_old Worker_old 1 1 3 3 1 3 2 7 2 7 2 4 4 '7 4 7 5 6 5 6 5 6 Worker_new Worker_new Worker_new 22/34
Migrating Vertices with Large Message Size Introduce delayed migration for very large vertices: At SS t : only ownership of vertex v is moved to worker new At SS t + 1: worker new receives vertex 7 messages worker old process vertex 7 Superstep t Superstep t+1 Worker_old Worker_old Worker_old 1 3 1 3 1 3 2 2 7 2 7 4 4 '7 4 7 5 6 6 5 6 5 Worker_new Worker_new Worker_new 22/34
Migrating Vertices with Large Message Size Introduce delayed migration for very large vertices: At SS t : only ownership of vertex v is moved to worker new At SS t + 1: worker new receives vertex 7 messages worker old process vertex 7 After SS t + 1: worker old moves state of vertex 7 to worker new Superstep t Superstep t+1 Superstep t+2 Worker_old Worker_old Worker_old 1 3 1 3 3 1 2 2 7 2 7 4 4 '7 4 7 5 5 6 6 6 5 Worker_new Worker_new Worker_new 22/34
Recommend
More recommend
