Coded MapReduce Mohammad Ali Maddah-Ali Bell Labs, Alcatel-Lucent - - PowerPoint PPT Presentation

Coded MapReduce

Mohammad Ali Maddah-Ali Bell Labs, Alcatel-Lucent

joint work with

Sonze Li (USC) and Salman Avestimehr (USC)

DIMACS

• Dec. 2015

Computing Communication Storage

The interaction among major components is the limiting barrier!

Infrastructure for big data

Computing Communication

Fundamental tradeoff between Computing and Communication

In this talk

Minimum communication for a specific computation task?

Computer Science (Yao 1979) Information Theory (Korner and Marton 1979)

Shortcomings:

• Problem oriented
• Does not scale

Need a framework that is

• General
• Scalable

Formulation

What does data companies are using? Challenge: right formulation

Hadoop Distributed File Systems (HDFS)

Communication Load Storage

Refer to Yesterdays’ Talks:

• Alexander Barg
• Alexander Dimakis

MapReduce

Storage Computation

Computation Load Communication Load

Map

1 1 1 2 2 2 1 3 5 2 4 6

Reduce Map

3 3 3 4 4 4 1 3 5 2 4 6

Reduce Map

5 5 5 6 6 6 1 3 5 2 4 6

Reduce

Input File 1 2 3 4 5 6 N Subfiles K Servers Q Keys

1

(Blue, , )

1

N Subfiles, K Servers, Q Keys Shuffling Phase

MapReduce: A General Framework

Intermediate (Key, Value)

# of A’s

Map

1 1 1 2 2 2

Map

3 3 3 4 4 4

Map

5 5 5 6 6 6 1 3 5 2 4 6

Reduce

1 3 5 2 4 6

Reduce

1 3 5 2 4 6

Reduce

A Books 1 2 3 4 5 6 N=6 Chapters K=3 Servers Q=3 Keys

Example: Word Counting

N Subfiles, K Servers, Q Keys

# of B’s # of C’s

1

(A, , )

1

Intermediate (Key, Value) Number of A’s in chapter one

Map

1 1 1 2 2 2 1 3 5 2 4 6

Reduce Map

3 3 3 4 4 4 1 3 5 2 4 6

Reduce Map

5 5 5 6 6 6 1 3 5 2 4 6

Reduce

1 2 3 4 4 6 N Subfiles, K Servers, Q Keys

MapReduce: A General Framework

General Framework

• Matrix Multiplication
• Distributed

Optimization

• Page Rank
• ….

Active Research Area: How to fit different jobs into this framework.

N=6 Subfiles, K=3 Servers, Q=3 Keys

1 2 3 4 5 6

1 1 1 2 2 2 1 3 5 2 4 6 3 3 3 4 5 4 1 3 5 2 4 6 5 5 5 6 6 6 1 3 5 2 4 6

Communication Load (MR)

Communication is a bottleneck!

1 2 1 2 3 4 3 4 5 6 5 6

Communication Load

Can we reduce communication load at the cost of computation?

1 2 3 4 5 6 1 2 3 4 1 2

1 3 5 2 4 6 1 3 5 2 4 6 1 3 5 2 4 6 1 1 1 2 2 2 3 3 3 4 4 4 3 3 3 4 5 4 1 1 1 2 2 2 5 5 5 6 6 6 1 1 1 2 2 2 3 4 5 6

Communication Load

N Subfiles, K Servers, Q Keys, Comp. Load r

1 2

• Comm. Load (Uncoded)

Locally available

Computation Load Communication Load

Communication Load

• Comm. Load (Map Reduce)
• Comm. Load (Uncoded)

Can we do better? Can we get a non-vanishing gain for large K?

N Subfiles, K Servers, Q Keys, Comp. Load r

2 6

3 4 5 6 5 6 1 2

1 1 1 2 2 2 1 3 5 2 4 6 3 3 3 4 4 4 3 3 3 4 4 4 1 3 5 2 4 6 5 5 5 6 6 6 5 5 5 6 6 6 1 3 5 2 4 6 1 1 1 2 2 2 1 3

⊕

5 4

⊕ ⊕

Coded MapReduce

Each Coded (key,value) pairs are useful for two servers

• Comm. Load (Coded)
• Comm. Load (Uncoded)

N Subfiles, K Servers, Q Keys, Comp. Load r

1 2 3 4

Computation Load Communication Load

Communication Load

• Comm. Load (Uncoded)
• Comm. Load (Map Reduce)
• Comm. Load (Coded)

N Subfiles, K Servers, Q Keys, Comp. Load r

Communication Load x Computation Load ~ constant

Objective: Each server can coded intermediate (Key, Value) pairs that are

useful for r other servers Need to assign the sub-files such that:

• for every subset S of r+1 servers,
• and for every subset T of S with r servers,
• Servers in T share an intermediate (Key, Value) pairs useful for server S\T

Proposed Scheme

⊕ ⊕ ⊕ S T N Subfiles, K Servers, Q Keys, Comp. Load r

• N sub-files: W1 , W2 , …, WN
• Each subset of size r of the servers takes a unique batch
• f subfiles.

Proposed Scheme

• Split the set of subfiles to batch of subfiles.

N Subfiles, K Servers, Q Keys, Comp. Load r

N=1200 Subfiles, K=10 Servers, Q=10 Keys

Coded MapReduce-Delay Profile

As soon as r copes of a mapping is done, kills that mapping on other servers. Map time duration: Exponential random variable

r=1 r=2 r=3 r=4 r=5 r=6 r=7

Connection with Coded Caching

A1# A2# A3# B1# B2# B3# C1# C2# C3# A1# B1# C1# A2# B2# C2# A3# B3# C3# A2 B1% A3 C1% B3 C2%

Maddah-Ali-Niessen, 2012 Ji-Caire-Molisch, 2014

• In coded caching, in placement phase, the demand of the each user is not known
• In coded MapReduce, in job assignment, the server which reduces a key is known!

Why it works!

Key Idea:

• When a subfile is assigned to a server, that server computes

all (key,value) pairs for that subfiles.

• This imposes a symmetry to the problem.

N Subfiles, K Servers, Q Keys, Comp. Load r

Can We Do Better?

The proposed scheme is optimum within a constant factor in rate.

Theorem:

• Comm. Load (Coded)

Outer Bound

N=3 Subfiles, K=3 Servers, Q=3 Keys, Comp. Load r

Server 1 Server 2 Server 3 Server 1 Server 2 Server 3 Server 1 Server 2 Server 3

Outer Bound

N=3 Subfiles, K=3 Servers, Q=3 Keys, Comp. Load r

Server 1 Server 2 Server 3 Server 1 Server 2 Server 3

Conclusion

• Communication-Computation tradeoff is of great interests and challenging
• Coded MapReduce provides a near optimal framework for trading

“computing” with “communication” in distributed computing

• Communication load x Computation load is approximately constant
• Many future directions:

– Impact of Coded MapReduce on the overall run-time of MapReduce – General server topologies – Applications to wireless distributed computing (“wireless Hadoop”)

• Papers available on arxiv.