Improved Computation-Communication Trade-Off for Coded Distributed - - PowerPoint PPT Presentation

▶
improved computation communication trade off for coded
SMART_READER_LITE
LIVE PREVIEW

Improved Computation-Communication Trade-Off for Coded Distributed - - PowerPoint PPT Presentation

Jun 18, 2023 133 likes •408 views

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

slide-1
SLIDE 1

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Shunsuke Horii

Waseda University

June 5, 2020

Shunsuke Horii | Waseda University | June 5, 2020 1 / 26

slide-2
SLIDE 2

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Background

The amount of data processed by machine learning/statistical analysis has been increasing Too large to process with a single computer/processor Distributed computing is becoming increasingly important

Shunsuke Horii | Waseda University | June 5, 2020 2 / 26

slide-3
SLIDE 3

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

A bottleneck in distributed computing

Map-Reduce

Input Node 1 Node K Node 2 Node 1 Node K Node 2 Intermediate file Output

Shunsuke Horii | Waseda University | June 5, 2020 3 / 26

slide-4
SLIDE 4

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

A bottleneck in distributed computing

Map-Reduce

Input Node 1 Node K Node 2 Node 1 Node K Node 2 Intermediate file Output

The more nodes, the more traffic

Shunsuke Horii | Waseda University | June 5, 2020 4 / 26

slide-5
SLIDE 5

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Contribution

There is a trade-off relationship between the computational load of each node and the amount of communication among the nodes Coded Map-Reduce: A framework to improve the trade-off curve by utilizing coding technique [S. Li et al. 2016]

Contributions of our work

Utilizing linear dependence of intermediate files to further improve the trade-off curve

Shunsuke Horii | Waseda University | June 5, 2020 5 / 26

slide-6
SLIDE 6

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Map-Reduce

input files: w1, . . . , wN ∈ F2F

  • utput functions: φ1, . . . , φQ : (F2F )N → F2B, q ∈ {1, . . . , Q}

number of nodes: K

Input files

!!" # " !" $! % &!'!!" # " !"( $# % &#'!!" # " !"(

! computing nodes

compute

Shunsuke Horii | Waseda University | June 5, 2020 6 / 26

slide-7
SLIDE 7

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Map-Reduce

Map functions: gq,n : F2F → (F2T ), q ∈ {1, . . . , Q} , n ∈ {1, . . . , N}

Intermediate values: vq,n = gq,n(wn) ∈ F2T

Ruduce functions: hq : (F2T )N → F2B

Output values: uq = hq(vq,1, . . . , vq,N)

! "!

!! !" "!#! # $!#!%!!& "$#! # $$#!%!!& "!#" # $!#"%!"& "$#" # $$#"%!"&

#! #" ! "#

'! # (! "!#!) * ) "!#" +# ,!%!!) * ) !"& '$ # ($ "$#!) * ) "$#" +# ,$%!!) * ) !"& Map functions Reduce functions

Shunsuke Horii | Waseda University | June 5, 2020 7 / 26

slide-8
SLIDE 8

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Coded Map-Reduce

Coding: Not used, Communication: Not used

Map Files 1

has needs

Node 1 Node 3 Node 2 Node 4 1 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 Map Files 1

has needs

1 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 Map Files 1

has needs

1 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 Map Files 1

has needs

1 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6 2 3 4 5 6 1 2 3 4 5 6

computation load: K

k=1 |Mk|

N = 24 6 communication load: K

k=1 bk

QNT = 0

Shunsuke Horii | Waseda University | June 5, 2020 8 / 26

slide-9
SLIDE 9

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Coded Map-Reduce

Coding: Not used, Communication: Used

Map Files 1

has needs

Node 1 Node 3 Node 2 Node 4 1 1 2 3 2 3 1 2 3 2 3 1 2 3 Map Files 1

has needs

1 4 5 4 5 2 3 6 1 4 5 1 4 5 1 4 5 Map Files 2

has needs

2 2 4 6 4 6 2 4 6 4 6 2 4 6 Map Files

has needs

3 5 6 3 5 6 3 5 6 3 5 6 3 5 6 4 5 6 1 3 5 1 2 4

sends sends sends sends

1 2 1 4 5 4 6 6 2 5 3 3

computation load: K

k=1 |Mk|

N = 12 6 communication load: K

k=1 bk

QNT = 12 24

Shunsuke Horii | Waseda University | June 5, 2020 9 / 26

slide-10
SLIDE 10

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Coded Map-Reduce

Coding: Used, Communication: Used [S. Li, et al. 2017]

Map Files 1

has needs

Node 1 Node 3 Node 2 Node 4 1 1 2 3 2 3 1 2 3 2 3 1 2 3 Map Files 1

has needs

1 4 5 4 5 2 3 6 1 4 5 1 4 5 1 4 5 Map Files 2

has needs

2 2 4 6 4 6 2 4 6 4 6 2 4 6 Map Files

has needs

3 5 6 3 5 6 3 5 6 3 5 6 3 5 6 4 5 6 1 3 5 1 2 4

sends sends sends sends

2 ! 1 3 ! 1 3 ! 2

! ! !

4 1 5 1 5 4

!

4 2 6 ! 2 4 ! 6

!

5 3

! !

6 3 6 5

computation load: K

k=1 |Mk|

N = 12 6 communication load: K

k=1 bk

QNT = 6 24

Shunsuke Horii | Waseda University | June 5, 2020 10 / 26

slide-11
SLIDE 11

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Coded Map-Reduce

L∗ CDC(r): The communication load of the coded Map-Reduce for the computation load r

Theorem [Li et al. 2017]

If r is an integer, L∗ CDC(r) is given by L∗ CDC(r) = 1 r

  • 1 − r

K

  • .

(1) When r is not an integer, it is given by the lower convex envelope of

  • r, 1

r

  • 1 − r

K

  • : r ∈ {1, . . . , K}
  • .

Shunsuke Horii | Waseda University | June 5, 2020 11 / 26

slide-12
SLIDE 12

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Coded Map-Reduce

The trade-off curves between computation load and communication load

2 4 6 8 10 computation load 0.0 0.2 0.4 0.6 0.8 communication load uncoded coded

Shunsuke Horii | Waseda University | June 5, 2020 12 / 26

slide-13
SLIDE 13

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Motivation

Previous studies have not made any assumptions about the intermediate values (outputs

  • f the Map functions).

In some cases, they have some structures, and the trade-off curve may be further improved by utilizing this structure. We focus on the linear dependency of the intermediate values.

Map Files 1 1 1 2 3 2 3 1 2 3 2 3 1 2 3

linearly dependent

Shunsuke Horii | Waseda University | June 5, 2020 13 / 26

slide-14
SLIDE 14

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

A problem to count the number of appearance of the numbers in a sequence 1212231

  • w1

2111121

  • w2

2312131

  • w3

3112132

  • w4

1131414

  • w5

1141231

  • w6

Map Files !!

has needs

Node 1 1 1 2 3 !" !# 1 2 3 2 3 1 2 3 4 5 6

sends

2 ! 1 3 ! 1 3 ! 2 3 ! " 3 ! " 1 ! 2 ! #

Shunsuke Horii | Waseda University | June 5, 2020 14 / 26

slide-15
SLIDE 15

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

A problem to count the number of appearance of the numbers in a sequence 1212231

  • w1

2111121

  • w2

2312131

  • w3

3112132

  • w4

1131414

  • w5

1141231

  • w6

Map Files !!

has needs

Node 1 1 1 2 3 !" !# 1 2 3 2 3 1 2 3 4 5 6

sends

2 ! 1 3 ! 1 3 ! 2 3 ! " 3 ! " 1 ! 2 ! # 3 ! 1 ! 3 ! 2

Shunsuke Horii | Waseda University | June 5, 2020 15 / 26

slide-16
SLIDE 16

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

A problem to count the number of appearance of the numbers in a sequence 1212231

  • w1

2111121

  • w2

2312131

  • w3

3112132

  • w4

1131414

  • w5

1141231

  • w6

Map Files !!

has needs

Node 1 1 1 2 3 !" !# 1 2 3 2 3 1 2 3 4 5 6

sends

2 ! 1 3 ! 1 3 ! 2 3 ! " 3 ! " 1 ! 2 ! # 3 ! 1 ! 3 ! 2

Expressed as two basis vectors and linear combination coefficients

Shunsuke Horii | Waseda University | June 5, 2020 16 / 26

slide-17
SLIDE 17

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

A problem to compute Ax1, . . . , AxN by Map-Reduce

! " # $!

"#$

%% %& %' &

Shunsuke Horii | Waseda University | June 5, 2020 17 / 26

slide-18
SLIDE 18

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

A problem to compute Ax1, . . . , AxN by Map-Reduce A is divided by row into submatrices {Ak : k = 1, . . . , K}

! "! # $"

# $ %&

%! %' %( & "' # $"

! " %&

' "$ # $"

! " %&

Shunsuke Horii | Waseda University | June 5, 2020 18 / 26

slide-19
SLIDE 19

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Illustrative example

Coded messages sent by a node are K

r+1

  • vectors of length m

K

The rank of the subspace constructed from these messages is smaller than the number of messages K

r+1

  • for some cases

! " #

Intermediate values

" $ % $ %

Coded message

" $ % &'( % ) * +

Shunsuke Horii | Waseda University | June 5, 2020 19 / 26

slide-20
SLIDE 20

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Proposed scheme

The way to construct the messages sent by a node

1 Compute the intermediate values 2 Construct coded messages 3 Compute the basis of the subspace spanned by coded messages 4 Send the basis and linear combination coefficients Map Files

! ! ! ! ! !

! ! !

!

Coded message

!

Shunsuke Horii | Waseda University | June 5, 2020 20 / 26

slide-21
SLIDE 21

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Proposed scheme

Theorem

If r is an integer, LCDC-LD(r, T, ρ) is given by LCDC-LD(r, T, ρ) =

  • 1

r K

r

+ K K−1

r

  • QNT
  • ρ

(2) r : computation load T : bit size required to represent intermediate values ρ = 1 K

K

  • k=1

ρk, where ρk is the rank of the subspace of node k

Shunsuke Horii | Waseda University | June 5, 2020 21 / 26

slide-22
SLIDE 22

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Proposed scheme

K = 4, N = 6, Q = 4, r = 2

  • T
  • ρℓ = 2

ρℓ = 1

Shunsuke Horii | Waseda University | June 5, 2020 22 / 26

slide-23
SLIDE 23

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Proposed scheme

K = 10, N = 2520, Q = 360, T = 64

  • r
  • ρℓ = 10

ρℓ = 5

Shunsuke Horii | Waseda University | June 5, 2020 23 / 26

slide-24
SLIDE 24

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Related works

Compressed coded distributed computing [S. Li et al. 2018]

Map Files 1

has needs

Node 1 Node 3 Node 2 1 1 2 3 2 3 1 2 3 2 3 Files 5 6

sends

! !

4 4 4 4 Map 3 3 3 4 5 4 5 3 4 5 4 5 6 6 6 6

has needs

!

1 2 Files Map 1 1 1 2 5 2 5 1 2 5 2 5 6 6 6 6

has needs

!

3 4

" 1 ! 2

3 4

! "

3 4

! " ! " 5

6

sends

! ! " 5

6

" 1 ! 2 !

sends

It utilizes the structure

  • f reduce functions.

Our method utilizes the structure of map functions.

Shunsuke Horii | Waseda University | June 5, 2020 24 / 26

slide-25
SLIDE 25

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Discussion

Our method compresses the coded messages. Ideally, they could be compressed to its entropy. However, a good compression method requires a large computational complexity. ⇒ a new trade-off between computation load and communication load

Map Files

! ! ! ! ! !

! ! !

!

Coded message

!

Compressed coded message

Shunsuke Horii | Waseda University | June 5, 2020 25 / 26

slide-26
SLIDE 26

Improved Computation-Communication Trade-Off for Coded Distributed Computing using Linear Dependence of Intermediate Values

Conclusion

Conclusion

We have developed a new coded distributed computing (CDC) scheme that improves the computation-communication trade-off. The central idea is that the messages constructed in the CDC scheme have linear dependency for some applications.

Future works

Detailed analysis of the additional computational complexity of the proposed method for specific applications Use of other compression methods

Shunsuke Horii | Waseda University | June 5, 2020 26 / 26