C O C O A Communication-Efficient Coordinate Ascent Virginia Smith - - PowerPoint PPT Presentation

COCOA Communication-Efficient Coordinate Ascent

Virginia Smith

Martin Jaggi, Martin Takáč, Jonathan Terhorst,
 Sanjay Krishnan, Thomas Hofmann, & Michael I. Jordan

LARGE-SCALE OPTIMIZATION

LARGE-SCALE OPTIMIZATION COCOA

LARGE-SCALE OPTIMIZATION COCOA RESULTS

LARGE-SCALE OPTIMIZATION COCOA RESULTS

Machine Learning with Large Datasets

Machine Learning with Large Datasets

Machine Learning with Large Datasets

image/music/video tagging document categorization item recommendation click-through rate prediction sequence tagging protein structure prediction sensor data prediction spam classification fraud detection

Machine Learning Workflow

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

MACHINE LEARNING MODEL

logistic regression, lasso, support vector machines, …

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

MACHINE LEARNING MODEL

logistic regression, lasso, support vector machines, …

OPTIMIZATION ALGORITHM

gradient descent, coordinate descent, Newton’s method, …

Example: SVM Classification

Example: SVM Classification

Example: SVM Classification

Example: SVM Classification

Example: SVM Classification

Example: SVM Classification

Example: SVM Classification

w x

• b

= 1 w x

• b

=

• 1

w x

• b

= w 2 / ||w||

Example: SVM Classification

w x

• b

= 1 w x

• b

=

• 1

w x

• b

= w 2 / ||w||

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`hinge(yiwT xi)

Example: SVM Classification

w x

• b

= 1 w x

• b

=

• 1

w x

• b

= w 2 / ||w||

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`hinge(yiwT xi)

Descent algorithms and line search methods Acceleration, momentum, and conjugate gradients Newton and Quasi-Newton methods Coordinate descent Stochastic and incremental gradient methods SMO SVMlight LIBLINEAR

Linear Regularized Loss Minimization

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi)

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines logistic regression

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines logistic regression lasso regression

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines logistic regression lasso regression ridge regression

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines logistic regression lasso regression ridge regression etc…

Linear Regularized Loss Minimization

min

w∈Rd

• 2 ||w||2 + 1

n

n

X

i=1

`i(wT xi) support vector machines logistic regression lasso regression ridge regression etc…

image/music/video tagging document categorization item recommendation click-through rate prediction sequence tagging protein structure prediction sensor data prediction spam classification fraud detection

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

MACHINE LEARNING MODEL

logistic regression, lasso, support vector machines, …

OPTIMIZATION ALGORITHM

gradient descent, coordinate descent, Newton’s method, …

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

MACHINE LEARNING MODEL

logistic regression, lasso, support vector machines, …

OPTIMIZATION ALGORITHM

gradient descent, coordinate descent, Newton’s method, …

SYSTEMS SETTING

multi-core, cluster, cloud, supercomputer, …

Machine Learning Workflow

DATA & PROBLEM

classification, regression, collaborative filtering, …

MACHINE LEARNING MODEL

logistic regression, lasso, support vector machines, …

OPTIMIZATION ALGORITHM

gradient descent, coordinate descent, Newton’s method, …

SYSTEMS SETTING

multi-core, cluster, cloud, supercomputer, …

Open Problem: efficiently solving objective when data is distributed

Distributed Optimization

Distributed Optimization

Distributed Optimization

reduce: w = w − α P

k ∆w

Distributed Optimization

reduce: w = w − α P

k ∆w

Distributed Optimization

“always communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees “always communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication “always communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication “always communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication

average: w := 1

K

P

k wk

“always communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication

average: w := 1

K

P

k wk

“always communicate” “never communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication

average: w := 1

K

P

k wk

“always communicate” “never communicate”

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Distributed Optimization

✔ convergence 
 guarantees ✗ high communication ✔ low communication ✗ convergence 
 not guaranteed

average: w := 1

K

P

k wk

“always communicate” “never communicate” ZDWJ, 2012

reduce: w = w − α P

k ∆w

Mini-batch

Mini-batch

Mini-batch

reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch

reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch

✔ convergence guarantees reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch

✔ convergence guarantees ✔ tunable communication reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch

✔ convergence guarantees ✔ tunable communication a natural middle-ground reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch

✔ convergence guarantees ✔ tunable communication a natural middle-ground reduce: w = w − α

|b|

P

i∈b ∆w

Mini-batch Limitations

Mini-batch Limitations

• 1. METHODS BEYOND SGD

Mini-batch Limitations

• 1. METHODS BEYOND SGD
• 2. STALE UPDATES

Mini-batch Limitations

• 1. METHODS BEYOND SGD
• 2. STALE UPDATES
• 3. AVERAGE OVER BATCH SIZE

LARGE-SCALE OPTIMIZATION COCOA RESULTS

LARGE-SCALE OPTIMIZATION COCOA RESULTS

Mini-batch Limitations

• 1. METHODS BEYOND SGD
• 2. STALE UPDATES
• 3. AVERAGE OVER BATCH SIZE

Mini-batch Limitations

• 1. METHODS BEYOND SGD


Use Primal-Dual Framework

• 2. STALE UPDATES 


Immediately apply local updates

• 3. AVERAGE OVER BATCH SIZE


Average over K << batch size

• 1. METHODS BEYOND SGD


Use Primal-Dual Framework

• 2. STALE UPDATES 


Immediately apply local updates

• 3. AVERAGE OVER BATCH SIZE


Average over K << batch size

Communication-Efficient Distributed Dual Coordinate Ascent (CoCoA)

• 1. Primal-Dual Framework

PRIMAL DUAL

≥

• 1. Primal-Dual Framework

PRIMAL DUAL

≥

min

w∈Rd

" P(w) := 2 ||w||2 + 1 n

n

X

i=1

`i(wT xi) #

• 1. Primal-Dual Framework

PRIMAL DUAL

≥

min

w∈Rd

" P(w) := 2 ||w||2 + 1 n

n

X

i=1

`i(wT xi) # max

α∈Rn

" D(↵) := −||A↵||2 − 1 n

n

X

i=1

`∗

i (−↵i)

# Ai = 1 λnxi

• 1. Primal-Dual Framework

PRIMAL DUAL Stopping criteria given by duality gap

≥

min

w∈Rd

" P(w) := 2 ||w||2 + 1 n

n

X

i=1

`i(wT xi) # max

α∈Rn

" D(↵) := −||A↵||2 − 1 n

n

X

i=1

`∗

i (−↵i)

# Ai = 1 λnxi

• 1. Primal-Dual Framework

PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice

≥

min

w∈Rd

" P(w) := 2 ||w||2 + 1 n

n

X

i=1

`i(wT xi) # max

α∈Rn

" D(↵) := −||A↵||2 − 1 n

n

X

i=1

`∗

i (−↵i)

# Ai = 1 λnxi

• 1. Primal-Dual Framework

PRIMAL DUAL Stopping criteria given by duality gap Good performance in practice Default in software packages e.g. liblinear

≥

min

w∈Rd

" P(w) := 2 ||w||2 + 1 n

n

X

i=1

`i(wT xi) # max

α∈Rn

" D(↵) := −||A↵||2 − 1 n

n

X

i=1

`∗

i (−↵i)

# Ai = 1 λnxi

• 2. Immediately Apply Updates

• 2. Immediately Apply Updates

for i 2 b ∆w ∆w αriP(w) end w w + ∆w

STALE

• 2. Immediately Apply Updates

for i 2 b ∆w ∆w αriP(w) end w w + ∆w

STALE for i 2 b ∆w ∆w αriP(w) w w + ∆w end FRESH

• 3. Average over K

• 3. Average over K

reduce: w = w + 1

K

P

k ∆wk

CoCoA

Algorithm 1: CoCoA Input: T ≥ 1, scaling parameter 1 ≤ βK ≤ K (default: βK := 1). Data: {(xi, yi)}n

i=1 distributed over K machines

Initialize: α(0)

[k] ← 0 for all machines k, and w(0) ← 0

for t = 1, 2, . . . , T for all machines k = 1, 2, . . . , K in parallel (∆α[k], ∆wk) ← LocalDualMethod(α(t−1)

[k]

, w(t−1)) α(t)

[k] ← α(t−1) [k]

+ βK

K ∆α[k]

end reduce w(t) ← w(t−1) + βK

K

PK

k=1 ∆wk

end

Procedure A: LocalDualMethod: Dual algorithm on machine k Input: Local α[k] ∈ Rnk, and w ∈ Rd consistent with other coordinate blocks of α s.t. w = Aα Data: Local {(xi, yi)}nk

i=1

Output: ∆α[k] and ∆w := A[k]∆α[k]

CoCoA

Algorithm 1: CoCoA Input: T ≥ 1, scaling parameter 1 ≤ βK ≤ K (default: βK := 1). Data: {(xi, yi)}n

i=1 distributed over K machines

Initialize: α(0)

[k] ← 0 for all machines k, and w(0) ← 0

for t = 1, 2, . . . , T for all machines k = 1, 2, . . . , K in parallel (∆α[k], ∆wk) ← LocalDualMethod(α(t−1)

[k]

, w(t−1)) α(t)

[k] ← α(t−1) [k]

+ βK

K ∆α[k]

end reduce w(t) ← w(t−1) + βK

K

PK

k=1 ∆wk

end

Procedure A: LocalDualMethod: Dual algorithm on machine k Input: Local α[k] ∈ Rnk, and w ∈ Rd consistent with other coordinate blocks of α s.t. w = Aα Data: Local {(xi, yi)}nk

i=1

Output: ∆α[k] and ∆w := A[k]∆α[k]

✔ <10 lines of code in Spark

CoCoA

Algorithm 1: CoCoA Input: T ≥ 1, scaling parameter 1 ≤ βK ≤ K (default: βK := 1). Data: {(xi, yi)}n

i=1 distributed over K machines

Initialize: α(0)

[k] ← 0 for all machines k, and w(0) ← 0

for t = 1, 2, . . . , T for all machines k = 1, 2, . . . , K in parallel (∆α[k], ∆wk) ← LocalDualMethod(α(t−1)

[k]

, w(t−1)) α(t)

[k] ← α(t−1) [k]

+ βK

K ∆α[k]

end reduce w(t) ← w(t−1) + βK

K

PK

k=1 ∆wk

end

Procedure A: LocalDualMethod: Dual algorithm on machine k Input: Local α[k] ∈ Rnk, and w ∈ Rd consistent with other coordinate blocks of α s.t. w = Aα Data: Local {(xi, yi)}nk

i=1

Output: ∆α[k] and ∆w := A[k]∆α[k]

✔ <10 lines of code in Spark ✔ primal-dual framework allows for any internal optimization method

CoCoA

Algorithm 1: CoCoA Input: T ≥ 1, scaling parameter 1 ≤ βK ≤ K (default: βK := 1). Data: {(xi, yi)}n

i=1 distributed over K machines

Initialize: α(0)

[k] ← 0 for all machines k, and w(0) ← 0

for t = 1, 2, . . . , T for all machines k = 1, 2, . . . , K in parallel (∆α[k], ∆wk) ← LocalDualMethod(α(t−1)

[k]

, w(t−1)) α(t)

[k] ← α(t−1) [k]

+ βK

K ∆α[k]

end reduce w(t) ← w(t−1) + βK

K

PK

k=1 ∆wk

end

Procedure A: LocalDualMethod: Dual algorithm on machine k Input: Local α[k] ∈ Rnk, and w ∈ Rd consistent with other coordinate blocks of α s.t. w = Aα Data: Local {(xi, yi)}nk

i=1

Output: ∆α[k] and ∆w := A[k]∆α[k]

✔ <10 lines of code in Spark ✔ primal-dual framework allows for any internal optimization method ✔ local updates applied immediately


CoCoA

Algorithm 1: CoCoA Input: T ≥ 1, scaling parameter 1 ≤ βK ≤ K (default: βK := 1). Data: {(xi, yi)}n

i=1 distributed over K machines

Initialize: α(0)

[k] ← 0 for all machines k, and w(0) ← 0

for t = 1, 2, . . . , T for all machines k = 1, 2, . . . , K in parallel (∆α[k], ∆wk) ← LocalDualMethod(α(t−1)

[k]

, w(t−1)) α(t)

[k] ← α(t−1) [k]

+ βK

K ∆α[k]

end reduce w(t) ← w(t−1) + βK

K

PK

k=1 ∆wk

end

Procedure A: LocalDualMethod: Dual algorithm on machine k Input: Local α[k] ∈ Rnk, and w ∈ Rd consistent with other coordinate blocks of α s.t. w = Aα Data: Local {(xi, yi)}nk

i=1

Output: ∆α[k] and ∆w := A[k]∆α[k]

✔ <10 lines of code in Spark ✔ primal-dual framework allows for any internal optimization method ✔ local updates applied immediately
 ✔ average over K

Convergence

Convergence

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ

Convergence

E[D(α∗) − D(α(T ))] ≤ ✓ 1 − (1 − Θ) 1 K λnγ σ + λnγ ◆T ⇣ D(α∗) − D(α(0)) ⌘

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ

Convergence

E[D(α∗) − D(α(T ))] ≤ ✓ 1 − (1 − Θ) 1 K λnγ σ + λnγ ◆T ⇣ D(α∗) − D(α(0)) ⌘

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ applies also to duality gap measure of difficulty of data partition 0 ≤ σ ≤ n/K

Convergence

E[D(α∗) − D(α(T ))] ≤ ✓ 1 − (1 − Θ) 1 K λnγ σ + λnγ ◆T ⇣ D(α∗) − D(α(0)) ⌘

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ applies also to duality gap measure of difficulty of data partition 0 ≤ σ ≤ n/K

✔ it converges!

Convergence

E[D(α∗) − D(α(T ))] ≤ ✓ 1 − (1 − Θ) 1 K λnγ σ + λnγ ◆T ⇣ D(α∗) − D(α(0)) ⌘

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ applies also to duality gap measure of difficulty of data partition 0 ≤ σ ≤ n/K

✔ it converges! 
 ✔ inherits convergence rate of locally used method

Convergence

E[D(α∗) − D(α(T ))] ≤ ✓ 1 − (1 − Θ) 1 K λnγ σ + λnγ ◆T ⇣ D(α∗) − D(α(0)) ⌘

Assumptions: are -smooth LocalDualMethod makes improvement per step



 e.g. for SDCA

Θ

Θ = ✓ 1 − λnγ 1 + λnγ 1 ˜ n ◆H

`i 1/γ applies also to duality gap measure of difficulty of data partition 0 ≤ σ ≤ n/K

✔ it converges! 
 ✔ inherits convergence rate of locally used method ✔ convergence rate is linear for smooth losses

LARGE-SCALE OPTIMIZATION COCOA RESULTS!

LARGE-SCALE OPTIMIZATION COCOA RESULTS!

Empirical Results in

Dataset Training (n) Features (d) Sparsity Workers (K) Cov 522,911 54 22.22% 4 Rcv1 677,399 47,236 0.16% 8 Imagenet 32,751 160,000 100% 32

200 400 600 800 10

−6

10

−4

10

−2

10 10

2

Imagenet Time (s) Log Primal Suboptimality

200 400 600 800 10

−6

10

−4

10

−2

10 10

2

Imagenet

COCOA (H=1e3) mini−batch−CD (H=1) local−SGD (H=1e3) mini−batch−SGD (H=10)

200 400 600 800 10

−6

10

−4

10

−2

10 10

2

Imagenet Time (s) Log Primal Suboptimality

200 400 600 800 10

−6

10

−4

10

−2

10 10

2

Imagenet

COCOA (H=1e3) mini−batch−CD (H=1) local−SGD (H=1e3) mini−batch−SGD (H=10)

20 40 60 80 100 10

−6

10

−4

10

−2

10 10

2

Cov Time (s) Log Primal Suboptimality

20 40 60 80 100 10

−6

10

−4

10

−2

10 10

2

Cov

COCOA (H=1e5) minibatch−CD (H=100) local−SGD (H=1e5) batch−SGD (H=1) 100 200 300 400 10

−6

10

−4

10

−2

10 10

2

RCV1 Time (s) Log Primal Suboptimality

100 200 300 400 10

−6

10

−4

10

−2

10 10

2

COCOA (H=1e5) minibatch−CD (H=100) local−SGD (H=1e4) batch−SGD (H=100)

20 40 60 80 100 10

−6

10

−4

10

−2

10 10

2

Time (s) Log Primal Suboptimality 20 40 60 80 100 10

−6

10

−4

10

−2

10 10

2

1e5 1e4 1e3 100 1

Effect of H on COCOA

COCOA Take-Aways

COCOA Take-Aways

A framework for distributed optimization

COCOA Take-Aways

A framework for distributed optimization Uses state-of-the-art primal-dual methods

COCOA Take-Aways

A framework for distributed optimization Uses state-of-the-art primal-dual methods Reduces communication:


• applies updates immediately

• averages over # of machines

COCOA Take-Aways

A framework for distributed optimization Uses state-of-the-art primal-dual methods Reduces communication:


• applies updates immediately

• averages over # of machines

Strong convergence guarantees & results in practice

COCOA Take-Aways

A framework for distributed optimization Uses state-of-the-art primal-dual methods Reduces communication:


• applies updates immediately

• averages over # of machines

Strong convergence guarantees & results in practice 
 NIPS ‘14

COCOA Take-Aways

A framework for distributed optimization Uses state-of-the-art primal-dual methods Reduces communication:


• applies updates immediately

• averages over # of machines

Strong convergence guarantees & results in practice 
 NIPS ‘14 github.com/gingsmith/cocoa

Future Work

Future Work

• ptimal scaling between adding &

averaging

Future Work

• ptimal scaling between adding &

averaging similar rates for local-SGD?

Future Work

• ptimal scaling between adding &

averaging similar rates for local-SGD? additional experiments: models/datasets

Future Work

• ptimal scaling between adding &

averaging similar rates for local-SGD? additional experiments: models/datasets integration into Spark MLlib

Thanks!

github.com/gingsmith/cocoa


 NIPS ‘14