Sketched Ridge Regression: Optimization and Statistical Perspectives - - PowerPoint PPT Presentation

Sketched Ridge Regression:

Shusen Wang

UC Berkeley

Alex Gittens

RPI

Michael Mahoney

UC Berkeley

Optimization and Statistical Perspectives

Overview

Ridge Regression

Over-determined: 𝑜 ≫ 𝑒

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

𝑜×𝑒

Ridge Regression

𝑜×𝑒 min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

• Efficient and approximate solution?
• Use only part of the data?

Ridge Regression

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

Matrix Sketching ng:

• Random selection
• Random projection

Approximate Ridge Regression

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

• Sketched solution: 𝐱O

Approximate Ridge Regression

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

• Sketched solution: 𝐱O
• Sketch size 𝑃

R

S T sketch size

Approximate Ridge Regression

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

• Sketched solution: 𝐱O
• Sketch size 𝑃

R

S T

• 𝑔 𝐱O ≤ 1 + 𝜗 min

𝐱 𝑔 𝐱

sketch size

Optimization Perspective

Approximate Ridge Regression

min

𝐱 𝑔 𝐱 = 1

𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

Statistical Perspective

• Bias
• Variance

Related Work

• Least squares regression: min

𝐱

𝐘𝐱 − 𝐳

7 7

Reference

• Drineas, Mahoney, and Muthukrishnan: Sampling algorithms for l2 regression and applications. In SODA, 2006.
• Drineas, Mahoney, Muthukrishnan, and Sarlos: Faster least squares approximation. Numerische Mathematik, 2011.
• Clarkson and Woodruff: Low rank approximation and regression in input sparsity time. In STOC, 2013.
• Ma, Mahoney, and Yu: A statistical perspective on algorithmic leveraging. Journal of Machine Learning Research,

2015.

• Pilanci and Wainwright: Iterative Hessian sketch: fast and accurate solution approximation for constrained least
• squares. Journal of Machine Learning Research, 2015.
• Raskutti and Mahoney: A statistical perspective on randomized sketching for ordinary least-squares. Journal of

Machine Learning Research, 2016.

• Etc …

Sketched Ridge Regression

Matrix Sketching

𝐘 𝐓[𝐘

• Turn big matrix into smaller one.
• 𝐘 ∈ ℝ^×S ➡ 𝐓[𝐘 ∈ ℝ_×S.
• 𝐓 ∈ ℝ^×_ is called sketching matrix, e.g.,
• Uniform sampling
• Leverage score sampling
• Gaussian projection
• Subsampled randomized Hadamard transform (SRHT)
• Count sketch (sparse embedding)
• Etc.

Matrix Sketching

𝐘 𝐓[𝐘

• Some matrix sketching methods are efficient.
• Time cost is o(𝑜𝑒𝑡) — lower than multiplication.
• Examples:
• Leverage score sampling: 𝑃(𝑜𝑒 log 𝑜) time
• SRHT: 𝑃(𝑜𝑒 log 𝑡) time

Ridge Regression

• Objective function:

𝑔 𝐱 = 1 𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7

• Optimal solution:

𝐱⋆ = argmin

𝐱

𝑔 𝐱 = 𝐘[𝐘 + 𝑜𝛿𝐉S e 𝐘[𝐳

• Time cost: 𝑃 𝑜𝑒7 or 𝑃 𝑜𝑒𝑢

Sketched Ridge Regression

• Goal: efficiently and approximately solve

argmin

𝐱

𝑔 𝐱 = 1 𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7 .

Sketched Ridge Regression

• Goal: efficiently and approximately solve

argmin

𝐱

𝑔 𝐱 = 1 𝑜 𝐘𝐱 − 𝐳

7 7 + 𝛿 𝐱 7 7 .

• Approach: reduce the size of 𝐘 and 𝐳 by matrix sketching.

Sketched Ridge Regression

• Sketched solution:

𝐱O = argmin

𝐱

1 𝑜 𝐓[𝐘𝐱 − 𝐓[𝐳

7 7 + 𝛿 𝐱 7 7

= 𝐘[𝐓𝐓[𝐘 + 𝑜𝛿𝐉S e 𝐘[𝐓𝐓[𝐳

Sketched Ridge Regression

• Sketched solution:
• Time: 𝑃 𝑡𝑒7 + 𝑈

_

• 𝑈

_ is the cost of sketching 𝐓[𝐘

• E.g. 𝑈

_ = 𝑃(𝑜𝑒 log 𝑡) for SRHT.

• E.g. 𝑈

_ = 𝑃 𝑜𝑒 log 𝑜 for leverage score sampling.

𝐱O = argmin

𝐱

1 𝑜 𝐓[𝐘𝐱 − 𝐓[𝐳

7 7 + 𝛿 𝐱 7 7

= 𝐘[𝐓𝐓[𝐘 + 𝑜𝛿𝐉S e 𝐘[𝐓𝐓[𝐳

Theory: Optimization Perspective

Optimization Perspective

• Recall the objective function 𝑔 𝐱 = i

^ 𝐘𝐱 − 𝐳 7 7 + 𝛿 𝐱 7 7.

• Bound 𝑔 𝐱O − 𝑔 𝐱⋆ .
• i

^ 𝐘𝐱O − 𝐘𝐱⋆ 7 7 ≤ 𝑔 𝐱O − 𝑔 𝐱⋆ .

Optimization Perspective

For the sketching methods

• SRHT or leverage sampling with s = 𝑃

R

jS T ,

• uniform sampling with s = 𝑃

k jS lmn S T

, 𝑔 𝐱O − 𝑔 𝐱⋆ ≤ 𝜗𝑔 𝐱⋆ holds w.p. 0.9.

• 𝐘 ∈ ℝ^×S: the design matrix
• 𝛿: the regularization parameter
• 𝛾 =

𝐘 p

p

^qr 𝐘 p

p ∈ (0, 1]

• 𝜈 ∈ 1,

^ S : the row coherence of 𝐘

Optimization Perspective

For the sketching methods

• SRHT or leverage sampling with s = 𝑃

R

jS T ,

• uniform sampling with s = 𝑃

k jS lmn S T

, 𝑔 𝐱O − 𝑔 𝐱⋆ ≤ 𝜗𝑔 𝐱⋆ holds w.p. 0.9.

i ^ 𝐘𝐱O − 𝐘𝐱⋆ 7 7 ≤ 𝜗𝑔 𝐱⋆ .

• 𝐘 ∈ ℝ^×S: the design matrix
• 𝛿: the regularization parameter
• 𝛾 =

𝐘 p

p

^qr 𝐘 p

p ∈ (0, 1]

• 𝜈 ∈ 1,

^ S : the row coherence of 𝐘

Theory: Statistical Perspective

Statistical Model

• 𝐘 ∈ ℝ^×S: fixed design matrix
• 𝐱x ∈ ℝS: the true and unknown model
• 𝐳 = 𝐘𝐱x + 𝛆: observed response vector
• 𝜀i, ⋯ , 𝜀^ are random noise
• 𝔽 𝛆 = 𝟏 and

𝔽 𝛆𝛆[ = 𝜊7𝐉^

Bias-Variance Decomposition

• Risk: 𝑆 𝐱 =

i ^ 𝔽 𝐘𝐱 − 𝐘𝐱x 7 7

• 𝔽 is taken w.r.t. the random noise 𝛆.

Bias-Variance Decomposition

• Risk: 𝑆 𝐱 =

i ^ 𝔽 𝐘𝐱 − 𝐘𝐱x 7 7

• 𝔽 is taken w.r.t. the random noise 𝛆.
• Risk measures prediction error.

Bias-Variance Decomposition

• Risk: 𝑆 𝐱 =

i ^ 𝔽 𝐘𝐱 − 𝐘𝐱x 7 7

• R 𝐱 = bias7 𝐱 + var 𝐱

Bias-Variance Decomposition

• Risk: 𝑆 𝐱 =

i ^ 𝔽 𝐘𝐱 − 𝐘𝐱x 7 7

• R 𝐱 = bias7 𝐱 + var 𝐱
• bias 𝐱⋆ = 𝛿 𝑜
• 𝚻7 + 𝑜𝛿𝐉S ƒi𝚻𝐖[𝐱x

7,

• var 𝐱⋆ = …p

^

𝐉S + 𝑜𝛿𝚻ƒ7 ƒi

7 7,

• bias 𝐱O = 𝛿 𝑜
• 𝚻𝐕[𝐓𝐓[𝐕𝚻 + 𝑜𝛿𝐉S e𝚻𝐖[𝐱x

7,

• var 𝐱O = …p

^

𝐕[𝐓𝐓[𝐕 + 𝑜𝛿𝚻ƒ7 e𝐕[𝐓𝐓[

7 7

,

• Here 𝐘 = 𝐕𝚻𝐖[ is the SVD.

Optimal Solution Sketched Solution

Statistical Perspective

For the sketching methods

• SRHT or leverage sampling with s = 𝑃

R

S Tp ,

• uniform sampling with s = 𝑃

k S lmn S Tp

,

the followings hold w.p. 0.9:

1 − 𝜗 ≤ bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗, 1 − 𝜗 𝑜 𝑡 ≤ var 𝐱O var 𝐱⋆ ≤ 1 + 𝜗 𝑜 𝑡 .

• 𝐘 ∈ ℝ^×S: the design matrix
• 𝜈 ∈ 1,

^ S : the row coherence of 𝐘

Good! Bad! Because 𝑜 ≫ 𝑡.

Statistical Perspective

For the sketching methods

• SRHT or leverage sampling with s = 𝑃

R

S Tp ,

• uniform sampling with s = 𝑃

k S lmn S Tp

,

the followings hold w.p. 0.9:

1 − 𝜗 ≤ bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗, 1 − 𝜗 𝑜 𝑡 ≤ var 𝐱O var 𝐱⋆ ≤ 1 + 𝜗 𝑜 𝑡 .

• 𝐘 ∈ ℝ^×S: the design matrix
• 𝜈 ∈ 1,

^ S : the row coherence of 𝐘

If 𝐳 is noisy

variance dominates bias 𝑆 𝐱_ ≫ 𝑆(𝐱⋆).

Conclusions

• Use sketched solution to initialize numerical
• ptimization.
• 𝐘𝐱O is close to 𝐘𝐱⋆.

Optimization Perspective

Conclusions

• Use sketched solution to initialize numerical
• ptimization.
• 𝐘𝐱O is close to 𝐘𝐱⋆.
• 𝐱 ‡ : output of the 𝑢-th iteration of CG algorithm.
• 𝐘𝐱 Š ƒ𝐘𝐱⋆

p p

𝐘𝐱 ‹ ƒ𝐘𝐱⋆

p p ≤ 2

• 𝐘Ž𝐘
• ƒi
• 𝐘Ž𝐘
• ri

‡

.

• Initialization is important.

Optimization Perspective

Conclusions

• Use sketched solution to initialize numerical
• ptimization.
• 𝐘𝐱O is close to 𝐘𝐱⋆.
• Never use sketched solution to replace the
• ptimal solution.
• Much higher variance è bad generalization.

Optimization Perspective Statistical Perspective

Model Averaging

Model Averaging

• Independently draw 𝐓i, ⋯ , 𝐓•.
• Compute the sketched solutions 𝐱i

O, ⋯ , 𝐱• O.

• Model averaging: 𝐱O =

i

• ∑

𝐱‘

O

• ‘’i

.

Optimization Perspective

• For sufficiently large 𝑡,

“ 𝐱”

• ƒ“ 𝐱⋆

“ 𝐱⋆

≤ 𝜗 holds w.h.p.

Without model averaging

Optimization Perspective

• For sufficiently large 𝑡,

“ 𝐱”

• ƒ“ 𝐱⋆

“ 𝐱⋆

≤ 𝜗 holds w.h.p.

• Using the same matrix sketching and same 𝑡,

“ 𝐱• ƒ“ 𝐱⋆ “ 𝐱⋆

≤ T

• + 𝜗7

holds w.h.p.

Without model averaging With model averaging

Optimization Perspective

• For sufficiently large 𝑡,

“ 𝐱”

• ƒ“ 𝐱⋆

“ 𝐱⋆

≤ 𝜗 holds w.h.p.

• Using the same matrix sketching and same 𝑡,

“ 𝐱• ƒ“ 𝐱⋆ “ 𝐱⋆

≤ T

• + 𝜗7

holds w.h.p.

Without model averaging With model averaging

Optimization Perspective

• For sufficiently large 𝑡,

“ 𝐱”

• ƒ“ 𝐱⋆

“ 𝐱⋆

≤ 𝜗 holds w.h.p.

• Using the same matrix sketching and same 𝑡,

“ 𝐱• ƒ“ 𝐱⋆ “ 𝐱⋆

≤ T

• + 𝜗7

holds w.h.p.

Without model averaging With model averaging

If 𝑡 ≫ 𝑒 𝜗7 is very small error bound ∝

T

• .

Statistical Perspective

• Risk: 𝑆 𝐱 = i

^ 𝔽 𝐘𝐱 − 𝐘𝐱x 7 7 = bias7 𝐱 + var 𝐱

• Model averaging :
• bias 𝐱O = 𝛿 𝑜
• i
• ∑

𝚻𝐕[𝐓‘𝐓‘

[𝐕𝚻 + 𝑜𝛿𝐉S e𝚻𝐖[𝐱x

• ‘’i

7

,

• var 𝐱O = …p

^ i

• ∑

𝐕[𝐓‘𝐓‘

[𝐕 + 𝑜𝛿𝚻ƒ7 e𝐕[𝐓‘𝐓‘ [

• ‘’i

7 7

.

• Here 𝐘 = 𝐕𝚻𝐖[ is the SVD.

Statistical Perspective

• For sufficiently large 𝑡, the followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≤ 𝑜 𝑡 1 + 𝜗 .

• Using the same sketching methods and same 𝑡, the

followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≲ 𝑜 𝑡 1 𝑕

• + 𝜗

𝟑

Without model averaging

Statistical Perspective

• For sufficiently large 𝑡, the followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≤ 𝑜 𝑡 1 + 𝜗 .

• Using the same sketching methods and same 𝑡, the

followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≲ 𝑜 𝑡 1 𝑕

• + 𝜗

𝟑

Without model averaging With model averaging

Statistical Perspective

• For sufficiently large 𝑡, the followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≤ 𝑜 𝑡 1 + 𝜗 .

• Using the same sketching methods and same 𝑡, the

followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≲ 𝑜 𝑡 1 𝑕

• + 𝜗

𝟑

Without model averaging With model averaging

Statistical Perspective

• For sufficiently large 𝑡, the followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≤ 𝑜 𝑡 1 + 𝜗 .

• Using the same sketching methods and same 𝑡, the

followings hold w.h.p.:

bias 𝐱O bias 𝐱⋆ ≤ 1 + 𝜗 and var 𝐱O var 𝐱⋆ ≲ 𝑜 𝑡 1 𝑕

• + 𝜗

𝟑

Without model averaging With model averaging

If 𝜗 is small, then var 𝐱O ∝ i

• .

Applications to Distributed Optimization

• 𝐲i, 𝑧i , ⋯ , 𝐲^, 𝑧^

are (randomly) split among 𝑕 machines.

• Equivalent to uniform

sampling with 𝑡 = ^

• .

Optimization Perspective

• Application to distributed optimization:
• If 𝑡 = ^
• ≫ 𝑒, 𝐱O is very close to 𝐱⋆ (provably).
• 𝐱O is good initialization of distributed optimization algorithms.

Statistical Perspective

• Application to distributed machine learning:
• If 𝑡 = ^
• ≫ 𝑒, then 𝑆 𝐱O is comparable to 𝑆 𝐱⋆ .
• If low-precision solution suffices, then 𝐱O is a good substitute of 𝐱⋆.
• One-shot solution.

Thank You!

The paper is at arXiv:1702.04837