Ra Randomized SV SVD, CU CUR De Decom ompos osition on, and - - PowerPoint PPT Presentation

Ra Randomized SV SVD, CU CUR De Decom

• mpos
• sition
• n,

and and SPSD SPSD Ma Matri trix Ap Approximati tion

• n

Shusen Wang

Outline

• CX Decomposition & Approximate SVD
• CUR Decomposition
• SPSD Matrix Approximation

• Given any matrix 𝐁 ∈ ℝ$×&
• The CX decomposition of 𝐁
• 1. Sketching: 𝐃 = 𝐁𝐐 ∈ ℝ$×*
• 2. Find 𝐘 such that 𝐁 ≈ 𝐃𝐘
• E.g. 𝐘⋆ = argmin𝐘 𝐁 − 𝐃𝐘

5 6 = 𝐃7𝐁

• It costs 𝑃 𝑛𝑜𝑑
• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J

CX Decomposition

• Let the sketching matrix 𝐐 ∈ ℝ&×* be defined in the table.
• min

WXYZ 𝐘 [\ 𝐁 − 𝐃𝐘 ] 6 ≤ 1 + 𝜗 𝐁 − 𝐁\ ] 6 Uniform sampling Leverage score sampling Gaussian projection SRHT Count sketch

c ≥

O 𝜉𝑙 log 𝑙 + 1 𝜗 O 𝑙 log 𝑙 + 1 𝜗 O 𝑙 𝜗 O 𝑙 + log 𝑜 log 𝑙 + 1 𝜗 O 𝑙6 + 𝑙 𝜗

𝜉 is the column coherence of 𝐁Z

CX Decomposition

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J SVD: 𝐃 = 𝐕G 𝚻G𝐖G

J ∈ ℝ$×*

Time cost: 𝑃(𝑛𝑑6)

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J SVD: 𝐃 = 𝐕G 𝚻G𝐖G

J ∈ ℝ$×*

Let 𝚻G𝐖G

J𝐘 = 𝐚 ∈ ℝ*×&

Time cost: 𝑃(𝑛𝑑6 + 𝑜𝑑6)

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J SVD: 𝐃 = 𝐕G 𝚻G𝐖G

J ∈ ℝ$×*

Let 𝚻G𝐖G

J𝐘 = 𝐚 ∈ ℝ*×&

SVD: 𝐚 = 𝐕L𝚻L𝐖L

J ∈ ℝ*×&

Time cost: 𝑃(𝑛𝑑6 + 𝑜𝑑6 + 𝑜𝑑6)

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J SVD: 𝐃 = 𝐕G 𝚻G𝐖G

J ∈ ℝ$×*

Let 𝚻G𝐖G

J𝐘 = 𝐚 ∈ ℝ*×&

SVD: 𝐚 = 𝐕L𝚻L𝐖L

J ∈ ℝ*×&

𝑛×𝑡 matrix with

• rthonormal columns

diagonal matrix 𝑡×𝑜 matrix with

• rthonormal rows

Time cost: 𝑃(𝑛𝑑6 + 𝑜𝑑6 + 𝑜𝑑6 + 𝑛𝑑6)

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD
• Done! Approximate rank 𝑑 SVD: 𝐁 ≈ (𝐕G𝐕L)𝚻L𝐖L

J

𝐁 ≈ 𝐃𝐘 = 𝐕G𝚻G𝐖G

J𝐘 = 𝐕G𝐚 = 𝐕G𝐕L𝚻L𝐖L J 𝑛×𝑡 matrix with

• rthonormal columns

diagonal matrix 𝑡×𝑜 matrix with

• rthonormal rows

Time cost: 𝑃 𝑛𝑑6 + 𝑜𝑑6 + 𝑜𝑑6 + 𝑛𝑑6 = 𝑃(𝑛𝑑6 + 𝑜𝑑6)

CX Decomposition ⇔ Approximate SVD

• CX decomposition ⇔ approximate SVD
• Given 𝐁 ∈ ℝ$×& and 𝐃 ∈ ℝ$×*, the approximate SVD costs
• 𝑃 𝑛𝑜𝑑 time
• 𝑃 𝑛𝑑 + 𝑜𝑑 memory

CX Decomposition

• The CX decomposition of 𝐁 ∈ ℝ$×&
• Optimal solution: 𝐘⋆ = argmin𝐘 𝐁 − 𝐃𝐘

5 6 = 𝐃7𝐁

• How to make it more efficient?

CX Decomposition

• The CX decomposition of 𝐁 ∈ ℝ$×&
• Optimal solution: 𝐘⋆ = argmin𝐘 𝐁 − 𝐃𝐘

5 6 = 𝐃7𝐁

• How to make it more efficient?

A regression problem!

Fast CX Decomposition

• Fast CX [Drineas, Mahoney, Muthukrishnan, 2008][Clarkson & Woodruff, 2013]
• Draw another sketching matrix 𝐓 ∈ ℝ$×m
• Compute 𝐘

n = argmin𝐘 𝐓o 𝐁 − 𝐃𝐘

5 6 = 𝐓J𝐃 7 𝐓J𝐁

• Time cost: 𝑃 𝑜𝑑𝑡 + TimeOfSketch
• When 𝑡 = 𝑃

q 𝑑/𝜗 ,

𝐁 − 𝐃𝐘 n

5 6

≤ 1 + 𝜗 ⋅ min𝐘 𝐁 − 𝐃𝐘

5 6

Outline

• CX Decomposition & Approximate SVD
• CUR Decomposition
• SPSD Matrix Approximation

CUR Decomposition

• Sketching
• 𝐃 = 𝐁𝐐𝐃 ∈ ℝ$×*
• 𝐒 = 𝐐𝐒

J𝐁 ∈ ℝv×&

• Find 𝐕 such that 𝐃𝐕𝐒 ≈ 𝐁
• CUR ⇔ Approximate SVD
• In the same way as “CX⇔ Approximate SVD”

CUR Decomposition

• Sketching
• 𝐃 = 𝐁𝐐𝐃 ∈ ℝ$×*
• 𝐒 = 𝐐𝐒

J𝐁 ∈ ℝv×&

• Find 𝐕 such that 𝐃𝐕𝐒 ≈ 𝐁
• CUR ⇔ Approximate SVD
• In the same way as “CX⇔ Approximate SVD”
• 3 types of 𝐕

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

• 𝐁𝐐𝐃

7

𝐃 𝐁 𝐕 𝐒

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

• 𝐁𝐐𝐃

7

• Recall the fast CX decomposition

𝐁 ≈ 𝐃𝐘 n = 𝐃 𝐐𝐒

• 𝐃

7 𝐐𝐒

• 𝐁 = 𝐃𝐕𝐒

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

• 𝐁𝐐𝐃

7

• Recall the fast CX decomposition

𝐁 ≈ 𝐃𝐘 n = 𝐃 𝐐𝐒

• 𝐃

7 𝐐𝐒

• 𝐁 = 𝐃𝐕𝐒

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

• 𝐁𝐐𝐃

7

• Recall the fast CX decomposition

𝐁 ≈ 𝐃𝐘 n = 𝐃 𝐐𝐒

• 𝐃

7 𝐐𝐒

• 𝐁 = 𝐃𝐕𝐒
• They’re equivalent: 𝐃 𝐘

n = 𝐃 𝐕 𝐒

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

• 𝐁𝐐𝐃

7

• Recall the fast CX decomposition

𝐁 ≈ 𝐃𝐘 n = 𝐃 𝐐𝐒

• 𝐃

7 𝐐𝐒

• 𝐁 = 𝐃𝐕𝐒
• They’re equivalent: 𝐃 𝐘

n = 𝐃 𝐕 𝐒

• Require 𝑑 = 𝑃

q

\ w and 𝑠 = 𝑃

q

* w such that

𝐁 − 𝐃𝐕𝐒

5 6 ≤ 1 + 𝜗 𝐁 − 𝐁\ 5 6

CUR Decomposition

• Type 1 [Drineas, Mahoney, Muthukrishnan, 2008]:

𝐕 = 𝐐𝐒

𝐔𝐁𝐐𝐃 7

• Efficient
• O 𝑠𝑑6 + TimeOfSketch
• Loose bound
• Sketch size ∝ 𝜗{6
• Bad empirical performance

CUR Decomposition

• Type 2: Optimal CUR

𝐕⋆ = min

𝐕

𝐁 − 𝐃𝐕𝐒

] 6 = 𝐃7𝐁𝐒7

CUR Decomposition

• Type 2: Optimal CUR

𝐕⋆ = min

𝐕

𝐁 − 𝐃𝐕𝐒

] 6 = 𝐃7𝐁𝐒7

• Theory [W & Zhang, 2013], [Boutsidis & Woodruff, 2014]:
• 𝐃 and 𝐒 are selected by the adaptive sampling algorithm
• 𝑑 = 𝑃

\ w and 𝑠 = 𝑃 \ w

• 𝐁 − 𝐃𝐕𝐒

] 6 ≤ 1 + 𝜗 𝐁 − 𝐁\ 5 6

CUR Decomposition

• Type 2: Optimal CUR

𝐕⋆ = min

𝐕

𝐁 − 𝐃𝐕𝐒

] 6 = 𝐃7𝐁𝐒7

• Inefficient
• O 𝑛𝑜𝑑 + TimeOfSketch

CUR Decomposition

• Type 3: Fast CUR [W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐓𝐃 and 𝐓𝐒
• Solve the problem

𝐕 n = min

𝐕

𝑻𝑫

• 𝐁 − 𝐃𝐕𝐒 𝐓𝐒

] 6

= 𝐓𝐃

J𝐃 7 𝐓𝐃

• 𝐁𝐓𝑺

𝐒𝐓𝐒 7

• Intuition?

CUR Decomposition

• The optimal 𝐕 matrix is obtained by the optimization problem

𝐕⋆ = min

𝐕

𝐃𝐕𝐒 − 𝐁

] 6

CUR Decomposition

• Approximately solve the optimization problem, e.g. by column

selection

CUR Decomposition

• Solve the small scale problem

CUR Decomposition

• Type 3: Fast CUR [W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐓𝐃 ∈ ℝ$×m€ and 𝐓𝐒 ∈ ℝ&×m•
• Solve the problem

𝐕 n = min

𝐕

𝐓𝑫

• 𝐁 − 𝐃𝐕𝐒 𝐓𝐒

] 6

= 𝐓𝐃

J𝐃 7 𝐓𝐃

• 𝐁𝐓𝑺

𝐒𝐓𝐒 7

• Theory
• 𝑡* = 𝑃

* w and sv = 𝑃 v w

• 𝐁 − 𝐃𝐕

n𝐒

] 6

≤ 1 + 𝜗 ⋅ min

𝐕

𝐁 − 𝐃𝐕𝐒

] 6

CUR Decomposition

• Type 3: Fast CUR [W, Zhang, Zhang, 2015]
• Draw 2 sketching matrices 𝐓𝐃 ∈ ℝ$×m€ and 𝐓𝐒 ∈ ℝ&×m•
• Solve the problem

𝐕 n = min

𝐕

𝐓𝑫

• 𝐁 − 𝐃𝐕𝐒 𝐓𝐒

] 6

= 𝐓𝐃

J𝐃 7 𝐓𝐃

• 𝐁𝐓𝑺

𝐒𝐓𝐒 7

• Efficient
• 𝑃 𝑡*𝑡v 𝑑 + 𝑠

+ TimeOfSketch

• Good empirical performance

Type 2: Optimal CUR Original Type 1: Fast CX Type 3: Fast CUR 𝑡* = 2𝑑, 𝑡v = 2𝑠 Type 3: Fast CUR 𝑡* = 4𝑑, 𝑡v = 4𝑠 𝐁: 𝑛 = 1920 𝑜 = 1168 𝐃 and 𝐒:

• 𝑑 = 𝑠 = 100
• uniform sampling

Conclusions

• Approximate truncated SVD
• CX decomposition
• CUR decomposition (3 types)
• Fast CUR is the best

Outline

• CX Decomposition & Approximate SVD
• CUR Decomposition
• SPSD Matrix Approximation

Motivation 1: Kernel Matrix

• Given 𝑜 samples 𝐲‰, ⋯ , 𝐲& ∈ ℝ‹ and kernel function 𝜆 ⋅,⋅ .
• E.g. Gaussian RBF kernel

𝜆 𝐲•, 𝐲Ž = exp − 𝐲• − 𝐲Ž

6 6

𝜏6 .

• Computing the kernel matrix 𝐋 ∈ ℝ&×&
• where 𝑙•Ž = 𝜆 𝐲•, 𝐲Ž
• costs O(𝑜6𝑒) time

Motivation 1: Kernel Matrix

• Given 𝑜 samples 𝐲‰, ⋯ , 𝐲& ∈ ℝ‹ and kernel function 𝜆 ⋅,⋅ .
• E.g. Gaussian RBF kernel

𝜆 𝐲•, 𝐲Ž = exp − 𝐲• − 𝐲Ž

6 6

𝜏6 .

• Computing the kernel matrix 𝐋 ∈ ℝ&×&
• where 𝑙•Ž = 𝜆 𝐲•, 𝐲Ž
• costs 𝑃(𝑜6𝑒) time

Motivation 2: Matrix Inversion

• Solve the linear system

𝐋 + 𝛽𝐉& 𝐱 = 𝐳 to find 𝐱 ∈ ℝ&.

• It costs O(𝑜–) time and O(𝑜6) memory.
• Performed by
• Gaussian process regression (equivalently, kernel ridge regression)
• Least squares kernel SVM
• 𝐋 ∈ ℝ&×& is the kernel matrix
• 𝐳 = 𝑧‰, ⋯ , 𝑧& ∈ ℝ& contains the labels

Motivation 2: Matrix Inversion

• Solve the linear system

𝐋 + 𝛽𝐉& 𝐱 = 𝐳 to find 𝐱 ∈ ℝ&.

• Solution: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳

Motivation 2: Matrix Inversion

• Solve the linear system

𝐋 + 𝛽𝐉& 𝐱 = 𝐳 to find 𝐱 ∈ ℝ&.

• Solution: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳
• It costs
• 𝑃(𝑜–) time
• 𝑃(𝑜6) memory.

Motivation 2: Matrix Inversion

• Solve the linear system

𝐋 + 𝛽𝐉& 𝐱 = 𝐳 to find 𝐱 ∈ ℝ&.

• Solution: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳
• It costs
• 𝑃(𝑜–) time
• 𝑃(𝑜6) memory.
• Performed by
• Kernel ridge regression
• Least squares kernel SVM

Motivation 3: Eigenvalue Decomposition

• Find the top 𝑙 (≪ 𝑜) eigenvectors of 𝐋.
• It costs
• 𝑃

q(𝑜6𝑙) time

• 𝑃(𝑜6) memory.

Motivation 3: Eigenvalue Decomposition

• Find the top 𝑙 (≪ 𝑜) eigenvectors of 𝐋.
• It costs
• 𝑃

q(𝑜6𝑙) time

• 𝑃(𝑜6) memory.
• Performed by
• Kernel PCA (𝑙 is the target rank)
• Manifold learning (𝑙 is the target rank)

Computational Challenges

• Time costs
• Computing kernel matrix: 𝑃(𝑜6𝑒)
• Matrix inversion: 𝑃(𝑜–)
• Rank 𝑙 eigenvalue decomposition: 𝑃(𝑜6𝑙)

Computational Challenges

• Time costs
• Computing kernel matrix: 𝑃(𝑜6𝑒)
• Matrix inversion: 𝑃(𝑜–)
• Rank 𝑙 eigenvalue decomposition: 𝑃(𝑜6𝑙)

At least quadratic time!

Computational Challenges

• Time costs
• Computing kernel matrix: 𝑃(𝑜6𝑒)
• Matrix inversion: 𝑃(𝑜–)
• Rank 𝑙 eigenvalue decomposition: 𝑃(𝑜6𝑙)
• Memory costs
• Inversion and eigenvalue decomposition: 𝑃(𝑜6)

Computational Challenges

• Time costs
• Computing kernel matrix: 𝑃(𝑜6𝑒)
• Matrix inversion: 𝑃(𝑜–)
• Rank 𝑙 eigenvalue decomposition: 𝑃(𝑜6𝑙)
• Memory costs
• Inversion and eigenvalue decomposition: 𝑃(𝑜6)
• Because
• the numerical algorithms are pass-inefficient
• è form 𝐋 and keep it in memory

Computational Challenges

• Time costs
• Computing kernel matrix: 𝑃(𝑜6𝑒)
• Matrix inversion: 𝑃(𝑜–)
• Rank 𝑙 eigenvalue decomposition: 𝑃(𝑜6𝑙)
• Memory costs
• Inversion and eigenvalue decomposition: 𝑃(𝑜6)
• Because
• the numerical algorithms are pass-inefficient
• è form 𝐋 and keep it in memory

When 𝑜 = 10™, the 𝑜×𝑜 matrix costs 80GB memory!

How to Speedup?

• Efficiently form the low-rank approximation

𝐋 ≈ 𝐃 𝐕 𝐃J 𝐋 𝐃 𝐃J 𝐕

How to Speedup?

• Efficiently form the low-rank approximation

𝐋 ≈ 𝐃 𝐕 𝐃J

• Equivalent 𝐋 ≈ 𝐌 𝐌J

Efficient Matrix Inversion

• Solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳:
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

Efficient Matrix Inversion

• Approximately solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳:
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

Efficient Matrix Inversion

• Approximately solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

• Expand the inversion by the Woodbury identity

Efficient Matrix Inversion

• Approximately solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

• Expand the inversion by the Woodbury identity

𝐁 + 𝐂𝐃𝐄 {‰ = 𝐁{‰ − 𝐁{‰𝐂 𝐃{‰ + 𝐄𝐁{‰𝐂 {‰𝐄𝐁{‰

Efficient Matrix Inversion

• Approximately solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

• Expand the inversion by the Woodbury identity

𝐱⋆ ≈ 𝛽{‰𝐳 + 𝛽{‰𝐌 𝛽𝐉 + 𝐌o𝐌 {‰𝐌𝐔𝐳

Efficient Matrix Inversion

• Approximately solve the linear system 𝐋 + 𝛽𝐉& 𝐱 = 𝐳
• Replace 𝐋 by 𝐌𝐌J: 𝐱⋆ = 𝐋 + 𝛽𝐉& {‰𝐳 ≈ 𝐌𝐌J + 𝛽𝐉&

{‰𝐳

• Expand the inversion by the Woodbury identity

𝐱⋆ ≈ 𝛽{‰𝐳 + 𝛽{‰𝐌 𝛽𝐉 + 𝐌o𝐌 {‰𝐌𝐔𝐳

• Time cost: 𝑃 𝑜𝑑6

Linear in 𝑜, much better than 𝑃 𝑜–

Efficient Eigenvalue Decomposition

• Approximately compute the 𝑙-eigenvalue decomposition of 𝐋
• SVD: 𝐌 = 𝐕𝐌𝚻𝐌𝐖𝐌
• 𝐋 ≈ 𝐌𝐌𝐔 = 𝐕𝐌𝚻𝐌

𝟑𝐕𝐌 𝐔

• Approximate 𝑙- eigenvalue decomposition of 𝐋
• eigenvectors: the first 𝑙 vectors in 𝐕𝐌
• eigenvalues: the first 𝑙 vectors diagonal entries in 𝚻𝐌

Efficient Eigenvalue Decomposition

• Approximately compute the 𝑙-eigenvalue decomposition of 𝐋
• SVD: 𝐌 = 𝐕𝐌𝚻𝐌𝐖𝐌
• 𝐋 ≈ 𝐌𝐌𝐔 = 𝐕𝐌𝚻𝐌

𝟑𝐕𝐌 𝐔

• Approximate 𝑙- eigenvalue decomposition of 𝐋
• eigenvectors: the first 𝑙 vectors in 𝐕𝐌
• Time cost: 𝑃 𝑜𝑑6

Efficient Eigenvalue Decomposition

• Approximately compute the 𝑙-eigenvalue decomposition of 𝐋
• SVD: 𝐌 = 𝐕𝐌𝚻𝐌𝐖𝐌
• 𝐋 ≈ 𝐌𝐌𝐔 = 𝐕𝐌𝚻𝐌

𝟑𝐕𝐌 𝐔

• Approximate 𝑙- eigenvalue decomposition of 𝐋
• eigenvectors: the first 𝑙 vectors in 𝐕𝐌
• Time cost: 𝑃 𝑜𝑑6
• Much lower than 𝑃

q 𝑜6𝑙

Sketching Based Models

• How to find such an approximation?

𝐋 ≈ 𝐃 𝐕 𝐃J

𝐋 𝐃 𝐃J 𝐕

Sketching Based Models

• How to find such an approximation?
• Sketching based Methods: 𝐃 = 𝐋𝐓 ∈ ℝ&×* is a sketch of 𝐋.
• 𝐓 ∈ ℝ&×* can be column selection or random projection matrix

𝐋 ≈ 𝐃 𝐕 𝐃J

Sketching Based Models

• How to find such an approximation?
• Sketching based Methods: 𝐃 = 𝐋𝐓 ∈ ℝ&×* is a sketch of 𝐋.
• 𝐓 ∈ ℝ&×* can be column selection or random projection matrix
• Three methods:
• The prototype model [HMT11, WZ13, WLZ16]
• The fast model [WZZ15]
• The Nyström method [WS15, GM13]

𝐋 ≈ 𝐃 𝐕 𝐃J

The Prototype Model

• Objective: 𝐋 ≈ 𝐃𝐕𝐃J
• Minimize the approximation error by

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

] 6

= 𝐃7𝐋 𝐃7 J.

The Prototype Model

• Objective: 𝐋 ≈ 𝐃𝐕𝐃J
• Minimize the approximation error by

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

] 6

= 𝐃7𝐋 𝐃7 J. Extension of the random SVD to SPSD matrix [HMT11]

The Prototype Model

• Objective: 𝐋 ≈ 𝐃𝐕𝐃J
• Minimize the approximation error by

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

] 6

= 𝐃7𝐋 𝐃7 J.

• Time: 𝑃(𝑜6𝑑)
• The time complexity is nearly the same to the 𝑙-eigenvalue decomposition.
• It is much faster than the 𝑙- eigenvalue decomposition in practice.

The Prototype Model

• Objective: 𝐋 ≈ 𝐃𝐕𝐃J
• Minimize the approximation error by

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

] 6

= 𝐃7𝐋 𝐃7 J.

• Time: 𝑃(𝑜6𝑑)
• #Passes: one

The Prototype Model

• Objective: 𝐋 ≈ 𝐃𝐕𝐃J
• Minimize the approximation error by

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

] 6

= 𝐃7𝐋 𝐃7 J.

• Time: 𝑃(𝑜6𝑑)
• #Passes: one
• Memory: 𝑃(𝑜𝑑)
• Put 𝑙•Ž in memory only when it is visited
• Keep 𝐃7 in memory

The Prototype Model

• Error Bound
• 𝑙 ≪ 𝑜 is arbitrary integer
• 𝐐 samples 𝑑 = 𝑃

\ w columns by adaptive sampling

• 𝔽

𝐋 − 𝐃𝐕⋆𝐃J

] 6

≤ 1 + 𝜗 𝐋 − 𝐋\

] 6

The Prototype Model

• Limitations
• 𝐕⋆ = 𝐃7𝐋 𝐃7 J
• Time cost is 𝑃 𝑜6𝑑
• Requires observing the whole of 𝐋

The Prototype Model

• Prototype model: 𝐋 ≈ 𝐃 𝐕⋆ 𝐃J, where

𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

5 6

. 𝐋 𝐃 𝐃J 𝐕

The Fast Model

• Column/row selection
• Form 𝐐J𝐋𝐐 and 𝐐J𝐃

𝐐J𝐋𝐐 𝐐J𝐃 𝐃J𝐐 𝐕

The Fast Model

• Column/row selection
• Form 𝐐J𝐋𝐐 and 𝐐J𝐃

𝐐J𝐋𝐐 𝐐J𝐃 𝐃J𝐐 𝐕

The Fast Model

• 𝐋 ≈ 𝐃 𝐕

n 𝐃J, where 𝐕 n = argmin

𝐕

𝐐J 𝐋 − 𝐃𝐕𝐃J 𝐐

𝑮 𝟑

. 𝐐J𝐋𝐐 𝐐J𝐃 𝐃J𝐐 𝐕

The Fast Model

• Prototype model: 𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

5 6

= 𝐃7𝐋 𝐃7 J

• Fast model: 𝐕

n = argmin

𝐕

𝐐J 𝐋 − 𝐃𝐕𝐃J 𝐐

] 𝟑

= 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7.

The Fast Model

• Prototype model: 𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

5 6

= 𝐃7𝐋 𝐃7 J

• Fast model: 𝐕

n = argmin

𝐕

𝐐J 𝐋 − 𝐃𝐕𝐃J 𝐐

] 𝟑

= 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7.

• Theory
• 𝑞 = 𝑃

&* w

• 𝐐 is column selection matrix (according to the row leverage scores of 𝐃)
• Then 𝐋 − 𝐃𝐕

n𝐃J

] 6

≤ 1 + 𝜗 𝐋 − 𝐃𝐕⋆𝐃J

] 6

The faster model is nearly as good as the prototype model!

The Fast Model

• Prototype model: 𝐕⋆ = argmin

𝐕

𝐋 − 𝐃𝐕𝐃J

5 6

= 𝐃7𝐋 𝐃7 J

• Fast model: 𝐕

n = argmin

𝐕

𝐐J 𝐋 − 𝐃𝐕𝐃J 𝐐

] 𝟑

= 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7.

• Theory
• 𝑞 = 𝑃

&* w

• 𝐐 is column selection matrix (according to the row leverage scores of 𝐃)
• Then 𝐋 − 𝐃𝐕

n𝐃J

] 6

≤ 1 + 𝜗 𝐋 − 𝐃𝐕⋆𝐃J

] 6

• Overall time cost: 𝑃 𝑞6𝑑 + 𝑜𝑑6 = 𝑃 𝑜𝑑–/𝜗

linear in 𝑜

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃𝐗7𝐃J

𝐋

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃𝐗7𝐃J

𝐋 𝐃 𝐃J

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃𝐗7𝐃J

𝐋 𝐃 𝐃J 𝐗

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J

𝐋 𝐃 𝐃J 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

• The Nystrom method is special

instance of the fast model.

• It is approximate solution to the

prototype model

The Nyström Method

• 𝐓 (𝑜×𝑑): column selection matrix
• 𝐃 = 𝐋𝐓 (𝑜×𝑑), 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 (𝑑×𝑑)
• The Nyström method: 𝐋 ≈ 𝐃 𝐗7 𝐃J
• New explanation:
• Recall the fast model: 𝐘

n = argmin

𝐘

𝐐J 𝐋 − 𝐃𝐘𝐃J 𝐐

5 6

• Setting 𝐐 = 𝐓, then

𝐘 n = argmin

𝐘

𝐓J 𝐋 − 𝐃𝐘𝐃J 𝐓

5 6

= 𝐓J𝐃

7 𝐓J𝐋𝐓

𝐃J𝐓

7

= 𝐗7𝐗𝐗7 = 𝐗7

• The Nystrom method is special

instance of the fast model.

• It is approximate solution to the

prototype model

The Nyström Method

• Cost
• Time: 𝑃 𝑜𝑑6
• Memory: 𝑃 𝑜𝑑

The Nyström Method

• Cost
• Time: 𝑃 𝑜𝑑6
• Memory: 𝑃 𝑜𝑑

Very efficient!

The Nyström Method

• Cost
• Time: 𝑃 𝑜𝑑6
• Memory: 𝑃 𝑜𝑑
• Error bound: weak

Very efficient!

Comparisons

• 𝐃 = 𝐋𝐓 ∈ ℝ&×*, 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 ∈ ℝ*×*
• SPSD matrix approximation: 𝐋 ≈ 𝐃𝐕𝐃J
• The prototype model: 𝐕 = 𝐃7𝐋 𝐃7 J
• The fast model: 𝐕 = 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7

• The Nyström method: 𝐕 = 𝐗7

Comparisons

• 𝐃 = 𝐋𝐓 ∈ ℝ&×*, 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 ∈ ℝ*×*
• SPSD matrix approximation: 𝐋 ≈ 𝐃𝐕𝐃J
• The prototype model: 𝐕 = 𝐃7𝐋 𝐃7 J
• The fast model: 𝐕 = 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7

• The Nyström method: 𝐕 = 𝐗7

When 𝐐 = 𝐉&, the prototype model ⇔ the fast model

Comparisons

• 𝐃 = 𝐋𝐓 ∈ ℝ&×*, 𝐗 = 𝐓J𝐋𝐓 = 𝐓J𝐃 ∈ ℝ*×*
• SPSD matrix approximation: 𝐋 ≈ 𝐃𝐕𝐃J
• The prototype model: 𝐕 = 𝐃7𝐋 𝐃7 J
• The fast model: 𝐕 = 𝐐J𝐃

7(𝐐J𝐋𝐐) 𝐃J𝐐 7

• The Nyström method: 𝐕 = 𝐗7

When 𝐐 = 𝐓, the Nyström method ⇔ the fast model

Comparisons

• 𝑑 = 150, 𝑜 = 100𝑑, vary 𝑞 from 2𝑑 to 40𝑑

𝑞/𝑑

𝐋 − 𝐃𝐕𝐃J

] 6

𝐋

] 6

The Nyström Method

𝑃 𝑜𝑑6 time

The Fast Model 𝑃 𝑜𝑑6 + 𝑞6𝑑 time The Prototype Model 𝑃 𝑜6𝑑 time

Conclusions

• Motivations
• Avoid forming the kernel matrix
• Avoid inversion/decomposition
• Prototype model, fast model, Nystrom
• They have connections
• The fast model and Nystrom are practical