Low Rank Matrix Completion: A Smoothed 0 -Search Wei Dai Jointly - - PowerPoint PPT Presentation

Low Rank Matrix Completion: A Smoothed ℓ0-Search

Wei Dai

Jointly with Guangyu Zhou and Xiaochen Zhao

Imperial College London

Queen Mary University 2012

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 1 / 19

Outline

Low rank matrix completion Why ℓ0-search? Major issue: singular points. Solution: smoothed ℓ0-search.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 2 / 19

Low-Rank Matrix Completion

Applications: Online recommendation. Robust PCA for machine learning. Video surveillance.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 3 / 19

The Formulation

Given Ω ⊂ [m] × [n]: the index set of the observed entries XΩ: partial observations r: the matrix rank Find an ˆ X such that rank

• ˆ

X

• ≤ r and
• ˆ

X

• Ω = XΩ.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 4 / 19

Approaches

LRMC: sparse in the eigen-space. ℓ1-minimization (Recht, et al., 2010; Candes-Recht, 2009; Candes et al. 2009; Toh-Yun, 2009; ...) minX′ X′∗ s.t. (X′)Ω = XΩ. Greedy algorithms

◮ SP⇒ADMiRA (Lee-Bresler, 2010). ◮ IHT⇒SVP (Meka, et al., 2009).

Specific algorithms

◮ Power factorization (Haldar-Hernando, 2009). ◮ OptSpace (Keshavan, et al., 2009). Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 5 / 19

Why ℓ0-Search? An Example

An example: XΩ =     1 ? ? −2 ? 1 ? 1 1 ?

1 2

?     ⇒ ˆ X =     1 −2 −2 −1    

• 1

− 1

2

− 1

2

• .

Rank-one case: ℓ0-search always works.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 6 / 19

Why ℓ0-Search? An Example

An example: XΩ =     1 ? ? −2 ? 1 ? 1 1 ?

1 2

?     ⇒ ˆ X =     1 −2 −2 −1    

• 1

− 1

2

− 1

2

• .

Rank-one case: ℓ0-search always works. All other existing algorithms may fail.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 6 / 19

Why ℓ0-Search?

Compressed sensing: Integer programming. Low-rank matrix completion: Optimization on continuous spaces.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 7 / 19

ℓ0-Search: How?

The optimization framework: Subspace Evolution (SE)

Let U ∈ Rm×r contain r orthonormal columns (U ∈ Um,r) and W ∈ Rr×n. minU,W XΩ − (UW )Ω2

F

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 8 / 19

ℓ0-Search: How?

The optimization framework: Subspace Evolution (SE)

Let U ∈ Rm×r contain r orthonormal columns (U ∈ Um,r) and W ∈ Rr×n. minU,W XΩ − (UW )Ω2

F

Observe failures in simulations. Main reason: singular points.

◮ Gradient does not vanish. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 8 / 19

An Example of Singular Points

minU,W XΩ − (UW )Ω2

F

= minU min

W

XΩ − (UW )Ω2

F

• f(U)

. = minU f (U).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 9 / 19

An Example of Singular Points

minU,W XΩ − (UW )Ω2

F

= minU min

W

XΩ − (UW )Ω2

F

• f(U)

. = minU f (U).

An example

f (U) = min

w

• 

 ? 1 1  

XΩ

−          √ 1 − 2ǫ2 ǫ ǫ  

• U

w       

Ω

• 2

= if ǫ = 0 2 if ǫ = 0 .

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 9 / 19

Why Singular Points are Bad

When every contour region is convex.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 10 / 19

Why Singular Points are Bad

When every contour region is convex. Even when some contour regions are not convex.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 10 / 19

Why Singular Points are Bad

When every contour region is convex. Even when some contour regions are not convex. Singular points form barriers.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 10 / 19

How to Handle Singular Points

SET algorithm Dai et al. 2011:

◮ Detect and jump across singular points.

Regularization Boumal-Absil, 2011:

◮ minU minW XΩ − (UW )Ω2

F + µ W 2 F .

◮ Discontinuous ⇒ continuous.

Geometric objective function Dai et al. 2012:

◮ Performance guarantees for special cases. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 11 / 19

More Recently

• G. Zhou, X. Zhao, and W. Dai, “Low Rank Matrix Completion: A Smoothed ℓ0-Search”, Allerton

Conference, 2012.

Rigorously show under what conditions the regularization methods may fail. Propose a new objective function.

◮ Continuous.

Implement a second order optimization method.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 12 / 19

To Address the Singularity Issue,

Original objective function. f (U) = minW XΩ − (UW )Ω2

F

=

i minwi

• xΩ,i − (Uwi)Ωi
• 2

F

=

i min wi

xΩ,i − UΩiwi2

F

• fi(UΩi)

=

i fi (UΩi).

Singular points ⇔ ∃i s.t. UΩi is column-rank deficient. Continuous objective function: ˜ fρ (U) =

i fi (UΩi) · gρ (λmin (UΩi)).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 13 / 19

To Address the Singularity Issue,

Original objective function. f (U) = minW XΩ − (UW )Ω2

F

=

i minwi

• xΩ,i − (Uwi)Ωi
• 2

F

=

i min wi

xΩ,i − UΩiwi2

F

• fi(UΩi)

=

i fi (UΩi).

Singular points ⇔ ∃i s.t. UΩi is column-rank deficient. Continuous objective function: ˜ fρ (U) =

i fi (UΩi) · gρ (λmin (UΩi)).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 13 / 19

To Address the Singularity Issue,

Original objective function. f (U) = minW XΩ − (UW )Ω2

F

=

i minwi

• xΩ,i − (Uwi)Ωi
• 2

F

=

i min wi

xΩ,i − UΩiwi2

F

• fi(UΩi)

=

i fi (UΩi).

Singular points ⇔ ∃i s.t. UΩi is column-rank deficient. Continuous objective function: ˜ fρ (U) =

i fi (UΩi) · gρ (λmin (UΩi)).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 13 / 19

To Address the Singularity Issue,

Original objective function. f (U) = minW XΩ − (UW )Ω2

F

=

i minwi

• xΩ,i − (Uwi)Ωi
• 2

F

=

i min wi

xΩ,i − UΩiwi2

F

• fi(UΩi)

=

i fi (UΩi).

Singular points ⇔ ∃i s.t. UΩi is column-rank deficient. Continuous objective function: ˜ fρ (U) =

i fi (UΩi) · gρ (λmin (UΩi)).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 13 / 19

To Address the Singularity Issue,

Original objective function. f (U) = minW XΩ − (UW )Ω2

F

=

i minwi

• xΩ,i − (Uwi)Ωi
• 2

F

=

i min wi

xΩ,i − UΩiwi2

F

• fi(UΩi)

=

i fi (UΩi).

Singular points ⇔ ∃i s.t. UΩi is column-rank deficient. Continuous objective function: ˜ fρ (U) =

i fi (UΩi) · gρ (λmin (UΩi)).

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 13 / 19

Properties of ˜ fρ (U)

˜ fρ is continuous ∀ρ > 0. When ρ → 0, ˜ fρ is the best lower semi-continuous approx. of f.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 14 / 19

Properties of ˜ fρ (U)

˜ fρ is continuous ∀ρ > 0. When ρ → 0, ˜ fρ is the best lower semi-continuous approx. of f. Geometric intuitions: f: Barriers ⇒ ˜ fρ: Tunnels

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 14 / 19

Solving the Optimization Problem

Smoothed Subspace Evolution (SSE): minU ˜ fρ (U) for a small ρ > 0. Options: Gradient descent: slow convergence. Newton methods: fast convergence.

◮ Hessian matrix. ◮ Computationally very expensive.

Quasi-Newton method: fast convergence.

◮ Computationally much easier. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 15 / 19

A Quasi-Newton Method on Manifold

In our problem, U ∈ Um,r is in a manifold. Implementation: consider the manifold structure.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 16 / 19

A Quasi-Newton Method on Manifold

In our problem, U ∈ Um,r is in a manifold. Implementation: consider the manifold structure.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 16 / 19

A Quasi-Newton Method on Manifold

In our problem, U ∈ Um,r is in a manifold. Implementation: consider the manifold structure.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 16 / 19

Numerical Results - Noiseless Case

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sampling Rate Success Rate 9−by−9 matrices, rank=1, # of realizations=200 SSE SET SE(no transfer step)

Success:

• ˆ

XΩ − XΩ

• 2

F / XΩ2 F ≤ 10−6.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 17 / 19

Numerical Results - Noiseless Case

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sampling Rate Success Rate 9−by−9 matrices, rank=2, # of realizations=200 SSE SET SE(no transfer step)

Success:

• ˆ

XΩ − XΩ

• 2

F / XΩ2 F ≤ 10−6.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 17 / 19

Numerical Results - Noiseless Case

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sampling Rate Success Rate 50−by−50 matrices,rank=2,# of realizations=500 SSE SET ADMiRA OptSpace PF APGL

Success:

• ˆ

XΩ − XΩ

• 2

F / XΩ2 F ≤ 10−6.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 17 / 19

Numerical Results - Noisy Case

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Sampling Rate Success Rate 50−by−50 matrices,rank=2,# of realizations=500,SNR=30dB SSE SET ADMiRA OptSpace PF APGL

Model: YΩ = XΩ + ZΩ Success:

• ˆ

XΩ − YΩ

• 2

F / XΩ2 F ≤

• ˆ

ZΩ

• 2

F / XΩ2 F = 10−3.

Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 18 / 19

Summary

Low rank matrix completion Why ℓ0-search. Major issue: Singular points. Solution: smoothed ℓ0-search.

◮ A quasi-Newton method on a manifold. ◮ Good performance. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ0-Search Queen Mary 2012 19 / 19