Low Rank Matrix Completion: A Smoothed 0 -Search Wei Dai Jointly - PowerPoint PPT Presentation

Low Rank Matrix Completion: A Smoothed ℓ 0 -Search Wei Dai Jointly with Guangyu Zhou and Xiaochen Zhao Imperial College London Queen Mary University 2012 Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 1 / 19

Outline Low rank matrix completion Why ℓ 0 -search? Major issue: singular points. Solution: smoothed ℓ 0 -search. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 2 / 19

Low-Rank Matrix Completion Applications: Online recommendation. Robust PCA for machine learning. Video surveillance. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 3 / 19

The Formulation Given Ω ⊂ [ m ] × [ n ] : the index set of the observed entries X Ω : partial observations r : the matrix rank � � � � Find an ˆ ˆ ˆ ≤ r and Ω = X Ω . X such that rank X X Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 4 / 19

Approaches LRMC: sparse in the eigen-space. ℓ 1 -minimization (Recht, et al., 2010; Candes-Recht, 2009; Candes et al. 2009; Toh-Yun, 2009; ...) min X ′ � X ′ � ∗ s.t. ( X ′ ) Ω = X Ω . Greedy algorithms ◮ SP ⇒ ADMiRA (Lee-Bresler, 2010) . ◮ IHT ⇒ SVP (Meka, et al., 2009) . Specific algorithms ◮ Power factorization (Haldar-Hernando, 2009) . ◮ OptSpace (Keshavan, et al., 2009) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 5 / 19

Why ℓ 0 -Search? An Example An example:     1 ? ? 1 − 2 ? 1 − 2     � �  ⇒ ˆ − 1 − 1 X Ω = X = 1 .     ? 1 1 − 2 2 2    1 ? ? − 1 2 Rank-one case: ℓ 0 -search always works. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 6 / 19

Why ℓ 0 -Search? An Example An example:     1 ? ? 1 − 2 ? 1 − 2     � �  ⇒ ˆ − 1 − 1 X Ω = X = 1 .     ? 1 1 − 2 2 2    1 ? ? − 1 2 Rank-one case: ℓ 0 -search always works. All other existing algorithms may fail. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 6 / 19

Why ℓ 0 -Search? Compressed sensing: Integer programming. Low-rank matrix completion: Optimization on continuous spaces. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 7 / 19

ℓ 0 -Search: How? The optimization framework: Subspace Evolution (SE) Let U ∈ R m × r contain r orthonormal columns ( U ∈ U m,r ) and W ∈ R r × n . min U , W � X Ω − ( UW ) Ω � 2 F Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 8 / 19

ℓ 0 -Search: How? The optimization framework: Subspace Evolution (SE) Let U ∈ R m × r contain r orthonormal columns ( U ∈ U m,r ) and W ∈ R r × n . min U , W � X Ω − ( UW ) Ω � 2 F Observe failures in simulations. Main reason: singular points. ◮ Gradient does not vanish. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 8 / 19

An Example of Singular Points min U , W � X Ω − ( UW ) Ω � 2 F � X Ω − ( UW ) Ω � 2 = min U min . F W � �� � f ( U ) = min U f ( U ) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 9 / 19

An Example of Singular Points min U , W � X Ω − ( UW ) Ω � 2 F � X Ω − ( UW ) Ω � 2 = min U min . F W � �� � f ( U ) = min U f ( U ) . An example �   � 2 � � √ � �     �  1 − 2 ǫ 2  � ? �   � �   � f ( U ) = min 1 − ǫ w     �   � w �   � 1 ǫ �   � � � � �� � � �� � � � X Ω U � 0 Ω if ǫ � = 0 = if ǫ = 0 . 2 Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 9 / 19

Why Singular Points are Bad When every contour region is convex. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 10 / 19

Why Singular Points are Bad When every contour Even when some region is convex. contour regions are not convex. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 10 / 19

Why Singular Points are Bad When every contour Even when some Singular points form region is convex. contour regions are barriers. not convex. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 10 / 19

How to Handle Singular Points SET algorithm Dai et al. 2011 : ◮ Detect and jump across singular points. Regularization Boumal-Absil, 2011 : ◮ min U min W � X Ω − ( UW ) Ω � 2 F + µ � W � 2 F . ◮ Discontinuous ⇒ continuous. Geometric objective function Dai et al. 2012 : ◮ Performance guarantees for special cases. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 11 / 19

More Recently G. Zhou, X. Zhao, and W. Dai, “Low Rank Matrix Completion: A Smoothed ℓ 0 -Search”, Allerton Conference, 2012. Rigorously show under what conditions the regularization methods may fail. Propose a new objective function. ◮ Continuous. Implement a second order optimization method. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 12 / 19

To Address the Singularity Issue, Original objective function. f ( U ) = min W � X Ω − ( UW ) Ω � 2 F � � = � � 2 i min w i � x Ω ,i − ( Uw i ) Ω i F = � � x Ω ,i − U Ω i w i � 2 = � i min i f i ( U Ω i ) . F w i � �� � f i ( U Ω i ) Singular points ⇔ ∃ i s.t. U Ω i is column-rank deficient. Continuous objective function: f ρ ( U ) = � ˜ i f i ( U Ω i ) · g ρ ( λ min ( U Ω i )) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 13 / 19

To Address the Singularity Issue, Original objective function. f ( U ) = min W � X Ω − ( UW ) Ω � 2 F � � = � � 2 i min w i � x Ω ,i − ( Uw i ) Ω i F = � � x Ω ,i − U Ω i w i � 2 = � i min i f i ( U Ω i ) . F w i � �� � f i ( U Ω i ) Singular points ⇔ ∃ i s.t. U Ω i is column-rank deficient. Continuous objective function: f ρ ( U ) = � ˜ i f i ( U Ω i ) · g ρ ( λ min ( U Ω i )) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 13 / 19

To Address the Singularity Issue, Original objective function. f ( U ) = min W � X Ω − ( UW ) Ω � 2 F � � = � � 2 i min w i � x Ω ,i − ( Uw i ) Ω i F = � � x Ω ,i − U Ω i w i � 2 = � i min i f i ( U Ω i ) . F w i � �� � f i ( U Ω i ) Singular points ⇔ ∃ i s.t. U Ω i is column-rank deficient. Continuous objective function: f ρ ( U ) = � ˜ i f i ( U Ω i ) · g ρ ( λ min ( U Ω i )) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 13 / 19

To Address the Singularity Issue, Original objective function. f ( U ) = min W � X Ω − ( UW ) Ω � 2 F � � = � � 2 i min w i � x Ω ,i − ( Uw i ) Ω i F = � � x Ω ,i − U Ω i w i � 2 = � i min i f i ( U Ω i ) . F w i � �� � f i ( U Ω i ) Singular points ⇔ ∃ i s.t. U Ω i is column-rank deficient. Continuous objective function: f ρ ( U ) = � ˜ i f i ( U Ω i ) · g ρ ( λ min ( U Ω i )) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 13 / 19

To Address the Singularity Issue, Original objective function. f ( U ) = min W � X Ω − ( UW ) Ω � 2 F � � = � � 2 i min w i � x Ω ,i − ( Uw i ) Ω i F = � � x Ω ,i − U Ω i w i � 2 = � i min i f i ( U Ω i ) . F w i � �� � f i ( U Ω i ) Singular points ⇔ ∃ i s.t. U Ω i is column-rank deficient. Continuous objective function: f ρ ( U ) = � ˜ i f i ( U Ω i ) · g ρ ( λ min ( U Ω i )) . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 13 / 19

Properties of ˜ f ρ ( U ) ˜ f ρ is continuous ∀ ρ > 0 . When ρ → 0 , ˜ f ρ is the best lower semi-continuous approx. of f . Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 14 / 19

Properties of ˜ f ρ ( U ) ˜ f ρ is continuous ∀ ρ > 0 . When ρ → 0 , ˜ f ρ is the best lower semi-continuous approx. of f . Geometric intuitions: ˜ f : Barriers ⇒ f ρ : Tunnels Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 14 / 19

Solving the Optimization Problem Smoothed Subspace Evolution (SSE): min U ˜ f ρ ( U ) for a small ρ > 0 . Options: Gradient descent: slow convergence. Newton methods: fast convergence. ◮ Hessian matrix. ◮ Computationally very expensive. Quasi-Newton method: fast convergence. ◮ Computationally much easier. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 15 / 19

A Quasi-Newton Method on Manifold In our problem, U ∈ U m,r is in a manifold. Implementation: consider the manifold structure. Zhou, Zhao, and Dai (Imperial College) Matrix Completion: ℓ 0 -Search Queen Mary 2012 16 / 19