Homotopy Analysis for Tensor PCA Yuan Deng Duke University Joint - - PowerPoint PPT Presentation

Homotopy Analysis for Tensor PCA

Yuan Deng Duke University

Joint work with Anima Anandkumar, Rong Ge, Hossein Mobahi

Non-convex Optimization

• Optimizing smooth function f(x).

Local Optimum Global Optimum

How to get rid of local optima

Gaussian Smoothing

• Idea: Smooth the function
• convolve with 𝒪(0, 𝑢)

Gaussian Smoothing

• Idea: Smooth the function
• convolve with 𝒪(0, 𝑢)

Gaussian Smoothing

• Idea: Smooth the function
• convolve with 𝒪(0, 𝑢)

Local Minimum Disappears!

Gaussian Smoothing

• Idea: Smooth the function
• convolve with 𝒪(0, 𝑢)

Local Minimum Disappears! Shifted Global Optimum

Gaussian Smoothing

• Idea: Smooth the function
• convolve with 𝒪(0, 𝑢)
• How to decide how much to smooth?
• How to recover the original global optium?

Local Minimum Disappears! Shifted Global Optimum

Homotopy Method

• Try all level of smoothing!

Homotopy Method

Computer Vision

• image deblurring

[Boccuto et al., 2002]

• image restoration

[Nikolova et al., 2010]

• optical flow

[Brox & Malik, 2011] Clustering [Gold, 1994] Graph matching [Zaslavskiy et al., 2009]

• No theoretical guarantees on the solution
• too restrictive

[Mobahi and Fisher III, 2015]

• difficult to check

[Hazan et al., 2016]

Homotopy Method

• Handcrafted the choice of smoothing levels
• Slow: Local search is repeated

for each smoothing level

Tensor PCA [Richard and Montanari 2014]

Probabilistic model for PCA 𝑤 ∈ ℝ*, 𝜐 ≥ 0 is the signal-to-noise ratio Tensor PCA: 𝑈 = 𝜐𝒘 ⊗ 𝒘 ⊗ 𝒘 + 𝐵 Objective:

• Design an efficient algorithm for as small 𝜐 as possible

𝑁 = 𝜐𝒘𝒘4 + 𝐵

Signal Gaussian Noise

Previous Work

• [Richard & Montanari 2014] Can find 𝑤 when 𝜐 = Ω 𝑒 in

poly time, and 𝜐 = Ω( 𝑒

• ) in exp. time.
• [Hopkins, Shi & Steurer 2015] Sum-of-Squares technique,

can find 𝑤 when 𝜐 = Ω 9(𝑒:/<) in poly time

• Basic Sum-of-Squares algorithm is very slow.
• Running time can be improved Ω

9 𝑒: , nearly linear

Our Results

Method Bound on 𝜐 Time Extra Space Ours Ω 9(𝑒:/<) Ω 9(𝑒:) 𝑃(𝑒) State-of-Art Ω 9(𝑒:/<) Ω 9(𝑒:) 𝑃(𝑒>)

Guarantee matches best known result Better convergence rate when 𝜐 is closed to 𝑒:/< One of the first results on provably analyzing homotopy method

Optimization for tensor PCA

• Recall: for matrix PCA, we optimize
• For tensor PCA, we optimize

max 𝒚4𝑁𝒚 = 𝜐 𝒘, 𝒚 > + 𝒚4𝐵𝒚 𝒚 = 1 max 𝑈 𝒚, 𝒚, 𝒚 = 𝜐 𝑤, 𝒚 : + 𝐵(𝒚, 𝒚, 𝒚) 𝒚 = 1

𝑤

Infinite Smoothing

unique optimum 𝑦∗ : correlation 𝜐 /𝑒 = Ω(𝑒FG.>I)

[random unit vector : 𝑒FG.I]

𝑤

𝑢 = ∞

Phase Transition in Homotopy Method

• Lemma*: there is a threshold 𝜄,
• If using infinite steps, i.e., continuously ∞ → 0
• 𝑢 > 𝜄, ||𝑦O − 𝑦∗|| ≤ 𝑝(1)||𝑦∗||
• 𝑢 < 𝜄, 𝑦O, 𝑤 = Ω(1)

𝑤

𝑢 > 𝜄 𝑢 < 𝜄

𝑤

𝑢 = 𝜄

• 1.0
• 0.5

0.5 1.0

• 0.4
• 0.3
• 0.2
• 0.1

0.1

Phase Transition 𝑔 𝑦 = −𝑦< + 0.8𝑦>

𝑕(𝑦, 0.2)

• 1.0
• 0.5

0.5 1.0

• 1.2
• 1.0
• 0.8
• 0.6
• 0.4
• 0.2

𝑕(𝑦, 0.4)

• 1.0
• 0.5

0.5 1.0

• 0.6
• 0.4
• 0.2
• 1.0
• 0.5

0.5 1.0

• 0.20
• 0.15
• 0.10
• 0.05

0.05 0.10 0.15

𝑕(𝑦, 0.3) 𝑕(𝑦, 0)

Phase Transition

• If using infinite steps, i.e., continuously ∞ → 0
• 𝑢 > 𝜄, ||𝑦O − 𝑦∗|| ≤ 𝑝(1)||𝑦∗||

𝑢Z = ∞

• 𝑢 < 𝜄, 𝑦O, 𝑤 = Ω(1)

𝑢> = 0

Infinite smoothing Power Method at 0 smoothing

Conclusions

• Homotopy method gives near-optimal results for

tensor PCA.

• Possible to analyze non-convex functions even

when they really have bad local optima.

Open Problems

• More examples of Homotopy method?
• When the tensor has higher rank?
• General results for effects of smoothing
• What kind of local optima will disappear?
• Different way of smoothing/regularization?