Non-convex Optimization for Machine Learning Prateek Jain - - PowerPoint PPT Presentation

Non-convex Optimization for Machine Learning

Prateek Jain

Microsoft Research India

Outline

• Optimization for Machine Learning
• Non-convex Optimization
• Convergence to Stationary Points
• First order stationary points
• Second order stationary points
• Non-convex Optimization in ML
• Neural Networks
• Learning with Structure
• Alternating Minimization
• Projected Gradient Descent

Relevant Monograph (Shameless Ad)

Optimization in ML

Supervised Learning

• Given points (𝑦𝑗, 𝑧𝑗)
• Prediction function: ෝ

𝑧𝑗 = 𝜚(𝑦𝑗, 𝑥)

• Minimize loss: min

𝑥 σ𝑗 ℓ(𝜚 𝑦𝑗, 𝑥 , 𝑧𝑗)

Unsupervised Learning

Given points (𝑦1, 𝑦2 … 𝑦𝑂) Find cluster center or train GANs Represent ෝ 𝑦𝑗 = 𝜚(𝑦𝑗, 𝑥) Minimize loss: min

𝑥 σ𝑗 ℓ(𝜚 𝑦𝑗, 𝑥 , 𝑦𝑗)

Optimization Problems

• Unconstrained optimization

min

𝑥∈𝑆𝑒 𝑔(𝑥)

• Deep networks
• Regression
• Gradient Boosted Decision

Trees

• Constrained optimization

min

𝑥 𝑔(𝑥) 𝑡. 𝑢. 𝑥 ∈ 𝐷

• Support Vector Machines
• Sparse regression
• Recommendation system
• …

Convex Optimization

Convex function Convex set

𝑔 𝜇𝑥1 + 1 − 𝜇 𝑥2 ≤ 𝜇𝑔 𝑥1 + 1 − 𝜇 𝑔 𝑥2 , 0 ≤ 𝜇 ≤ 1 ∀𝑥1, 𝑥2 ∈ 𝐷, 𝜇𝑥1 + 1 − 𝜇 𝑥2 ∈ 𝐷 0 ≤ 𝜇 ≤ 1

min

𝑥 𝑔(𝑥)

𝑡. 𝑢. 𝑥 ∈ 𝐷

Slide credit: Purushottam Kar

Examples

Linear Programming Quadratic Programming Semidefinite Programming

Slide credit: Purushottam Kar

Convex Optimization

• Unconstrained optimization

min

𝑥∈𝑆𝑒 𝑔(𝑥)

Optima: just ensure ∇𝑥𝑔 𝑥 = 0

• Constrained optimization

min

𝑥 𝑔(𝑥) 𝑡. 𝑢. 𝑥 ∈ 𝐷

Optima: KKT conditions

In this talk, lets assume 𝑔 is 𝑀 −smooth => 𝑔 is differentiable 𝑔 𝑦 ≤ 𝑔 𝑧 + ∇𝑔 𝑧 , 𝑦 − 𝑧 + 𝑀 2 ||𝑦 − 𝑧||2 OR, ||∇𝑔 𝑦 − ∇𝑔 𝑧 || ≤ 𝑀||𝑦 − 𝑧||

Gradient Descent Methods

• Projected gradient descent method:
• For t=1, 2, … (until convergence)
• 𝑥𝑢+1 = 𝑄

𝐷(𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 )

• 𝜃: step-size

Convergence Proof

𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥𝑢 + ∇𝑔 𝑥𝑢 , 𝑥𝑢+1 − 𝑥𝑢 + 𝑀 2 ||𝑥𝑢+1 − 𝑥𝑢||2 𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥𝑢 − 1 − 𝑀𝜃 2 𝜃||∇𝑔 𝑥𝑢 ||2 ≤ 𝑔 𝑥𝑢 − 𝜃 2 ||∇𝑔 𝑥𝑢 ||2 𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥∗ + ∇𝑔 𝑥𝑢 , 𝑥𝑢 − 𝑥∗ − 1 2𝜃 ||𝑥𝑢+1 − 𝑥𝑢||2 𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥∗ + 1 2𝜃 ||𝑥𝑢 − 𝑥∗||2 − ||𝑥𝑢+1 − 𝑥∗||2 𝑔 𝑥𝑈 ≤ 𝑔 𝑥∗ + 1 𝑈 ⋅ 2𝜃 ||𝑥0 − 𝑥∗||2 ⇒ 𝑔 𝑥𝑈 ≤ 𝑔 𝑥∗ + 𝜗 𝑈 = 𝑃 𝑀 ⋅ ||𝑥0 − 𝑥∗||2 𝜗

𝑥𝑢

Non-convexity?

• Critical points: ∇𝑔 𝑥 = 0
• But: ∇f w = 0 ⇏ Optimality

min

𝑥∈𝑆𝑒 𝑔(𝑥)

Local Optima

• 𝑔 𝑥 ≤ 𝑔 𝑥′ , ∀||𝑥 − 𝑥′|| ≤ 𝜗

Local Minima

image credit: academo.org

First Order Stationary Points

• Defined by: ∇𝑔 𝑥 = 0
• But ∇2𝑔(𝑥) need not be positive semi-definite

First Order Stationary Point (FOSP)

image credit: academo.org

First Order Stationary Points

• E.g., 𝑔 𝑥 = 0.5(𝑥1

2 − 𝑥2 2)

• ∇𝑔 𝑥 =

𝑥1 −𝑥2

• ∇𝑔 0 = 0
• But, ∇2𝑔 𝑥 = 1

−1 ⇒ 𝑗𝑜𝑒𝑓𝑔𝑗𝑜𝑗𝑢𝑓

• 𝑔

𝜗 2 , 𝜗

= −

3 8 𝜗2 ⇒ 𝑔 0,0

is not a local minima

First Order Stationary Point (FOSP)

image credit: academo.org

Second Order Stationary Points

Second Order Stationary Point (SOSP) if:

• ∇𝑔 𝑥 = 0
• ∇2𝑔 𝑥 ≽ 0

Does it imply local optimality?

Second Order Stationary Point (SOSP)

image credit: academo.org

Second Order Stationary Points

• 𝑔 𝑥 =

1 3 (𝑥1 3 − 3 𝑥1𝑥2 2)

• ∇ 𝑔 𝑥 = (𝑥1

2 − 𝑥2 2)

−2 𝑥1𝑥2

• ∇2𝑔 𝑥 =

2𝑥1 −2𝑥2 −2𝑥2 −2𝑥1

• ∇𝑔 0 = 0, ∇2𝑔 0 = 0 ⇒ 0 𝑗𝑡 𝑇𝑃𝑇𝑄
• 𝑔 𝜗, 𝜗

= −

2 3 𝜗3 < 𝑔(0)

Second Order Stationary Point (SOSP)

image credit: academo.org

Stationarity and local optima

• 𝑥 is local optima implies: 𝑔 𝑥 ≤ 𝑔 𝑥′ , ∀||𝑥 − 𝑥′|| ≤ 𝜗
• 𝑥 is FOSP implies:

𝑔 𝑥 ≤ 𝑔 𝑥′ + 𝑃(||𝑥 − 𝑥||2)

• 𝑥 is SOSP implies:

𝑔 𝑥 ≤ 𝑔 𝑥′ + 𝑃(||𝑥 − 𝑥′||3)

• 𝑥 is p-th order SP implies:

𝑔 𝑥 ≤ 𝑔 𝑥′ + 𝑃(||𝑥 − 𝑥′||𝑞+1)

• That is, local optima: 𝑞 = ∞

Computability?

First Order Stationary Point Second Order Stationary Point Third Order Stationary Point 𝑞 ≥ 4 Stationary Point NP-Hard Local Optima NP-Hard

𝑔 𝑥 ≤ 𝑔 𝑥′ + 𝑃(||𝑥 − 𝑥′||𝑞+1)

Anandkumar and Ge-2016

Does Gradient Descent Work for Local Optimality?

• Yes!
• In fact, with high probability converges

to a “local minimizer”

• If initialized randomly!!!
• But no rates known 
• NP-hard in general!!
• Big open problem ☺

image credit: academo.org

Finding First Order Stationary Points

• Defined by: ∇𝑔 𝑥 = 0
• But ∇2𝑔(𝑥) need not be positive semi-definite

First Order Stationary Point (FOSP)

image credit: academo.org

Gradient Descent Methods

• Gradient descent:
• For t=1, 2, … (until convergence)
• 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢
• 𝜃: step-size
• Assume:

||∇𝑔 𝑦 − ∇𝑔 𝑧 || ≤ 𝑀||𝑦 − 𝑧||

Convergence to FOSP

𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥𝑢 + ∇𝑔 𝑥𝑢 , 𝑥𝑢+1 − 𝑥𝑢 + 𝑀 2 ||𝑥𝑢+1 − 𝑥𝑢||2 𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥𝑢 − 1 − 𝑀𝜃 2 𝜃||∇𝑔 𝑥𝑢 ||2 ≤ 𝑔 𝑥𝑢 − 1 2𝑀 ||∇𝑔 𝑥𝑢 ||2 ||∇𝑔 𝑥𝑢 ||2 ≤ 𝑔 𝑥𝑢 − 𝑔 𝑥𝑢+1 1 2𝑀 ෍

𝑢

||∇𝑔 𝑥𝑢 ||2 ≤ 𝑔 𝑥0 − 𝑔(𝑥∗) min

𝑢

||∇𝑔 𝑥𝑢 || ≤ 2𝑀 (𝑔 𝑥0 − 𝑔(𝑥∗)) 𝑈 ≤ 𝜗

𝑈 = 𝑃 𝑀 ⋅ (𝑔 𝑥0 − 𝑔 𝑥∗ ) 𝜗2

Accelerated Gradient Descent for FOSP?

• For t=1, 2….T
• 𝑥𝑢+1

𝑛𝑒 = 1 − 𝛽𝑢 𝑥𝑢 𝑏𝑕 + 𝛽𝑢𝑥𝑢

• 𝑥𝑢+1 = 𝑥𝑢 − 𝜃𝑢∇𝑔(𝑥𝑢+1

𝑛𝑒)

• 𝑥𝑢+1

𝑏𝑕 = 𝑥𝑢 𝑛𝑒 − 𝛾𝑢∇𝑔(𝑥𝑢+1 𝑛𝑒)

• Convergence? min

𝑢

||∇𝑔 𝑥𝑢 || ≤ 𝜗

• For 𝑈 = 𝑃(

𝑀⋅(𝑔 𝑥0 −𝑔 𝑥∗ ) 𝜗

)

• If convex: 𝑈 = 𝑃(

𝑀⋅(𝑔 𝑥0 −𝑔 𝑥∗ ) 1/4 𝜗

)

Ghadimi and Lan - 2013

𝑥𝑢 𝑥𝑢

𝑏𝑕

𝑥𝑢

𝑛𝑒

𝑥𝑢+1 𝑥𝑢+1

𝑏𝑕

Non-convex Optimization: Sum of Functions

• What if the function has more structure?

min

𝑥

𝑔 𝑥 = 1 𝑜 ෍

𝑗=1 𝑜

𝑔

𝑗(𝑥)

• ∇𝑔 𝑥 = σ𝑗=1

𝑜

∇𝑔

𝑗(𝑥)

• I.e., computing gradient would require 𝑃(𝑜) computation

Does Stochastic Gradient Descent Work?

• For t=1, 2, … (until convergence)
• Sample 𝑗𝑢 ∼ 𝑉𝑜𝑗𝑔[1, 𝑜]
• 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔

𝑗𝑢 𝑥𝑢

Proof? 𝐹𝑗𝑢 𝑥𝑢+1 − 𝑥𝑢 = 𝜃∇𝑔(𝑥𝑢) 𝑔 𝑥𝑢+1 ≤ 𝑔 𝑥𝑢 + ∇𝑔 𝑥𝑢 , 𝑥𝑢+1 − 𝑥𝑢 + 𝑀 2 ||𝑥𝑢+1 − 𝑥𝑢||2 E 𝑔 𝑥𝑢+1 ≤ 𝐹 𝑔 𝑥𝑢 − 𝜃 2 ||∇𝑔 𝑥𝑢 ||2 + 𝑀 2 𝜃2 ⋅ 𝑊𝑏𝑠 min

𝑢 ||∇𝑔 𝑥𝑢 || ≤ 𝑀 𝑔 𝑥0 − 𝑔 𝑥∗

⋅ 𝑊𝑏𝑠

1 4

𝑈

1 4

≤ 𝜗 𝑈 = 𝑃 𝑀 ⋅ 𝑊𝑏𝑠 ⋅ (𝑔 𝑥0 − 𝑔 𝑥∗ ) 𝜗4

Summary: Convergence to FOSP

Algorithm

• No. of Gradient Calls (Non-convex)
• No. of Gradient Calls (Convex)

GD [Folkore; Nesterov] 𝑃 1 𝜗2 𝑃 1 𝜗 AGD [Ghadimi & Lan-2013] 𝑃 1 𝜗 𝑃 1 𝜗 Algorithm

• No. of Gradient Calls

Convex Case GD [Folkore] 𝑃( 𝑜 𝜗2) 𝑃(𝑜 𝜗) AGD [Ghadimi & Lan’2013] 𝑃 𝑜 𝜗 𝑃 𝑜 𝜗 SGD [Ghadimi & Lan’2013] 𝑃( 1 𝜗4) 𝑃( 1 𝜗2) SVRG [Reddi et al-2016, Allen- Zhu&Hazan-2016] 𝑃(𝑜 + 𝑜

2 3/𝜗2)

𝑃(𝑜 + 𝑜/𝜗2) MSVRG [Reddi et al-2016] 𝑃(min( 1 𝜗4 , 𝑜

2 3

𝜗2)) 𝑃 𝑜 + 𝑜 𝜗2

𝑔 𝑥 = 1 𝑜 ෍

𝑗=1 𝑜

𝑔

𝑗(𝑥)

Finding Second Order Stationary Points (SOSP)

Second Order Stationary Point (SOSP) if:

• ∇𝑔 𝑥 = 0
• ∇2𝑔 𝑥 ≽ 0

Approximate SOSP:

• ||∇𝑔 𝑥 || ≤ 𝜗
• 𝜇𝑛𝑗𝑜 ∇2𝑔 𝑥

≥ − 𝜍𝜗

Second Order Stationary Point (SOSP)

image credit: academo.org

Cubic Regularization (Nesterov and Polyak-2006)

• For t=1, 2, … (until convergence)

𝑥𝑢+1 = arg min

𝑥 𝑔 𝑥𝑢 + 𝑥 − 𝑥𝑢, ∇𝑔 𝑥𝑢

+ 1 2 𝑥 − 𝑥𝑢 𝑈∇2𝑔 𝑥𝑢 𝑥 − 𝑥𝑢 + 𝜍 6 ||𝑥 − 𝑥𝑢||3

• Assumption: Hessian continuity, i.e., ||∇2𝑔 𝑦 − ∇2𝑔 𝑧 || ≤ 𝜍||𝑦 − 𝑧||
• Convergence to SOSP? 𝑈 = 𝑃(

1 𝜗1.5)

• But requires Hessian computation! (even storage is 𝑃(𝑒2)
• Can we find SOSP using only gradients?

Noisy Gradient Descent for SOSP

• For t=1, 2, … (until convergence)
• If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 )
• 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢
• Else
• 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽)
• Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations
• Claim: above algorithm converges to SOSP in 𝑃(1/𝜗2)

Ge et al-2015, Jin et al-2017

Proof

• FOSP analysis: convergence in 𝑃

1 𝜗2

iterations

• But, ∇2𝑔 𝑥𝑢

≱ 0

• That is, 𝜇𝑛𝑗𝑜 ∇2𝑔 𝑥𝑢

< − 𝜍𝜗

For t=1, 2, … (until convergence) If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 ) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 Else 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽) Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations

image credit: academo.org

Proof

• Random perturbation with Gradient descent

leads to decrease in objective function

For t=1, 2, … (until convergence) If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 ) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 Else 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽) Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations

image credit: academo.org

Proof?

• Random perturbation with Gradient descent leads to decrease in objective

function

• Hessian continuity => function nearly quadratic in small neighborhood
• 𝑔 𝑥 ≈ 𝑔 𝑥𝑢 + ∇𝑔 𝑥𝑢 , 𝑥 − 𝑥𝑢 + 𝑥 − 𝑥𝑢 𝑈∇2𝑔 𝑥𝑢 (𝑥 − 𝑥𝑢)

𝑥𝑠+𝑢 = 𝑥𝑠−1+𝑢 − 𝜃∇2𝑔 𝑥𝑢 𝑥𝑠−1+𝑢 − 𝑥𝑢 ⇒ 𝑥𝑠+𝑢 − 𝑥𝑢 = 𝐽 − 𝜃∇2𝑔 𝑥𝑢

𝑠 𝑥𝑢+1 − 𝑥𝑢

For t=1, 2, … (until convergence) If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 ) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 Else 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽) Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations

Proof?

• Random perturbation with Gradient descent leads to decrease in objective function
• Hessian continuity => function nearly quadratic in small neighborhood
• 𝑔 𝑥 ≈ 𝑔 𝑥𝑢 + ∇𝑔 𝑥𝑢 , 𝑥 − 𝑥𝑢 + 𝑥 − 𝑥𝑢 𝑈∇2𝑔 𝑥𝑢 (𝑥 − 𝑥𝑢)

𝑥𝑠+𝑢 = 𝑥𝑠−1+𝑢 − 𝜃∇2𝑔 𝑥𝑢 𝑥𝑠−1+𝑢 − 𝑥𝑢 ⇒ 𝑥𝑠+𝑢 − 𝑥𝑢 = 𝐽 − 𝜃∇2𝑔 𝑥𝑢

𝑠 𝑥𝑢+1 − 𝑥𝑢

• 𝑥𝑠+𝑢 − 𝑥𝑢 converge to largest eigenvector of 𝐽 − 𝜃∇2𝑔(𝑥𝑢)
• Which is smallest (most negative) eigenvector of ∇2𝑔(𝑥𝑢)
• Hence, 𝑥𝑠+𝑢 − 𝑥𝑢 𝑈∇2𝑔 𝑥𝑢

𝑥𝑠+𝑢 − 𝑥𝑢 ≤ −𝛿2 𝜍𝜗

• 𝑔 𝑥𝑠+𝑢 ≤ 𝑔 𝑥𝑢 − 𝛿2 𝜍𝜗

For t=1, 2, … (until convergence) If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 ) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 Else 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽) Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations

Proof

• Entrapment near SOSP

For t=1, 2, … (until convergence) If ( ||∇𝑔(𝑥𝑢|| ≥ 𝜗 ) 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 Else 𝑥𝑢+1 = 𝑥𝑢 + 𝜂, 𝜂 ∼ 𝛿 ⋅ 𝑂(0, 𝐽) Update 𝑥𝑢+1 = 𝑥𝑢 − 𝜃∇𝑔 𝑥𝑢 for next 𝑠 iterations

Final result: convergence to SOSP in 𝑃(1/𝜗2)

Ge et al-2015, Jin et al-2017 image credit: academo.org

Summary: Convergence to SOSP

Algorithm

• No. of Gradient Calls (Non-convex)
• No. of Gradient Calls (Convex)

Noisy GD [Jin et al-2017, Ge et al- 2015] 𝑃 1 𝜗2 𝑃 1 𝜗 Noisy Accelerated GD [Jin et al- 2017] 𝑃 1 𝜗1.75 𝑃 1 𝜗 Cubic Regularization [Nesterov & Polyak-2006] 𝑃 1 𝜗1.5 N/A Algorithm

• No. of Gradient Calls

Convex Case Noisy GD [Jin et al-2017, Ge et al-2015] 𝑃( 𝑜 𝜗2) 𝑃(𝑜 𝜗) Noisy AGD [Jin et al-2017] 𝑃 𝑜 𝜗1.75 𝑃 𝑜 𝜗 Noisy SGD [Jin et al-2017, Ge et al-2015] 𝑃( 1 𝜗4) 𝑃( 1 𝜗2) SVRG [Allen-Zhu-2018] 𝑃(𝑜 + 𝑜

3 4/𝜗2)

𝑃(𝑜 + 𝑜/𝜗2)

𝑔 𝑥 = 1 𝑜 ෍

𝑗=1 𝑜

𝑔

𝑗(𝑥)

Convergence to Global Optima?

• FOSP/SOSP methods can’t even guarantee

local convergence

• Can we guarantee global optimality for

some “nicer” non-convex problems?

• Yes!!!
• Use statistics ☺

image credit: academo.org

Can Statistics Help: Realizable models!

• Data points: 𝑦𝑗, 𝑧𝑗 ∼ 𝐸
• 𝐸: nice distribution
• 𝐹[𝑧𝑗] = 𝜚(𝑦𝑗, 𝑥∗)

ෝ 𝑥 = arg min

𝑥 ෍ 𝑗

𝑚𝑝𝑡𝑡(𝑧𝑗, 𝜚(𝑦𝑗, 𝑥))

• That is, 𝑥∗ is the optimal solution!
• Parameter learning

Learning Neural Networks: Provably

• 𝑧𝑗 = 1 ⋅ 𝜏 𝑋

∗𝑦𝑗

• 𝑦𝑗 ∼ 𝑂(0, 𝐽)

min

𝑋 ෍ 𝑗

𝑧𝑗−1 ⋅ 𝜏 𝑋𝑦𝑗

2

• Does gradient descent converge to global optima: 𝑋

∗?

• NO!!!
• The objective function has poor local minima [Shamir et al-2017, Lee et al-2017]

𝑦𝑗

1 1 1 1 1

𝑧𝑗 𝑋

∗

Learning Neural Networks: Provably

• But, no local minima within constant distance of 𝑋

∗

• If,

||𝑋

0 − 𝑋 ∗|| ≤ 𝑑

Then, Gradient Descent (𝑋

𝑢+1 = 𝑋 𝑢 − 𝜃∇f(Wt)) converges to 𝑋 ∗

• No. of iterations: log 1/𝜗

Can we get rid of initialization condition? Yes but by changing the network [Liang-Lee-Srikant’2018]

Zhong-Song-J-Bartlett-Dhillon’2017

Learning with Structure

• 𝑧𝑗 = 𝜚 𝑦𝑗, 𝑥∗ , 𝑦𝑗 ∼ 𝐸 ∈ 𝑆𝑒 ,

1 ≤ 𝑗 ≤ 𝑜

• But no. of samples are limited!
• For example, 𝑗𝑔 𝑜 ≤ 𝑒?
• Can we still recover 𝑥∗? In general, no!
• But, what if 𝑥∗ has some structure?

Sparse Linear Regression

• But: 𝑜 ≪ 𝑒
• 𝑥: 𝑡 −sparse (𝑡 non-zeros)
• Information theoretically: 𝑜 = 𝑡 log 𝑒 samples should suffice

0.1 1 ⋮ 0.9

𝑌 𝑥 = =

n

⋮

d

𝑧

41

Learning with structure

• Linear classification/regression
• 𝐷 = {𝑥, ||𝑥||0 ≤ 𝑡}
• 𝑡 ≪ 𝑒
• Matrix completion
• 𝐷 = {𝑋, 𝑠𝑏𝑜𝑙 𝑋 ≤ 𝑠}
• 𝑠 ≪ (𝑒1, 𝑒2)

min

𝑥 𝑔(𝑥)

𝑡. 𝑢. 𝑥 ∈ 𝐷

42

Other Examples

• Low-rank Tensor completion
• 𝐷 = {𝑋, 𝑢𝑓𝑜𝑡𝑝𝑠 − 𝑠𝑏𝑜𝑙 𝑋 ≤ 𝑠}
• 𝑠 ≪ (𝑒1, 𝑒2, 𝑒3)
• Robust PCA
• 𝐷 = {𝑋, 𝑋 = 𝑀 + 𝑇, 𝑠𝑏𝑜𝑙 𝑀 ≤ 𝑠, ||𝑇||0 ≤ 𝑡}
• 𝑠 ≪ 𝑒1, 𝑒2 , 𝑇 ≪ 𝑒1 × 𝑒2

43

Non-convex Structures

• Linear classification/regression
• 𝐷 = {𝑥, ||𝑥||0 ≤ 𝑡}
• 𝑡 ≪ 𝑒
• Matrix completion
• 𝐷 = {𝑋, 𝑠𝑏𝑜𝑙 𝑋 ≤ 𝑠}
• 𝑠 ≪ (𝑒1, 𝑒2)

44

• NP-Hard
• ||𝑥||0: Non-convex
• NP-Hard
• 𝑠𝑏𝑜𝑙(𝑋): Non-convex

Non-convex Structures

• Low-rank Tensor completion
• 𝐷 = {𝑋, 𝑢𝑓𝑜𝑡𝑝𝑠 − 𝑠𝑏𝑜𝑙 𝑋 ≤ 𝑠}
• 𝑠 ≪ (𝑒1, 𝑒2, 𝑒3)
• Robust PCA
• 𝐷 = {𝑋, 𝑋 = 𝑀 + 𝑇, 𝑠𝑏𝑜𝑙 𝑀 ≤ 𝑠, ||𝑇||0 ≤ 𝑡}
• 𝑠 ≪ 𝑒1, 𝑒2 , 𝑇 ≪ 𝑒1 × 𝑒2

45

• Indeterminate
• 𝑢𝑓𝑜𝑡𝑝𝑠𝑠𝑏𝑜𝑙 𝑋 : Non-convex
• NP-Hard
• 𝑠𝑏𝑜𝑙 𝑋 , ||𝑇||0: Non-convex

Technique: Projected Gradient Descent

min

𝑥 𝑔 𝑥

𝑡. 𝑢. 𝑥 ∈ 𝐷

• 𝑥𝑢+1 = 𝑥𝑢 − ∇𝑥𝑔(𝑥𝑢)
• 𝑥𝑢+1 = 𝑄𝐷(𝑥𝑢+1)

46

min

𝑥 ||𝑥 − 𝑥𝑢+1||2

𝑡. 𝑢. 𝑥 ∈ 𝐷

Results for Several Problems

• Sparse regression [Jain et al.’14, Garg and Khandekar’09]
• Sparsity
• Robust Regression [Bhatia et al.’15]
• Sparsity+output sparsity
• Vector-value Regression [Jain & Tewari’15]
• Sparsity+positive definite matrix
• Dictionary Learning [Agarwal et al.’14]
• Matrix Factorization + Sparsity
• Phase Sensing [Netrapalli et al.’13]
• System of Quadratic Equations

47

Results Contd…

• Low-rank Matrix Regression [Jain et al.’10, Jain et al.’13]
• Low-rank structure
• Low-rank Matrix Completion [Jain & Netrapalli’15, Jain et al.’13]
• Low-rank structure
• Robust PCA [Netrapalli et al.’14]
• Low-rank ∩ Sparse Matrices
• Tensor Completion [Jain and Oh’14]
• Low-tensor rank
• Low-rank matrix approximation [Bhojanapalli et al.’15]
• Low-rank structure

48

Sparse Linear Regression

• But: 𝑜 ≪ 𝑒
• 𝑥: 𝑡 −sparse (𝑡 non-zeros)

0.1 1 ⋮ 0.9

𝑌 𝑥 = =

n

⋮

d

𝑧

49

Sparse Linear Regression

min

𝑥 ||𝑧 − 𝑌𝑥||2

𝑡. 𝑢. ||𝑥||0 ≤ 𝑡

• ||𝑧 − 𝑌𝑥||2 = σ𝑗 𝑧𝑗 − 𝑦𝑗, 𝑥

2

• ||𝑥||0: number of non-zeros
• NP-hard problem in general 
• 𝑀0: non-convex function

50

Technique: Projected Gradient Descent

min

𝑥 𝑔 𝑥 = ||𝑧 − 𝑌𝑥||2

𝑡. 𝑢. ||𝑥||0 ≤ 𝑡

• 𝑥𝑢+1 = 𝑥𝑢 − ∇𝑥𝑔(𝑥𝑢)
• 𝑥𝑢+1 = 𝑄

𝑡(𝑥𝑢+1) [Jain, Tewari, Kar’2014]

51

min

𝑥 ||𝑥 − 𝑥𝑢+1||2

𝑡. 𝑢. ||𝑥||0 ≤ 𝑡

Statistical Guarantees

𝑧𝑗 = 〈𝑦𝑗, 𝑥∗〉 + 𝜃𝑗

• 𝑦𝑗 ∼ 𝑂(0, Σ)
• 𝜃𝑗 ∼ 𝑂(0, 𝜂2)
• 𝑥∗: 𝑡 −sparse

|| ෝ 𝑥 − 𝑥∗|| ≤ 𝜂𝜆3 𝑡 log 𝑒 𝑜

• 𝜆 = 𝜇1(Σ)/𝜇𝑒(Σ)

[Jain, Tewari, Kar’2014]

52

Low-rank Matrix Completion

min

𝑋

෍

𝑗,𝑘 ∈Ω

𝑋

𝑗𝑘 − 𝑁𝑗𝑘 2

𝑡. 𝑢 𝐬𝐛𝐨𝐥 𝑋 ≤ 𝑠 Ω: set of known entries

• Special case of low-rank matrix regression
• However, assumptions required by the regression analysis not satisfied

Technique: Projected Gradient Descent

• 𝑋

0 = 0

• For t=0:T-1

𝑋

𝑢+1 = 𝑄 𝑠 𝑋 𝑢 − 𝜃𝛂𝑔(𝑋 𝑢)

• 𝑄

𝑙(𝑎): projection onto set of rank-r projection

• Singular Value Projection
• Pros:
• Fast (always, rank-r SVD)
• Matrix completion: 𝑃(𝑒 ⋅ 𝑠3)!
• Cons: In general, might not even converge
• Our Result: Convergence under “certain” assumptions

[Jain, Tewari, Kar’2014], [Netrapalli, Jain’2014], [Jain, Meka, Dhillon’2009]

54

Guarantees

• Projected Gradient Descent:
• 𝑋

𝑢+1 = 𝑄 𝑠 𝑋 𝑢 − 𝜃𝛼𝑋𝑔 𝑋 𝑢

, ∀𝑢

• Show 𝜗-approximate recovery in log

1 𝜗 iterations

• Assuming:
• 𝑁: incoherent
• Ω: uniformly sampled
• Ω ≥ 𝑜 ⋅ 𝑠5 ⋅ log3 𝑜
• First near linear time algorithm for exact Matrix Completion with

finite samples

[J., Netrapalli’2015]

General Result for Any Function

• 𝑔: 𝑆𝑒 → 𝑆
• 𝑔: satisfies RSC/RSS, i.e.,

𝛽 ⋅ 𝐽𝑒×𝑒 ≼ 𝐼 𝑥 ≼ 𝑀 ⋅ 𝐽𝑒×𝑒, 𝑗𝑔, 𝑥 ∈ 𝐷

• PGD guarantee:

𝑔 𝑥𝑈 ≤ 𝑔 𝑥∗ + 𝜗

After 𝑈 = 𝑃(log

𝑔 𝑥0 𝜗

) steps

• If

𝑀 𝛽 ≤ 1.5

[J., Tewari, Kar’2014]

min

𝑥 𝑔(𝑥)

𝑡. 𝑢. 𝑥 ∈ 𝐷

Learning with Latent Variables

min

𝑥,𝑨 𝑔(𝑥, 𝑨)

• Typically, 𝑨 are latent variables
• E.g., clustering: 𝑥: means of clusters, 𝑨: cluster index
• 𝑔: non − convex
• NP-hard to solve in general

Alternating Minimization

𝑨𝑢+1 = arg min

𝑨

𝑔(𝑥𝑢, 𝑨) 𝑥𝑢+1 = arg min

𝑥 𝑔(𝑥, 𝑨𝑢+1)

• For example, if 𝑔(𝑥𝑢, 𝑨) is convex and 𝑔(𝑥, 𝑨𝑢) is convex
• Does that imply 𝑔(𝑥, 𝑨) is convex?
• No!!!
• 𝑔 𝑥, 𝑨 = 𝑥 ⋅ 𝑨
• Linear in both 𝑥, 𝑨 individually
• So can Alt. Min. converge to global optima?

image credit: academo.org

Low-rank Matrix Completion

min

𝑋

෍

𝑗,𝑘 ∈Ω

𝑋

𝑗𝑘 − 𝑁𝑗𝑘 2

𝑡. 𝑢 𝐬𝐛𝐨𝐥 𝑋 ≤ 𝑠 Ω: set of known entries

• Special case of low-rank matrix regression
• However, assumptions required by the regression analysis not satisfied

Matrix Completion: Alternating Minimization

W 𝑉 𝑊𝑈 ≅ ×

× F 2

y − 𝐘 ⋅

) (

𝑊𝑢+1 = min

𝑊 ||𝑧 − 𝑌 ⋅ (𝑉𝑢𝑊𝑈)||2 2

𝑉𝑢+1 = min

𝑉 ||𝑧 − 𝑌 ⋅ 𝑉 𝑊𝑢+1 𝑈 ||2 2

Results: Alternating Minimization

• Provable global convergence [J., Netrapalli, Sanghavi’13]
• Rate of convergence: geometric

||𝑋𝑈 − 𝑋∗|| ≤ 2−𝑈

• Assumptions:
• Matrix regression: RIP
• Matrix completion: uniform sampling and no. samples Ω ≥ 𝑃(𝑒𝑙6)

[Jain, Netrapalli, Sanghavi’13]

General Results

min

𝑥,𝑨 𝑔(𝑥, 𝑨)

• Alternating minimization: optimal?
• If:
• Joint Restricted Strong Convexity (Strong convexity close to the optimal)
• Restricted Smoothness (smoothness near optimal)
• Cross-product bound:

𝑥 − 𝑥∗, ∇𝑥𝑔 𝑥, 𝑨 − ∇𝑥𝑔 𝑥, 𝑨∗ − 𝑨 − 𝑨∗, ∇𝑨𝑔 𝑥, 𝑨 − ∇𝑨𝑔 𝑥∗, 𝑨 ≤ 𝑃(|𝑥 − 𝑥∗|2 + |𝑨 − 𝑨∗|2)

Ha and Barber-2017, Jain and Kar-2018

Summary I

Non-convex Optimization: two approaches

• 1. General non-convex functions
• a. First Order Stationary Point
• b. Second Order Stationary Point
• 2. Statistical non-convex functions: learning with structure
• a. Projected Gradient Descent (RSC/RSS)
• b. Alternating minimization/EM algorithms (RSC/RSS)

Summary II

• First Order Stationary Point :𝑔 𝑥 ≤ 𝑔 𝑥′ + ||𝑥 − 𝑥′||2
• Tools: gradient descent, acceleration, stochastic gd, variance reduction
• Key quantity: iteration complexity
• Several questions: for example, can we do better? Especially in finite sum setting
• Second order stationary point: 𝑔 𝑥 ≤ 𝑔 𝑥′ + ||𝑥 − 𝑥′||3
• Tools: noise+gd, noise+acceleration, noise+sgd, noise+variance reduction
• Several questions: better rates? Can we remove Lipschitz condition on Hessian?

Summary III

• Projected Gradient Descent
• Works under statistical conditions like RSC/RSS
• Still several open questions for most problems
• E.g., tight guarantees support recovery for sparse linear regression?
• Alternating minimization
• Works under some assumptions on 𝑔
• What is the weakest condition on 𝑔 for Alt. Min. to work?