Escaping Saddle Points with Adaptive Gradient Methods Matthew Staib - - PowerPoint PPT Presentation

Escaping Saddle Points with
 Adaptive Gradient Methods

Matthew Staib1, Sashank Reddi2, Satyen Kale2, Sanjiv Kumar2, Suvrit Sra1

• 1. MIT EECS
• 2. Google Research, New York

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Adam, RMSProp and friends

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Adam, RMSProp and friends

• Empirically: good non-convex performance

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Adam, RMSProp and friends

• Empirically: good non-convex performance
• Limited theory, some non-convergence results [e.g.

Reddi et al. ‘18]

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Adam, RMSProp and friends

• Empirically: good non-convex performance
• Limited theory, some non-convergence results [e.g.

Reddi et al. ‘18]

• Our take: adaptive methods escape saddles (in

words: via isotropic noise), reach SOSPs

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Adam, RMSProp and friends

• Empirically: good non-convex performance
• Limited theory, some non-convergence results [e.g.

Reddi et al. ‘18]

• Our take: adaptive methods escape saddles (in

words: via isotropic noise), reach SOSPs

This paper: The first second-order rates for adaptive methods

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

xt+1 ← xt − ηgt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

xt+1 ← xt − ηgt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

xt+1 ← xt − ηgt + ξt

E[ξt] = 0 Cov(ξt) ∝ I

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

xt+1 ← xt − ηgt + ξt

E[gt] = 0 Cov(gt) =???

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

E[gt] = 0 Cov(gt) =???

Cov(E[gtgT

t ]−1/2gt) = I

xt+1 ← xt − ηE[gtgT

t ]−1/2gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

RMSProp

E[gt] = 0 Cov(gt) =???

Cov(E[gtgT

t ]−1/2gt) = I

xt+1 ← xt − ηE[gtgT

t ]−1/2gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

xt+1 ← xt − η ˆ G−1/2

t

gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

ˆ Gt : = Pt

i=1 βt−igigT i

≈ E[gtgT

t ] =: Gt

xt+1 ← xt − η ˆ G−1/2

t

gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

ˆ Gt : = Pt

i=1 βt−igigT i

≈ E[gtgT

t ] =: Gt

xt+1 ← xt − η ˆ G−1/2

t

gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

ˆ Gt : = Pt

i=1 βt−igigT i

≈ E[gtgT

t ] =: Gt

(Theorem: w.h.p. if β chosen correctly given η)

xt+1 ← xt − η ˆ G−1/2

t

gt

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Theorem (informal): RMSProp converges to a (τ, τ1/2)-stationary point in time O(τ-5).

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

6

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

• Also works for standard diagonal approx.

6

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

• Also works for standard diagonal approx.

Concrete takeaways:

6

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

• Also works for standard diagonal approx.

Concrete takeaways:

• How to set β as a function of stepsize η

6

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

• Also works for standard diagonal approx.

Concrete takeaways:

• How to set β as a function of stepsize η
• How to set the ε in the RMSProp

denominator:

6

(E[gtgT

t ]1/2 + εI)−1

Escaping Saddle Points with Adaptive Gradient Methods — Matthew Staib

Summary

• New approach: theory for general

preconditioners

• Also works for standard diagonal approx.

Concrete takeaways:

• How to set β as a function of stepsize η
• How to set the ε in the RMSProp

denominator:

6

(E[gtgT

t ]1/2 + εI)−1

Poster #98