Tutorial on Gradient methods for non-convex problems Part 1 - - PowerPoint PPT Presentation

Tutorial on Gradient methods for non-convex problems

Part 1

Guillaume Garrigos – November 28th – ENS

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle?

General results

Let 0 ≪ 𝜇𝑙 ≪ 2/𝑀, then: 1) 𝑔 𝑦𝑙 is decreasing 2) if 𝑦𝑙𝑜 → 𝑦∞ then 𝛼𝑔 𝑦∞ = 0 3) Isolated local minima are attractive Proposition f ∶ Rn → R is of class C𝑀

1,1

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙

[Pro 1.2.3, 1.2.5 & Ex. 1.2.18] Bertsekas, Nonlinear Programming, 1999.

General results

Let 0 ≪ 𝜇𝑙 ≪ 2/𝑀, then: 1) 𝑔 𝑦𝑙 is decreasing 2) if 𝑦𝑙𝑜 → 𝑦∞ then 𝛼𝑔 𝑦∞ = 0 3) Isolated local minima are attractive Proposition f ∶ Rn → R is of class C𝑀

1,1

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙

[Pro 1.2.3, 1.2.5 & Ex. 1.2.18] Bertsekas, Nonlinear Programming, 1999.

𝑦𝑙 can have no limit !! No convergence ≠ Lack of regularity, but rather wild ildness

General results

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙

[Ex. 3] Palis, de Melo, Geometric Theory of Dynamical Systems: An Introduction, 1982. H.B.Curry, The method of steepest descent for nonlinear minimization problems, 1944.

f ∶ Rn → R is of class C𝑀

1,1

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

• It is a classic result that `Finite Length’ implies convergence
• Converse is not true (but tricky):

𝑦𝑜 ≔ σ𝑙=1

𝑜 −1 𝑙 𝑙

→ −log(2) but σ 𝑦𝑜+1 − 𝑦𝑜 = σ

1 𝑜

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

• Length is invariant up to a reparametrization in time

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

• Length is invariant up to a reparametrization in time
• We have a natural diffeomorphism 𝑔 ∘ 𝑦 ∶ [0, ∞ → 𝑡∞, 𝑡0] where

𝑡0 = 𝑔(𝑦0) and 𝑡∞ = lim

∞ 𝑔(𝑦 𝑢 )

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

• Length is invariant up to a reparametrization in time
• We have a natural diffeomorphism 𝑔 ∘ 𝑦 ∶ [0, ∞ → 𝑡∞, 𝑡0] where

𝑡0 = 𝑔(𝑦0) and 𝑡∞ = lim

∞ 𝑔(𝑦 𝑢 )

• With 𝑡 = 𝑔(𝑦 𝑢 ) we can define 𝑧 𝑡 = 𝑦

𝑔 ∘ 𝑦 −1 𝑡 s.t. ሶ 𝑧 𝑡 = 𝛼𝑔 𝑧 𝑡 𝛼𝑔 𝑧 𝑡

−2

How to guarantee convergence?

• A sufficient condition for 𝑦(𝑢) to converge is ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 < ∞

• Length is invariant up to a reparametrization in time
• We have a natural diffeomorphism 𝑔 ∘ 𝑦 ∶ [0, ∞ → 𝑡∞, 𝑡0] where

𝑡0 = 𝑔(𝑦0) and 𝑡∞ = lim

∞ 𝑔(𝑦 𝑢 )

• With 𝑡 = 𝑔(𝑦 𝑢 ) we can define 𝑧 𝑡 = 𝑦

𝑔 ∘ 𝑦 −1 𝑡 s.t. ሶ 𝑧 𝑡 = 𝛼𝑔 𝑧 𝑡 𝛼𝑔 𝑧 𝑡

−2

• So the length becomes ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡

Ignore 𝛼𝑔 𝑧 𝑡 = 0 Finite interval !

How to guarantee convergence?

• How to upper bound ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 = ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡 ?

How to guarantee convergence?

• How to upper bound ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 = ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡 ?

• ``Naive’’ hypothesis: 𝛼𝑔 𝑧

≥ 𝐷 i.e. sharpness

How to guarantee convergence?

• How to upper bound ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 = ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡 ?

• ``Naive’’ hypothesis: 𝛼𝑔 𝑧

≥ 𝐷 i.e. sharpness

How to guarantee convergence?

• How to upper bound ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 = ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡 ?

• ``Naive’’ hypothesis: 𝛼𝑔 𝑧

≥ 𝐷 i.e. sharpness

• ``Smart’’ hypothesis:

1 ‖𝛼𝑔 𝑧 𝑡 ‖ ≤ 𝜒′(𝑡) with 𝜒 ≥ 0, 𝜒 ↑

so the length is ≤ 𝜒 𝑡0 − 𝜒 𝑡∞ ≤ 𝜒(𝑡0)

How to guarantee convergence?

• How to upper bound ׬

∞

ሶ 𝑦 𝑢 𝑒𝑢 = ׬

𝑡∞ 𝑡0 1 ‖𝛼𝑔 𝑧 𝑡 ‖ 𝑒𝑡 ?

• ``Naive’’ hypothesis: 𝛼𝑔 𝑧

≥ 𝐷 i.e. sharpness

• ``Smart’’ hypothesis:

1 ‖𝛼𝑔 𝑧 𝑡 ‖ ≤ 𝜒′(𝑡) with 𝜒 ≥ 0, 𝜒 ↑

so the length is ≤ 𝜒 𝑡0 − 𝜒 𝑡∞ ≤ 𝜒(𝑡0)

• In other words 𝜒′ 𝑔 𝑦 𝑢

𝛼𝑔 𝑦 𝑢 ≥ 1 i.e. 𝜒 ∘ 𝑔 is sharp: 𝛼 𝜒 ∘ 𝑔 𝑦 ≥ 1

The Łojasiewicz property

We say that 𝑔 is Łojasiewicz at a critical point 𝑦∗ if 𝜒′ 𝑔 𝑦 − 𝑔 𝑦∗ 𝛼𝑔 𝑦 ≥ 1,

• with 𝜒: [0, ∞[→ [0, ∞[ s.t. 𝜒 0 = 0, 𝜒 ↑, 𝜒 concave
• for all 𝑦 ∈

𝑦′ ∈ 𝔺 𝑦∗, 𝜀 𝑔 𝑦∗ < 𝑔 𝑦′ < 𝑔 𝑦∗ + 𝑠 } Definition

• 𝑔 is Łojasiewicz if it is Łojasiewicz at every critical point
• 𝑔 is p-Łojasiewicz if it is Łojasiewicz at every critical point with

𝜒 𝑡 ≃ 𝑡1/𝑞 : 𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Definition

The Łojasiewicz property : convergence

Let 𝑔 be Łojasiewicz and 𝜇𝑙 ∈ ]0,2/𝑀[. If 𝑦𝑙 is bounded, then it converges to some critical point 𝑦∞. Theorem (convergence) 𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙

Łojasiewicz. Sur les trajectoires du gradient d’une fonction analytique, 1984. Absil, Mahony, Andrews. Convergence of the Iterates of Descent Methods for Analytic Cost Functions, 2005.

Let 𝑔 be Łojasiewicz and 𝜇𝑙 ∈ ]0,2/𝑀[. For every 𝑦∗ ∈ 𝑏𝑠𝑕𝑛𝑗𝑜 𝑔, if 𝑦0 ∼ 𝑦∗ then 𝑦𝑙 converges to 𝑦∞ ∈ 𝑏𝑠𝑕𝑛𝑗𝑜 𝑔. Theorem (capture) f ∶ Rn → R is of class C𝑀

1,1

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) ≥ 𝜒′(𝑔 𝑦𝑙 − 𝑔 𝑦∗ )(𝑔 𝑦𝑙 − 𝑔 𝑦𝑙+1 ) because 𝜒 concave Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) ≥ 𝜒′(𝑔 𝑦𝑙 − 𝑔 𝑦∗ )(𝑔 𝑦𝑙 − 𝑔 𝑦𝑙+1 ) because 𝜒 concave ≥ 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝑑𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙

2

with Descent Lemma Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) ≥ 𝜒′(𝑔 𝑦𝑙 − 𝑔 𝑦∗ )(𝑔 𝑦𝑙 − 𝑔 𝑦𝑙+1 ) because 𝜒 concave ≥ 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝑑𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙

2

with Descent Lemma = 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝐷𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙 𝛼𝑔 𝑦𝑙 Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) ≥ 𝜒′(𝑔 𝑦𝑙 − 𝑔 𝑦∗ )(𝑔 𝑦𝑙 − 𝑔 𝑦𝑙+1 ) because 𝜒 concave ≥ 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝑑𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙

2

with Descent Lemma = 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝐷𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙 𝛼𝑔 𝑦𝑙 Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙 f ∶ Rn → R is of class C𝑀

1,1

𝜒 𝑔 𝑦𝑙 − 𝑔 𝑦∗ − 𝜒 𝑔 𝑦𝑙+1) − 𝑔(𝑦∗) ≥ 𝜒′(𝑔 𝑦𝑙 − 𝑔 𝑦∗ )(𝑔 𝑦𝑙 − 𝑔 𝑦𝑙+1 ) because 𝜒 concave ≥ 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝑑𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙

2

with Descent Lemma = 𝜒′ 𝑔 𝑦𝑙 − 𝑔 𝑦∗ 𝐷𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙 𝛼𝑔 𝑦𝑙 ≥ 1 ⋅ 𝐷𝜇,𝑀 𝑦𝑙+1 − 𝑦𝑙 with: Sketch of proof : show that 𝜒′ 𝑡 ≥ ‖ ሶ 𝑦 𝑢 ‖

The Łojasiewicz property : convergence

Let 𝑔 be Łojasiewicz and 𝜇𝑙 ∈ ]0,2/𝑀[. If 𝑦𝑙 is bounded, then it converges to some critical point 𝑦∞. Theorem (convergence) 𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝑙𝛼𝑔 𝑦𝑙

Łojasiewicz. Sur les trajectoires du gradient d’une fonction analytique, 1984. Absil, Mahony, Andrews. Convergence of the Iterates of Descent Methods for Analytic Cost Functions, 2005.

Let 𝑔 be Łojasiewicz and 𝜇𝑙 ∈ ]0,2/𝑀[. For every 𝑦∗ ∈ 𝑏𝑠𝑕𝑛𝑗𝑜 𝑔, if 𝑦0 ∼ 𝑦∗ then 𝑦𝑙 converges to 𝑦∞ ∈ 𝑏𝑠𝑕𝑛𝑗𝑜 𝑔. Theorem (capture) f ∶ Rn → R is of class C𝑀

1,1

The Łojasiewicz property : in practice

Bolte, Daniilidis, Ley, Mazet, Characterizations of Łojasiewicz inequalities […], 2010.

f ∶ Rn → R is of class C𝑀

1,1

• If 𝑔 is 𝜈-strongly convex, then it is 2-Łojasiewicz with 𝜈 = 𝜈
• If 𝑔 convex and 𝜈 𝑒 𝑦, argmin 𝑔 𝑞 ≤ 𝑔 𝑦 − inf 𝑔

Examples 𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• There is a convex function 𝑔: ℝ2 → ℝ which is not Łojasiewicz

Counter-example

The Łojasiewicz property : in practice

Łojasiewicz, Ensembles semi-analytiques, 1965. Kurdyka, On gradients of functions definable in o-minimal structures, 1998. Bolte, Daniilidis, Lewis, Shiota, Clarke Subgradients of Stratifiable Functions, 2007.

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Any analytic function is p-Łojasiewicz at its critical points. Theorem

• Any semi-algebraic function is p-Łojasiewicz at its critical points.
• Any o-minimal function is Łojasiewicz.

Theorem

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Polynomials by parts

Examples of semi-algebraic functions

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Polynomials by parts

Examples of semi-algebraic functions The class of semi-algebraic functions is stable under:

• addition, multiplication, division, sup, inf
• restriction, composition, inverse 𝑔−1
• derivative

Theorem (``Tarski-Seidenberg’’)

Coste, An Introduction to O-minimal Geometry, 2000.

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Polynomials by parts
• 𝛽 𝑦 0 + 𝐵𝑦 − 𝑐 2, 𝑦 ∗, …

Examples of semi-algebraic functions The class of semi-algebraic functions is stable under:

• addition, multiplication, division, sup, inf
• restriction, composition, inverse 𝑔−1
• derivative

Theorem (``Tarski-Seidenberg’’)

Coste, An Introduction to O-minimal Geometry, 2000.

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Exponential/Logarithmic stuff

Counter-examples of semi-algebraic functions

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Exponential/Logarithmic stuff

Counter-examples of semi-algebraic functions There exists a class of functions (o-minimal structure) which:

• Includes the semi-algebraic structure
• Contains the exponential function
• Has the same stability property than the semi-algebraics
• Is also stable by integration (and resolution of 1st order ODEs)

Theorem

Speissegger, The Pfaffian closure of an o-minimal structure, 1999.

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Take-home message: Virtually an any fu function you can think about is Łojasiewicz, as long as it does not involve ℝ → ℝ 𝑦 ↦ sin(𝑦)

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Take-home message: Virtually an any fu function you can think about is Łojasiewicz, as long as it does not involve ℝ → ℝ 𝑦 ↦ sin(𝑦)

The Łojasiewicz property : in practice

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Take-home message: Virtually an any fu function you can think about is Łojasiewicz, as long as it does not involve ℝ → ℝ 𝑦 ↦ sin(𝑦) So gradient descent ``always converges’’ to a critical point

The Łojasiewicz property : rates for free

Polyak, Gradient methods for the minimisation of functionals, 1963.

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Let 𝑔 be globally 2-Łojasiewicz, 𝜇 ∈]]0,2/𝑀[[, and 𝑦𝑙 → 𝑦∗. Then we have linear convergence : 𝑔 𝑦𝑙+1 − 𝑔 𝑦∗ ≤ 𝜄 𝑔 𝑦𝑙 − 𝑔 𝑦∗ where 𝜄 ∈ [0,1[, and 𝜾 = 𝟐 − 𝝂/𝑴 if 𝜇 = 1/𝑀. Theorem (p=2)

• If 𝑔 strongly convex we have 𝜄 = 1 − 𝜆 2 for 𝜇 = 1/𝑀.
• If 𝑔 is [any weak s. convex notion] we have a better 𝜄.
• Rates become asymptotic if local Łojasiewicz only.

The Łojasiewicz property : rates for free

Attouch, Bolte, On the convergence of the proximal algorithm for nonsmooth functions […], 2009. Chouzenoux, Pesquet, Repetti, A block coordinate variable metric forward-backward algorithm, 2014.

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Let 𝑔 be globally p-Łojasiewicz, 𝜇 ∈]]0,2/𝑀[[, and 𝑦𝑙 → 𝑦∗. Then we have sublinear convergence : 𝑔 𝑦𝑙 − 𝑔 𝑦∗ = 𝑃 𝑙

−𝑞 𝑞−2

Theorem (p>2)

• 𝑞

𝑞−2 → +∞ when 𝑞 ↓ 2 ; 𝑞 𝑞−2 → 1 when 𝑞 ↑ ∞

• Rates are matched for 𝑔 𝑦 = 𝑦𝑞

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

Take-home message: Virtually an any fu function you can think about is Łojasiewicz, as long as it does not involve ℝ → ℝ 𝑦 ↦ sin(𝑦) So gradient descent ``always converges’’ to a critical point What about oth

• ther methods?

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same

Attouch, Bolte, Svaiter, Convergence of descent methods for semi-algebraic and tame problems […], 2013.

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same
• Projected gradient
• Forward Backward
• Douglas Rachford
• ADMM
• Adapts to Maximal Monotone theory (saddle point problems)

Attouch, Bolte, Svaiter, Convergence of descent methods for semi-algebraic and tame problems […], 2013.

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same
• Inertial (2nd order in time) methods :

Bégout, Bolte, Jendoubi, On damped second-order gradient systems, 2015. + refs within! Li et al., Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization, 2017.

ሷ 𝑦 𝑢 + 𝛽(𝑢) ሶ 𝑦 𝑢 + 𝛼𝑔 𝑦 𝑢 = 0 𝑦𝑙+1 = 𝑧𝑙 − 𝜇𝛼𝑔 𝑧𝑙 𝑧𝑙 = 𝑦𝑙 +

1 1+𝛽𝑙 (𝑦𝑙 − 𝑦𝑙−1)

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same
• Inertial (2nd order in time) methods :
• Heavy-Ball (𝛽 𝑢 ≡ 𝛽) ok
• Nesterov (𝛽 𝑢 ∼ 𝛽/𝑢) + Monotone OK pour p=2 global

Bégout, Bolte, Jendoubi, On damped second-order gradient systems, 2015. + refs within! Li et al., Convergence Analysis of Proximal Gradient with Momentum for Nonconvex Optimization, 2017.

ሷ 𝑦 𝑢 + 𝛽(𝑢) ሶ 𝑦 𝑢 + 𝛼𝑔 𝑦 𝑢 = 0 𝑦𝑙+1 = 𝑧𝑙 − 𝜇𝛼𝑔 𝑧𝑙 𝑧𝑙 = 𝑦𝑙 +

1 1+𝛽𝑙 (𝑦𝑙 − 𝑦𝑙−1)

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same
• Inertial (2nd order in time) methods : ok
• Newton-like (2nd order in space) methods: some results for trust-

region methods, Landweber iterations. IDK for Newton/BFGS.

Frankel, Garrigos, Peypouquet, Splitting Methods with Variable Metric […], 2015. + Absil et al., and many others.

How to guarantee convergence?

f ∶ Rn → R is of class C𝑀

1,1

𝜈(𝑔 𝑦 − 𝑔 𝑦∗ )𝑞−1 ≤ 𝛼𝑔 𝑦

𝑞

• Nonsmooth 1st-order methods : works the same
• Inertial (2nd order in time) methods : ok for Heavy-Ball
• Newton-like (2nd order in space) methods: some results for trust-

region methods, Landweber iterations. IDK for Newton/BFGS.

• Stochastic methods: (under global 2-Łojasiewicz)
• SAGA, SVRG: linear rates
• SGD: rates 𝑃( 1/𝑙 ), and linear if vanishing variance
• SVRG + monotone Nesterov: linear rates

Reddi, Hefny, Sra, Poczos, Smola, Stochastic variance reduction for nonconvex optimization, 2016. Karimi, Nutini, Schmidt, Linear Convergence of Gradient and Proximal-Gradient […], 2016. Lei et al., Stochastic Gradient Descent for Nonconvex Learning without Bounded Gradient Assumptions, 2019.

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle?

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle?

What is the limit? The linear case

𝐵 symmetric operator ሶ 𝑦 𝑢 + 𝐵𝑦 𝑢 = 0

What is the limit? The linear case

𝐵 symmetric operator ሶ 𝑦 𝑢 + 𝐵𝑦 𝑢 = 0 𝐵 = 1 1 𝐵 = −1 −1 𝐵 = 1 −1

• Positive eigs. are attractive, negative eigs. are repulsive.
• Converging to the saddle point requires starting from 𝐹−1(𝐵)

ሶ 𝑦

What is the limit? The linear case

𝐵 symmetric operator ሶ 𝑦 𝑢 + 𝐵𝑦 𝑢 = 0 Let ҧ 𝑦 be an equilibrium of the system. We define 𝑋 ҧ 𝑦 = 𝑦 𝑦 𝑢 → ҧ 𝑦 with 𝑦 0 = 𝑦} Definition 𝑋 ҧ 𝑦 ≃ ⊕𝜇>0 𝐹𝜇(𝐵) Theorem If 𝜇𝑛𝑗𝑜 𝐵 < 0, then 𝑋 ҧ 𝑦 has Lebesgue measure 0. Corollary

What is the limit? The potential case

ሶ 𝑦 𝑢 + 𝛼𝑔(𝑦 𝑢 ) = 0 f ∶ Rn → R is of class C2

What is the limit? The potential case

ሶ 𝑦 𝑢 + 𝛼𝑔(𝑦 𝑢 ) = 0 𝑋 ҧ 𝑦 is a submanifold of dimension smaller than the one of ⊕𝜇>0 𝐹𝜇 𝛼2𝑔 ҧ 𝑦 Theorem (Stable Manifold Lemma) If 𝜇𝑛𝑗𝑜 𝛼2𝑔 ҧ 𝑦 < 0, then 𝑋 ҧ 𝑦 has Lebesgue measure 0. Corollary f ∶ Rn → R is of class C2

Perron, Die stabilitätsfrage bei differentialgleichungen, 1930. Smale, Differentiable dynamical systems, 1967.

What is the limit? The linear case

The 3 kinds of critical points:

• The local minima (e.g. 𝜇𝑛𝑗𝑜 𝛼2𝑔

ҧ 𝑦 > 0) , attractive

• The strict saddles 𝜇𝑛𝑗𝑜 𝛼2𝑔

ҧ 𝑦 < 0, repulsive

• The degenerated ones (they have 𝜇𝑛𝑗𝑜 𝛼2𝑔

ҧ 𝑦 = 0), ??? ሶ 𝑦 ሶ 𝑦 𝑢 + 𝛼𝑔(𝑦 𝑢 ) = 0 f ∶ Rn → R is of class C2

What is the limit? The potential case

ሷ 𝑦 𝑢 + 𝛽 ሶ 𝑦 𝑢 + 𝛼𝑔(𝑦 𝑢 ) = 0 f ∶ Rn → R is of class C2

Goudou, Munier, The gradient and heavy ball with friction dynamical systems: the quasiconvex case, 2007.

What is the limit? The potential case

ሷ 𝑦 𝑢 + 𝛽 ሶ 𝑦 𝑢 + 𝛼𝑔(𝑦 𝑢 ) = 0 𝑋 ҧ 𝑦 is a submanifold of dimension smaller than the one of ⊕𝜇>0 𝐹𝜇 𝛼2𝑔 ҧ 𝑦 Theorem (Stable Manifold Lemma) If 𝜇𝑛𝑗𝑜 𝛼2𝑔 ҧ 𝑦 < 0, then 𝑋 ҧ 𝑦 has Lebesgue measure 0. Corollary f ∶ Rn → R is of class C2

Goudou, Munier, The gradient and heavy ball with friction dynamical systems: the quasiconvex case, 2007.

What is the limit? The potential case

Let 𝜇 ∈]0,1/𝑀[. Then 𝑋 ҧ 𝑦 is a submanifold of dimension smaller than the one of ⊕𝜇>0 𝐹𝜇 𝛼2𝑔 ҧ 𝑦 Theorem If 𝜇𝑛𝑗𝑜 𝛼2𝑔 ҧ 𝑦 < 0, then 𝑋 ҧ 𝑦 has Lebesgue measure 0. Corollary f ∶ Rn → R is of class C𝑀

1,1 ∩ 𝐷2

Lee, Simchowitz, Jordan, Recht, Gradient Descent Converges to Minimizers, 2016.

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝛼𝑔 𝑦𝑙 If 𝑔 has no degenerated critical points and is Łojasiewicz, then 𝑦𝑙 converges a.s. to a local minima with random initialization. Corollary

What is the limit? The potential case

f ∶ Rn → R is of class C𝑀

1,1 ∩ 𝐷2

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝛼𝑔 𝑦𝑙 It is time now for examples.

What is the limit? The potential case

f ∶ Rn → R is of class C𝑀

1,1 ∩ 𝐷2

Li et al., Symmetry, saddle points, and global geometry of nonconvex matrix factorization, 2016.

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝛼𝑔 𝑦𝑙 If 𝑔 has no degenerated critical points and is Łojasiewicz, then 𝑦𝑙 converges a.s. to a local minima with random initialization. Corollary Some problems have no degenrated critical points, like the matrix factorization problem a.k.a. two-layer-linear-neural-network min

𝑌∈ℝ𝑒×𝑠 𝑔 𝑌 =

𝑌𝑈𝑌 − 𝐵 𝐺

2

What is the limit? The potential case

f ∶ Rn → R is of class C𝑀

1,1 ∩ 𝐷2

𝑦𝑙+1 = 𝑦𝑙 − 𝜇𝛼𝑔 𝑦𝑙 + 𝜊𝑙 If 𝑔 has no degenerated critical points and is Łojasiewicz, then 𝑦𝑙 converges a.s. to a local minima with random initialization. Corollary The above result remains true for the noisy gradient method. Corollary The noise here must be isotropic! Not the case for SGD (proportional to eigenvalues), but proof can be adapted for RKHS learning with a loss s.t. ℓ′′ = 𝑃( ℓ′ ).

Daneshmand et al., Escaping saddles with stochastic gradients, 2018.

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle?

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle?

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle? Depends strongly on :

• What your problem is
• how you initialize

Yes, this bold statement will be my conclusion

What can we expect?

• Does my algorithm converge? 𝑦∞ ≔ lim

𝑙→+∞ 𝑦𝑙 exists?

• What is the nature of the limit 𝑦∞?

Global/Local minima? Saddle? Depends strongly on :

• What your problem is
• how you initialize

Yes, this bold statement will be my conclusion

Any questions ?