A framework for non-convex recovery of low dimensional models in - - PowerPoint PPT Presentation

A framework for non-convex recovery of low dimensional models in infinite dimension GDR MIA, October 2020 Yann Traonmilin (CNRS, IMB), Jean-Fran¸ cois Aujol (U. Bdx, IMB), Arthur Leclaire (U. Bdx, IMB)

GDR MIA 2020, Yann Traonmilin 1

What do these problems have in common?

Low rank matrix recovery: Off-the-grid sparse spike recovery:

Gaussian mixture estimation from random moments:

• 5

• 15
• 10
• 5

GDR MIA 2020, Yann Traonmilin 2

Non-convex inverse problems with low-dimensional models Linear inverse problem with low dimensional model Smooth parametrization of the model in Rd Recovery guarantees of a non-convex functional under a given number of measurements m 0(dlog(d)) Non-convex optimization techniques studied theoretically and used in practice (initialization + descent)

GDR MIA 2020, Yann Traonmilin 3

Objective

Study the performance of non-convex techniques under the same general non-convex framework. Give new results for examples of non-convex recovery.

GDR MIA 2020, Yann Traonmilin 4

Outline

Introduction Non-convex ideal decoder and the RIP Basins of attraction of global minimizers Application Conclusion

GDR MIA 2020, Yann Traonmilin 5

The setting

Measurement of x0 (e.g. signal, image, object of interest) yl = x0, αl + el with αl ∈ D. We summarize y = Ax0 + e This makes sense for D Banach space and x0 ∈ D∗ a locally convex topological vector space. A underdetermined ⇒ we need a low dimensional model set : x0 ∈ Σ.

GDR MIA 2020, Yann Traonmilin 6

Examples of low dimensional models

LR matrices: Σ = Σr := {ZZ T : Z ∈ Rp×r} Off-the-grid sparse spikes: Σ = Σk,ǫ := k

• i=1

aiδti : ti − tj2 > ǫ, ti2 < R

• Gaussian mixtures:

Σ = Σk,ǫ,Γ := k

• i=1

aiµti : ti − tjΓ > ǫ, ti2 < R

• ,

where dµti(t) = e− 1

2 t−ti2 Γ dt, u2

Γ = utΓ−1u and Γ is the

fixed known covariance matrix.

GDR MIA 2020, Yann Traonmilin 7

Ideal decoder and performance

We consider the estimation method (ideal decoder) : x∗ ∈ arg min

x∈Σ Ax − y2 2

Problem : how to quantify recovery when no norm is attached to D∗ ? We want x∗ − x02

H ≤ Ce2 2

where x∗ − x02

H is a metric measuring estimation

quality (Hilbert norm in our case).

GDR MIA 2020, Yann Traonmilin 8

Restricted isometries (RIP)

For x ∈ Σ − Σ: (1 − γ)x2 ≤ Ax2

2 ≤ (1 + γ)x2

Sufficient condition on A to guarantee the success of the ideal decoder (and convex methods in classical sparse recovery methods). Necessary for uniform recovery. Verified under a condition on number of (compressive) measurements in our 3 examples.

GDR MIA 2020, Yann Traonmilin 9

Wait a minute ....

The ideal decoder is far from convex. It can even be NP hard !

GDR MIA 2020, Yann Traonmilin 10

Outline

Introduction Non-convex ideal decoder and the RIP Basins of attraction of global minimizers Application Conclusion

GDR MIA 2020, Yann Traonmilin 11

Why consider the non-convex ideal decoder? The following strategy to perform the non-convex minimization can be successful (with theoretical guarantees). Perform a clever intialization Apply a descent algorithm

This strategy is found in phase recovery [Waldspurger, 2018], low rank matrix recovery [Zhao et al., 2015] and blind deconvolution [Li et al., 2018, Cambareri and Jacques, 2018], sparse spike estimation [Flinth and Weiss, 2019, Traonmilin, Aujol, Leclaire, 2019-2020].

GDR MIA 2020, Yann Traonmilin 12

How can success be guaranteed with such a strategy? Continuous parameter space Conditions on the number of measurements and dimensionality of the model set. Success if initialization fall in the basin of attraction

• f a global minimum.

◮ Let’s study the basins of attractions of our problem!

GDR MIA 2020, Yann Traonmilin 13

Parametrization of the model

When we look at the non-convex minimization in the parameter space it is in fact very smooth inside the constraints θ∗ ∈ arg min

θ∈Θ g(θ) = arg min θ∈Θ Aφ(θ) − y2 2.

where φ is the parametrization functional Θ is the parameter set Θ := φ−1(Σ) ⊂ Rd

GDR MIA 2020, Yann Traonmilin 14

Gradient descent in the parameter space

Consider the fixed step gradient descent : θn+1 = θn − τ∇g(θn) Basin of attraction: Λ ⊂ Rd is a g-basin of attraction

• f θ∗ if there exists τ > 0, such that if θ0 ∈ Λ then the

sequence g(θn) converges to g(θ∗). ◮ As g is smooth, make sure the Hessian is positive on a set Λ and θn stays in Λ. ◮ In the LR case, no local convexity. We need to look at the Hessian in relevant directions

GDR MIA 2020, Yann Traonmilin 15

Shape of the basin Λβ and indeterminacy Simple ℓ2 ball is not enough to manage all cases Instead use distance to a set of equivalent parametrizations d(θ, θ∗) := min

˜ θ∈Θ φ(˜ θ)=φ(θ∗)

˜ θ − θ2 Shape of basins of attraction Λβ := {θ ∈ Θ : d(θ, θ∗) < β}.

GDR MIA 2020, Yann Traonmilin 16

General theorem : regularity hypotheses

Regularity Hypotheses: Let A be a weak-* continuous linear map from the space D∗ to Cm. Suppose A has the RIP on Σ − Σ with constant γ and φ is weak-* continuous and twice weak-* Gateaux differentiable. Let θ∗ be a global minimizer of g on Θ. Let us assume that there exists β1 > 0 such that θ ∈ Λ2β1 implies φ(θ) ∈ Σ (local stability of the model set) and ˜ θ = p(θ, θ∗) is unique; there is Cφ,θ∗ > 0 such that ∀θ ∈ Λ2β1, φ(θ) − φ(θ∗)H ≤ Cφ,θ∗d(θ, θ∗); the first-order derivatives of Aφ are uniformly bounded on φ−1(θ∗): the second-order derivatives of Aφ are uniformly bounded on Λ2β1:

GDR MIA 2020, Yann Traonmilin 17

General Theorem : Basin of attraction

Theorem [Traonmilin, Aujol, Leclaire, 2020] Under the previous hypotheses, let β2 := (1 − γ) Cφ,θ∗√1 + γ inf

θ∈Λβ1

inf

z∈[θ,˜ θ]

• ∂˜

θ−θφ(z)2 H

A∂2

˜ θ−θφ(z)2

• −

1 Cφ,θ∗√1 + γ e2 > 0. (1) Then Λmin(β1,β2) is a g-basin of attraction of θ∗. With this theorem, with the right gradient descent step and given θ0 ∈ Λmin(β1,β2), we guarantee that: φ(θn) − x02

H ≤

4 1 − γ e2

2 + O

1 n

• .

(2)

GDR MIA 2020, Yann Traonmilin 18

Outline

Introduction Non-convex ideal decoder and the RIP Basins of attraction of global minimizers Application Conclusion

GDR MIA 2020, Yann Traonmilin 19

Application : Low rank matrix estimation

Study kind of outdated from a practical point of view (global convergence of stochastic gradient descent) Ideal decoder (Burer - Monteiro) : min

Z∈Rp×r A(ZZ T) − y2 2.

Basin of attraction with y = A(Z0Z T

0 ):

ΛβLR := {Z : inf

H∈O(r) ZH − Z0F < βLR}

With RIP on Σ2r with constant γ, basin of attraction with βLR := 1 8κ(Z0)−1 (1 − γ) (1 + γ)σmin(Z0)

GDR MIA 2020, Yann Traonmilin 20

Application : sparse spikes recovery

Ideal Decoder: min

ai∈R; ti∈B2(R);∀i=j,ti−tj2>ǫ

• A

k

• i=1

aiδti

• − y
• 2

2

Shape of the basin Λβspikes := {θ : θ − θ∗2 < βspikes} [Traonmilin and Aujol, 2019] With RIP on Σk, ǫ

2 − Σk, ǫ 2 , y = Ax0, basin of

attraction with βspikes := min ǫ 4, |a1| 2 , Cspikes

• (3)

where Cspikes ∝ 1−γ

1+γ and Cspikes ∝ |amin| |amax|, all constants can be explicit.

GDR MIA 2020, Yann Traonmilin 21

Application : Gaussian mixtures

New result !!! Ideal decoder min

ai∈R; ti∈B2(R);∀i=j,ti−tjΓ>ǫ

• A

k

• i=1

aiµti

• − y
• 2

2

Shape of the basin ΛβGMM := {θ : θ − θ∗2 < βGMM} With RIP on Σk, ǫ

2 ,Γ − Σk, ǫ 2 ,Γ, y = Ax0, basin of attraction with

βGMM = min

• ǫ
• λmin(Γ)

8 , |a1| 2 , CGMM

• where CGMM ∝ 1−γ

1+γ and CGMM ∝ |amin| |amax|

GDR MIA 2020, Yann Traonmilin 22

Initialization by back-projection: the real challenge?

Ideal backprojection z ∈ D with z :=

m

• l=1

ylαl. (4) with the RIP, (1 − γ)x02

H ≤ x0, z ≤ (1 + γ)x02 H.

(5) Problem : z ∈ D. How can we go from z to θinit ? Possible for 2D/3D spikes imaging [Traonmilin, Aujol, Leclaire, 2020]:

20 40 60 80 100 10 20 30 40 50 60 70 80 90 100

GDR MIA 2020, Yann Traonmilin 23

Outline

Introduction Non-convex ideal decoder and the RIP Basins of attraction of global minimizers Application Conclusion

GDR MIA 2020, Yann Traonmilin 24

Future work We gave: General theory of non-convex estimation for linear inverse problems with low-dimensional models. Explicit basins of attraction of global minimizers with link to number of measurements. Future work: GMM with covariance estimation? Tight results? Systematic approach to initialization?

GDR MIA 2020, Yann Traonmilin 25

Thanks ! yanntraonmilin.wordpress.com !!! Internship + Phd opportunity : ANR EFFIREG !!!

Available : [1] The basins of attraction of the global minimizers of non-convex inverse problems with low-dimensional models in infinite dimension, Traonmilin, Aujol and Leclaire, preprint HAL, 2020. Also: [2] The basins of attraction of the global minimizers of the non-convex sparse spike estimation problem, Traonmilin and Aujol, Inverse Problems; 2018. [3] Projected gradient descent for non-convex sparse spike estimation, Traonmilin, Aujol and Leclaire, Signal Processing Letters, 2020.

GDR MIA 2020, Yann Traonmilin 26

Questions?

Introduction Non-convex ideal decoder and the RIP Basins of attraction of global minimizers Application Conclusion

GDR MIA 2020, Yann Traonmilin 27