Relationships between necessary optimality conditions for the 2 - 0 - - PowerPoint PPT Presentation

Relationships between necessary optimality conditions for the ℓ2-ℓ0 minimization problem.

Emmanuel Soubies Imaging in Paris Seminar - October 3 2019

Signal and Communication Group, IRIT, Universit´ e de Toulouse, CNRS 1

• L. Blanc-F´

eraud

• G. Aubert

2

Outline of the talk

• 1. Introduction ℓ2-ℓ0 Minimization
• 2. Necessary optimality conditions
• 3. Relationship between optimality conditions
• 4. Quantifying “optimal” points
• 5. Algorithms and necessary optimality conditions
• 6. Concluding remarks

3

Introduction ℓ2-ℓ0 Minimization

4

Formulation

The ℓ2-ℓ0 minimization problem ˆ x ∈

• arg min

x∈RN

1 2Ax − y2 + λx0

• ,

◮ A ∈ RM×N with M ≪ N, ◮ Sparsity is modeled with the ℓ0 pseudo-norm: x0 = ♯ {xi, i ∈ [1, . . . , N] : xi = 0} ◮ Non-convex and NP-Hard problem [Natarajan, 1995, Nguyen et al., 2019]

5

Formulation

The ℓ2-ℓ0 minimization problem ˆ x ∈

• arg min

x∈RN

1 2Ax − y2 + λx0

• ,

◮ A ∈ RM×N with M ≪ N, ◮ Sparsity is modeled with the ℓ0 pseudo-norm: x0 = ♯ {xi, i ∈ [1, . . . , N] : xi = 0} ◮ Non-convex and NP-Hard problem [Natarajan, 1995, Nguyen et al., 2019] Applications

• Inverse problems,
• Statistical regression,
• Machine learning,
• Compressed sensing ...

5

A Brief Literature Review

Convex relaxations Greedy algorithms Iterative thresholding algorithms Global optimization

6

A Brief Literature Review

Convex relaxations ◮ Basic Pursuit De-Noising [Chen et al., 2001], LASSO, [Tibshirani, 1996], ˆ x ∈

• arg min

x∈RN

1 2Ax − d2

2 + λx1

• Under some conditions (RIP [Candes et al., 2006, Cand`

es and Wakin, 2008],

incoherence [Donoho, 2006, Gribonval and Nielsen, 2003] ...) → exact recovery by ℓ1-minimization, ◮ The convex non-convex strategy [Selesnick and Farshchian, 2017, Selesnick, 2017] Greedy algorithms Iterative thresholding algorithms Global optimization

6

A Brief Literature Review

Convex relaxations Greedy algorithms Idea: add one by one non-zero components to the solution: ◮ Matching Pursuit (MP) [Mallat and Zhang, 1993], Orthogonal Matching Pursuit (OMP) [Pati et al., 1993], Orthogonal Least Squares (OLS)

[Chen et al., 1991] ...

→ under some conditions, optimality guarantees for OMP

[Tropp, 2004] and OLS [Soussen et al., 2013],

◮ Forward-backward extensions: Single Best Replacement (SBR)

[Soussen et al., 2011] ...

Iterative thresholding algorithms Global optimization

6

A Brief Literature Review

Convex relaxations Greedy algorithms Iterative thresholding algorithms ◮ Iterative hard thresholding (IHT) [Blumensath and Davies, 2009], ◮ Subspace pursuit [Dai and Milenkovic, 2009], ◮ Hard thresholding pursuit [Foucart, 2011], ◮ Compressive Sampling Matching Pursuit (CoSaMP) [Needell and Tropp, 2009], Global optimization

6

A Brief Literature Review

Convex relaxations Greedy algorithms Iterative thresholding algorithms Global optimization Mixed integer programming together with branch and bounds algorithms

[Bourguignon et al., 2016] → limited to moderate size problems,

6

A Brief Literature Review

Continuous non-convex relaxations of the ℓ0-norm ˆ x ∈

• arg min

x∈RN

1 2Ax − y2 + Φ(x)

• .

◮ Adaptive Lasso [Zou, 2006], ◮ Nonnegative Garrote [Breiman, 1995], ◮ Exponential approximation [Mangasarian, 1996], ◮ Log-Sum Penalty [Cand`

es et al., 2008],

◮ Smoothly Clipped Absolute Deviation (SCAD) [Fan and Li, 2001], ◮ Minimax Concave Penalty (MCP) [Zhang, 2010], ◮ ℓp-norms 0 < p < 1 [Chartrand, 2007, Foucart and Lai, 2009] ◮ Smoothed ℓ0-norm Penalty (SL0) [Mohimani et al., 2009], ◮ Class of smooth non-convex penalties [Chouzenoux et al., 2013], ◮ Smoothed norm ratio [Repetti et al., 2015, Cherni et al., 2019].

7

A Brief Literature Review

Continuous non-convex relaxations of the ℓ0-norm ˆ x ∈

• arg min

x∈RN

1 2Ax − y2 + Φ(x)

• .

−2 2 0.5 1 1.5 ℓ0 ℓ1 Cap-ℓ1 ℓ0.5 Log-Sum SCAD MCP Exp

7

A Brief Literature Review

Continuous non-convex relaxations of the ℓ0-norm ˆ x ∈

• arg min

x∈RN

1 2Ax − y2 + Φ(x)

• .

There exist a class of penalties Φ that lead to exact continuous relaxation of the ℓ2-ℓ0 functional in the sense that their global minimizers coincide [Soubies et al., 2017, Carlsson, 2019]

7

Motivation of this work

Motivation of this work NP-hardness implies that ◮ one cannot expect, in general, to attain an optimal point ◮ verifying the optimality of a point ˆ x is also, in general, intractable Interest in studying the “restrictiveness” of tractable necessary (but not sufficient) optimality conditions

8

Motivation of this work

Motivation of this work NP-hardness implies that ◮ one cannot expect, in general, to attain an optimal point ◮ verifying the optimality of a point ˆ x is also, in general, intractable Interest in studying the “restrictiveness” of tractable necessary (but not sufficient) optimality conditions Some notations ◮ IN = {1, . . . , N}, ◮ σx = {i ∈ IN : xi = 0} denotes the support of x ∈ RN, ◮ xω ∈ R#ω is the restriction of x ∈ RN to the elements indexed by ω ◮ Aω ∈ RM×#ω is the restriction of A ∈ RM×N to the columns indexed by ω ◮ ai = A{i} ∈ RM

8

Necessary optimality conditions

9

Local optimality

Definition (Local optimality) A point x ∈ RN is a local minimizer of F0 if and only if x ∈

• arg min

u∈RN Au − y2 s.t. σu ⊆ σx

• ,
• r, equivalently, if x is such that

ai, Ax − y = 0 ∀i ∈ σx

10

Local optimality

Definition (Local optimality) A point x ∈ RN is a local minimizer of F0 if and only if x ∈

• arg min

u∈RN Au − y2 s.t. σu ⊆ σx

• ,
• r, equivalently, if x is such that

ai, Ax − y = 0 ∀i ∈ σx ◮ Local minimizers of F0 are independent of λ, ◮ When rank(A) < N (e.g., M < N), local minimizers of F0 are uncountable, ◮ An important subset contains strict local minimizers, ∃ε > 0, ∀u ∈ B2(x, ε), F0(x) < F0(u). ◮ Indeed, global minimizers of F0 are strict [Nikolova, 2013].

10

Local optimality

Theorem (Strict local optimality for F0 [Nikolova, 2013]) A local minimizer x ∈ RN of F0 is strict if and only if rank(Aσx) = #σx.

11

Local optimality

Theorem (Strict local optimality for F0 [Nikolova, 2013]) A local minimizer x ∈ RN of F0 is strict if and only if rank(Aσx) = #σx. ◮ A strict (local) minimizer of F0 can be easily computed:

• 1. choose a support ω ∈ Ωmax where

Ωmax =

M

• r=0

Ωr and Ωr :=

• ω ∈ IN : #ω = r = rank (Aω)
• (Ω0 = ∅)
• 2. solve (Aω)T Aωxω = (Aω)T y

⇒ Given A and y we can compute all the strict (local) minimizers of F0 by solving the restricted normal equations ∀ω ∈ Ωmax ◮ #Ωmax is finite (but huge)

11

Support-based optimality conditions

Definition (Partial support coordinate-wise points [Beck and Hallak, 2018]) A local minimizer x ∈ RN of F0 is said to be partial support coordinate-wise (CW) optimal for F0 if it verifies F0(x) ≤ min{F0(u) : u ∈ {u−

x , uswap x

, u+

x }}

where u−

x , uswap x

, and u+

x are local minimizers of F0 with supports

• σu−

x = σx\{ix}

• σuswap

x

= σx\{ix} ∪ {jx}

• σu+

x = σx ∪ {jx}

for ix ∈

• arg min

k∈σx |xk|

• ,

jx ∈

• arg max

k∈(σx)c | ak, Ax − y |

• .

12

L-Stationarity

Definition (L-stationarity [Tropp, 2006, Beck and Hallak, 2018]) A point x ∈ RN is said to be L-stationary for F0 (L > 0), if x ∈

• arg min

u∈RN

1 2TL(x) − u2 + λ L u0

• ,

where TL(x) = x − L−1AT(Ax − y).

13

L-Stationarity

Definition (L-stationarity [Tropp, 2006, Beck and Hallak, 2018]) A point x ∈ RN is said to be L-stationary for F0 (L > 0), if x ∈

• arg min

u∈RN

1 2TL(x) − u2 + λ L u0

• ,

where TL(x) = x − L−1AT(Ax − y). ◮ For L ≥ A2, L-stationarity points are fixed points of the IHT algorithm [Blumensath and Davies, 2009, Attouch et al., 2013].

13

Conditions based on exact relaxations

Exact continuous relaxations: Motivation Are there continuous relaxations of F0 of the form ˜ F(x) = 1 2Ax − y2 +

N

• i=1

φi(xi), such that, for all y ∈ RM, arg min

x∈RN

˜ F(x) = arg min

x∈RN F0(x),

(P1) x local minimizer of ˜ F = ⇒ x local minimizer of F0 (P2)

14

Conditions based on exact relaxations

Exact continuous relaxations: Motivation Are there continuous relaxations of F0 of the form ˜ F(x) = 1 2Ax − y2 +

N

• i=1

φi(xi), such that, for all y ∈ RM, arg min

x∈RN

˜ F(x) = arg min

x∈RN F0(x),

(P1) x local minimizer of ˜ F = ⇒ x local minimizer of F0 (P2) ◮ Properties (P1) and (P2) imply that local optimality for ˜ F is a necessary

• ptimality condition for F0.

◮ Moreover, there is no converse property for (P2) − → ˜ F can potentially remove local (not global) minimizers of F0

14

Conditions based on exact relaxations

Theorem ([Soubies et al., 2017]) Properties (P1) and (P2) are satisfied ∀y ∈ RM if and only if Φ verifies the following conditions: ∀i ∈ {1, . . . , N}, φi(0) = 0, ∀u ∈ R \ (βi−, βi+), φi(u) = λ, ∀u ∈ (βi−, βi+) \ {0}, φi(u) > φcelo(ai, λ; u), ∀u ∈ Bi \ {0}, limv→u

v<u φ′ i(v) > limv→u v>u φ′ i(v),

∀u ∈]βi−, βi+[\Bi,        φ′′

i (u) ≤ −ai2 and

∀ε > 0, ∃vε ∈ (u − ε, u + ε) s.t. φ′′

i (vε) < −ai2

u

Moreover, global minimizers of ˜ F are strict.

15

Conditions based on exact relaxations

Theorem ([Soubies et al., 2017]) Properties (P1) and (P2) are satisfied ∀y ∈ RM if and only if Φ verifies the following conditions: ∀i ∈ {1, . . . , N}, φi(0) = 0, ∀u ∈ R \ (βi−, βi+), φi(u) = λ, ∀u ∈ (βi−, βi+) \ {0}, φi(u) > φcelo(ai, λ; u), ∀u ∈ Bi \ {0}, limv→u

v<u φ′ i(v) > limv→u v>u φ′ i(v),

∀u ∈]βi−, βi+[\Bi,        φ′′

i (u) ≤ −ai2 and

∀ε > 0, ∃vε ∈ (u − ε, u + ε) s.t. φ′′

i (vε) < −ai2

u

Moreover, global minimizers of ˜ F are strict.

15

Conditions based on exact relaxations

Theorem ([Soubies et al., 2017]) Properties (P1) and (P2) are satisfied ∀y ∈ RM if and only if Φ verifies the following conditions: ∀i ∈ {1, . . . , N}, φi(0) = 0, ∀u ∈ R \ (βi−, βi+), φi(u) = λ, ∀u ∈ (βi−, βi+) \ {0}, φi(u) > φcelo(ai, λ; u), ∀u ∈ Bi \ {0}, limv→u

v<u φ′ i(v) > limv→u v>u φ′ i(v),

∀u ∈]βi−, βi+[\Bi,        φ′′

i (u) ≤ −ai2 and

∀ε > 0, ∃vε ∈ (u − ε, u + ε) s.t. φ′′

i (vε) < −ai2

λ u −

√ 2λ a √ 2λ a

β− β+

Moreover, global minimizers of ˜ F are strict.

15

Conditions based on exact relaxations

Theorem ([Soubies et al., 2017]) Properties (P1) and (P2) are satisfied ∀y ∈ RM if and only if Φ verifies the following conditions: ∀i ∈ {1, . . . , N}, φi(0) = 0, ∀u ∈ R \ (βi−, βi+), φi(u) = λ, ∀u ∈ (βi−, βi+) \ {0}, φi(u) > φcelo(ai, λ; u), ∀u ∈ Bi \ {0}, limv→u

v<u φ′ i(v) > limv→u v>u φ′ i(v),

∀u ∈]βi−, βi+[\Bi,        φ′′

i (u) ≤ −ai2 and

∀ε > 0, ∃vε ∈ (u − ε, u + ε) s.t. φ′′

i (vε) < −ai2

λ u φcelo −

√ 2λ a √ 2λ a

β− β+

Moreover, global minimizers of ˜ F are strict.

15

Conditions based on exact relaxations

Theorem ([Soubies et al., 2017]) Properties (P1) and (P2) are satisfied ∀y ∈ RM if and only if Φ verifies the following conditions: ∀i ∈ {1, . . . , N}, φi(0) = 0, ∀u ∈ R \ (βi−, βi+), φi(u) = λ, ∀u ∈ (βi−, βi+) \ {0}, φi(u) > φcelo(ai, λ; u), ∀u ∈ Bi \ {0}, limv→u

v<u φ′ i(v) > limv→u v>u φ′ i(v),

∀u ∈]βi−, βi+[\Bi,        φ′′

i (u) ≤ −ai2 and

∀ε > 0, ∃vε ∈ (u − ε, u + ε) s.t. φ′′

i (vε) < −ai2

λ u φcelo −

√ 2λ a √ 2λ a

β− β+

Moreover, global minimizers of ˜ F are strict.

15

Conditions based on exact relaxations

The continuous exact ℓ0 penalty (CEL0) [Soubies et al., 2015] Φcelo(x) =

N

• i=1

φcelo(ai, λ; xi), φcelo(a, λ; u) = λ − a2 2

• |u| −

√ 2λ a 2 1{|u|≤

√ 2λ a

} 16

Conditions based on exact relaxations

The continuous exact ℓ0 penalty (CEL0) [Soubies et al., 2015] Φcelo(x) =

N

• i=1

φcelo(ai, λ; xi), φcelo(a, λ; u) = λ − a2 2

• |u| −

√ 2λ a 2 1{|u|≤

√ 2λ a

}

Properties of the CEL0 relaxation ˜ Fcelo ◮ Inferior limit of the derived class of penalties, ◮ Convex hull of F0 when A has nonzero orthogonal columns, ◮ Convex w.r.t. each variable xi for all A ∈ RN×M, ◮ Potentially eliminates the larger amount of local minimizers of F0,

16

Conditions based on exact relaxations

Properties of the CEL0 relaxation ˜ Fcelo ◮ Inferior limit of the derived class of penalties, ◮ Convex hull of F0 when A has nonzero orthogonal columns, ◮ Convex w.r.t. each variable xi for all A ∈ RN×M, ◮ Potentially eliminates the larger amount of local minimizers of F0, −2 2 2 4 F0 ˜ Fcelo −2 2 4 6 2 4 6 −2 2 4 6 2 4 6

16

Conditions based on exact relaxations

Theorem (Link between global minimizers of F0 and ˜ Fcelo) (i) The set of global minimizers of F0 is included in the one of ˜ Fcelo, arg min

x∈RN F0(x) ⊆ arg min x∈RN

˜ Fcelo(x) (ii) Conversely if ˆ x ∈ RN is a global minimizer of ˜ Fcelo, ˆ x0 defined by ∀ i ∈ IN, ˆ x0

i = ˆ

xi1{|ˆ

xi |≥

√ 2λ ai } ,

is a global minimizer of F0 and ˜ Fcelo(ˆ x) = ˜ Fcelo(ˆ x0) = F0(ˆ x0).

17

Conditions based on exact relaxations

Theorem (Link between global minimizers of F0 and ˜ Fcelo) (i) The set of global minimizers of F0 is included in the one of ˜ Fcelo, arg min

x∈RN F0(x) ⊆ arg min x∈RN

˜ Fcelo(x) (ii) Conversely if ˆ x ∈ RN is a global minimizer of ˜ Fcelo, ˆ x0 defined by ∀ i ∈ IN, ˆ x0

i = ˆ

xi1{|ˆ

xi |≥

√ 2λ ai } ,

is a global minimizer of F0 and ˜ Fcelo(ˆ x) = ˜ Fcelo(ˆ x0) = F0(ˆ x0).

17

4 1 1.5 2 2.5 3

ˆ xi

√ 2λ ai

Conditions based on exact relaxations

Theorem (Link between global minimizers of F0 and ˜ Fcelo) (i) The set of global minimizers of F0 is included in the one of ˜ Fcelo, arg min

x∈RN F0(x) ⊆ arg min x∈RN

˜ Fcelo(x) (ii) Conversely if ˆ x ∈ RN is a global minimizer of ˜ Fcelo, ˆ x0 defined by ∀ i ∈ IN, ˆ x0

i = ˆ

xi1{|ˆ

xi |≥

√ 2λ ai } ,

is a global minimizer of F0 and ˜ Fcelo(ˆ x) = ˜ Fcelo(ˆ x0) = F0(ˆ x0). Theorem (Link between local minimizers of F0 and ˜ Fcelo) ˆ x ∈ RN local minimizer of ˜ Fcelo = ⇒ ˆ x0 local minimizer of F0 and ˜ Fcelo(ˆ x) = ˜ Fcelo(ˆ x0) = F0(ˆ x0)

17

Conditions based on exact relaxations

Assumption 1 When F0 does not admit a unique global minimizer, every pair (ˆ x1, ˆ x2) of global minimizers (ˆ x1 = ˆ x2) verify ˆ x1 − ˆ x20 > 1.

18

Conditions based on exact relaxations

Assumption 1 When F0 does not admit a unique global minimizer, every pair (ˆ x1, ˆ x2) of global minimizers (ˆ x1 = ˆ x2) verify ˆ x1 − ˆ x20 > 1. Corollary ([Soubies et al., 2019]) Under Assumption 1, global minimizers of F0 and ˜ Fcelo coincide. Moreover, they are stricts for both F0 and ˜ Fcelo.

18

Conditions based on exact relaxations

−0.5 0.5 1 1.5 −0.5 0.5 1 1.5 Global minimizers −0.5 0.5 1 1.5 −0.5 0.5 1 1.5 Global minimizers

A =  1 1  , d =  1 1  , λ = 0.5 F0 ˜ Fcelo

19

Conditions based on exact relaxations

−0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 −0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Local minimizers Global minimizer −0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 −0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 Global minimizer

A =  0.5 2 2 1  , d =   2 1.5  , λ = 0.5 F0 ˜ Fcelo

19

Conditions based on exact relaxations

−0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 Local minimizers Global minimizer −0.6 −0.4 −0.2 0.2 0.4 0.6 0.8 1 −0.4 −0.2 0.2 0.4 0.6 0.8 1 1.2 Local minimizer Global minimizer

A =  3 2 1 3  , d =  1 2  , λ = 1 F0 ˜ Fcelo

19

Relationship between optimality conditions

20

Relationship between optimality conditions

We introduced four necessary (not sufficient) optimality conditions for F0 ◮ Strict local optimality for F0 ◮ L-stationarity ◮ Partial support coordinate-wise optimality ◮ Strict local optimality for ˜ F Are there any inclusion properties between the sets of points associated to these conditions ?

21

Relationship between optimality conditions

minglob{F0} minloc{F0}

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0}

Theorem (L-stationary ⇒ minloc{F0} [Beck and Hallak, 2018]) Let x ∈ RN be a L-stationary point of F0 for some L > 0. Then x is a local minimizer of F0.

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

Theorem (SuppCWpartial{F0} ⇒ L-stationary [Beck and Hallak, 2018]) If x ∈ RN is a partial support CW point of F0, then it is a L-stationary point of F0 for any L ≥ A2.

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

Theorem (SuppCWpartial{F0} ⇒ minst

loc{F0} [Soubies et al., 2019])

Let A satisfy the unique representation property (URP)a. Let x ∈ RN be a partial support CW point of F0. Then it is a strict local minimizer of F0.

aA matrix A ∈ RM×N satisfies the URP [Gorodnitsky and Rao, 1997] if any min{M, N} columns

• f A are linearly independent.

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

Theorem (minst

loc{˜

F} ⇒ minst

loc{F0} [Soubies et al., 2015])

Let x be a strict local minimizer of ˜ F, then x is a strict local minimizer of F0.

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

Theorem (minst

loc{˜

F} ⇒ L-stationary [Soubies et al., 2019]) Let x ∈ RN be a strict local minimizer of ˜

• F. Then it is a L-stationary point of

F0 for any L ≥ maxi∈IN ai2.

22

Relationship between optimality conditions

minglob{F0} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

Theorem (minglob{F0} ⇒ SuppCWpartial{F0} [Beck and Hallak, 2018]) Let x ∈ RN be a global minimizer of F0. Then it is partial support CW point of F0.

22

Relationship between optimality conditions

minglob{F0} = minglob{˜ F} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

Theorem (minglob{F0} = minglob{˜ F} [Soubies et al., 2019]) Global minimizers of F0 and ˜ F coincide. Moreover, they are stricts for both F0 and ˜ F.

22

Relationship between optimality conditions

minglob{F0} = minglob{˜ F} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

No inclusion property between L-Stat and minst

loc{F0} [Soubies et al., 2019] 22

Relationship between optimality conditions

minglob{F0} = minglob{˜ F} minloc{F0} minst

loc{F0}

L–Stat{F0} SuppCWpartial{F0}

U R P

minst

loc{˜

F}

URP (for almost all λ)

Theorem (SuppCWpartial{F0} ⇒ minst

loc{˜

F} [Soubies et al., 2019]) Let A satisfy the URP and have unit norm columns. Then, for all λ ∈ R>0\Λ (where Λ is a subset of R>0 whose Lebesgue measure is zero), each partial support CW point of F0 is a strict local minimizer of ˜ F.

22

Quantifying “optimal” points

23

Quantifying “optimal” points

Objective Let S0 be the set of strict local minimizers of F0 and define the three following subsets ◮ SCW = {x ∈ S0 : x partial support CW point} ◮ ˜ S = {x ∈ S0 : x strict local minimizer of ˜ F} ◮ SL = {x ∈ S0 : x L-stationary point} Then, our goal is to quantify the cardinality of these sets, i.e., # ˜ S, #SCW, and #SL.

24

Quantifying “optimal” points

Experiment Given A ∈ R5×10 and y ∈ R5, we proceed as follows:

• 1. Compute all strict local minimizers of F0 → S0 (independent of λ),
• 2. For each λ ∈ {λ1, . . . , λP}, determine the subset of S0 that contains

points verifying a given necessary optimality condition (i.e., ˜ S, SCW, SL),

• 3. Repeat 1-2 for different A ∈ R5×10 and y ∈ R5, and draw the average

evolution of # ˜ S, #SCW, and #SL, with respect to λ.

25

Quantifying “optimal” points

Experiment Given A ∈ R5×10 and y ∈ R5, we proceed as follows:

• 1. Compute all strict local minimizers of F0 → S0 (independent of λ),
• 2. For each λ ∈ {λ1, . . . , λP}, determine the subset of S0 that contains

points verifying a given necessary optimality condition (i.e., ˜ S, SCW, SL),

• 3. Repeat 1-2 for different A ∈ R5×10 and y ∈ R5, and draw the average

evolution of # ˜ S, #SCW, and #SL, with respect to λ. Considered scenario

• 1. The entries of A and y are drawn from a standard normal distribution,
• 2. The entries of A and y are drawn from a uniform distribution on [0, 1],
• 3. A is a “sampled Toeplitz” matrix built from a Gaussian kernel with

σ2 = 0.04. The entries of y are drawn from a standard normal distribution.

25

Quantifying “optimal” points

10−5 10−3 10−1 101 103 200 400 600 λ Cardinality A: Random Normal

#S0 # ˜ S #SCW #SL

10−5 10−3 10−1 101 103 λ A: Random Uniform 10−5 10−3 10−1 101 103 λ A: Sampled Toeplitz

26

Theorem ([Soubies et al., 2019]) Let S0 be the set of strict local minimizers of F0. Let ˜ S, SCW, and SL be the subsets of S0 containing the strict local minimizers of ˜ F, the partial support CW points, and the L-stationary points, respectively. Finally, define XLS = arg min

x∈RN Ax − y2,

(1) the solution set of the un-penalized least-squares problem. Then, for all S ∈ { ˜ S, SL, SCW}, there exists (under the URP of A for SCW) λ0 > 0 and λ∞ > 0 such that

• 1. ∀λ ∈ (0, λ0), S = (S0 ∩ XLS),
• 2. ∀λ ∈ (λ∞, +∞), S = {0RN }.

27

Algorithms and necessary optimality conditions

28

Algorithms and necessary optimality conditions

One can expect that the efficiency of a given algorithm A to minimize F0 depends on the “restrictiveness” of the necessary optimality condition it guarantees to converge to.

29

Algorithms and necessary optimality conditions

One can expect that the efficiency of a given algorithm A to minimize F0 depends on the “restrictiveness” of the necessary optimality condition it guarantees to converge to. Numerical Experiment We consider four algorithms: ◮ CowS: the CW support optimatity (CowS) algorithm. Greedy method that converges to a partial support CW point [Beck and Hallak, 2018]. ◮ IHT: the iterative hard thresholding (IHT) algorithm that ensures the convergence to an L-stationary point [Attouch et al., 2013]

[Beck and Hallak, 2018] [Blumensath and Davies, 2009].

◮ FBS-CEL0: the forward-backward splitting (FBS) algorithm applied to the CEL0 relaxation ˜

• F. FBS ensures the convergence to a stationary

point of ˜ F [Attouch et al., 2013]. ◮ IRL1-CEL0: the iterative reweighted-ℓ1 (IRL1) algorithm [Ochs et al., 2015] also used to obtain a stationary point of ˜ F.

29

Algorithms and necessary optimality conditions

Numerical Experiment ◮ K = 50 instances of the problem (i.e., instances of A and y), ◮ M = 100, N = 256, λ ∈ {10−8, 10−3}, ◮ Initial point x0 = 0RN , ◮ Generation of A

◮ i.i.d. entries drawn from a standard normal distribution, ◮ i.i.d. entries drawn from a uniform distribution, ◮ “sampled Toeplitz” matrix with a Gaussian kernel

◮ Measurements y ∈ RM are generated according to y = Ax⋆ + n, where x⋆ is a 30-sparse vector (i.e., x⋆0 = 30) with non-zero entries drawn from a normal distribution. n is a vector of Gaussian noise with standard deviation 10−2.

30

Algorithms and necessary optimality conditions

2 4 6 8 · 10−6 F0(ˆ x)/F0(x0) (λ = 10−8) A: Random Normal CowS IRL1-CEL0 IHT FBS-CEL0 · 10−6 A: Random Uniform · 10−6 A: Sampled Toeplitz 10 20 30 40 50 0.05 0.1 0.15 Instance F0(ˆ x)/F0(x0) (λ = 10−3) 10 20 30 40 50 Instance 10 20 30 40 50 Instance 31

Concluding remarks

32

Concluding remarks

Support-based optimality conditions Exact continuous relaxations

33

Concluding remarks

Support-based optimality conditions ◮ The more restrictive (stronger) among the conditions studied in this work ◮ Trade-off between restrictiveness and computational burden Exact continuous relaxations

33

Concluding remarks

Support-based optimality conditions ◮ The more restrictive (stronger) among the conditions studied in this work ◮ Trade-off between restrictiveness and computational burden Exact continuous relaxations ◮ Open the door to a variety of nonsmooth nonconvex optimization algorithms to minimize F0, ◮ Although the derived inclusion properties plays in favor of greedy-based conditions, numerical experiments reveal that the associated algorithms are comparable in terms of their ability to minimize F0. − → specific analysis of fixed points of algorithms that minimize ˜ F ◮ For moderate-size problems, exact continuous relaxations ˜ F can be globally minimized using Lasserres hierarchies [Marmin et al., 2019]

33

◮ New Insights on the Optimality Conditions of the ℓ2-ℓ0 Minimization Problem. Submitted, 2019. Emmanuel Soubies, Laure Blanc-F´ eraud and Gilles Aubert ◮ Proximal Mapping for Symmetric Penalty and Sparsity. SIAM Journal on Optimization 28-1, pp. 496-527, 2018. Amir Beck and Nadav Hallak ◮ Description of the Minimizers of Least Squares Regularized with ℓ0-norm. Uniqueness of the Global Minimizer. SIAM Journal on Imaging Science 6-2, pp. 904-937, 2013. Mila Nikolova ◮ A unified view of exact continuous penalties for ℓ2-ℓ0 minimization. SIAM Journal on Optimization 27-3, pp. 2034-2060, 2017. Emmanuel Soubies, Laure Blanc-F´ eraud and Gilles Aubert ◮ A Continuous Exact ℓ0 penalty (CEL0) for least squares regularized problem. SIAM Journal on Imaging Science 8-3, pp. 1574-1606, 2015. Emmanuel Soubies, Laure Blanc-F´ eraud and Gilles Aubert.

Thank you !

33

References i

Attouch, H., Bolte, J., and Svaiter, B. F. (2013). Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, forward–backward splitting, and regularized Gauss–Seidel methods. Mathematical Programming, 137(1):91–129. Beck, A. and Hallak, N. (2018). Proximal Mapping for Symmetric Penalty and Sparsity. SIAM Journal on Optimization, 28(1):496–527. Blumensath, T. and Davies, M. E. (2009). Iterative hard thresholding for compressed sensing. Applied and Computational Harmonic Analysis, 27(3):265–274. Bourguignon, S., Ninin, J., Carfantan, H., and Mongeau, M. (2016). Exact Sparse Approximation Problems via Mixed-Integer Programming: Formulations and Computational Performance. IEEE Transactions on Signal Processing, 64(6):1405–1419.

34

References ii

Breiman, L. (1995). Better Subset Regression Using the Nonnegative Garrote. Technometrics, 37(4):373–384. Candes, E. J., Romberg, J., and Tao, T. (2006). Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2):489–509. Cand` es, E. J. and Wakin, M. B. (2008). An Introduction To Compressive Sampling [A sensing/sampling paradigm that goes against the common knowledge in data acquisition]. IEEE Signal Processing Magazine, 25:21–30. Cand` es, E. J., Wakin, M. B., and Boyd, S. P. (2008). Enhancing Sparsity by Reweighted ℓ1 Minimization. Journal of Fourier Analysis and Applications, 14(5):877–905.

35

References iii

Carlsson, M. (2019). On Convex Envelopes and Regularization of Non-convex Functionals Without Moving Global Minima. Journal of Optimization Theory and Applications, 183(1):66–84. Chartrand, R. (2007). Exact Reconstruction of Sparse Signals via Nonconvex Minimization. IEEE Signal Processing Letters, 14(10):707–710. Chen, S., Cowan, C. F. N., and Grant, P. M. (1991). Orthogonal least squares learning algorithm for radial basis function networks. IEEE Transactions on Neural Networks, 2(2):302–309. Chen, S., Donoho, D., and Saunders, M. (2001). Atomic Decomposition by Basis Pursuit. SIAM Review, 43(1):129–159.

36

References iv

Cherni, A., Chouzenoux, E., Duval, L., and Pesquet, J.-C. (2019). Forme liss´ ee de rapports de normes lp/lq (SPOQ) pour la reconstruction des signaux avec p´ enalisation parcimonieuse. In GRETSI 2019, Lille, France. Chouzenoux, E., Jezierska, A., Pesquet, J., and Talbot, H. (2013). A Majorize-Minimize Subspace Approach for ℓ2 − ℓ0 Image Regularization. SIAM Journal on Imaging Sciences, 6(1):563–591. Dai, W. and Milenkovic, O. (2009). Subspace Pursuit for Compressive Sensing Signal Reconstruction. IEEE Transactions on Information Theory, 55(5):2230–2249. Donoho, D. L. (2006). For most large underdetermined systems of linear equations the minimal ℓ1-norm solution is also the sparsest solution. Communications on Pure and Applied Mathematics, 59(6):797–829.

37

References v

Fan, J. and Li, R. (2001). Variable Selection via Nonconcave Penalized Likelihood and Its Oracle Properties. Journal of the American Statistical Association, 96(456):1348–1360. Foucart, S. (2011). Hard Thresholding Pursuit: An Algorithm for Compressive Sensing. SIAM Journal on Numerical Analysis, 49(6):2543–2563. Foucart, S. and Lai, M.-J. (2009). Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0<q ≤ 1. Applied and Computational Harmonic Analysis, 26(3):395–407. Gorodnitsky, I. F. and Rao, B. D. (1997). Sparse signal reconstruction from limited data using FOCUSS: A re-weighted minimum norm algorithm. IEEE Transactions on Signal Processing, pages 600–616.

38

References vi

Gribonval, R. and Nielsen, M. (2003). Sparse representations in unions of bases. IEEE Transactions on Information Theory, 49(12):3320–3325. Mallat, S. G. and Zhang, Z. (1993). Matching pursuits with time-frequency dictionaries. IEEE Transactions on Signal Processing, 41(12):3397–3415. Mangasarian, O. L. (1996). Machine Learning via Polyhedral Concave Minimization. In Fischer, H., Riedm¨ uller, B., and Sch¨ affler, S., editors, Applied Mathematics and Parallel Computing: Festschrift for Klaus Ritter, pages 175–188. Physica-Verlag HD, Heidelberg.

39

References vii

Marmin, A., Castella, M., and Pesquet, J. (2019). How to Globally Solve Non-convex Optimization Problems Involving an Approximate ℓ0 Penalization. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5601–5605. Mohimani, H., Babaie-Zadeh, M., and Jutten, C. (2009). A Fast Approach for Overcomplete Sparse Decomposition Based on Smoothed ℓ0-Norm. IEEE Transactions on Signal Processing, 57(1):289–301. Natarajan, B. (1995). Sparse Approximate Solutions to Linear Systems. SIAM Journal on Computing, 24(2):227–234.

40

References viii

Needell, D. and Tropp, J. A. (2009). CoSaMP: Iterative signal recovery from incomplete and inaccurate samples. Applied and Computational Harmonic Analysis, 26(3):301–321. Nguyen, T. T., Soussen, C., Idier, J., and Djermoune, E.-H. (2019). NP-hardness of ℓ0 minimization problems: revision and extension to the non-negative setting. In International Conference on Sampling Theory and Applications (SampTa), Bordeaux. Nikolova, M. (2013). Description of the Minimizers of Least Squares Regularized with ℓ0-norm. Uniqueness of the Global Minimizer. SIAM Journal on Imaging Sciences, 6(2):904–937.

41

References ix

Ochs, P., Dosovitskiy, A., Brox, T., and Pock, T. (2015). On Iteratively Reweighted Algorithms for Nonsmooth Nonconvex Optimization in Computer Vision. SIAM Journal on Imaging Sciences, 8(1):331–372. Pati, Y. C., Rezaiifar, R., and Krishnaprasad, P. S. (1993). Orthogonal matching pursuit: recursive function approximation with applications to wavelet decomposition. In Proceedings of 27th Asilomar Conference on Signals, Systems and Computers, pages 40–44 vol.1. Repetti, A., Pham, M. Q., Duval, L., Chouzenoux, ´ E., and Pesquet, J. (2015). Euclid in a Taxicab: Sparse Blind Deconvolution with Smoothed ℓ1/ℓ2 Regularization. IEEE Signal Processing Letters, 22(5):539–543.

42

References x

Selesnick, I. (2017). Sparse Regularization via Convex Analysis. IEEE Transactions on Signal Processing, 65(17):4481–4494. Selesnick, I. and Farshchian, M. (2017). Sparse Signal Approximation via Nonseparable Regularization. IEEE Transactions on Signal Processing, 65(10):2561–2575. Soubies, E., Blanc-F´ eraud, L., and Aubert, G. (2015). A Continuous Exact ℓ0 Penalty (CEL0) for Least Squares Regularized Problem. SIAM Journal on Imaging Sciences, 8(3):1607–1639. Soubies, E., Blanc-F´ eraud, L., and Aubert, G. (2017). A Unified View of Exact Continuous Penalties for ℓ2-ℓ0 Minimization. SIAM Journal on Optimization, 27(3):2034–2060.

43

References xi

Soubies, E., Blanc-F´ eraud, L., and Aubert, G. (2019). New Insights on the Optimality Conditions of the l2-l0 Minimization Problem. Soussen, C., Gribonval, R., Idier, J., and Herzet, C. (2013). Joint K-Step Analysis of Orthogonal Matching Pursuit and Orthogonal Least Squares. IEEE Transactions on Information Theory, 59(5):3158–3174. Soussen, C., Idier, J., Brie, D., and Duan, J. (2011). From Bernoulli–Gaussian Deconvolution to Sparse Signal Restoration. IEEE Transactions on Signal Processing, 59(10):4572–4584. Tibshirani, R. (1996). Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288.

44

References xii

Tropp, J. A. (2004). Greed is good: algorithmic results for sparse approximation. IEEE Transactions on Information Theory, 50(10):2231–2242. Tropp, J. A. (2006). Just relax: convex programming methods for identifying sparse signals in noise. IEEE Transactions on Information Theory, 52(3):1030–1051. Zhang, C.-H. (2010). Nearly unbiased variable selection under minimax concave penalty. The Annals of Statistics, 38(2):894–942. Zou, H. (2006). The Adaptive Lasso and Its Oracle Properties. Journal of the American Statistical Association, 101(476):1418–1429.

45