Support Recovery for Orthogonal Matching Pursuit: Upper and Lower - - PowerPoint PPT Presentation

Support Recovery for Orthogonal Matching Pursuit:

Upper and Lower bounds Raghav Somani 1, Chirag Gupta 2, Prateek Jain 1 & Praneeth Netrapalli 1

1Microsoft Research Lab - India 2Machine Learning Department,

Carnegie Mellon University

December 6, 2018

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 1 / 7

Sparse Linear Regression (SLR)

Sparse Linear Regression (SLR)

A y x = arg x ¯ min

‖x ≤ ‖0 s∗

2 2

Unconditionally, NP hard. Tractable under the assumption of Restricted Strong Convexity (RSC). Fundamental quantity capturing hardness :-

Standard optimization : Condition number κ = smoothness strong convexity Sparse optimization : Restricted Condition number ˜ κ = restricted smoothness restricted strong convexity

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 2 / 7

Sparse Linear Regression (SLR)

Sparse Linear Regression (SLR)

A y x = arg x ¯ min

‖x ≤ ‖0 s∗

2 2

Unconditionally, NP hard. Tractable under the assumption of Restricted Strong Convexity (RSC). Fundamental quantity capturing hardness :-

Standard optimization : Condition number κ = smoothness strong convexity Sparse optimization : Restricted Condition number ˜ κ = restricted smoothness restricted strong convexity

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 2 / 7

Sparse Linear Regression (SLR)

Sparse Linear Regression (SLR)

A y x = arg x ¯ min

‖x ≤ ‖0 s∗

2 2

Unconditionally, NP hard. Tractable under the assumption of Restricted Strong Convexity (RSC). Fundamental quantity capturing hardness :-

Standard optimization : Condition number κ = smoothness strong convexity Sparse optimization : Restricted Condition number ˜ κ = restricted smoothness restricted strong convexity

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 2 / 7

Sparse Linear Regression (SLR)

Sparse Linear Regression (SLR)

A y x = arg x ¯ min

‖x ≤ ‖0 s∗

2 2

Unconditionally, NP hard. Tractable under the assumption of Restricted Strong Convexity (RSC). Fundamental quantity capturing hardness :-

Standard optimization : Condition number κ = smoothness strong convexity Sparse optimization : Restricted Condition number ˜ κ = restricted smoothness restricted strong convexity

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 2 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Sparse Linear Regression (SLR)

Setup and Goals

We work under the model where Observations Measurement matrix y = A ¯ x + η s∗-sparse vector Noise Goals of SLR

1

Bounding Generalization error/Excess Risk - G(x) ∶= 1 n ∥A(x − ¯ x)∥2

2.

2

Support Recovery - Recover the support of ¯ x. We study SLR under RSC assumption for OMP .

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 3 / 7

Orthogonal Matching Pursuit

Orthogonal Matching Pursuit

Incremental Greedy algorithm Popular and easy to implement Widely studied in literature

iteration

(k + 1)th

x̂

k+1

Selected greedily iteration

kth

x̂

k

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 4 / 7

Results Known results and our contribution

Known results and our contribution

Upper bound Lower bound Known Generalization bound ∝ 1 nσ2s∗̃ κ2 1 nσ2s∗̃ κ Our Generalization bound ∝ 1 nσ2s∗̃ κ log̃ κ 1 nσ2s∗̃ κ Unconditional lower bounds for OMP . Support recovery guarantees and its lower bounds. Support Expansion Known ∝ s∗̃ κ2 Our’s ∝ s∗̃ κ log̃ κ

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 5 / 7

Results Known results and our contribution

Known results and our contribution

Upper bound Lower bound Known Generalization bound ∝ 1 nσ2s∗̃ κ2 1 nσ2s∗̃ κ Our Generalization bound ∝ 1 nσ2s∗̃ κ log̃ κ 1 nσ2s∗̃ κ Unconditional lower bounds for OMP . Support recovery guarantees and its lower bounds. Support Expansion Known ∝ s∗̃ κ2 Our’s ∝ s∗̃ κ log̃ κ

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 5 / 7

Results Known results and our contribution

Known results and our contribution

Upper bound Lower bound Known Generalization bound ∝ 1 nσ2s∗̃ κ2 1 nσ2s∗̃ κ Our Generalization bound ∝ 1 nσ2s∗̃ κ log̃ κ 1 nσ2s∗̃ κ Unconditional lower bounds for OMP . Support recovery guarantees and its lower bounds. Support Expansion Known ∝ s∗̃ κ2 Our’s ∝ s∗̃ κ log̃ κ

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 5 / 7

Results Key ideas

A key idea

f(x) = ∥Ax − y∥2

2

If any support is unrecovered, then there is a large additive decrease. f(x) ≥ 0 ⇒ support recovery will happen soon. Recovery with small support ⇒ small generalization error.

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 6 / 7

Results Key ideas

A key idea

f(x) = ∥Ax − y∥2

2

If any support is unrecovered, then there is a large additive decrease. f(x) ≥ 0 ⇒ support recovery will happen soon. Recovery with small support ⇒ small generalization error.

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 6 / 7

Results Key ideas

A key idea

f(x) = ∥Ax − y∥2

2

If any support is unrecovered, then there is a large additive decrease. f(x) ≥ 0 ⇒ support recovery will happen soon. Recovery with small support ⇒ small generalization error.

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 6 / 7

Results Key ideas

A key idea

f(x) = ∥Ax − y∥2

2

If any support is unrecovered, then there is a large additive decrease. f(x) ≥ 0 ⇒ support recovery will happen soon. Recovery with small support ⇒ small generalization error.

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 6 / 7

Results Thank You

Thank You!

• R. Somani, C. Gupta, P

. Jain & P . Netrapalli Support Recovery bounds of OMP December 6, 2018 7 / 7