Sparse Reconstruction for Compressed Sensing using Stagewise - - PowerPoint PPT Presentation

Sparse Reconstruction for Compressed Sensing using Stagewise Polytope Faces Pursuit

Mark D. Plumbley and Marco Bevilacqua

{mark.plumbley, marco.bevilacqua}@elec.qmul.ac.uk

Queen Mary University of London

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 1/17

Overview

Introduction Greedy and Stagewise algorithms Polytope Geometry of ℓ1 sparse recovery (Stepwise) Polytope Faces Pursuit algorithm Stagewise Polytope Faces Pursuit Experiments Conclusions

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 2/17

Introduction

We want to represent signal y = [y1, . . . , yd]T as a linear combination y = As with basis matrix A = [ai] ∈ Rd×n, coefficient vector s ∈ Rn. Sparsest (minimum ℓ0 solution)

min

s

s0

such that

As = y

But this is hard, so instead use . . . Basis Pursuit (minimum ℓ1 solution)

min

s

s1

such that

As = y

which can be solved using linear programming (LP) (Chen et al., 1998)

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 3/17

Greedy and Stagewise Algorithms

Greedy (stepwise) approximate algorithms exists which add

• ne basis vector at a time:

Matching Pursuits (MP) (Mallat and Zhang, 1993) Orthogonal Matching Pursuits (OMP) (Pati et al., 1993) Recently, Stagewise algorithms introduced, add several vectors at a time Stagewise OMP (StOMP) (Donoho et al, 2006) Stagewise Weak OMP (SWOMP) (Blumensath & Davies, 2008) This paper: Stagewise version of prevous Polytope Faces Pursuit algorithm (Plumbley, 2006)

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 4/17

Polytope Geometry of ℓ1 Soln

Using ˜

A = [A, −A], ˜ s ≥ 0 get standard dual LPs for ℓ1 soln: min

˜ s {1T˜

s | y = ˜ A˜ s, ˜ s ≥ 0} = max

c {yTc | ˜

ATc ≤ 1}

−1.5 −1 −0.5 0.5 1 1.5 −0.5 0.5 1 1.5 a1 a2 c1,y1 c2,y2 a1

+

a3,a3

+

y O c*

Optimum point c∗ within polar polytope P ∗ (shaded). Polytope: d-dimensional generalization of bounded polygon.

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 5/17

(Stepwise) Polytope Faces Pursuit

• 1. Start at zero.
• 2. Move in the direction of y until we ‘hit’ a face of the

constraining polytope

• 3. Add a constraint (basis vector) corresponding to face

just hit

• 4. Now move in direction of y projected onto new

constraint face

• 5. If movement would take us away from a face, remove

corresponding constraint

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 6/17

Polytope Faces Pursuit Algorithm

1: Input: y, ˜

A [˜ ai] = [A, −A]. Set kmax > 0, θmin ≥ 0

2: Initialize: k ← 0, Ik ← ∅, ˜

Ak ← ∅, ck ← 0, rk ← y

3: while k < kmax and maxi ˜

aT

i rk−1 > θmin do

4:

k ← k + 1

5:

{Find face} ik ← arg maxi/

∈Ik−1{ ˜ aT

i rk−1

1−˜ aT

i ck−1 | ˜

aT

i rk−1 > 0}

6:

˜ Ak ← [ ˜ Ak−1, ˜ aik], Ik ← Ik−1 ∪ {ik}, ˜ sk ← ( ˜ Ak)†y

7:

while ˜

sk 0 do {Release retarding constraints}

8:

Select some j ∈ Ik for which ˜

xk

j < 0;

˜ Ak ← ˜ Ak \ ˜ aj, Ik ← Ik \ {j}, ˜ sk ← ( ˜ Ak)†y

9:

end while

10:

ck ← ( ˜ Ak)†T1, ˆ yk ← ˜ Ak˜ sk, rk ← y − ˆ yk

11: end while 12: Output: ˜

s ← 0 + corresp elements of ˜ sk

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 7/17

Polytope Faces Pursuit vs OMP

PFP is very similar to Orthogonal Matching Pursuit (OMP), except Adjusted correlations with residual:

arg max

˜ ai

• ˜

aT

i rk−1

1 − ˜ aT

i ck−1

• vs

arg max

˜ ai {˜

aT

i rk−1}

Switch out of ‘retarding’ constraints, with ˜

sj < 0

Note: At first step k = 0 we start at c = 0, so:

˜ aT

i r/(1 − ˜

aT

i c) = ˜

aT

i r

i.e. first step of PFP chooses same basis as MP/OMP

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 8/17

Stagewise Algorithms

MP , OMP , PFP add one atom per step to the active set. Recently, exploration of ‘Stagewise’ sparse recovery algorithms, able to add several atoms per stage: Stagewise OMP (StOMP)

(Donoho, Tsaig, Driori & Starck, 2006)

Stagewise Weak OMP (SWOMP)

(Davies & Blumensath, 2008)

Stagewise version of PFP: At stage k select atoms with qk ≥ 1 adjusted correlations Polytope perspective: Move in direction of y until we ‘pierce’ qk hyperplanes

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 9/17

Basis selection strategies

Fixed q method: select a fixed number qk = q of basis vectors with largest adjusted correlations at each stage Weak stagewise selection method: select all basis vectors ai such that:

|θ(ak

i )| ≥ β max i

|θ(ak

i )|

where θ(ak

i ) is adjusted correlation for basis vector ai at

step k, 0 < β ≤ 1 is threshold control parameter.

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 10/17

Stagewise PFP Algorithm

1: Input: y, ˜

A [˜ ai] = [A, −A]. Set kmax > 0, q ≥ 1

2: Initialize: k ← 0, Ik ← ∅, ˜

Ak ← ∅, ck ← 0, rk ← y

3: while k < kmax and maxi ˜

aT

i rk−1 > θmin do

4:

k ← k + 1

5:

{Find faces} J k ← arg maxq

i/ ∈Ik−1 { ˜ aT

i rk−1

1−˜ aT

i ck−1 | ˜

aT

i rk−1 > 0}

6:

˜ Ak ← [ ˜ Ak−1, ˜ ai | i ∈ J k], Ik ← Ik−1 ∪J k, ˜ sk ← ( ˜ Ak)†y

7:

while ˜

sk 0 do {Release retarding constraints}

8:

Select some j ∈ Ik for which ˜

xk

j < 0;

˜ Ak ← ˜ Ak \ ˜ aj, Ik ← Ik \ {j}, ˜ sk ← ( ˜ Ak)†y

9:

end while

10:

ck ← ( ˜ Ak)†T1, ˆ yk ← ˜ Ak˜ sk, rk ← y − ˆ yk

11: end while 12: Output: ˜

s ← 0 + corresp elements of ˜ sk

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 11/17

Possible degenerate case

Basis vertex c no longer strictly guaranteed to stay inside the constrant surface.

−1 −0.5 0.5 1 −0.5 0.5 1 1.5 c1,y1 c2,y2 a1

+

a2

+

a3

+

y O c0 h0 h1 h2 c1

(But, experimental results so far suggest this is not a significant issue on problems we have tried.)

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 12/17

Experiments: Compressed Sensing

Sparco problem 5 (gcosspike) Cosine-like function with spikes, in union of DCT and Dirac bases, measured with Gaussian ensemble.

500 1000 1500 2000 2500 −30 −20 −10 10 20 30 40 50 60 70

Coefficients of the reconstructed signal

Coefficient 200 400 600 800 1000 1200 −8 −6 −4 −2 2 4 6

Original signal

Signal successfully recovered

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 13/17

Sparco Problem 5: Running Times

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 0.5 1 1.5 2

Max number of atoms per iteration Running time (sec) (a) Fixed−q selection

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 42 44 46 48 50

Max number of atoms per iteration SNR (dB)

0.1 0.16 0.22 0.28 0.34 0.4 0.46 0.52 0.58 0.64 0.7 0.76 0.82 0.88 0.94 1 0.5 1 1.5

beta Running time (sec) (b) Weak stagewise selection

0.1 0.16 0.22 0.28 0.34 0.4 0.46 0.52 0.58 0.64 0.7 0.76 0.82 0.88 0.94 1 20 30 40 50

beta SNR (dB)

Running time and Signal-to-noise ration (SNR) against basis selection strategy. Stagewise updates faster by factor of 3 SNR of recovered signal still good

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 14/17

Sparco Problems 6 and 11

Problem 6: piecewise cubic polynomial, wavelet basis, measured by Gaussian ensemble. Problem 11: spikes with gaussian amplitude, identity (Dirac) basis, measured by Gaussian ensemble.

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 5 10 15

Max number of atoms per iteration Running time (sec) (a) Sparco problem 6

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 15.5 16 16.5 17

Max number of atoms per iteration SNR (dB)

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 0.2 0.4 0.6

Max number of atoms per iteration Running time (sec) (b) Sparco problem 11

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 295 300 305 310

Max number of atoms per iteration SNR (dB)

Stagewise algorithm can be faster, but not always

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 15/17

Future Work

Many remaining issues to be investigated: Can we prove unit norm general position A never produces degenerate case? How to dynamically choose qk based on thresholding? (c.f Donoho et al, 2006) How best to ‘pull back’ to constraint polytope in degenerate case? Replace Cholesky-based pseudo-inverse with approximate conjugate gradient?

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 16/17

Conclusions

New stagewise greedy algorithm for sparse representations Method based on previous Polytope Faces Pursuit Adds several basis vectors per iteration Stagewise selection strategies: fixed-q and weak selection Applied to test problems in Sparco framework Can speed up by factor of 3, with similar recovery SNR Future speed up via conjugate gradient instead of Cholesky?

Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 17/17