Sparse Reconstruction for Compressed Sensing using Stagewise - PowerPoint PPT Presentation

Sparse Reconstruction for Compressed Sensing using Stagewise Polytope Faces Pursuit Mark D. Plumbley and Marco Bevilacqua { mark.plumbley, marco.bevilacqua } @elec.qmul.ac.uk Queen Mary University of London Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 1/17

Overview Introduction Greedy and Stagewise algorithms Polytope Geometry of ℓ 1 sparse recovery (Stepwise) Polytope Faces Pursuit algorithm Stagewise Polytope Faces Pursuit Experiments Conclusions Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 2/17

Introduction We want to represent signal y = [ y 1 , . . . , y d ] T as a linear combination y = As with basis matrix A = [ a i ] ∈ R d × n , coefficient vector s ∈ R n . Sparsest (minimum ℓ 0 solution) min � s � 0 such that As = y s But this is hard, so instead use . . . Basis Pursuit (minimum ℓ 1 solution) � s � 1 such that min As = y s which can be solved using linear programming (LP) (Chen et al., 1998) Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 3/17

Greedy and Stagewise Algorithms Greedy (stepwise) approximate algorithms exists which add one basis vector at a time: Matching Pursuits (MP) (Mallat and Zhang, 1993) Orthogonal Matching Pursuits (OMP) (Pati et al., 1993) Recently, Stagewise algorithms introduced, add several vectors at a time Stagewise OMP (StOMP) (Donoho et al, 2006) Stagewise Weak OMP (SWOMP) (Blumensath & Davies, 2008) This paper: Stagewise version of prevous Polytope Faces Pursuit algorithm (Plumbley, 2006) Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 4/17

Polytope Geometry of ℓ 1 Soln Using ˜ A = [ A , − A ] , ˜ s ≥ 0 get standard dual LPs for ℓ 1 soln: s { 1 T ˜ s | y = ˜ c { y T c | ˜ A T c ≤ 1 } min A ˜ s , ˜ s ≥ 0 } = max ˜ a 1 1.5 y a 2 1 c 2 ,y 2 c* + a 3 ,a 3 0.5 + a 1 0 O −0.5 −1.5 −1 −0.5 0 0.5 1 1.5 c 1 ,y 1 Optimum point c ∗ within polar polytope P ∗ (shaded). Polytope : d -dimensional generalization of bounded polygon. Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 5/17

(Stepwise) Polytope Faces Pursuit 1. Start at zero. 2. Move in the direction of y until we ‘hit’ a face of the constraining polytope 3. Add a constraint (basis vector) corresponding to face just hit 4. Now move in direction of y projected onto new constraint face 5. If movement would take us away from a face, remove corresponding constraint Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 6/17

Polytope Faces Pursuit Algorithm 1: Input: y , ˜ A � [˜ a i ] = [ A , − A ] . Set k max > 0 , θ min ≥ 0 2: Initialize: k ← 0 , I k ← ∅ , ˜ A k ← ∅ , c k ← 0 , r k ← y i r k − 1 > θ min do a T 3: while k < k max and max i ˜ k ← k + 1 4: {Find face} i k ← arg max i/ a T i r k − 1 i r k − 1 > 0 } ˜ a T ∈I k − 1 { i c k − 1 | ˜ 5: a T 1 − ˜ A k ← [ ˜ a i k ] , I k ← I k − 1 ∪ { i k } , ˜ s k ← ( ˜ ˜ A k − 1 , ˜ A k ) † y 6: s k � 0 do {Release retarding constraints} while ˜ 7: Select some j ∈ I k for which ˜ x k j < 0 ; 8: A k ← ˜ A k \ ˜ a j , I k ← I k \ { j } , ˜ s k ← ( ˜ ˜ A k ) † y end while 9: c k ← ( ˜ y k ← ˜ s k , r k ← y − ˆ A k ) † T 1 , ˆ A k ˜ y k 10: 11: end while s k s ← 0 + corresp elements of ˜ 12: Output: ˜ Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 7/17

Polytope Faces Pursuit vs OMP PFP is very similar to Orthogonal Matching Pursuit (OMP), except Adjusted correlations with residual: a T i r k − 1 � ˜ � a T i r k − 1 } arg max vs arg max a i { ˜ a T i c k − 1 1 − ˜ ˜ ˜ a i Switch out of ‘retarding’ constraints, with ˜ s j < 0 Note: At first step k = 0 we start at c = 0 , so: a T a T a T i r / (1 − ˜ ˜ i c ) = ˜ i r i.e. first step of PFP chooses same basis as MP/OMP Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 8/17

Stagewise Algorithms MP , OMP , PFP add one atom per step to the active set. Recently, exploration of ‘Stagewise’ sparse recovery algorithms, able to add several atoms per stage: Stagewise OMP (StOMP) (Donoho, Tsaig, Driori & Starck, 2006) Stagewise Weak OMP (SWOMP) (Davies & Blumensath, 2008) Stagewise version of PFP: At stage k select atoms with q k ≥ 1 adjusted correlations Polytope perspective: Move in direction of y until we ‘pierce’ q k hyperplanes Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 9/17

Basis selection strategies Fixed q method: select a fixed number q k = q of basis vectors with largest adjusted correlations at each stage Weak stagewise selection method: select all basis vectors a i such that: | θ ( a k | θ ( a k i ) | ≥ β max i ) | i where θ ( a k i ) is adjusted correlation for basis vector a i at step k , 0 < β ≤ 1 is threshold control parameter. Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 10/17

Stagewise PFP Algorithm 1: Input: y , ˜ A � [˜ a i ] = [ A , − A ] . Set k max > 0 , q ≥ 1 2: Initialize: k ← 0 , I k ← ∅ , ˜ A k ← ∅ , c k ← 0 , r k ← y i r k − 1 > θ min do a T 3: while k < k max and max i ˜ k ← k + 1 4: {Find faces} J k ← arg max q a T i r k − 1 i r k − 1 > 0 } ˜ a T ∈I k − 1 { i c k − 1 | ˜ 5: i/ a T 1 − ˜ A k ← [ ˜ a i | i ∈ J k ] , I k ← I k − 1 ∪J k , ˜ s k ← ( ˜ ˜ A k − 1 , ˜ A k ) † y 6: s k � 0 do {Release retarding constraints} while ˜ 7: Select some j ∈ I k for which ˜ x k j < 0 ; 8: A k ← ˜ A k \ ˜ a j , I k ← I k \ { j } , ˜ s k ← ( ˜ ˜ A k ) † y end while 9: c k ← ( ˜ y k ← ˜ s k , r k ← y − ˆ A k ) † T 1 , ˆ A k ˜ y k 10: 11: end while s k s ← 0 + corresp elements of ˜ 12: Output: ˜ Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 11/17

Possible degenerate case Basis vertex c no longer strictly guaranteed to stay inside the constrant surface. 1.5 y 1 c 1 h 2 + a 2 h 1 c 2 ,y 2 0.5 + a 3 + a 1 0 O c 0 h 0 −0.5 −1 −0.5 0 0.5 1 c 1 ,y 1 (But, experimental results so far suggest this is not a significant issue on problems we have tried.) Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 12/17

Experiments: Compressed Sensing Sparco problem 5 (gcosspike) Cosine-like function with spikes, in union of DCT and Dirac bases, measured with Gaussian ensemble. Coefficients of the reconstructed signal Original signal 70 6 60 4 50 2 40 30 0 20 −2 10 0 −4 −10 −6 −20 −30 −8 0 500 1000 1500 2000 2500 0 200 400 600 800 1000 1200 Coefficient Signal successfully recovered Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 13/17

Sparco Problem 5: Running Times (a) Fixed−q selection (b) Weak stagewise selection 2 Running time (sec) Running time (sec) 1.5 1.5 1 1 0.5 0.5 0 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 0.1 0.16 0.22 0.28 0.34 0.4 0.46 0.52 0.58 0.64 0.7 0.76 0.82 0.88 0.94 1 Max number of atoms per iteration beta 50 50 48 SNR (dB) SNR (dB) 40 46 30 44 42 20 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 0.1 0.16 0.22 0.28 0.34 0.4 0.46 0.52 0.58 0.64 0.7 0.76 0.82 0.88 0.94 1 Max number of atoms per iteration beta Running time and Signal-to-noise ration (SNR) against basis selection strategy. Stagewise updates faster by factor of 3 SNR of recovered signal still good Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 14/17

Sparco Problems 6 and 11 Problem 6: piecewise cubic polynomial, wavelet basis, measured by Gaussian ensemble. Problem 11: spikes with gaussian amplitude, identity (Dirac) basis, measured by Gaussian ensemble. (a) Sparco problem 6 (b) Sparco problem 11 Running time (sec) Running time (sec) 0.6 15 0.4 10 0.2 5 0 0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 Max number of atoms per iteration Max number of atoms per iteration 17 310 SNR (dB) SNR (dB) 16.5 305 16 300 15.5 295 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 Max number of atoms per iteration Max number of atoms per iteration Stagewise algorithm can be faster, but not always Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 15/17

Future Work Many remaining issues to be investigated: Can we prove unit norm general position A never produces degenerate case? How to dynamically choose q k based on thresholding? (c.f Donoho et al, 2006) How best to ‘pull back’ to constraint polytope in degenerate case? Replace Cholesky-based pseudo-inverse with approximate conjugate gradient? Sparse Reconstruction for Compressed Sensing usingStagewise Polytope Faces Pursuit – p. 16/17