Subsampling at Information Theoretically Optimal Rates Adel - PowerPoint PPT Presentation

Subsampling at Information Theoretically Optimal Rates Adel Javanmard, Andrea Montanari Stanford University July 5, 2012 Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 1 / 26

A classical compressive sensing application Sampling a random signal with sparse support in frequency domain. Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 2 / 26

Notation ◮ Time domain: x ❂ ✭ x ✭ 1 ✮ ❀ x ✭ 2 ✮ ❀ ✁ ✁ ✁ ❀ x ✭ t ✮ ❀ ✁ ✁ ✁ ❀ x ✭ n ✮✮ ✷ C n ✿ ◮ Fourier domain: x ❂ F x ❀ F : Fourier matrix ❜ n ❳ 1 ♣ n e � i ✦ t x ✭ t ✮ ❀ ✦ ✷ ❢ 2 ✙ k ❂ n ❣ n � 1 x ✭ ✦ ✮ ❂ ❜ k ❂ 0 ✿ t ❂ 1 Sparse structure: ❫ x has k nonzero entries ( k ✜ n ). Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 3 / 26

Notation ◮ Time domain: x ❂ ✭ x ✭ 1 ✮ ❀ x ✭ 2 ✮ ❀ ✁ ✁ ✁ ❀ x ✭ t ✮ ❀ ✁ ✁ ✁ ❀ x ✭ n ✮✮ ✷ C n ✿ ◮ Fourier domain: x ❂ F x ❀ F : Fourier matrix ❜ n ❳ 1 ♣ n e � i ✦ t x ✭ t ✮ ❀ ✦ ✷ ❢ 2 ✙ k ❂ n ❣ n � 1 x ✭ ✦ ✮ ❂ ❜ k ❂ 0 ✿ t ❂ 1 Sparse structure: ❫ x has k nonzero entries ( k ✜ n ). Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 3 / 26

Sampling mechanism y i ❂ ❤ a i ❀ x ✐ ❀ i ❂ 1 ❀ ✁ ✁ ✁ ❀ m ✿ We refer to m ❂ n as the sampling rate. (In time domain) y ❂ Ax . y ❂ A F ✄ ❜ (In frequency domain) x ❂ A F ❜ x . Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 4 / 26

Sampling mechanism y i ❂ ❤ a i ❀ x ✐ ❀ i ❂ 1 ❀ ✁ ✁ ✁ ❀ m ✿ We refer to m ❂ n as the sampling rate. (In time domain) y ❂ Ax . y ❂ A F ✄ ❜ (In frequency domain) x ❂ A F ❜ x . Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 4 / 26

Normalization ✦ m ❀ n ✦ ✶ , m ❂ n ❂ ✍ ✷ ✸ a 1 ✻ . ✼ . ✦ A ❂ ❦ a i ❦ 2 ❂ 1 ✹ ✺ . a m Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 5 / 26

Sampling schemes: ◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate [Shannon 1948] ◮ Instantaneous sampling at random times ✦ m ❂ Ck log n [Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme: ◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o ✭ n ✮ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

Sampling schemes: ◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate [Shannon 1948] ◮ Instantaneous sampling at random times ✦ m ❂ Ck log n [Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme: ◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o ✭ n ✮ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

Sampling schemes: ◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate [Shannon 1948] ◮ Instantaneous sampling at random times ✦ m ❂ Ck log n [Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme: ◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o ✭ n ✮ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

Classical compressive sensing scheme ✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization ( ❵ 1 minimization) Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

Classical compressive sensing scheme ✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization ( ❵ 1 minimization) Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

Classical compressive sensing scheme ✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization ( ❵ 1 minimization) Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

Classical compressive sensing scheme ✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization ( ❵ 1 minimization) Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

Our scheme A different solution! We ‘smear out’ the samples in the time domain ❢ t 1 ❀ ✁ ✁ ✁ ❀ t m ❣ ❀ ❢ ✦ 1 ❀ ✁ ✁ ✁ ❀ ✦ m ❣ ❀ ✦ i ❂ 2 ✙ i ❂ m ✿ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

Our scheme A different solution! We ‘smear out’ the samples in the time domain ❢ t 1 ❀ ✁ ✁ ✁ ❀ t m ❣ ❀ ❢ ✦ 1 ❀ ✁ ✁ ✁ ❀ ✦ m ❣ ❀ ✦ i ❂ 2 ✙ i ❂ m ✿ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

Our scheme A different solution! We ‘smear out’ the samples in the time domain ❢ t 1 ❀ ✁ ✁ ✁ ❀ t m ❣ ❀ ❢ ✦ 1 ❀ ✁ ✁ ✁ ❀ ✦ m ❣ ❀ ✦ i ❂ 2 ✙ i ❂ m ✿ modulate with ✦ i Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

Our scheme A different solution! We ‘smear out’ the samples in the time domain ❢ t 1 ❀ ✁ ✁ ✁ ❀ t m ❣ ❀ ❢ ✦ 1 ❀ ✁ ✁ ✁ ❀ ✦ m ❣ ❀ ✦ i ❂ 2 ✙ i ❂ m ✿ t i integrate over a window (of size ❵ ) around t i Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

Our scheme A different solution! We ‘smear out’ the samples in the time domain ❢ t 1 ❀ ✁ ✁ ✁ ❀ t m ❣ ❀ ❢ ✦ 1 ❀ ✁ ✁ ✁ ❀ ✦ m ❣ ❀ ✦ i ❂ 2 ✙ i ❂ m ✿ t i ✚ ✛ i ✦ ✄ t � ✭ t � t ✄ ✮ 2 i ❂ 1 ❀ ✁ ✁ ✁ ❀ m ✿ b ✦ ✄ ❀ t ✄ ✭ t ✮ ✑ exp y i ❂ ❤ b ✦ i ❀ t i ❀ x ✐ ❀ ✿ 2 ❵ 2 Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

Our scheme (Cont’d) Fourier domain: ... integrating over freq. within a window of size ❵ � 1 around ✦ ✄ . ❂ ✮ A F is roughly band-diagonal ! ◮ Reconstruction: Bayesian AMP Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

Our scheme (Cont’d) Fourier domain: ... integrating over freq. within a window of size ❵ � 1 around ✦ ✄ . ❂ ✮ A F is roughly band-diagonal ! ◮ Reconstruction: Bayesian AMP Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

Our scheme (Cont’d) Fourier domain: ... integrating over freq. within a window of size ❵ � 1 around ✦ ✄ . ❂ ✮ A F is roughly band-diagonal ! ◮ Reconstruction: Bayesian AMP Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

Why should it work? Classical scheme Our scheme Fourier coefficients Gabor coefficients (Delocalized measurements) ( Band-diagonal sensing matrix) This is an implementation of a broader idea ✦ Spatial Coupling! [Kudekar, Pfister, 2010] [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011] Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 10 / 26

Why should it work? Classical scheme Our scheme Fourier coefficients Gabor coefficients (Delocalized measurements) ( Band-diagonal sensing matrix) This is an implementation of a broader idea ✦ Spatial Coupling! [Kudekar, Pfister, 2010] [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011] Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 10 / 26

An overview on spatial coupling Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 11 / 26

Spatially coupled sensing matrix ✷ ✸ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✻ ✼ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✻ ✼ ✻ ✼ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✻ ✼ ✻ ✼ 0 0 0 0 0 0 0 0 0 0 0 0 a 1 a 2 a ❵ ✄ ✄ ✻ ✼ ✻ ✼ 0 0 0 0 0 0 0 0 0 0 0 0 A ❂ b 1 b 2 b ❵ ✄ ✄ ✻ ✼ ✻ ✼ 0 0 0 0 0 0 0 0 0 0 0 0 c 1 c 2 c ❵ ✄ ✄ ✻ ✼ ✻ ✼ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✻ ✼ ✹ ✺ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ◮ ✘ independent entries ◮ ✘ band diagonal ◮ m ❀ n ❀ ❵ ✦ ✶ , with m ❂ n ✦ ✍ ✷ ✭ 0 ❀ 1 ✮ , ❵❂ n ✦ 0 Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 12 / 26

How does spatial coupling work? ¡ ¡ ..... ¡ ¡ ¡ Coordinates ¡of ¡x ¡ ¡ Coordinates ¡of ¡y ¡ Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 13 / 26