Subsampling at Information Theoretically Optimal Rates Adel - - PowerPoint PPT Presentation

▶

Feb 06, 2024 179 likes •674 views

Subsampling at Information Theoretically Optimal Rates Adel Javanmard, Andrea Montanari Stanford University July 5, 2012 Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 1 / 26 A classical compressive sensing application

SLIDE 1

Subsampling at Information Theoretically Optimal Rates

Adel Javanmard, Andrea Montanari

Stanford University

July 5, 2012

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 1 / 26

SLIDE 2

A classical compressive sensing application

Sampling a random signal with sparse support in frequency domain.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 2 / 26

SLIDE 3

Notation

◮ Time domain:

x ❂ ✭x✭1✮❀ x✭2✮❀ ✁ ✁ ✁ ❀ x✭t✮❀ ✁ ✁ ✁ ❀ x✭n✮✮ ✷ Cn✿

◮ Fourier domain:

❜

x ❂ Fx❀ F: Fourier matrix

❜

x✭✦✮ ❂

n

❳

t❂1

1 ♣n ei✦tx✭t✮❀ ✦ ✷ ❢2✙k❂n❣n1

k❂0✿

Sparse structure: ❫ x has k nonzero entries (k ✜ n).

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 3 / 26

SLIDE 4

Notation

◮ Time domain:

x ❂ ✭x✭1✮❀ x✭2✮❀ ✁ ✁ ✁ ❀ x✭t✮❀ ✁ ✁ ✁ ❀ x✭n✮✮ ✷ Cn✿

◮ Fourier domain:

❜

x ❂ Fx❀ F: Fourier matrix

❜

x✭✦✮ ❂

n

❳

t❂1

1 ♣n ei✦tx✭t✮❀ ✦ ✷ ❢2✙k❂n❣n1

k❂0✿

Sparse structure: ❫ x has k nonzero entries (k ✜ n).

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 3 / 26

SLIDE 5

Sampling mechanism

yi ❂ ❤ai❀ x✐❀ i ❂ 1❀ ✁ ✁ ✁ ❀ m✿ We refer to m❂n as the sampling rate. (In time domain) y ❂ Ax. (In frequency domain) y ❂ AF✄❜ x ❂ AF❜ x.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 4 / 26

SLIDE 6

Sampling mechanism

yi ❂ ❤ai❀ x✐❀ i ❂ 1❀ ✁ ✁ ✁ ❀ m✿ We refer to m❂n as the sampling rate. (In time domain) y ❂ Ax. (In frequency domain) y ❂ AF✄❜ x ❂ AF❜ x.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 4 / 26

SLIDE 7

Normalization

✦ m❀ n ✦ ✶, m❂n ❂ ✍ ✦ A ❂

✷ ✻ ✹

a1 . . . am

✸ ✼ ✺

❦ai❦2 ❂ 1

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 5 / 26

SLIDE 8

Sampling schemes:

◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate

[Shannon 1948]

◮ Instantaneous sampling at random times ✦ m ❂ Ck log n

[Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme:

◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o✭n✮

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

SLIDE 9

Sampling schemes:

◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate

[Shannon 1948]

◮ Instantaneous sampling at random times ✦ m ❂ Ck log n

[Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme:

◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o✭n✮

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

SLIDE 10

Sampling schemes:

◮ Instantaneous sampling at equispaced times ✦ rate = Nyquist rate

[Shannon 1948]

◮ Instantaneous sampling at random times ✦ m ❂ Ck log n

[Candés, Romberg, Tao 2006, Candés, Plan 2011] Our scheme:

◮ Non-instantaneous sampling at random times ✦ m ❂ k ✰ o✭n✮

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 6 / 26

SLIDE 11

Classical compressive sensing scheme

✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization (❵1 minimization)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

SLIDE 12

Classical compressive sensing scheme

✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization (❵1 minimization)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

SLIDE 13

Classical compressive sensing scheme

✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization (❵1 minimization)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

SLIDE 14

Classical compressive sensing scheme

✎ Measurements: sample pointwise at random times Fourier domain: random rows of DFT matrix. ✎ Probes all freq. with the same weight. (Delocalized measurements) ✳ Reconstruction: Convex minimization (❵1 minimization)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 7 / 26

SLIDE 15

Our scheme

A different solution! We ‘smear out’ the samples in the time domain ❢t1❀ ✁ ✁ ✁ ❀ tm❣❀ ❢✦1❀ ✁ ✁ ✁ ❀ ✦m❣❀ ✦i ❂ 2✙i❂m✿

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

SLIDE 16

Our scheme

A different solution! We ‘smear out’ the samples in the time domain ❢t1❀ ✁ ✁ ✁ ❀ tm❣❀ ❢✦1❀ ✁ ✁ ✁ ❀ ✦m❣❀ ✦i ❂ 2✙i❂m✿

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

SLIDE 17

Our scheme

A different solution! We ‘smear out’ the samples in the time domain ❢t1❀ ✁ ✁ ✁ ❀ tm❣❀ ❢✦1❀ ✁ ✁ ✁ ❀ ✦m❣❀ ✦i ❂ 2✙i❂m✿ modulate with ✦i

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

SLIDE 18

Our scheme

A different solution! We ‘smear out’ the samples in the time domain ❢t1❀ ✁ ✁ ✁ ❀ tm❣❀ ❢✦1❀ ✁ ✁ ✁ ❀ ✦m❣❀ ✦i ❂ 2✙i❂m✿

integrate over a window (of size ❵) around ti

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

SLIDE 19

Our scheme

A different solution! We ‘smear out’ the samples in the time domain ❢t1❀ ✁ ✁ ✁ ❀ tm❣❀ ❢✦1❀ ✁ ✁ ✁ ❀ ✦m❣❀ ✦i ❂ 2✙i❂m✿

yi ❂ ❤b✦i❀ti❀ x✐❀ i ❂ 1❀ ✁ ✁ ✁ ❀ m✿ b✦✄❀t✄✭t✮ ✑ exp

✚

i✦✄t ✭tt✄✮2

2❵2

✛

✿

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 8 / 26

SLIDE 20

Our scheme (Cont’d)

Fourier domain: ... integrating over freq. within a window of size ❵1 around ✦✄. ❂ ✮ AF is roughly band-diagonal !

◮ Reconstruction: Bayesian AMP

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

SLIDE 21

Our scheme (Cont’d)

Fourier domain: ... integrating over freq. within a window of size ❵1 around ✦✄. ❂ ✮ AF is roughly band-diagonal !

◮ Reconstruction: Bayesian AMP

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

SLIDE 22

Our scheme (Cont’d)

Fourier domain: ... integrating over freq. within a window of size ❵1 around ✦✄. ❂ ✮ AF is roughly band-diagonal !

◮ Reconstruction: Bayesian AMP

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 9 / 26

SLIDE 23

Why should it work?

Classical scheme Our scheme Fourier coefficients Gabor coefficients (Delocalized measurements) ( Band-diagonal sensing matrix) This is an implementation of a broader idea ✦ Spatial Coupling!

[Kudekar, Pfister, 2010] [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011]

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 10 / 26

SLIDE 24

Why should it work?

Classical scheme Our scheme Fourier coefficients Gabor coefficients (Delocalized measurements) ( Band-diagonal sensing matrix) This is an implementation of a broader idea ✦ Spatial Coupling!

[Kudekar, Pfister, 2010] [Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011]

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 10 / 26

SLIDE 25

An overview on spatial coupling

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 11 / 26

SLIDE 26

Spatially coupled sensing matrix

A ❂ ✷ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✹ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ a1 a2 ✄ ✄ a❵ b1 b2 ✄ ✄ b❵ c1 c2 ✄ ✄ c❵ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✸ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✺

◮ ✘ independent entries ◮ ✘ band diagonal ◮ m❀ n❀ ❵ ✦ ✶, with m❂n ✦ ✍ ✷ ✭0❀ 1✮, ❵❂n ✦ 0

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 12 / 26

SLIDE 27

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Coordinates ¡of ¡x ¡ ¡ Coordinates ¡of ¡y ¡

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 13 / 26

SLIDE 28

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

Addi%onal ¡measurements ¡associated ¡ to ¡the ¡first ¡few ¡coordinates ¡ First ¡few ¡coordinates ¡are ¡oversampled! ¡

.....

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 14 / 26

SLIDE 29

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 15 / 26

SLIDE 30

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 16 / 26

SLIDE 31

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 17 / 26

SLIDE 32

Bayes-optimal AMP

[Donoho, Maleki, Montanari 2009] [Donoho, Javanmard, Montanari 2011] x t✰1 ❂ ✑t✭x t ✰ ✭Qt ☞ AF✮✄r t✮ ❀ r t ❂ y AFx t ✰ bt ☞ r t1 ✰ dt ☞ ✖ r t1 ✿ Qt❀ bt❀ dt explicitly given normalizations ✑t✭y✮ ✑ E❢X ❥X ✰ ✜tZ ❂ y❣ (reduces to simple expression in most cases)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 18 / 26

SLIDE 33

Bayes-optimal AMP

[Donoho, Maleki, Montanari 2009] [Donoho, Javanmard, Montanari 2011] x t✰1 ❂ ✑t✭x t ✰ ✭Qt ☞ AF✮✄r t✮ ❀ r t ❂ y AFx t ✰ bt ☞ r t1 ✰ dt ☞ ✖ r t1 ✿ Qt❀ bt❀ dt explicitly given normalizations ✑t✭y✮ ✑ E❢X ❥X ✰ ✜tZ ❂ y❣ (reduces to simple expression in most cases)

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 18 / 26

SLIDE 34

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 19 / 26

SLIDE 35

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 19 / 26

SLIDE 36

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 19 / 26

SLIDE 37

Rényi information dimension

Characterization of d✭X ✮ (Rényi)

Let pX be a probability measure over R, and X ✘ pX . Let pX ❂ ✭1 ✧✮✗d ✰ ✧⑦ ✗ with ✗d: a discrete distribution (i.e. with countable support) ⑦ ✗d: an absolutely continuous then d✭X ✮ ❂ ✧. In particular, if P❢X ✻❂ 0❣ ✔ ✧ then d✭X ✮ ✔ ✧. [cf. Wu, Verdú]

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 20 / 26

SLIDE 38

Rényi information dimension

Characterization of d✭X ✮ (Rényi)

Let pX be a probability measure over R, and X ✘ pX . Let pX ❂ ✭1 ✧✮✗d ✰ ✧⑦ ✗ with ✗d: a discrete distribution (i.e. with countable support) ⑦ ✗d: an absolutely continuous then d✭X ✮ ❂ ✧. In particular, if P❢X ✻❂ 0❣ ✔ ✧ then d✭X ✮ ✔ ✧. [cf. Wu, Verdú]

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 20 / 26

SLIDE 39

Question

Does the spatial coupling phenomenon survive for physically constrained sensing matrices?

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 21 / 26

SLIDE 40

Experiments

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 22 / 26

SLIDE 41

Experiment

◮ x✭1✮❀ ✿ ✿ ✿ ❀ x✭n✮ ✘i✿i✿d✿ ✭1 ✧✮✍0 ✰ ✧ Normal✭0❀ 1✮ ◮ Will it work for m ✕ n✧ ✰ o✭n✮?

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 23 / 26

SLIDE 42

✧ ❂ 0✿1, m ❂ 0✿15 n

200 400 600 800 0.01 0.02 0.03 0.04 0.05 0.06 i t=5 t=50 t=100 t=150 t=200 t=250 t=300 t=350 t=400

MSE✭t✮ MSE✭t✮ ✷ Rn❀ MSE✭t✮✭i✮ ❂ E❢❥❜ x t

i ❜

xi❥2❣✿

◮ ❵1 minimization requires m 0✿33 n!

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 24 / 26

SLIDE 43

✧ ❂ 0✿1, m ❂ 0✿15 n

200 400 600 800 0.01 0.02 0.03 0.04 0.05 0.06 i t=5 t=50 t=100 t=150 t=200 t=250 t=300 t=350 t=400

MSE✭t✮ MSE✭t✮ ✷ Rn❀ MSE✭t✮✭i✮ ❂ E❢❥❜ x t

i ❜

xi❥2❣✿

◮ ❵1 minimization requires m 0✿33 n!

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 24 / 26

SLIDE 44

Phase transition

0.2 0.4 0.6 0.8 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 ε δ

Scheme III (empirical) δℓ1() Scheme II (empirical) ˜ δ() Scheme I (empirical) δ =

◮ Scheme I : Bayesian AMP, Random Gabor. ◮ Scheme II: Bayesian AMP, Random Fourier. ◮ Scheme III: ❵1, Random Gabor.

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 25 / 26

SLIDE 45

Conclusion

◮ “Spatially-couplled measurements + Bayesian AMP” achieves the

information theoretically optimal rate.

◮ The power of this scheme also applies to the physically

constrained sensing matrices. Thanks!

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 26 / 26

SLIDE 46

Conclusion

◮ “Spatially-couplled measurements + Bayesian AMP” achieves the

information theoretically optimal rate.

◮ The power of this scheme also applies to the physically

constrained sensing matrices. Thanks!

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 26 / 26

SLIDE 47

Conclusion

◮ “Spatially-couplled measurements + Bayesian AMP” achieves the

information theoretically optimal rate.

◮ The power of this scheme also applies to the physically

constrained sensing matrices. Thanks!

Javanmard, Montanari (Stanford) Optimal subsampling July 5, 2012 26 / 26