Information-Theoretically Optimal Compressed Sensing via Spatial - - PowerPoint PPT Presentation

▶

Jan 28, 2023 10 likes •472 views

Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing Adel Javanmard with David Donoho and Andrea Montanari Stanford University November 10, 2012 Donoho, Javanmard, Montanari Compressed

SLIDE 1

Information-Theoretically Optimal Compressed Sensing via Spatial Coupling and Approximate Message Passing

Adel Javanmard with David Donoho and Andrea Montanari

Stanford University

November 10, 2012

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 1 / 28

SLIDE 2

General problem

y ❂ Ax ✰ noise ❀ n n m y ❂ A x ✰ w

◮ x high-dimensional but highly structured ◮ How many linear measurements are needed?

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 2 / 28

SLIDE 3

Normalization

✦ w ✘ N✭0❀ ✛2 Im✂m✮ ✦ m❀ n ✦ ✶, m❂n ❂ ✍ ✦ A ❂ ❬A1❥ ✁ ✁ ✁ ❥An❪ ❦Ai❦2 ❂ ✂✭1✮

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 3 / 28

SLIDE 4

Compressed sensing: Basic insights

Donoho, Candés, Romberg, Tao, Indyk, Gilbert, . . . [2005-. . . ]

Structure ✦ ❦x❦0 ✔ k adversarial Rate ✦ m ❂ C k log✭n❂k✮ Reconstruction ✦ Convex optimization Measurements ✦ Random isotropic vectors Robustness ✦ MSE ✔ C✛2 Is this the optimal compression rate?

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 4 / 28

SLIDE 5

Compressed sensing: Basic insights

Donoho, Candés, Romberg, Tao, Indyk, Gilbert, . . . [2005-. . . ]

Structure ✦ ❦x❦0 ✔ k adversarial Rate ✦ m ❂ C k log✭n❂k✮ Reconstruction ✦ Convex optimization Measurements ✦ Random isotropic vectors Robustness ✦ MSE ✔ C✛2 Is this the optimal compression rate?

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 4 / 28

SLIDE 6

This paper

Structure ✦ x ❂ xdiscr ✰ xother; ❦xother❦0 ✔ k oblivious Rate ✦ m ❂ k ✰ o✭n✮ Reconstruction ✦ Bayesian AMP Measurements ✦ Spatially coupled matrices Robustness ✦ MSE ✔ C✭x✮✛2

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 5 / 28

SLIDE 7

This paper

Structure ✦ x ❂ xdiscr ✰ xother; ❦xother❦0 ✔ k oblivious Rate ✦ m ❂ k ✰ o✭n✮ Reconstruction ✦ Bayesian AMP Measurements ✦ Spatially coupled matrices Robustness ✦ MSE ✔ C✭x✮✛2

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 5 / 28

SLIDE 8

Outline

◮ A toy example (random signal). ◮ Results.

◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP

◮ Proof technique.

◮ State evolution

◮ Supercooling.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

SLIDE 9

Outline

◮ A toy example (random signal). ◮ Results.

◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP

◮ Proof technique.

◮ State evolution

◮ Supercooling.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

SLIDE 10

Outline

◮ A toy example (random signal). ◮ Results.

◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP

◮ Proof technique.

◮ State evolution

◮ Supercooling.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

SLIDE 11

Outline

◮ A toy example (random signal). ◮ Results.

◮ ‘Spatially coupled’ sensing matrices ◮ How does spatial coupling work? ◮ Bayes-optimal AMP

◮ Proof technique.

◮ State evolution

◮ Supercooling.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 6 / 28

SLIDE 12

A toy example (random signal)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 7 / 28

SLIDE 13

A toy example (random signal)

x ❂ ✭x1❀ ✿ ✿ ✿ ❀ xn✮❀ xi ✘i✿i✿d✿ pX ❀ y ❂ Ax ❀ y ✷ Rm ❀ pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿ pX is known! Non-universal!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

SLIDE 14

A toy example (random signal)

x ❂ ✭x1❀ ✿ ✿ ✿ ❀ xn✮❀ xi ✘i✿i✿d✿ pX ❀ y ❂ Ax ❀ y ✷ Rm ❀ pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿ pX is known! Non-universal!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

SLIDE 15

A toy example (random signal)

x ❂ ✭x1❀ ✿ ✿ ✿ ❀ xn✮❀ xi ✘i✿i✿d✿ pX ❀ y ❂ Ax ❀ y ✷ Rm ❀ pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿ pX is known! Non-universal!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 8 / 28

SLIDE 16

How many measurements are needed?

pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿

◮ Classical compressed sensing: m ❂ 0✿97 n ✰ o✭n✮

(Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust)

◮ This talk: m ❂ 0✿1 n ✰ o✭n✮

(non-universal, robust)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

SLIDE 17

How many measurements are needed?

pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿

◮ Classical compressed sensing: m ❂ 0✿97 n ✰ o✭n✮

(Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust)

◮ This talk: m ❂ 0✿1 n ✰ o✭n✮

(non-universal, robust)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

SLIDE 18

How many measurements are needed?

pX ❂ 0✿2 ✍0 ✰ 0✿3 ✍1 ✰ 0✿2 ✍1 ✰ 0✿2 ✍3 ✰ 0✿1 Uniform✭2❀ 2✮✿

◮ Classical compressed sensing: m ❂ 0✿97 n ✰ o✭n✮

(Donoho 2006, universal, Donoho-Maleki-M. 2011 uniformly robust)

◮ This talk: m ❂ 0✿1 n ✰ o✭n✮

(non-universal, robust)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 9 / 28

SLIDE 19

What is 0✿1 here?

Definition (Renyi’s Information Dimension)

For X ✘ pX , let ❤X ✐m ❂ ❜2mX ❝❂2m be an m-digits rounding of X d✭X ✮ ✑ lim sup

m✦✶

H✭❤X ✐m✮ m ✿ Alternative characterization: ✎ If pX ❂ ✭1 ✧✮ ✁ discrete ✰ ✧ ✁ abs. continuous, then d✭X ✮ ❂ ✧.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 10 / 28

SLIDE 20

What is 0✿1 here?

Definition (Renyi’s Information Dimension)

For X ✘ pX , let ❤X ✐m ❂ ❜2mX ❝❂2m be an m-digits rounding of X d✭X ✮ ✑ lim sup

m✦✶

H✭❤X ✐m✮ m ✿ Alternative characterization: ✎ If pX ❂ ✭1 ✧✮ ✁ discrete ✰ ✧ ✁ abs. continuous, then d✭X ✮ ❂ ✧.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 10 / 28

SLIDE 21

Why is this important?

Theorem (Verdú, Wu, 2010)

Under mild regularity hypotheses, non-adaptive compressed sensing is possible if and only if m ❃ d✭X ✮ n ✰ o✭n✮ ✿ (equivalently, ✍ ❃ d✭X ✮ ✰ o✭1✮). Shannon-theoretic argument. Exhaustive-search reconstruction :-(

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 11 / 28

SLIDE 22

Results

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 12 / 28

SLIDE 23

Two tricks

◮ ‘Spatially coupled’ sensing matrix.

[Kudekar, Pfister, 2010] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011]

◮ AMP reconstruction, Posterior-expectation denoiser

[Donoho, Maleki, Montanari 2009]

◮ Spatial coupling + MP reconstruction

[Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [no proof :-(]

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 13 / 28

SLIDE 24

Two tricks

◮ ‘Spatially coupled’ sensing matrix.

[Kudekar, Pfister, 2010] [cf. also Felstrom, Zigangirov, 1999; Kudekar, Richardson, Urbanke 2009-2011]

◮ AMP reconstruction, Posterior-expectation denoiser

[Donoho, Maleki, Montanari 2009]

◮ Spatial coupling + MP reconstruction

[Krzakala, Mézard, Sausset, Sun, Zdeborova, 2011] [no proof :-(]

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 13 / 28

SLIDE 25

Our contributions

◮ Construction ◮ A rigorous proof ◮ Beyond random signals ◮ Robustness

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 14 / 28

SLIDE 26

Spatially coupled sensing matrix

A ❂ ✷ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✻ ✹ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ a1 a2 ✄ ✄ a❵ b1 b2 ✄ ✄ b❵ c1 c2 ✄ ✄ c❵ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✁ ✸ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✼ ✺

◮ ✘ independent entries ◮ ✘ band diagonal ◮ m❀ n❀ ❵ ✦ ✶, with m❂n ✦ ✍ ✷ ✭0❀ 1✮, ❵❂n ✦ 0

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 15 / 28

SLIDE 27

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Coordinates ¡of ¡x ¡ ¡ Coordinates ¡of ¡y ¡

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 16 / 28

SLIDE 28

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

Addi%onal ¡measurements ¡associated ¡ to ¡the ¡first ¡few ¡coordinates ¡ First ¡few ¡coordinates ¡are ¡oversampled! ¡

.....

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 17 / 28

SLIDE 29

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 18 / 28

SLIDE 30

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 19 / 28

SLIDE 31

How does spatial coupling work?

¡ ¡ ¡ ¡ ¡

.....

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 20 / 28

SLIDE 32

Bayes-optimal AMP

x t✰1 ❂ ✑t✭x t ✰ ✭Qt ☞ A✮✄r t✮ ❀ r t ❂ y Ax t ✰ bt ☞ r t1 ✿ Qt❀ bt explicitly given normalizations ✑t✭y✮ ✑ E❢X ❥X ✰ ✜tZ ❂ y❣ (reduces to simple expression in most cases)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 21 / 28

SLIDE 33

Bayes-optimal AMP

x t✰1 ❂ ✑t✭x t ✰ ✭Qt ☞ A✮✄r t✮ ❀ r t ❂ y Ax t ✰ bt ☞ r t1 ✿ Qt❀ bt explicitly given normalizations ✑t✭y✮ ✑ E❢X ❥X ✰ ✜tZ ❂ y❣ (reduces to simple expression in most cases)

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 21 / 28

SLIDE 34

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements. Further, ifa m ❃ D✭X ✮n ✰ o✭n✮, and measurements are noisy MSE ✔ C✭pX ✮✛2 ✿

aD✭X ✮ ❂ d✭X ✮ in most cases.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 22 / 28

SLIDE 35

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements. Further, ifa m ❃ D✭X ✮n ✰ o✭n✮, and measurements are noisy MSE ✔ C✭pX ✮✛2 ✿

aD✭X ✮ ❂ d✭X ✮ in most cases.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 22 / 28

SLIDE 36

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements. Further, ifa m ❃ D✭X ✮n ✰ o✭n✮, and measurements are noisy MSE ✔ C✭pX ✮✛2 ✿

aD✭X ✮ ❂ d✭X ✮ in most cases.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 22 / 28

SLIDE 37

A theorem

Theorem (Donoho, Javanmard, Montanari, 2011)

Let ❢✭x✭n✮❀ y✭n✮✮❣n✕0 be a sequence of instances and assume the empirical distributions converge px✭n✮ ✦ pX . Using Gaussian spatially-coupled matrices, Bayes-optimal AMP recovers x✭n✮ with high probability from m ❃ d✭X ✮ n ✰ o✭n✮ ❀ noiseless measurements. Further, ifa m ❃ D✭X ✮n ✰ o✭n✮, and measurements are noisy MSE ✔ C✭pX ✮✛2 ✿

aD✭X ✮ ❂ d✭X ✮ in most cases.

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 22 / 28

SLIDE 38

Proof technique

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 23 / 28

SLIDE 39

State evolution

A block Gaussian sensing matrix

¡ ¡ ¡ ¡ ¡

     

k blocks ❂ A x t❂ MSE✭t✮ ✷ Rk❀ MSE✭t✮✭i✮ ❂ lim

n✦✶

n k ❦x t

Bi xBi❦2✿

We show a state evolution recursion: MSE✭t✰1✮ ❂ ❋✭MSE✭t✮❀ pX ✮

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 24 / 28

SLIDE 40

State evolution

A block Gaussian sensing matrix

¡ ¡ ¡ ¡ ¡

     

k blocks ❂ A x t❂ MSE✭t✮ ✷ Rk❀ MSE✭t✮✭i✮ ❂ lim

n✦✶

n k ❦x t

Bi xBi❦2✿

We show a state evolution recursion: MSE✭t✰1✮ ❂ ❋✭MSE✭t✮❀ pX ✮

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 24 / 28

SLIDE 41

An illustration

200 400 600 800 0.01 0.02 0.03 0.04 0.05 0.06 i t=5 t=50 t=100 t=150 t=200 t=250 t=300 t=350 t=400

MSE✭t✮

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 25 / 28

SLIDE 42

Steps of the proof

◮ Analysis of the state evolution ◮ Continuum state evolution ◮ An energy functional ❊✭✁✮

✎ Fixed point of the state evolution ✟✶ ✦ r❊✭✟✶✮ ❂ 0

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 26 / 28

SLIDE 43

Supercooling

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 27 / 28

SLIDE 44

Question

Does the spatial coupling phenomenon survive for physically constrained sensing matrices? I will discuss it in my talk on Thursday! Thanks!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 28 / 28

SLIDE 45

Question

Does the spatial coupling phenomenon survive for physically constrained sensing matrices? I will discuss it in my talk on Thursday! Thanks!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 28 / 28

SLIDE 46

Question

Does the spatial coupling phenomenon survive for physically constrained sensing matrices? I will discuss it in my talk on Thursday! Thanks!

Donoho, Javanmard, Montanari Compressed Sensing–Spatial Coupling November 10, 2012 28 / 28