Sparse Recovery and Fourier Sampling Eric Price MIT Eric Price - - PowerPoint PPT Presentation

Sparse Recovery and Fourier Sampling

Eric Price

MIT

Eric Price (MIT) Sparse Recovery and Fourier Sampling 1 / 37

The Fourier Transform

Conversion between time and frequency domains

Time Domain Frequency Domain Fourier Transform Displacement of Air Concert A

Eric Price (MIT) Sparse Recovery and Fourier Sampling 2 / 37

The Fourier Transform is Ubiquitous

Audio Video Medical Imaging Radar GPS Oil Exploration

Eric Price (MIT) Sparse Recovery and Fourier Sampling 3 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2).

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965] [T]he method greatly reduces the tediousness of mechanical calculations. – Carl Friedrich Gauss, 1805

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965] [T]he method greatly reduces the tediousness of mechanical calculations. – Carl Friedrich Gauss, 1805 By hand: 22n log n seconds. [Danielson-Lanczos, 1942]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965] [T]he method greatly reduces the tediousness of mechanical calculations. – Carl Friedrich Gauss, 1805 By hand: 22n log n seconds. [Danielson-Lanczos, 1942] Can we do better?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965] [T]he method greatly reduces the tediousness of mechanical calculations. – Carl Friedrich Gauss, 1805 By hand: 22n log n seconds. [Danielson-Lanczos, 1942] Can we do much better?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Computing the Discrete Fourier Transform

How to compute x = Fx? Naive multiplication: O(n2). Fast Fourier Transform: O(n log n) time. [Cooley-Tukey, 1965] [T]he method greatly reduces the tediousness of mechanical calculations. – Carl Friedrich Gauss, 1805 By hand: 22n log n seconds. [Danielson-Lanczos, 1942] Can we do much better? When can we compute the Fourier Transform in sublinear time?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 4 / 37

Idea: Leverage Sparsity

Often the Fourier transform is dominated by a small number of peaks: Time Signal Frequency (Exactly sparse) Frequency (Approximately sparse)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 5 / 37

Idea: Leverage Sparsity

Often the Fourier transform is dominated by a small number of peaks: Time Signal Frequency (Exactly sparse) Frequency (Approximately sparse) Sparsity is common:

Audio Video Medical Imaging Radar GPS Oil Exploration

Eric Price (MIT) Sparse Recovery and Fourier Sampling 5 / 37

Idea: Leverage Sparsity

Often the Fourier transform is dominated by a small number of peaks: Time Signal Frequency (Exactly sparse) Frequency (Approximately sparse) Sparsity is common:

Audio Video Medical Imaging Radar GPS Oil Exploration

Goal of this work: a sparse Fourier transform

Faster Fourier Transform on sparse data.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 5 / 37

Talk Outline

1

Sparse Fourier Transform Overview Technical Details

Eric Price (MIT) Sparse Recovery and Fourier Sampling 6 / 37

Talk Outline

1

Sparse Fourier Transform Overview Technical Details

2

Beyond: Sparse Recovery / Compressive Sensing Overview Adaptivity Conclusion

Eric Price (MIT) Sparse Recovery and Fourier Sampling 6 / 37

Talk Outline

1

Sparse Fourier Transform Overview Technical Details

2

Beyond: Sparse Recovery / Compressive Sensing Overview Adaptivity Conclusion

Eric Price (MIT) Sparse Recovery and Fourier Sampling 7 / 37

My Contributions

Goal: Compute the Fourier transform x = Fx when x is k-sparse. Theory:

◮ The fastest algorithm for Fourier transforms of sparse data. ◮ The only algorithms faster than FFT for all k = o(n). Eric Price (MIT) Sparse Recovery and Fourier Sampling 8 / 37

My Contributions

Goal: Compute the Fourier transform x = Fx when x is k-sparse. Theory:

◮ The fastest algorithm for Fourier transforms of sparse data. ◮ The only algorithms faster than FFT for all k = o(n).

Practice:

◮ Implementation is faster than FFTW for a wide range of inputs. ◮ Orders of magnitude faster than previous sparse Fourier transforms. ◮ Useful in multiple applications. Eric Price (MIT) Sparse Recovery and Fourier Sampling 8 / 37

Applications of ideas

http://groups.csail.mit.edu/netmit/sFFT/workshop.html

GPS [HAKI]: 2× faster

Eric Price (MIT) Sparse Recovery and Fourier Sampling 9 / 37

Applications of ideas

http://groups.csail.mit.edu/netmit/sFFT/workshop.html

GPS [HAKI]: 2× faster Spectrum sensing [HSAHK]: 6× lower sampling rate

Eric Price (MIT) Sparse Recovery and Fourier Sampling 9 / 37

Applications of ideas

http://groups.csail.mit.edu/netmit/sFFT/workshop.html

GPS [HAKI]: 2× faster Spectrum sensing [HSAHK]: 6× lower sampling rate Dense FFT over clusters [TPKP]: 2× faster

Eric Price (MIT) Sparse Recovery and Fourier Sampling 9 / 37

Applications of ideas

http://groups.csail.mit.edu/netmit/sFFT/workshop.html

GPS [HAKI]: 2× faster Spectrum sensing [HSAHK]: 6× lower sampling rate Dense FFT over clusters [TPKP]: 2× faster ...

Eric Price (MIT) Sparse Recovery and Fourier Sampling 9 / 37

Talk Outline

1

Sparse Fourier Transform Overview Technical Details

2

Beyond: Sparse Recovery / Compressive Sensing Overview Adaptivity Conclusion

Eric Price (MIT) Sparse Recovery and Fourier Sampling 10 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

◮ All take at least k log4 n time. ◮ Only better than FFT if k ≪ n/ log3 n. Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

◮ All take at least k log4 n time. ◮ Only better than FFT if k ≪ n/ log3 n.

Our results [HIKP12a, HIKP12b]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

◮ All take at least k log4 n time. ◮ Only better than FFT if k ≪ n/ log3 n.

Our results [HIKP12a, HIKP12b]

◮ Exactly k-sparse: O(k log n) ⋆ Optimal if FFT is optimal. Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

◮ All take at least k log4 n time. ◮ Only better than FFT if k ≪ n/ log3 n.

Our results [HIKP12a, HIKP12b]

◮ Exactly k-sparse: O(k log n) ⋆ Optimal if FFT is optimal. ◮ Approximately k-sparse: O(k log(n/k) log n)

result − x2 (1 + ǫ) min

k-sparse x(k)

• x(k) −

x2

Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Theoretical Results

For a signal of size n with k large frequencies

First on Boolean cube [GL89, KM92, L93] Adapted to complexes [Mansour ’92, GGIMS02, AGS03, GMS05, Iwen ’10, Akavia ’10]

◮ All take at least k log4 n time. ◮ Only better than FFT if k ≪ n/ log3 n.

Our results [HIKP12a, HIKP12b]

◮ Exactly k-sparse: O(k log n) ⋆ Optimal if FFT is optimal. ◮ Approximately k-sparse: O(k log(n/k) log n)

result − x2 (1 + ǫ) min

k-sparse x(k)

• x(k) −

x2

◮ Better than FFT for any k = o(n) Eric Price (MIT) Sparse Recovery and Fourier Sampling 11 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n (where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

• x = F x

for Fij = ω−ij/n (where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

• x = F x

for Fij = ω−ij/n Inverse transform almost identical: (where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

• x = F x

for Fij = ω−ij/n Inverse transform almost identical: xi =

• j

ωij xj

◮ ω → ω−1, scale

(where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

• x = F x

for Fij = ω−ij/n Inverse transform almost identical: xi =

• j

ωij xj

◮ ω → ω−1, scale

Lots of nice properties (where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Discrete Fourier Transform (DFT) Definition

Given x ∈ Cn, compute Fourier transform x:

• xi = 1

n

• j

ω−ijxj for ω = eτi/n

• x = F x

for Fij = ω−ij/n Inverse transform almost identical: xi =

• j

ωij xj

◮ ω → ω−1, scale

Lots of nice properties

◮ Convolution ←

→ Multiplication

(where τ is the circle constant 6.283...)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 12 / 37

Algorithm

Simpler case: x is exactly k-sparse.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 13 / 37

Algorithm

Simpler case: x is exactly k-sparse.

Theorem

We can compute x in O(k log n) expected time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 13 / 37

Algorithm

Simpler case: x is exactly k-sparse.

Theorem

We can compute x in O(k log n) expected time. Still kind of hard.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 13 / 37

Algorithm

Simpler case: x is exactly k-sparse.

Theorem

We can compute x in O(k log n) expected time. Still kind of hard. Simplest case: x is exactly 1-sparse.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 13 / 37

Algorithm

Simpler case: x is exactly k-sparse.

Theorem

We can compute x in O(k log n) expected time. Still kind of hard. Simplest case: x is exactly 1-sparse.

Lemma

We can compute a 1-sparse x in O(1) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 13 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t).

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t). x0 = a

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t). x0 = a x1 = aωt

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t). x0 = a x1 = aωt x1/x0 = ωt = ⇒ t.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t). x0 = a x1 = aωt x1/x0 = ωt = ⇒ t.

• Eric Price (MIT)

Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for k = 1

• x:

t a

Lemma

We can compute a 1-sparse x in O(1) time.

• xi =

a if i = t

• therwise

Then x = (a, aωt, aω2t, aω3t, . . . , aω(n−1)t). x0 = a x1 = aωt x1/x0 = ωt = ⇒ t.

• (Related to OFDM, Prony’s method, matrix pencil.)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 14 / 37

Algorithm for general k

Reduce general k to k = 1.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket. Random permutation

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket. Random permutation

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket. Random permutation

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Algorithm for general k

Reduce general k to k = 1. “Filters”: partition frequencies into O(k) buckets.

◮ Sample from time domain of each

bucket with O(log n) overhead.

◮ Recovered by k = 1 algorithm

Most frequencies alone in bucket. Random permutation

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Recovers most of x:

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 15 / 37

Overall outline

• x

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Theorem

We can compute x in O(k log n) expected time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

Overall outline

• x −

x ′

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial k-sparse recovery x

• x ′

Lemma (Partial sparse recovery)

In O(k log n) expected time, we can compute an estimate x ′ such that

• x −

x ′ is k/2-sparse. Repeat, k → k/2 → k/4 → · · ·

Theorem

We can compute x in O(k log n) expected time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 16 / 37

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x n-dimensional DFT of first k terms: O(n log n) x · rect → x ∗ sinc.

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x n-dimensional DFT of first k terms: O(n log n) x · rect → x ∗ sinc.

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x n-dimensional DFT of first k terms: O(n log n) x · rect → x ∗ sinc. k-dimensional DFT of first k terms: O(B log B) alias(x · rect) → subsample( x ∗ sinc).

How can you isolate frequencies?

Time Frequency

× = ∗ =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 17 / 37

n-dimensional DFT: O(n log n) x → x n-dimensional DFT of first k terms: O(n log n) x · rect → x ∗ sinc. k-dimensional DFT of first k terms: O(B log B) alias(x · rect) → subsample( x ∗ sinc).

The issue

Frequency We want to isolate frequencies.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 18 / 37

The issue

Frequency We want to isolate frequencies. The sinc filter “leaks”. Contamination from other buckets.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 18 / 37

The issue

Frequency We want to isolate frequencies. The sinc filter “leaks”. Contamination from other buckets. We introduce a better filter: (Gaussian / prolate spheroidal sequence) convolved with rectangle.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 18 / 37

Algorithm for exactly sparse signals

Original signal x Goal ˆ x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

Computed F ·x Filtered signal c F ∗c x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Filtered signal c F ∗c x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Computed samples of c F ∗c x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Computed samples of c F ∗c x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Knowledge about ˆ x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Knowledge about ˆ x

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

F ·x aliased to k terms Knowledge about ˆ x

Lemma

If t is isolated in its bucket and in the “super-pass” region, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 19 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt

Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt = ⇒ can compute t.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt = ⇒ can compute t.

◮ Just like our 1-sparse recovery algorithm, x1/x0 = ωt. Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt = ⇒ can compute t.

◮ Just like our 1-sparse recovery algorithm, x1/x0 = ωt.

Gives partial sparse recovery: x ′ such that x − x ′ is k/2-sparse.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt = ⇒ can compute t.

◮ Just like our 1-sparse recovery algorithm, x1/x0 = ωt.

Gives partial sparse recovery: x ′ such that x − x ′ is k/2-sparse.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Repeat k → k/2 → k/4 → · · ·

Eric Price (MIT) Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for exactly sparse signals

Lemma

For most t, the value b we compute for its bucket satisfies b = xt. Computing the b for all O(k) buckets takes O(k log n) time. Time-shift x by one and repeat: b′ = xtωt. Divide to get b′/b = ωt = ⇒ can compute t.

◮ Just like our 1-sparse recovery algorithm, x1/x0 = ωt.

Gives partial sparse recovery: x ′ such that x − x ′ is k/2-sparse.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Repeat k → k/2 → k/4 → · · · O(k log n) time sparse Fourier transform.

• Eric Price (MIT)

Sparse Recovery and Fourier Sampling 20 / 37

Algorithm for approximately sparse signals

Eric Price (MIT) Sparse Recovery and Fourier Sampling 21 / 37

Algorithm for approximately sparse signals

What changes with noise?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 21 / 37

Algorithm for approximately sparse signals

What changes with noise? Identical architecture:

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial sparse recovery x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 21 / 37

Algorithm for approximately sparse signals

What changes with noise? Identical architecture:

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

Partial sparse recovery x

• x ′

Just requires robust 1-sparse recovery.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 21 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

x1/x0 = ωt

With exact sparsity: log n bits in a single measurement.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

x1/x0 = ωt + noise

With exact sparsity: log n bits in a single measurement. With noise: only constant number of useful bits.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

x1/x0 = ωt + noise

With exact sparsity: log n bits in a single measurement. With noise: only constant number of useful bits. Choose Θ(log n) time shifts c to recover i.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

xc2/x0 = ωc2t + noise

With exact sparsity: log n bits in a single measurement. With noise: only constant number of useful bits. Choose Θ(log n) time shifts c to recover i.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

xc3/x0 = ωc3t + noise

With exact sparsity: log n bits in a single measurement. With noise: only constant number of useful bits. Choose Θ(log n) time shifts c to recover i.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: k = 1

Lemma

Suppose x is approximately 1-sparse: | xt|/ x2 90%. Then we can recover it with O(log n) samples and O(log2 n) time.

xc3/x0 = ωc3t + noise

With exact sparsity: log n bits in a single measurement. With noise: only constant number of useful bits. Choose Θ(log n) time shifts c to recover i. Error correcting code with efficient recovery = ⇒ Lemma.

• Eric Price (MIT)

Sparse Recovery and Fourier Sampling 22 / 37

Algorithm for approximately sparse signals: general k

Lemma

If x is approximately 1-sparse, we can recover it with O(log n) samples and O(log2 n) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 23 / 37

Algorithm for approximately sparse signals: general k

Lemma

If x is approximately 1-sparse, we can recover it with O(log n) samples and O(log2 n) time.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Reduce k-sparse to 1-sparse on buckets of size n/k, with log n

• verhead per sample.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 23 / 37

Algorithm for approximately sparse signals: general k

Lemma

If x is approximately 1-sparse, we can recover it with O(log n) samples and O(log2 n) time.

Permute Filters O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Reduce k-sparse to 1-sparse on buckets of size n/k, with log n

• verhead per sample.

Theorem

If x is approximately k-sparse, we can recover it in O(k log(n/k) log n) time.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 23 / 37

Empirical performance

Compare to

◮ FFTW, the “Fastest Fourier Transform in the West” ◮ AAFFT, the [GMS05] sparse Fourier transform.

0.001 0.01 0.1 1 10 26 27 28 29 210 211 212 213 214 215 216 217 218

Run Time (sec) Sparsity (K) Run Time vs Signal Sparsity (N=222)

sFFT 3.0 (Exact) FFTW AAFFT 0.9 1e-05 0.0001 0.001 0.01 0.1 1 10 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

Run Time (sec) Signal Size (n) Run Time vs Signal Size (k=50)

sFFT 3.0 (Exact) FFTW AAFFT 0.9

Eric Price (MIT) Sparse Recovery and Fourier Sampling 24 / 37

Empirical performance

Compare to

◮ FFTW, the “Fastest Fourier Transform in the West” ◮ AAFFT, the [GMS05] sparse Fourier transform.

0.001 0.01 0.1 1 10 26 27 28 29 210 211 212 213 214 215 216 217 218

Run Time (sec) Sparsity (K) Run Time vs Signal Sparsity (N=222)

sFFT 3.0 (Exact) FFTW AAFFT 0.9 1e-05 0.0001 0.001 0.01 0.1 1 10 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

Run Time (sec) Signal Size (n) Run Time vs Signal Size (k=50)

sFFT 3.0 (Exact) FFTW AAFFT 0.9

Eric Price (MIT) Sparse Recovery and Fourier Sampling 24 / 37

Empirical performance

Compare to

◮ FFTW, the “Fastest Fourier Transform in the West” ◮ AAFFT, the [GMS05] sparse Fourier transform.

0.001 0.01 0.1 1 10 26 27 28 29 210 211 212 213 214 215 216 217 218

Run Time (sec) Sparsity (K) Run Time vs Signal Sparsity (N=222)

sFFT 3.0 (Exact) FFTW AAFFT 0.9 1e-05 0.0001 0.001 0.01 0.1 1 10 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

Run Time (sec) Signal Size (n) Run Time vs Signal Size (k=50)

sFFT 3.0 (Exact) FFTW AAFFT 0.9

Eric Price (MIT) Sparse Recovery and Fourier Sampling 24 / 37

Empirical performance

Compare to

◮ FFTW, the “Fastest Fourier Transform in the West” ◮ AAFFT, the [GMS05] sparse Fourier transform.

0.001 0.01 0.1 1 10 26 27 28 29 210 211 212 213 214 215 216 217 218

Run Time (sec) Sparsity (K) Run Time vs Signal Sparsity (N=222)

sFFT 3.0 (Exact) FFTW AAFFT 0.9 1e-05 0.0001 0.001 0.01 0.1 1 10 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224

Run Time (sec) Signal Size (n) Run Time vs Signal Size (k=50)

sFFT 3.0 (Exact) FFTW AAFFT 0.9

Faster than FFTW for wide range of values.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 24 / 37

Recap of Sparse Fourier Transform

Theory:

◮ The fastest algorithm for Fourier transforms of sparse data. ◮ The only algorithms faster than FFT for all k = o(n). Eric Price (MIT) Sparse Recovery and Fourier Sampling 25 / 37

Recap of Sparse Fourier Transform

Theory:

◮ The fastest algorithm for Fourier transforms of sparse data. ◮ The only algorithms faster than FFT for all k = o(n).

Practice:

◮ Implementation is faster than FFTW for a wide range of inputs. ◮ Orders of magnitude faster than previous sparse Fourier transforms. ◮ Useful in multiple applications. Eric Price (MIT) Sparse Recovery and Fourier Sampling 25 / 37

Talk Outline

1

Sparse Fourier Transform Overview Technical Details

2

Beyond: Sparse Recovery / Compressive Sensing Overview Adaptivity Conclusion

Eric Price (MIT) Sparse Recovery and Fourier Sampling 26 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax. x A y =

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax. x A y =

Sparse Fourier

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax.

Sparse Fourier

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax.

Sparse Fourier MRI

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax.

Sparse Fourier MRI Single-Pixel Camera

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax.

Sparse Fourier MRI Single-Pixel Camera Streaming Algorithms A(x + ∆) = Ax + A∆

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

Sparse Recovery / Compressive Sensing

Robustly recover sparse x from linear measurements y = Ax.

Sparse Fourier MRI Single-Pixel Camera Streaming Algorithms A(x + ∆) = Ax + A∆ Genetic Testing

Eric Price (MIT) Sparse Recovery and Fourier Sampling 27 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a] MRI: minimize Fourier sample complexity [GHIKPS13, IKP14]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a] MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a] MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] Streaming: improved analysis of Count-Sketch [MP14, PW11, P11]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a] MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] Streaming: improved analysis of Count-Sketch [MP14, PW11, P11] Genetic testing: first asymptotic gain using adaptivity [IPW11, PW13]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

My Contributions

Sparse Fourier: minimize time complexity [HIKP12b, HIKP12a] MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] Streaming: improved analysis of Count-Sketch [MP14, PW11, P11] Genetic testing: first asymptotic gain using adaptivity [IPW11, PW13]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 28 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed. Output x ′ satisfying x ′ − x2 < (1 + ǫ) min

k-sparse x(k)

x − x(k)2

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed. Output x ′ satisfying x ′ − x2 < (1 + ǫ) min

k-sparse x(k)

x − x(k)2 Nonadaptively: Θ(k log(n/k)) measurements necessary and

• sufficient. [Cand`

es-Romberg-Tao ’06, DIPW ’10]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed. Output x ′ satisfying x ′ − x2 < (1 + ǫ) min

k-sparse x(k)

x − x(k)2 Nonadaptively: Θ(k log(n/k)) measurements necessary and

• sufficient. [Cand`

es-Romberg-Tao ’06, DIPW ’10] Natural question: does adaptivity help?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed. Output x ′ satisfying x ′ − x2 < (1 + ǫ) min

k-sparse x(k)

x − x(k)2 Nonadaptively: Θ(k log(n/k)) measurements necessary and

• sufficient. [Cand`

es-Romberg-Tao ’06, DIPW ’10] Natural question: does adaptivity help?

◮ Studied in [MSW08, JXC08, CHNR08, AWZ08, HCN09, ACD11, ...] Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Adaptive Sparse Recovery Model

Unknown approximately k-sparse vector x ∈ Rn. Choose v ∈ Rn, observe y = v, x. Choose another v and repeat as needed. Output x ′ satisfying x ′ − x2 < (1 + ǫ) min

k-sparse x(k)

x − x(k)2 Nonadaptively: Θ(k log(n/k)) measurements necessary and

• sufficient. [Cand`

es-Romberg-Tao ’06, DIPW ’10] Natural question: does adaptivity help?

◮ Studied in [MSW08, JXC08, CHNR08, AWZ08, HCN09, ACD11, ...]

First asymptotic improvement: O(k log log(n/k)) measurements. [IPW ’11]

Eric Price (MIT) Sparse Recovery and Fourier Sampling 29 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Applications of Adaptivity

Eric Price (MIT) Sparse Recovery and Fourier Sampling 30 / 37

Outline of Algorithm

Theorem

Adaptive k-sparse recovery is possible with O(k log log(n/k)) measurements.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 31 / 37

Outline of Algorithm

Theorem

Adaptive k-sparse recovery is possible with O(k log log(n/k)) measurements.

Permute Partition O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Eric Price (MIT) Sparse Recovery and Fourier Sampling 31 / 37

Outline of Algorithm

Theorem

Adaptive k-sparse recovery is possible with O(k log log(n/k)) measurements.

Permute Partition O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Suffices to solve for k = 1:

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 31 / 37

Outline of Algorithm

Theorem

Adaptive k-sparse recovery is possible with O(k log log(n/k)) measurements.

Permute Partition O(k) 1-sparse recovery 1-sparse recovery 1-sparse recovery 1-sparse recovery

x

• x ′

Suffices to solve for k = 1:

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 31 / 37

1-sparse recovery: non-adaptive lower bound

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 32 / 37

1-sparse recovery: non-adaptive lower bound

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements. Non-adaptive lower bound: why is this hard?

Eric Price (MIT) Sparse Recovery and Fourier Sampling 32 / 37

1-sparse recovery: non-adaptive lower bound

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements. Non-adaptive lower bound: why is this hard? Hard case: x is random ei plus Gaussian noise w with w2 ≈ 1.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 32 / 37

1-sparse recovery: non-adaptive lower bound

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements. Non-adaptive lower bound: why is this hard? Hard case: x is random ei plus Gaussian noise w with w2 ≈ 1. Robust recovery must locate i.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 32 / 37

1-sparse recovery: non-adaptive lower bound

Lemma

Adaptive 1-sparse recovery is possible with O(log log n) measurements. Non-adaptive lower bound: why is this hard? Hard case: x is random ei plus Gaussian noise w with w2 ≈ 1. Robust recovery must locate i. Observations v, x = vi + v, w = vi + v2

√n z, for z ∼ N(0, 1).

Eric Price (MIT) Sparse Recovery and Fourier Sampling 32 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Shannon 1948: information capacity I(i, v, x) 1 2 log(1 + SNR) where SNR denotes the “signal-to-noise ratio,” SNR = E[signal2] E[noise2] = E[v2

i ]

v2

2/n = 1

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: non-adaptive lower bound

Observe v, x = vi + v2

√n z, where z ∼ N(0, 1)

Shannon 1948: information capacity I(i, v, x) 1 2 log(1 + SNR) where SNR denotes the “signal-to-noise ratio,” SNR = E[signal2] E[noise2] = E[v2

i ]

v2

2/n = 1

Finding i needs Ω(log n) non-adaptive measurements.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 33 / 37

1-sparse recovery: changes in adaptive setting

Information capacity I(i, v, x) 1 2 log(1 + SNR). where SNR denotes the “signal-to-noise ratio,” SNR = E[v2

i ]

v2

2/n.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 34 / 37

1-sparse recovery: changes in adaptive setting

Information capacity I(i, v, x) 1 2 log(1 + SNR). where SNR denotes the “signal-to-noise ratio,” SNR = E[v2

i ]

v2

2/n.

If i is independent of v, this is O(1).

Eric Price (MIT) Sparse Recovery and Fourier Sampling 34 / 37

1-sparse recovery: changes in adaptive setting

Information capacity I(i, v, x) 1 2 log(1 + SNR). where SNR denotes the “signal-to-noise ratio,” SNR = E[v2

i ]

v2

2/n.

If i is independent of v, this is O(1). As we learn about i, we can increase the SNR.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 34 / 37

1-sparse recovery: idea

x = ei + w

0 bits v Candidate set Signal

SNR = 2 I(i, v, x) log SNR = 1 v, x = vi + v, w

Eric Price (MIT) Sparse Recovery and Fourier Sampling 35 / 37

1-sparse recovery: idea

x = ei + w

0 bits 1 bit v Candidate set Signal

SNR = 22 I(i, v, x) log SNR = 2 v, x = vi + v, w

Eric Price (MIT) Sparse Recovery and Fourier Sampling 35 / 37

1-sparse recovery: idea

x = ei + w

0 bits 1 bit 2 bits v Candidate set Signal

SNR = 24 I(i, v, x) log SNR = 4 v, x = vi + v, w

Eric Price (MIT) Sparse Recovery and Fourier Sampling 35 / 37

1-sparse recovery: idea

x = ei + w

0 bits 1 bit 2 bits 4 bits v Candidate set Signal

SNR = 28 I(i, v, x) log SNR = 8 v, x = vi + v, w

Eric Price (MIT) Sparse Recovery and Fourier Sampling 35 / 37

1-sparse recovery: idea

x = ei + w

0 bits 1 bit 2 bits 4 bits 8 bits v Candidate set Signal

SNR = 216 I(i, v, x) log SNR = 16 v, x = vi + v, w

Eric Price (MIT) Sparse Recovery and Fourier Sampling 35 / 37

1-sparse recovery

Lemma (IPW11)

Adaptive 1-sparse recovery takes O(log log n) measurements.

5 10 15 20 25 30

log n

5 10 15 20 25 30

Number of measurements

m as a function of n (SNR=10db,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

10 20 30 40 50

SNR (dB)

5 10 15 20 25 30

Number of measurements

m as a function of SNR (n=8192,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

Eric Price (MIT) Sparse Recovery and Fourier Sampling 36 / 37

1-sparse recovery

Lemma (IPW11, PW13)

Adaptive 1-sparse recovery takes Θ(log log n) measurements.

5 10 15 20 25 30

log n

5 10 15 20 25 30

Number of measurements

m as a function of n (SNR=10db,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

10 20 30 40 50

SNR (dB)

5 10 15 20 25 30

Number of measurements

m as a function of SNR (n=8192,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

Eric Price (MIT) Sparse Recovery and Fourier Sampling 36 / 37

1-sparse recovery

Lemma (IPW11, PW13)

Adaptive 1-sparse recovery takes Θ(log log n) measurements.

5 10 15 20 25 30

log n

5 10 15 20 25 30

Number of measurements

m as a function of n (SNR=10db,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

10 20 30 40 50

SNR (dB)

5 10 15 20 25 30

Number of measurements

m as a function of SNR (n=8192,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

Eric Price (MIT) Sparse Recovery and Fourier Sampling 36 / 37

1-sparse recovery

Lemma (IPW11, PW13)

Adaptive 1-sparse recovery takes Θ(log log n) measurements.

5 10 15 20 25 30

log n

5 10 15 20 25 30

Number of measurements

m as a function of n (SNR=10db,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

10 20 30 40 50

SNR (dB)

5 10 15 20 25 30

Number of measurements

m as a function of SNR (n=8192,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

Eric Price (MIT) Sparse Recovery and Fourier Sampling 36 / 37

1-sparse recovery

Lemma (IPW11, PW13)

Adaptive 1-sparse recovery takes Θ(log log n) measurements.

5 10 15 20 25 30

log n

5 10 15 20 25 30

Number of measurements

m as a function of n (SNR=10db,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

10 20 30 40 50

SNR (dB)

5 10 15 20 25 30

Number of measurements

m as a function of SNR (n=8192,k=1)

Gaussian measurements, L1 minimization Adaptive measurements

Gives Θ(k log log(n/k)) k-sparse recovery via general framework.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 36 / 37

Summary

Sparse Fourier transform

◮ Fastest algorithm for Fourier transforms on sparse data ◮ Already has applications with substantial improvements Eric Price (MIT) Sparse Recovery and Fourier Sampling 37 / 37

Summary

Sparse Fourier transform

◮ Fastest algorithm for Fourier transforms on sparse data ◮ Already has applications with substantial improvements

Broader sparse recovery theory

◮ Sparse Fourier: minimize time complexity [HIKP12] ◮ MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] ◮ Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] ◮ Streaming: improved analysis of Count-Sketch [MP14, PW11, P11] ◮ Genetic testing: first asymptotic gain using adaptivity [IPW11, PW13] Eric Price (MIT) Sparse Recovery and Fourier Sampling 37 / 37

Summary

Sparse Fourier transform

◮ Fastest algorithm for Fourier transforms on sparse data ◮ Already has applications with substantial improvements

Broader sparse recovery theory

◮ Sparse Fourier: minimize time complexity [HIKP12] ◮ MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] ◮ Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] ◮ Streaming: improved analysis of Count-Sketch [MP14, PW11, P11] ◮ Genetic testing: first asymptotic gain using adaptivity [IPW11, PW13]

Lower bounds

◮ Based on Gaussian channel capacity: tight bounds, extensible to

adaptive settings.

◮ Based on communication complexity: extends to ℓ1 setting. Eric Price (MIT) Sparse Recovery and Fourier Sampling 37 / 37

Summary

Sparse Fourier transform

◮ Fastest algorithm for Fourier transforms on sparse data ◮ Already has applications with substantial improvements

Broader sparse recovery theory

◮ Sparse Fourier: minimize time complexity [HIKP12] ◮ MRI: minimize Fourier sample complexity [GHIKPS13, IKP14] ◮ Camera: use Earth-Mover Distance metric [IP11, GIP10, GIPR11] ◮ Streaming: improved analysis of Count-Sketch [MP14, PW11, P11] ◮ Genetic testing: first asymptotic gain using adaptivity [IPW11, PW13]

Lower bounds

◮ Based on Gaussian channel capacity: tight bounds, extensible to

adaptive settings.

◮ Based on communication complexity: extends to ℓ1 setting.

Thank You

Eric Price (MIT) Sparse Recovery and Fourier Sampling 37 / 37

The Future

Make sparse Fourier applicable to more problems

Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

The Future

Make sparse Fourier applicable to more problems

◮ Better sample complexity Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

The Future

Make sparse Fourier applicable to more problems

◮ Better sample complexity ◮ Incorporate stronger notions of structure Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

The Future

Make sparse Fourier applicable to more problems

◮ Better sample complexity ◮ Incorporate stronger notions of structure

Tight constants in compressive sensing

Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

The Future

Make sparse Fourier applicable to more problems

◮ Better sample complexity ◮ Incorporate stronger notions of structure

Tight constants in compressive sensing

◮ Analogous to channel capacity in coding theory. Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

The Future

Make sparse Fourier applicable to more problems

◮ Better sample complexity ◮ Incorporate stronger notions of structure

Tight constants in compressive sensing

◮ Analogous to channel capacity in coding theory. ◮ Lower bound techniques, from information theory, should be strong

enough.

Eric Price (MIT) Sparse Recovery and Fourier Sampling 38 / 37

Eric Price (MIT) Sparse Recovery and Fourier Sampling 39 / 37

Eric Price (MIT) Sparse Recovery and Fourier Sampling 40 / 37