(Nearly) Sample Optimal Sparse Fourier Transform Piotr Indyk 1 - - PowerPoint PPT Presentation

(Nearly) Sample Optimal Sparse Fourier Transform

Piotr Indyk1 Michael Kapralov 1 Eric Price2

1MIT 2MIT→ IBM Almaden → UT Austin

SODA’14

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 1 / 28

Fourier Transform and Sparsity

Discrete Fourier Transform

Given x ∈ Cn, compute

• xi =
• j∈[n]

xjωij, where ω is the n-th root of unity. Fundamental tool: Compression (image, audio, video) Signal processing Data analysis Medical imaging (MRI, NMR)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 2 / 28

Fourier Transform and Sparsity

Sparse Fourier Transform

The fast algorithm for DFT is FFT, runs in O(nlogn) time improving on FFT in full generality a major open problem

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 3 / 28

Fourier Transform and Sparsity

Sparse Fourier Transform

The fast algorithm for DFT is FFT, runs in O(nlogn) time improving on FFT in full generality a major open problem Most interesting signals are sparse (have few nonzero entries) or approximately sparse in the Fourier domain. k-sparse=at most k non-zeros

Hassanieh-Indyk-Katabi-Price’12 compute

approximate sparse FFT in O(k lognlog(n/k)) time

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 3 / 28

Fourier Transform and Sparsity

Sample complexity

Sample complexity=number of samples accessed in time domain. In some applications at least as important as runtime Magnetic Resonance Imaging

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 4 / 28

Fourier Transform and Sparsity

Sample complexity

Sample complexity=number of samples accessed in time domain. In some applications at least as important as runtime Magnetic Resonance Imaging Given access to x ∈ Cn, find y such that

||

x − y||2 ≤ C ·mink−sparse

z||

x − z||2 Optimal sample complexity? ...and small runtime?

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 4 / 28

Fourier Transform and Sparsity

Uniform bounds (for all):

Candes-Tao’06 Rudelson-Vershynin’08 Cheraghchi-Guruswami-Velingker’12 Deterministic, Ω(n) runtime O(k log3 k logn)

Non-uniform bounds (for each):

Goldreich-Levin’89 Mansour’92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss’02 Gilbert-Muthukrishnan-Strauss’05 Hassanieh-Indyk-Katabi-Price’12a Hassanieh-Indyk-Katabi-Price’12b Randomized, O(k ·poly(logn)) runtime O(k lognlog(n/k)) Lower bound: Ω(k logn/loglogn) even for adaptive algorithms Hassanieh-Indyk-Katabi-Price’12

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 5 / 28

Fourier Transform and Sparsity

Uniform bounds (for all):

Candes-Tao’06 Rudelson-Vershynin’08 Cheraghchi-Guruswami-Velingker’12 Deterministic, Ω(n) runtime O(k log3 k logn)

Non-uniform bounds (for each):

Goldreich-Levin’89 Mansour’92 Gilbert-Guha-Indyk-Muthukrishnan-Strauss’02 Gilbert-Muthukrishnan-Strauss’05 Hassanieh-Indyk-Katabi-Price’12a Hassanieh-Indyk-Katabi-Price’12b Randomized, O(k ·poly(logn)) runtime O(k lognlog(n/k)) Lower bound: Ω(k logn/loglogn) even for adaptive algorithms Hassanieh-Indyk-Katabi-Price’12

Theorem There exists an algorithm for ℓ2/ℓ2 sparse recovery from Fourier measurements using O∗(k logn) samples and O∗(k log2n) runtime. O∗() hides (loglogn)O(1) factors. Optimal up to O((loglogn)C) factor

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 5 / 28

Sparse FFT in sublinear time

ℓ2/ℓ2 sparse recovery guarantees: ||

x − y||2 ≤ C ·mink−sparse

z||

x − z||2

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 6 / 28

Sparse FFT in sublinear time

ℓ2/ℓ2 sparse recovery guarantees: ||

x − y||2 ≤ C ·Err2

k(

x)

|

x1| ≥ ... ≥ | xk| ≥ | xk+1| ≥ | xk+2| ≥ ... Err2

k(

x) = n

j=k+1 |

xj|2 Residual error bounded by noise energy Err2

k(

x)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 7 / 28

Sparse FFT in sublinear time

ℓ2/ℓ2 sparse recovery guarantees:

Signal to noise ratio R = || x − y||2/Err2

k(

x) ≤ C

|

x1| ≥ ... ≥ | xk| ≥ | xk+1| ≥ | xk+2| ≥ ... Err2

k(

x) = n

j=k+1 |

xj|2 Residual error bounded by noise energy Err2

k(

x)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 8 / 28

Sparse FFT in sublinear time

ℓ2/ℓ2 sparse recovery guarantees:

Signal to noise ratio R = || x − y||2/Err2

k(

x) ≤ C

|

x1| ≥ ... ≥ | xk| ≥ | xk+1| ≥ | xk+2| ≥ ... Err2

k(

x) = n

j=k+1 |

xj|2 Residual error bounded by noise energy Err2

k(

x) Sufficient to ensure that most elements are below average noise level:

|

xi − yi|2 ≤ c ·Err2

k(

x)/k =: µ2

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 8 / 28

Sparse FFT in sublinear time

Iterative recovery

Many algorithms use the iterative recovery scheme: Input: x ∈ Cn

• y0 ← 0

For t = 1 to L

• z ← REFINEMENT(x,

yt−1) ⊲Takes random samples of x −y Update yt ← yt−1 + z REFINEMENT(x, y) return dominant Fourier coefficients z of x −y (approximately) dominant coefficients≈ | xi − yi|2 ≥ µ2(above average noise level)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 9 / 28

Sparse FFT in sublinear time

REFINEMENT(x, y) return dominant Fourier coefficients z of x −y (approximately) dominant coefficients≈ | xi − yi|2 ≥ µ2(above average noise level) Main questions: How many samples per SNR reduction step? How many iterations?

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 10 / 28

Sparse FFT in sublinear time

REFINEMENT(x, y) return dominant Fourier coefficients z of x −y (approximately) dominant coefficients≈ | xi − yi|2 ≥ µ2(above average noise level) Main questions: How many samples per SNR reduction step? How many iterations? Summary of techniques from

Gilbert-Guha-Indyk-Muthukrishnan-Strauss’02, Akavia-Goldwasser-Safra’03, Gilbert-Muthukrishnan-Strauss’05, Iwen’10, Akavia’10, Hassanieh-Indyk-Katabi-Price’12a, Hassanieh-Indyk-Katabi-Price’12b

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 10 / 28

Sparse FFT in sublinear time

1-sparse recovery from Fourier measurements

−1000 −800 −600 −400 −200 200 400 600 800 1000 −1.5 −1 −0.5 0.5 1 1.5 time magnitude

xa = ωa·f +noise O(logSNR n) measurements for random a

−1000 −800 −600 −400 −200 200 400 600 800 1000 0.2 0.4 0.6 0.8 1 1.2 1.4 frequency magnitude

2πf/n

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 11 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Permute with a random linear transformation and phase shift

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude −1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 12 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Permute with a random linear transformation and phase shift

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude −1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 13 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Permute with a random linear transformation and phase shift

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude −1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 14 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Partition frequency space into B = k/α buckets for constant α ∈ (0,1)

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude

Choose a filter G, G such that

• G approximates the buckets

G has small support

−1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Compute x ∗ G = (x ·G)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 15 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Partition frequency space into B = k/α buckets for constant α ∈ (0,1)

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude

Choose a filter G, G such that

• G approximates the buckets

G has small support

−1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Compute x ∗ G = (x ·G)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 16 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Partition frequency space into B = k/α buckets for constant α ∈ (0,1)

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude

Choose a filter G, G such that

• G approximates the buckets

G has small support

−1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Compute x ∗ G = (x ·G)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 17 / 28

Sparse FFT in sublinear time

Reducing k-sparse recovery to 1-sparse recovery

Partition frequency space into B = k/α buckets for constant α ∈ (0,1)

−1000 −800 −600 −400 −200 200 400 600 800 1000 −2.5 −2 −1.5 −1 −0.5 0.5 1 1.5 2 2.5 time magnitude

Choose a filter G, G such that

• G approximates the buckets

G has small support

−1000 −800 −600 −400 −200 200 400 600 800 1000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Compute x ∗ G = (x ·G) Sample complexity=supp G!

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 18 / 28

Sparse FFT in sublinear time

REFINEMENT step

REFINEMENT(x, y) Make measurements (independent permutation+filtering) Locate and estimate large frequencies (1-sparse recovery) return dominant Fourier coefficients z of x −y (approximately) Sample complexity = support of G

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 19 / 28

Sparse FFT in sublinear time

REFINEMENT step

REFINEMENT(x, y) Make measurements (independent permutation+filtering) Locate and estimate large frequencies (1-sparse recovery) return dominant Fourier coefficients z of x −y (approximately) Sample complexity = support of G How many measurements do we need? How effective is a refinement step? Both determined by signal to noise ratio in each bucket – function of filter choice

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 19 / 28

Sparse FFT in sublinear time

Time domain: support O(k) [GMS’05] Frequency domain:

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

SNR = O(1) Reduce SNR by O(1) factor

Ω(k log2n) samples

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 20 / 28

Sparse FFT in sublinear time

Time domain: support O(k) [GMS’05] Frequency domain:

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

SNR = O(1) Reduce SNR by O(1) factor

Ω(k log2n) samples

Time domain: support Θ(k logn) [HIKP12] Frequency domain:

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

SNR = can by poly(n) Reduce sparsity by O(1) factor

Ω(k log2n) samples

This paper: interpolate between the two extremes, get all benefits, avoid shortcomings

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 21 / 28

Sparse FFT in sublinear time

Our approach

A new family of filters that adapt to current upper bound on SNR. Sharp filters initially, more blurred later

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 22 / 28

Sparse FFT in sublinear time

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

When SNR is bounded by R: filter support O(k logR) (≈ convolve boxcar with itself logR times)

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 23 / 28

Sparse FFT in sublinear time

200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude 200 400 600 800 1000 1200 1400 1600 1800 2000 −0.2 0.2 0.4 0.6 0.8 1 frequency magnitude

When SNR is bounded by R: filter support O(k logR) (≈ convolve boxcar with itself logR times) (most) 1-sparse recovery subproblems for dominant frequencies have high SNR (about R) so O∗(logR n) measurements! O∗(k logR ·logR n) = O∗(k logn) samples per step!

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 23 / 28

Sparse FFT in sublinear time

R → R1/2 → R1/4 → ... → C2 → C

• O(loglogn) iterations

REFINEMENT(x, y,R) k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2 Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 24 / 28

Sparse FFT in sublinear time

R → R1/2 → R1/4 → ... → C2 → C

• O(loglogn) iterations

REFINEMENT(x, y,R) k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2 Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 24 / 28

Sparse FFT in sublinear time

R → R1/2 → R1/4 → ... → C2 → C

• O(loglogn) iterations

REFINEMENT(x, y,R

1 2 )

k

µ2 = tail noise/B

R

1 2 ·µ2

R

1 4 ·µ2 Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 24 / 28

Sparse FFT in sublinear time

R → R1/2 → R1/4 → ... → C2 → C

• O(loglogn) iterations

REFINEMENT(x, y,C2) k

µ2 = tail noise/B

C2 ·µ2 C ·µ2

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 24 / 28

Sparse FFT in sublinear time

Algorithm

Input: x ∈ Cn

• y0 ← 0

R0 ← poly(n) For t = 1 to O(loglogn)

• z ← REFINEMENT(x,

yt−1,Rt−1) ⊲Takes random samples of x −y Update yt ← yt−1 + z Rt ←

• Rt−1

REFINEMENT step: Takes O∗(k logn) samples independent of R Is very effective: reduces R → R

1 2 , so O(loglogn) iterations suffice Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 25 / 28

Sparse FFT in sublinear time

Refinement step analysis

REFINEMENT(x, y,R) k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2

Need to reduce most ‘large’ frequencies, i.e. | xi|2 ≥

• Rµ2

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 26 / 28

Sparse FFT in sublinear time

Refinement step analysis

REFINEMENT(x, y,R) k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2

Need to reduce most ‘large’ frequencies, i.e. | xi|2 ≥

• Rµ2

Most=1−1/poly(R) fraction

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 26 / 28

Sparse FFT in sublinear time

Refinement step analysis

REFINEMENT(x, y,R) k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2

Need to reduce most ‘large’ frequencies, i.e. | xi|2 ≥

• Rµ2

Most=1−1/poly(R) fraction Iterative process, O(loglogn) steps

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 26 / 28

Sparse FFT in sublinear time

k

µ2 = tail noise/B

R ·µ2 R

1 2 ·µ2

partition elements into geometric weight classes write down recursion that governs the dynamics top half classes are reduced at double exponentialy rate∗ if we use Ω(loglogR) levels

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 27 / 28

Sparse FFT in sublinear time

Conclusions

Achieved O(k logn(loglogn)C) samples and O(k log2n(loglogn)C), within O((loglogn)C) of lower bound. Recent work [IK’14?]

• ptimal O(k logn) sample complexity in

O(n) time Open questions: O(k logn) in O(k) time?

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 28 / 28

Sparse FFT in sublinear time

Conclusions

Achieved O(k logn(loglogn)C) samples and O(k log2n(loglogn)C), within O((loglogn)C) of lower bound. Recent work [IK’14?]

• ptimal O(k logn) sample complexity in

O(n) time Open questions: O(k logn) in O(k) time?

Thank you!

Indyk, Kapralov, Price (MIT, IBM Almaden) (Nearly) Optimal Fourier Sampling SODA’14 28 / 28