Applied Harmonic Analysis meets Compressed Sensing Gitta Kutyniok - PowerPoint PPT Presentation

Applied Harmonic Analysis meets Compressed Sensing Gitta Kutyniok (Technische Universit¨ at Berlin) ICERM Program “Network Science and Graph Algorithms” February 4, 2014 Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 1 / 50

Outline Imaging Sciences 1 Inspiring Empirical Results Goal for Today Review of Compressed Sensing 2 Separation via Compressed Sensing Inpainting via Compressed Sensing Algorithmic Aspects 3 Shearlet Systems Numerical Results Theoretical Analysis 4 Geometrically Clustered Sparsity Analysis of Separation (joint with D. L. Donoho) Analysis of Inpainting (joint with E. King and X. Zhuang) Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 2 / 50

Ill-posed Inverse Problems in Data Analysis Two Challenges: Modern Data in general is often composed of two or more morphologically distinct constituents. � Task: Separation of components given the composed data. Applications often cause loss of information or necessary information can not be collected. � Task: Recovery of missing data given the observed data. Novel Approach: Applied Harmonic Analysis Compressed Sensing � First empirical results by J. L. Starck, M. Elad, and D. L. Donoho. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 3 / 50

Separating Artifacts in Images, I + (Source: Starck, Elad, and Donoho; 2006) Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 4 / 50

Separating Artifacts in Images, II Neurobiological Imaging: Detection of characteristics of Alzheimer. Separation of spines and dendrites. + (Source: Brandt, K, Lim, and S¨ undermann; 2010) Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 5 / 50

Inpainting, I (Source: Hennenfent and Herrmann; 2008) Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 6 / 50

Inpainting, II (Source: King, K, Lim, Zhuang; 2012) Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 7 / 50

Applied Harmonic Analysis Approach Methodology: Exploit a carefully designed representation system ( ψ λ ) λ ⊆ H : � H ⊇ C ∋ f − → ( � f , ψ λ � ) λ − → � f , ψ λ � ψ λ = f λ Two Main Goals: (1) Decomposition (2) Efficient representations Main Desiderata: Multiscale representation system. Partition of Fourier domain. Fast decomposition and reconstruction algorithm. Optimally sparse approximation of the considered class. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 8 / 50

Goal for Today Challenges for Today: Methodology to derive the empirical results! ◮ Applied Harmonic Analysis. ◮ Compressed Sensing. Improvement of the methodology! ◮ Shearlets as sparsifying system. Analysis of the methodology! ◮ Continuum model. ◮ Geometrically clustered sparsity. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 9 / 50

Goal for Today Challenges for Today: Methodology to derive the empirical results! ◮ Applied Harmonic Analysis. ◮ Compressed Sensing. Improvement of the methodology! ◮ Shearlets as sparsifying system. Analysis of the methodology! ◮ Continuum model. ◮ Geometrically clustered sparsity. Another Path to Imaging Science are Variational Approaches: Contributors: Bertozzi, Burger, Chan, Esedoglu, Kang, Osher, Sapiro, Setzer, Shen, Steidl, Vese, Weikert, ... Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 9 / 50

How does Compressed Sensing come into play? Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 10 / 50

Underdetermined Situations Separation: Observe a signal x composed of two subsignals x 1 and x 2 : x = x 1 + x 2 . Extract the two subsignals x 1 and x 2 from x , if only x is known. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50

Underdetermined Situations Separation: Observe a signal x composed of two subsignals x 1 and x 2 : x = x 1 + x 2 . Extract the two subsignals x 1 and x 2 from x , if only x is known. � The two components are geometrically different. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50

Underdetermined Situations Separation: Observe a signal x composed of two subsignals x 1 and x 2 : x = x 1 + x 2 . Extract the two subsignals x 1 and x 2 from x , if only x is known. � The two components are geometrically different. Inpainting: Given a signal x = x K + x M ∈ H K ⊕ H M . Recover x , if only x K is known. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50

Underdetermined Situations Separation: Observe a signal x composed of two subsignals x 1 and x 2 : x = x 1 + x 2 . Extract the two subsignals x 1 and x 2 from x , if only x is known. � The two components are geometrically different. Inpainting: Given a signal x = x K + x M ∈ H K ⊕ H M . Recover x , if only x K is known. � The original signal is sparse within a frame. Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 11 / 50

Birth of Separation via Compressed Sensing Composition of Sinusoids and Spikes sampled at n points: � c 0 � x = x 0 1 + x 0 2 = Φ 1 c 0 1 + Φ 2 c 0 1 2 = [ Φ 1 | Φ 2 ] , c 0 2 where x , c 0 1 , and c 0 2 are n × 1. Φ 1 is the n × n -Fourier matrix ((Φ 1 ) t , k = e 2 π itk / n ). Φ 2 is the n × n -Identity matrix. 1 0.5 0 -0.5 -1 0 50 100 150 200 250 Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 12 / 50

Sparsity and ℓ 1 Assumption: Letting A be an n × N -matrix, n << N , the seeked solution c 0 of x = Ac 0 satisfies: i � = 0 } is ‘small’, i.e., c 0 is sparse. � c 0 � 0 = # { i : c 0 Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50

Sparsity and ℓ 1 Assumption: Letting A be an n × N -matrix, n << N , the seeked solution c 0 of x = Ac 0 satisfies: i � = 0 } is ‘small’, i.e., c 0 is sparse. � c 0 � 0 = # { i : c 0 Ideal: Solve... ( P 0 ) min c � c � 0 subject to x = Ac Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50

Sparsity and ℓ 1 Assumption: Letting A be an n × N -matrix, n << N , the seeked solution c 0 of x = Ac 0 satisfies: i � = 0 } is ‘small’, i.e., c 0 is sparse. � c 0 � 0 = # { i : c 0 Ideal: Solve... ( P 0 ) min c � c � 0 subject to x = Ac Basis Pursuit (Chen, Donoho, Saunders; 1998) ( P 1 ) min c � c � 1 subject to x = Ac Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50

Sparsity and ℓ 1 Assumption: Letting A be an n × N -matrix, n << N , the seeked solution c 0 of x = Ac 0 satisfies: i � = 0 } is ‘small’, i.e., c 0 is sparse. � c 0 � 0 = # { i : c 0 Ideal: Solve... ( P 0 ) min c � c � 0 subject to x = Ac Basis Pursuit (Chen, Donoho, Saunders; 1998) ( P 1 ) min c � c � 1 subject to x = Ac Meta-Result: If the solution c 0 is sufficiently sparse, and A is sufficiently incoherent, then c 0 can be recovered from x via ( P 1 ). Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 13 / 50

First Results of Compressed Sensing Composition of Sinusoids and Spikes sampled at n points: � c 0 � x = x 0 1 + x 0 2 = Φ 1 c 0 1 + Φ 2 c 0 1 2 = [ Φ 1 | Φ 2 ] . c 0 2 Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50

First Results of Compressed Sensing Composition of Sinusoids and Spikes sampled at n points: � c 0 � x = x 0 1 + x 0 2 = Φ 1 c 0 1 + Φ 2 c 0 1 2 = [ Φ 1 | Φ 2 ] . c 0 2 Theorem (Donoho, Huo; 2001) 2 ) � 0 < (1 + √ n ) / 2, then If #(Sinusoids) + #(Spikes) = � ( c 0 1 ) � 0 + � ( c 0 ( c 0 1 , c 0 2 ) = argmin( � c 1 � 1 + � c 2 � 1 ) subject to x = Φ 1 c 1 + Φ 2 c 2 . Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50

First Results of Compressed Sensing Composition of Sinusoids and Spikes sampled at n points: � c 0 � x = x 0 1 + x 0 2 = Φ 1 c 0 1 + Φ 2 c 0 1 2 = [ Φ 1 | Φ 2 ] . c 0 2 Theorem (Donoho, Huo; 2001) 2 ) � 0 < (1 + √ n ) / 2, then If #(Sinusoids) + #(Spikes) = � ( c 0 1 ) � 0 + � ( c 0 ( c 0 1 , c 0 2 ) = argmin( � c 1 � 1 + � c 2 � 1 ) subject to x = Φ 1 c 1 + Φ 2 c 2 . Theorem (Bruckstein, Elad; 2002)(Donoho, Elad; 2003) Let A = ( a i ) N i =1 be an n × N -matrix with normalized columns, n << N , and let c 0 satisfy � c 0 � 0 < 1 1 + µ ( A ) − 1 � � , 2 with coherence µ ( A ) = max i � = j |� a i , a j �| . Then c 0 = argmin � c � 1 subject to x = Ac . Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 14 / 50

Birth of Inpainting via Compressed Sensing Main Idea: Let x 0 ∈ H be a signal. Φ be an ONB ( x 0 = Φ c 0 ). H = H M ⊕ H K with orthogonal projections P M and P K . ℓ 1 Minimization Problem (Elad, Starck, Querre, Donoho; 2005): c = argmin � c � 1 subject to P K x 0 = P K Φ c ˆ x = Φˆ ˆ c � Theorem (Donoho, Elad; 2003) 2 (1 + µ ( P K Φ) − 1 ), then x 0 = ˆ ⇒ If � c 0 � 0 < 1 = x . Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 15 / 50

Two Paths Gitta Kutyniok (TU Berlin) Applied Harmonic Analysis meets CS ICERM, Feb. 2014 16 / 50