Sparse Coding and Dictionary Learning for Image Analysis Part II: - PowerPoint PPT Presentation

Sparse Coding and Dictionary Learning for Image Analysis Part II: Dictionary Learning for signal reconstruction Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro ICCV’09 tutorial, Kyoto, 28th September 2009 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 1/43

What this part is about The learning of compact representations of images adapted to restoration tasks. A fast online algorithm for learning dictionaries and factorizing matrices in general. Various formulations for image and video processing. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 2/43

The Image Denoising Problem y x orig + w = ���� ���� ���� noise measurements original image Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 3/43

Sparse representations for image restoration y = x orig + w ���� ���� ���� noise measurements original image Energy minimization problem - MAP estimation || y − x || 2 E ( x ) = + Pr ( x ) 2 � �� � � �� � relation to measurements p rior Some classical priors Smoothness λ ||L x || 2 2 Total variation λ ||∇ x || 2 1 Wavelet sparsity λ || Wx || 1 . . . Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 4/43

Sparse representations for image restoration Sparsity and redundancy Pr ( x ) = λ || α || 0 for x ≈ D α   α [1]      α [2]   d 1 d 2 · · · d p    x = .    .  .   α [ p ] � �� � � �� � x ∈ R m D ∈ R m × p � �� � α ∈ R p , sparse Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 5/43

Sparse representations for image restoration Designed dictionaries [Haar, 1910], [Zweig, Morlet, Grossman ∼ 70s], [Meyer, Mallat, Daubechies, Coifman, Donoho, Candes ∼ 80s-today]. . . (see [Mallat, 1999]) Wavelets, Curvelets, Wedgelets, Bandlets, . . . lets Learned dictionaries of patches [Olshausen and Field, 1997], [Engan et al., 1999], [Lewicki and Sejnowski, 2000], [Aharon et al., 2006] , [Roth and Black, 2005], [Lee et al., 2007] 1 � 2 || x i − D α i || 2 min + λψ ( α i ) 2 α i , D ∈C � �� � i � �� � sparsity reconstruction ψ ( α ) = || α || 0 (“ ℓ 0 pseudo-norm”) ψ ( α ) = || α || 1 ( ℓ 1 norm) Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 6/43

Sparse representations for image restoration Solving the denoising problem [Elad and Aharon, 2006] Extract all overlapping 8 × 8 patches y i . Solve a matrix factorization problem: n 1 � 2 || y i − D α i || 2 min + λψ ( α i ) , 2 α i , D ∈C � �� � i =1 � �� � sparsity reconstruction with n > 100 , 000 Average the reconstruction of each patch. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 7/43

Sparse representations for image restoration K-SVD: [Elad and Aharon, 2006] Figure: Dictionary trained on a noisy version of the image boat. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 8/43

Sparse representations for image restoration Inpainting, Demosaicking 1 � 2 || β i ⊗ ( y i − D α i ) || 2 min 2 + λ i ψ ( α i ) D ∈C , α i RAW Image Processing (see our poster) White balance. Denoising Black substraction. Conversion to sRGB. Demosaicking Gamma correction. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 9/43

Sparse representations for image restoration [Mairal, Bach, Ponce, Sapiro, and Zisserman, 2009c] Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 10/43

Sparse representations for image restoration [Mairal, Sapiro, and Elad, 2008b] Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 11/43

Sparse representations for image restoration Inpainting, [Mairal, Elad, and Sapiro, 2008a] Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 12/43

Sparse representations for image restoration Inpainting, [Mairal, Elad, and Sapiro, 2008a] Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 13/43

Sparse representations for video restoration Key ideas for video processing [Protter and Elad, 2009] Using a 3D dictionary. Processing of many frames at the same time. Dictionary propagation. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 14/43

Sparse representations for image restoration Inpainting, [Mairal, Sapiro, and Elad, 2008b] Figure: Inpainting results. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 15/43

Sparse representations for image restoration Inpainting, [Mairal, Sapiro, and Elad, 2008b] Figure: Inpainting results. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 16/43

Sparse representations for image restoration Inpainting, [Mairal, Sapiro, and Elad, 2008b] Figure: Inpainting results. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 17/43

Sparse representations for image restoration Inpainting, [Mairal, Sapiro, and Elad, 2008b] Figure: Inpainting results. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 18/43

Sparse representations for image restoration Inpainting, [Mairal, Sapiro, and Elad, 2008b] Figure: Inpainting results. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 19/43

Sparse representations for image restoration Color video denoising, [Mairal, Sapiro, and Elad, 2008b] Figure: Denoising results. σ = 25 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 20/43

Sparse representations for image restoration Color video denoising, [Mairal, Sapiro, and Elad, 2008b] Figure: Denoising results. σ = 25 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 21/43

Sparse representations for image restoration Color video denoising, [Mairal, Sapiro, and Elad, 2008b] Figure: Denoising results. σ = 25 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 22/43

Sparse representations for image restoration Color video denoising, [Mairal, Sapiro, and Elad, 2008b] Figure: Denoising results. σ = 25 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 23/43

Sparse representations for image restoration Color video denoising, [Mairal, Sapiro, and Elad, 2008b] Figure: Denoising results. σ = 25 Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 24/43

Optimization for Dictionary Learning n 1 � 2 || x i − D α i || 2 min 2 + λ || α i || 1 α ∈ R p × n i =1 D ∈C = { D ∈ R m × p s.t. ∀ j = 1 , . . . , p , △ C || d j || 2 ≤ 1 } . Classical optimization alternates between D and α . Good results, but very slow! Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 25/43

Optimization for Dictionary Learning [Mairal, Bach, Ponce, and Sapiro, 2009a] Classical formulation of dictionary learning n 1 � D ∈C f n ( D ) = min min l ( x i , D ) , n D ∈C i =1 where 1 l ( x , D ) △ 2 || x − D α || 2 = min 2 + λ || α || 1 . α ∈ R p Which formulation are we interested in? n � � 1 � min f ( D ) = E x [ l ( x , D )] ≈ lim l ( x i , D ) n n → + ∞ D ∈C i =1 [Bottou and Bousquet, 2008]: Online learning can handle potentially infinite or dynamic datasets, be dramatically faster than batch algorithms. Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 26/43

Optimization for Dictionary Learning Require: D 0 ∈ R m × p (initial dictionary); λ ∈ R 1: A 0 = 0, B 0 = 0. 2: for t=1,. . . ,T do Draw x t 3: Sparse Coding 4: 1 2 || x t − D t − 1 α || 2 α t ← arg min 2 + λ || α || 1 , α ∈ R p Aggregate sufficient statistics 5: A t ← A t − 1 + α t α T t , B t ← B t − 1 + x t α T t Dictionary Update (block-coordinate descent) 6: t � 1 � 1 � 2 || x i − D α i || 2 D t ← arg min 2 + λ || α i || 1 . t D ∈C i =1 7: end for Francis Bach, Julien Mairal, Jean Ponce and Guillermo Sapiro Dictionary Learning for signal reconstruction 27/43