Outlier-aware Dictionary Learning for Sparse Representation Sajjad - - PowerPoint PPT Presentation

Outlier-aware Dictionary Learning for Sparse Representation

Sajjad Amini∗ Mostafa Sadeghi∗ Mohsen Joneidi∗ Massoud Babaie-Zadeh∗ Christian Jutten∗∗

∗Department of Electrical Engineering, Sharif University of Technology, Tehran, Iran. ∗∗GIPSA-Lab, Grenoble, and Institut Universitaire de France, France.

September 2014

Sparse Representation

Underdetermind Linear System of Equations y = Dx + e

✑ y ∈ Rn, D ∈ Rn×K, x ∈ RK, e ∈ Rn ✑ D = [d1 d2 . . . dK]: dictionary, di: atom ✑ The dictionary is usually overcomplete: K > n

Sparse representation problem: x∗ = argmin

x

x0 subject to y − Dx2 ≤ ǫ

2

Choosing the Dictionary

Pre-defined and fixed dictionaries: Fourier, Gabor, DCT, wavelet, . . .

✦ Fast computations ✪ Unable to sparsely represent a given signal class

Learned dictionaries

✦ More efficient for sparse representation ✦ Very promising results in many applications: image enhancement, pattern recognition, . . . ✪ High computational load 3

Dictionary Learning (DL)

. . . . . . . . .

Noisy Sample

Y

Given a noisy training data matrix, Y = [y1, . . . , yL], the goal is to find an

• ver-complete set of basis functions (atoms) over which each data can be

sparsely represented Training Data Model yi = Dxi + ni i = 1, . . . , L ✑ p(x) ∝ exp(−x1

β1 )

✑ p(n) ∝ exp(−n2

2

β2 ) 4

Dictionary Learning (DL)

MAP Estimation of Dictionary and Representations minD,{xi}L

i=1

• i yi − Dxi2

2

subject to di2 = 1, i = 1, . . . , K xj0 ≤ T0, j = 1, . . . , L x0 |supp(x) = {i : xi = 0} | Solution to the Dictionary Learning Problem Alternating Minimization Starting with an initial dictionary, the following two stages are repeated several times:

1 Sparse representation:

X(k+1) = argminX∈X Y − D(k)X2

F subject to xj0 ≤ T0, j = 1, . . . , L =

⇒ OMP

2 Dictionary update:

D(k+1) = argmin

D∈D

Y−DX(k+1)2

F subject to di2 = 1, i = 1, . . . , K =

⇒ Differentiating 5

Image Denoising Using Dictionary Learning

Convert Image Patch to Vector

Image Y

1 4 7 2 3 5 6 8 9 1 2 3 4 5 6 7 8 9

Noisy Pixel

Image Denoising Training Image Dictionary (D⋆) Estimating Denoised Image Patch Representation

✑ ˆ x = OMPD⋆(y)

Reconstructing Denoised Image Patch

✑ ˆ y = D⋆ˆ x 6

Robust Dictionary Learning

. . . . . . . . .

Y

Noiseless Sample Outlier

Problem Formulation Training Data Model: yi = Dxi + ni i = 1, . . . , L

✑ p(x) ∝ exp(−x1

β1 )

✑ p(n) ∝ exp(−n1

β2 )

MAP Estimation of Dictionary and Representations minD,{xi}L

i=1

• i(yi − Dxi1 + λxi1) subject to di2 = 1, i = 1, . . . , K

✑ Strategy to Solve= ⇒ w.r.t. D ⇒ Iteratively Reweighted Least Squares w.r.t. {xi}L

i=1 ⇒ Iteratively Reweighted Least Squares

7

Robust Dictionary Learning by Error Source Decomposition

. . . . . . . . .

Noisy Sample

Y

Outlier

Problem Formulation Training Data Model: yi = Dxi + ni + oi i = 1, . . . , L

✑ p(x) ∝ exp(−x1

β1 )

✑ p(n) ∝ exp(−n2

2

β2 )

✑ p(o) ∝ exp(−o1

β3 )

MAP Estimation of Dictionary and Representations

minD,{xi}L

i=1,{oi}L i=1

• i(yi−Dxi−oi2

2+λoi1) subject to

di2 = 1, i = 1, . . . , K xj0 ≤ T0, j = 1, . . . , L ✑ Strategy to Solve = ⇒ w.r.t. D ⇒ Differentiating w.r.t. {xi}L

i=1, {oi}L i=1 ⇒ Shrinkage

8

Outlier Aware Dictionary Learning (Proposed)

. . . . . . . . .

Noisy Sample

Y

Outlier

Problem Formulation Training Data Model: yi = Dxi + ni + oi i = 1, . . . , L

✑ p(x) ∝ exp(−x1

β1 )

✑ p(n) ∝ exp(−n2

2

β2 )

✑ p(o) ∝ exp(−o2

β3 )

MAP Estimation of Dictionary and Representations

minD,{xi}L

i=1,{oi}L i=1

• i(yi−Dxi−oi2

2+λoi2) subject to

di2 = 1, i = 1, . . . , K xj0 ≤ T0, j = 1, . . . , L minD,X,O Y − DXi − O2

F + λO21 subject to

di2 = 1, i = 1, . . . , K xj0 ≤ T0, j = 1, . . . , L 9

Outlier Aware Dictionary Learning (Proposed)

Solution Strategy Alternating Minimization:

✑ w.r.t. {xi}L

i=1 ⇒ xi = OMPD(yi − oi), i = 1, . . . , L

✑ w.r.t. {oi}L

i=1 ⇒ oi =

(1 −

λ 2ri2)ri,

if ri2 > λ

2

0,

• therwise

ri = yi − Dxi, i = 1, . . . , L ✑ w.r.t. D ⇒ D = (Y − O)XT(XXT)−1

Initialization

✑ D ⇒ Overcomplete DCT ✑ {oi}L

i=1 ⇒ oi = 0, i = 1, . . . , L

At the beginning, all training signal are considered not to be an outlier.

10

Image Denoising Based on OADL

Convert Image Patch to Vector

Image Y

4 5 1 7 2 3 6 8 9 3 4 7 1 2 5 6 8 9

Noisy Pixel Outlier

Image Denoising Training Image Dictionary Using OADL(D⋆) Estimating Denoised Image Patch Representation

✑ (ˆ x, ˆ

• ) = Alternating Minimization

minx x0 subject to (y − o) − Dx2 ≤ ǫ mino (y − Dx) − o2

2 + o2

Reconstructing Denoised Image Patch

✑ ˆ y = D⋆ˆ x 11

Simulation Results

Synthetic Data

Generate a random dictionary (D) Generate 2500 training signals and 500 test signals using 3 atoms of D Add Gaussian noise (N(0, 0.012I)) to each of training signals Add Gaussian noise (N(0, 0.042I)) to p% of randomly selected training signals (outlier) Train a dictionary using training signals (D⋆) Evaluate the ability of dictionary to code test signals using 3 atoms of D⋆

Image Denoising

Select 6 benchmark images (256 × 256) Add Gaussian noise (N(0, 102)) to each of image pixels Add Gaussian noise (N(0, 202)) to B blocks of pixels according to the following pattern Denoise the resultant image

1 4 5 6 9 8 7 2 3

Figure: Outlier block pattern 12

Simulation Results - Synthetic Data

20 40 60 80 0.05 0.1 0.15 0.2 Iteration Representation RMSE MOD OADL

(a) p = 2%

20 40 60 80 0.05 0.1 0.15 0.2 Iteration Representation RMSE MOD OADL

(b) p = 6%

20 40 60 80 0.05 0.1 0.15 0.2 Iteration Representation RMSE MOD OADL

(c) p = 12%

2 4 6 8 10 12 2 4 6 8x 10

• 3

Percentage of corrupted samples ∆RMSE

(d) Final RMSE difference Figure: (a)-(c) Test data representation RMSE along iterations for p = 2, 4 and 6, respctively. (d) Final

representation RMSE of test data versus p

13

Simulation Results - Synthetic Data

2 4 6 8 10 12 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Percentage of corrupted samples λopt Figure: Best λ, which minimizes test data representation RMSE, versus percentage of outliers. p = 2% to 6% p directly related to β3 λ inversely related to β3 ⇒ p inversely related to λ p = 6% to 12% Violation of outlier sparsity assumption OADL approaches regular Dictionary Learning ⇒ p directly related to λ 14

Simulation Results - Image Denoising

2 4 6 8 30 31 32 33 34 Number of outlier blocks PSNR (dB) OADL MOD DCT

Figure: Averaged PSNR over 6 different test images versus number of outlier blocks.

2 4 6 8 70 75 80 85 90 95 100 Number of outlier blocks λopt

Figure: Best λ, which maximizes PSNR, versus percentage of outliers. 15

Conclusions

A new and practical placement of outlier was considered. We introduce a new model for training signals based on separating noise and

• utlier source.

We formulate DL problem using MAP estimation. We introduce a fast and efficient algorithm to solve the proposed robust dictionary learning problem. Simulation results showed that our new method leads to considerable improvements over traditional methods

16

References

1 K. Engan, S. O. Aase, and J. Hakon Husoy, “Method of optimal directions for

frame design,” IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 5, pp. 2443–2446, 1999.

2 M. Aharon, M. Elad, and A. Bruckstein, “K-SVD: An algorithm for designing

• vercomplete dictionaries for sparse representation,” IEEE Trans. on Signal

Processing, vol. 54, no. 11, pp. 4311–4322, 2006.

3 M. Elad and M. Aharon, “Image Denoising Via Sparse and Redundant

representations over Learned Dictionaries,” IEEE Trans. on Image Processing,

• vol. 15, no. 12, pp. 3736-3745, December 2006.

4 C. Lu, J. Shi, and J. Jia, “Online robust dictionary learning,” IEEE Conference

• n Computer Vision and Pattern Recognition (CVPR), pp. 415–422. June

2013.

5 Z. Chen, and Y. Wu, “Robust Dictionary Learning by Error Source

Decomposition,” IEEE International Conference on Computer Vision (ICCV),

• pp. 2216–2223, December 2013.

17