Finding the Right Exemplars for Reconstructing Single Image - - PowerPoint PPT Presentation

Motivations and Important Issues New Algorithm Details Experimental Results

Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Jiahuan Zhou, Ying Wu September 28, 2016

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Single Image Super-Resolution

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Traditional Single Image Super-Resolution Formulation

Formulation

Y ∗ = arg min

Y

D(X, (Y ∗ K) ↓) + λP(Y ),

◮ X is the low-resolution input; ◮ Y is the high-resoluton output;

Data Fidelity

(1) D(X, (Y ∗ K) ↓) is the fidelity term.

Image Prior

(2) P(Y ) is the image prior term.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Previous Work - Explicit Methods

Explicit Methods

◮

Y ∗ = arg min

Y

D(X, (Y ∗ K) ↓) + λP(Y ),

◮ Focus more on how to design a good image prior.

◮ Smooth-Edge, Dai et al. CVPR2007 ◮ Transform-Invariant, Granda et al. ICCV2013 ◮ Sparse-Coding, Yang et al. TIP2010 ◮ Combination, Villena et al. DSP2013, Villena et al.

DSP2014

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Previous Work - Implicit Methods

Implicit Methods

◮

Y ∗ = arg min

Y

D(X, (Y ∗ K) ↓) + λP(Y ),

◮ Focus more on how to reconstruct the HR images better. ◮ Learning-Based Algorithm:

◮ Freeman et al. ICCV1999 ◮ Pickup et al. NIPS2003 ◮ Kim et al. CVPR2016

◮ Reconstruction-Based Algorithm: a

◮ Chang et al. CVPR2004 ◮ Belekos et al. ITC-CSCC2011 ◮ Yang et al. TIP2012 aImplicit methods are almost patch-based methods, so some integration

schemes e.g., MRF, are needed to integrate all the small patches into one single image output.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Reconstruction from Low-Resolution Exemplars

• 1. Nearest neighbors Li for LR input X;
• 2. Local linear reconstruction from the set of nearest neighbors:

w∗ = arg min

• X −

i∈k

wiLi

• 2

s.t.

i

wi = 1 where w = [w1, ..., wk]T (1)

• 3. Closed-form solution:

w = C−11 1TC−11, where Cij = (X − Li)T(X − Lj) (2)

• 4. Directly transfered HR reconstruction: Y =

i

wiHi.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Reconstruction from Low-Resolution Exemplars

• 1. Nearest neighbors Li for LR input X;
• 2. Local linear reconstruction from the set of nearest neighbors:

w∗ = arg min

• X −

i∈k

wiLi

• 2

s.t.

i

wi = 1 where w = [w1, ..., wk]T (1)

• 3. Closed-form solution:

w = C−11 1TC−11, where Cij = (X − Li)T(X − Lj) (2)

• 4. Directly transfered HR reconstruction: Y =

i

wiHi.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Reconstruction from Low-Resolution Exemplars

• 1. Nearest neighbors Li for LR input X;
• 2. Local linear reconstruction from the set of nearest neighbors:

w∗ = arg min

• X −

i∈k

wiLi

• 2

s.t.

i

wi = 1 where w = [w1, ..., wk]T (1)

• 3. Closed-form solution:

w = C−11 1TC−11, where Cij = (X − Li)T(X − Lj) (2)

• 4. Directly transfered HR reconstruction: Y =

i

wiHi.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Reconstruction from Low-Resolution Exemplars

• 1. Nearest neighbors Li for LR input X;
• 2. Local linear reconstruction from the set of nearest neighbors:

w∗ = arg min

• X −

i∈k

wiLi

• 2

s.t.

i

wi = 1 where w = [w1, ..., wk]T (1)

• 3. Closed-form solution:

w = C−11 1TC−11, where Cij = (X − Li)T(X − Lj) (2)

• 4. Directly transfered HR reconstruction: Y =

i

wiHi.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Outline

Motivations and Important Issues New Algorithm Details Experimental Results

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Outline

Motivations and Important Issues New Algorithm Details Experimental Results

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Two Important Issues in Existing Methods

Issues

• 1. What if the obtained exemplars are not correct?
• 2. How many exemplars should we use?

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Two Important Issues in Existing Methods

Issues

• 1. What if the obtained exemplars are not correct?
• 2. How many exemplars should we use?

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Failure Case I - Wrong Exemplars

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Failure Case II - Wrong Number of Exemplars

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Outline

Motivations and Important Issues New Algorithm Details Experimental Results

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Similarity Structure Refinement

◮ Idea: Project the original LR images into another transformed

space, and expect right exemplars can be found after projection;

◮ A Mahalanobis distance S will be learned to perform the

projection;

◮ In order to learn S, LR-HR patch pairs are needed for learning;

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Pipeline Illustration

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Formulation and Optimization

Pair-wise Affinity Computation

vij = exp

• −DP(hi,hj)

2σhr

• and pij =

vij

• k=i

vik ,

pii = 0, P = [pij] uij = exp

• −(li−lj)T S(li−lj)

2σlr

• and qij =

uij

• k=i

uik ,

qii = 0, Q = [qij] (3)

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Formulation and Optimization

◮ Find the best S =

⇒ the affinity structure in the transformed LR space is as close to that in the HR space as possible. S∗ = arg min

S

• i,j

KL[pij|qij] s.t. S ∈ PSD (4)

◮ Gradient-based optimization.

∇f (S) =

1 2σlr

• ij

(pij − qij) (li − lj) (li − lj)T St+1 ← St − ǫ∇f (St) (5)

◮ In each iteration, project S back to the PSD cone.

S◦ =

• k

max(λk, 0)vkvT

k

(6)

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Formulation and Optimization

◮ Find the best S =

⇒ the affinity structure in the transformed LR space is as close to that in the HR space as possible. S∗ = arg min

S

• i,j

KL[pij|qij] s.t. S ∈ PSD (4)

◮ Gradient-based optimization.

∇f (S) =

1 2σlr

• ij

(pij − qij) (li − lj) (li − lj)T St+1 ← St − ǫ∇f (St) (5)

◮ In each iteration, project S back to the PSD cone.

S◦ =

• k

max(λk, 0)vkvT

k

(6)

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 1st Issue: Formulation and Optimization

◮ Find the best S =

⇒ the affinity structure in the transformed LR space is as close to that in the HR space as possible. S∗ = arg min

S

• i,j

KL[pij|qij] s.t. S ∈ PSD (4)

◮ Gradient-based optimization.

∇f (S) =

1 2σlr

• ij

(pij − qij) (li − lj) (li − lj)T St+1 ← St − ǫ∇f (St) (5)

◮ In each iteration, project S back to the PSD cone.

S◦ =

• k

max(λk, 0)vkvT

k

(6)

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 2nd Issue: Automatic Number Determination

◮ The number of used exemplars influences SR results

significantly:

◮ Too few =

⇒ Insufficient

◮ Too many =

⇒ Noises

◮ Appropriate =

⇒ Hard

◮ Inspired by the common similar structure among the initial

exemplars, utilizing R-PCA technique: min

T,E T∗ + λE0 subj T + E = N

(7)

◮ The appropriate number of exemplars can be determined by

the intrinsic similarity structure and error distribution.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 2nd Issue: Automatic Number Determination

◮ The number of used exemplars influences SR results

significantly:

◮ Too few =

⇒ Insufficient

◮ Too many =

⇒ Noises

◮ Appropriate =

⇒ Hard

◮ Inspired by the common similar structure among the initial

exemplars, utilizing R-PCA technique: min

T,E T∗ + λE0 subj T + E = N

(7)

◮ The appropriate number of exemplars can be determined by

the intrinsic similarity structure and error distribution.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 2nd Issue: Automatic Number Determination

◮ The number of used exemplars influences SR results

significantly:

◮ Too few =

⇒ Insufficient

◮ Too many =

⇒ Noises

◮ Appropriate =

⇒ Hard

◮ Inspired by the common similar structure among the initial

exemplars, utilizing R-PCA technique: min

T,E T∗ + λE0 subj T + E = N

(7)

◮ The appropriate number of exemplars can be determined by

the intrinsic similarity structure and error distribution.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Solve the 2nd Issue: Algorithm Details

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Outline

Motivations and Important Issues New Algorithm Details Experimental Results

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Experiment Setup

◮ The LR-HR exemplar database is collected off-line. ◮ The Berkeley Segmentation Dataset (BSD300), contains

300 natural images from various scenes;

◮ Compared with the State-of-the-arts:

◮ Bi-cubic Interpolation; ◮ Dai’s Softcuts method; ◮ Chang’s neighbor embedding method; ◮ Yang’s sparse coding algorithm; ◮ Dong’s ASDS method; Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Similarity Structure Refinement

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Super-resolution Results with Magnification Factor 2

Figure: SR results on a paddy field image with magnification factor 2.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Super-resolution Results with Magnification Factor 2

Figure: SR results on the Parthenon image with magnification factor 2.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Super-resolution Results with Magnification Factor 2

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Super-resolution Results with Magnification Factor 3

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Super-resolution Results with Magnification Factor 3

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Quantitative Evaluation

Images Parthenon Paddy field Girl BC 2.323 5.347 4.479 NE[CYX04] 2.326 5.395 5.234 Soft[DHX+09] 2.368 5.284 4.634 SP[YWHM10] 2.321 5.255 4.478 AS[DZSW11] 2.317 5.213 4.452 Ours 2.312 5.154 4.344

Table: The RMS errors of SR results by different SR methods W.R.T the ground truth images.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Conclusion and Future Work

Summation

• 1. Learn to re-align the similarity structure in the LR feature

space to make it as close as possible to the similarity structure in the HR space;

• 2. Automatically determine the appropriate number of exemplars

for reconstruction;

Future Work

• 1. How to handle a large-scale exemplar database for learning in

a more efficient and effective way.

• 2. The LR-HR exemplar pairs are tough to obtain in practice, is

it possible to use a large number of non-corresponded exemplars to facilitate the learning.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Conclusion and Future Work

Summation

• 1. Learn to re-align the similarity structure in the LR feature

space to make it as close as possible to the similarity structure in the HR space;

• 2. Automatically determine the appropriate number of exemplars

for reconstruction;

Future Work

• 1. How to handle a large-scale exemplar database for learning in

a more efficient and effective way.

• 2. The LR-HR exemplar pairs are tough to obtain in practice, is

it possible to use a large number of non-corresponded exemplars to facilitate the learning.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

References I

Hong Chang, Dit-Yan Yeung, and Yimin Xiong, Super-resolution through neighbor embedding, Computer Vision and Pattern Recognition, 2004. CVPR

• 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 1,

IEEE, 2004, pp. I–I. Shengyang Dai, Mei Han, Wei Xu, Ying Wu, Yihong Gong, and Aggelos K Katsaggelos, Softcuts: a soft edge smoothness prior for color image super-resolution, Image Processing, IEEE Transactions on 18 (2009), no. 5, 969–981. Weisheng Dong, D Zhang, Guangming Shi, and Xiaolin Wu, Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization, Image Processing, IEEE Transactions on 20 (2011), no. 7, 1838–1857. Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma, Image super-resolution via sparse representation, Image Processing, IEEE Transactions

• n 19 (2010), no. 11, 2861–2873.

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Thanks for your attending ! AQ ?

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution

Motivations and Important Issues New Algorithm Details Experimental Results

What if such a Mahalanobis distance S doesn’t exist?

Solution I - Nonlinear Metric Learning by Kernelization

◮ The exemplar distribution is significantly complicated so that

such a global Mahalanobis metric may not exist;

◮ Take advantage of kernelization, propose to learn a nonlinear

Mahalanobis metric in the projected high-dimensional space;

Solution II - Piece-wise Local Metric Learning

◮ In stead of learning only one global Mahalanobis metric,

learning multiple metrics for different exemplar distribution;

◮ Bu clustering or other technique, group the similar HR

exemplars together;

Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution