Motivations and Important Issues New Algorithm Details Experimental Results Finding the Right Exemplars for Reconstructing Single Image Super-Resolution Jiahuan Zhou , Ying Wu September 28, 2016 Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Single Image Super-Resolution Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Traditional Single Image Super-Resolution Formulation Formulation Y ∗ = arg min D ( X , ( Y ∗ K ) ↓ ) + λ P ( Y ) , Y ◮ X is the low-resolution input; ◮ Y is the high-resoluton output; Data Fidelity (1) D ( X , ( Y ∗ K ) ↓ ) is the fidelity term. Image Prior (2) P ( Y ) is the image prior term. Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Previous Work - Explicit Methods Explicit Methods ◮ Y ∗ = arg min D ( X , ( Y ∗ K ) ↓ ) + λ P ( Y ) , Y ◮ Focus more on how to design a good image prior. ◮ Smooth-Edge, Dai et al. CVPR2007 ◮ Transform-Invariant, Granda et al. ICCV2013 ◮ Sparse-Coding, Yang et al. TIP2010 ◮ Combination, Villena et al. DSP2013 , Villena et al. DSP2014 Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Previous Work - Implicit Methods Implicit Methods ◮ Y ∗ = arg min D ( X , ( Y ∗ K ) ↓ ) + λ P ( Y ) , Y ◮ Focus more on how to reconstruct the HR images better. ◮ Learning-Based Algorithm: ◮ Freeman et al. ICCV1999 ◮ Pickup et al. NIPS2003 ◮ Kim et al. CVPR2016 ◮ Reconstruction-Based Algorithm : a ◮ Chang et al. CVPR2004 ◮ Belekos et al. ITC-CSCC2011 ◮ Yang et al. TIP2012 a Implicit methods are almost patch-based methods, so some integration schemes e.g., MRF, are needed to integrate all the small patches into one single image output. Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Reconstruction from Low-Resolution Exemplars 1. Nearest neighbors L i for LR input X ; 2. Local linear reconstruction from the set of nearest neighbors: 2 � � w ∗ = arg min � � � X − � w i L i � � � � (1) i ∈ k � w i = 1 where w = [ w 1 , ..., w k ] T s . t . � i 3. Closed-form solution: C − 1 1 where C ij = ( X − L i ) T ( X − L j ) w = 1 T C − 1 1 , (2) 4. Directly transfered HR reconstruction: Y = � w i H i . i Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Reconstruction from Low-Resolution Exemplars 1. Nearest neighbors L i for LR input X ; 2. Local linear reconstruction from the set of nearest neighbors: 2 � � w ∗ = arg min � � � X − � w i L i � � � � (1) i ∈ k � w i = 1 where w = [ w 1 , ..., w k ] T s . t . � i 3. Closed-form solution: C − 1 1 where C ij = ( X − L i ) T ( X − L j ) w = 1 T C − 1 1 , (2) 4. Directly transfered HR reconstruction: Y = � w i H i . i Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Reconstruction from Low-Resolution Exemplars 1. Nearest neighbors L i for LR input X ; 2. Local linear reconstruction from the set of nearest neighbors: 2 � � w ∗ = arg min � � � X − � w i L i � � � � (1) i ∈ k � w i = 1 where w = [ w 1 , ..., w k ] T s . t . � i 3. Closed-form solution: C − 1 1 where C ij = ( X − L i ) T ( X − L j ) w = 1 T C − 1 1 , (2) 4. Directly transfered HR reconstruction: Y = � w i H i . i Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Reconstruction from Low-Resolution Exemplars 1. Nearest neighbors L i for LR input X ; 2. Local linear reconstruction from the set of nearest neighbors: 2 � � w ∗ = arg min � � � X − � w i L i � � � � (1) i ∈ k � w i = 1 where w = [ w 1 , ..., w k ] T s . t . � i 3. Closed-form solution: C − 1 1 where C ij = ( X − L i ) T ( X − L j ) w = 1 T C − 1 1 , (2) 4. Directly transfered HR reconstruction: Y = � w i H i . i Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Outline Motivations and Important Issues New Algorithm Details Experimental Results Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Outline Motivations and Important Issues New Algorithm Details Experimental Results Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Two Important Issues in Existing Methods Issues 1. What if the obtained exemplars are not correct? 2. How many exemplars should we use? Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Two Important Issues in Existing Methods Issues 1. What if the obtained exemplars are not correct? 2. How many exemplars should we use? Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Failure Case I - Wrong Exemplars Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Failure Case II - Wrong Number of Exemplars Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Outline Motivations and Important Issues New Algorithm Details Experimental Results Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Solve the 1st Issue: Similarity Structure Refinement ◮ Idea: Project the original LR images into another transformed space, and expect right exemplars can be found after projection; ◮ A Mahalanobis distance S will be learned to perform the projection; ◮ In order to learn S , LR-HR patch pairs are needed for learning; Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Solve the 1st Issue: Pipeline Illustration Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Solve the 1st Issue: Formulation and Optimization Pair-wise Affinity Computation � − D P ( h i , h j ) � v ij v ij = exp and p ij = v ik , p ii = 0 , P = [ p ij ] � 2 σ hr k � = i (3) � − ( l i − l j ) T S ( l i − l j ) � u ij u ij = exp and q ij = q ii = 0 , Q = [ q ij ] u ik , 2 σ lr � k � = i Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Solve the 1st Issue: Formulation and Optimization ◮ Find the best S = ⇒ the affinity structure in the transformed LR space is as close to that in the HR space as possible. S ∗ = arg min � KL [ p ij | q ij ] (4) S i , j s . t . S ∈ PSD ◮ Gradient-based optimization. 1 ( p ij − q ij ) ( l i − l j ) ( l i − l j ) T ∇ f ( S ) = � 2 σ lr (5) ij S t − ǫ ∇ f ( S t ) S t +1 ← ◮ In each iteration, project S back to the PSD cone. S ◦ = � max( λ k , 0) v k v T (6) k k Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Motivations and Important Issues New Algorithm Details Experimental Results Solve the 1st Issue: Formulation and Optimization ◮ Find the best S = ⇒ the affinity structure in the transformed LR space is as close to that in the HR space as possible. S ∗ = arg min � KL [ p ij | q ij ] (4) S i , j s . t . S ∈ PSD ◮ Gradient-based optimization. 1 ( p ij − q ij ) ( l i − l j ) ( l i − l j ) T ∇ f ( S ) = � 2 σ lr (5) ij S t − ǫ ∇ f ( S t ) S t +1 ← ◮ In each iteration, project S back to the PSD cone. S ◦ = � max( λ k , 0) v k v T (6) k k Jiahuan Zhou, Ying Wu Finding the Right Exemplars for Reconstructing Single Image Super-Resolution
Recommend
More recommend
