Face Hallucination via Face Hallucination via Sparse Coding - PowerPoint PPT Presentation

Face Hallucination via Face Hallucination via Sparse Coding Jianchao Yang, Hao Tang, Yi Ma, and Thomas Huang Dept. of Electrical & Computer Engineering University of Illinois at Urbana Champaign University of Illinois at Urbana-Champaign

Outline • What is Face Hallucination? What is Face Hallucination? • Related Works • Our Framework O F k – Localized subspace to capture global structure – Local patch recovery from sparse representation • Experiment Results

Definition • Super-Resolution (SR) Super Resolution (SR) Low resolution images High resolution images • Face Hallucination Super-resolution on human faces • Constraints – Reconstruction constraints Close to input image when blurred and down-sampled – Global structure constraints – Global structure constraints Recovered image should be like a face – Sparsity assumption Sparse coefficients are preserved during down-sampling

Example Wei Liu etc. CVPR 2005 CVPR 2008

SR Approaches • Interpolation-Based Approaches Interpolation Based Approaches – Bilinear or Bicubic interpolation – Interpolation by combining multiple frames Interpolation by combining multiple frames – Results are blurred with large zoom factor • Patch-Based Approaches – Work on image patches – Use priors on patches for regularization – Better priors are needed

Our Approach • Novel approach from compressed sensing Novel approach from compressed sensing – Sparse representation is preserved during down-sampling down sampling • Two steps – Recover the global structure using localized Recover the global structure using localized face subspace obtained from Non-negative Matrix Factorization (NMF) Matrix Factorization (NMF) – Recover the local face details using sparse prior for image patches prior for image patches

Non-negative Matrix Non negative Matrix Factorization • Object function • Update rules p

NMF vs. PCA • PCA bases are holistic, not intuitive and hard to explain. p • NMF bases are part-based, psychological and physiological evidence and physiological evidence. • Reconstruction results are sharper using NMF than using PCA NMF than using PCA.

Recover Global Structure • Super-resolution Model: • MAP Estimation: • MAP Estimation:

Recover Global Structure • Estimation function: where where is the basis obtained from NMF, and is the basis obtained from NMF and is a prior term on the desired image • Intermediate high resolution image is • Intermediate high-resolution image is given by

Sparse Representation • Given the over-complete dictionary Given the over complete dictionary which is composed of K prototype signals. Assume a signal has a sparse representation: g p p • In practice we may only observe the low • In practice, we may only observe the low resolution version of the signal: Try to recover the sparse solution from .

Recover Face Details • Suppose we have two dictionaries: and , where is the down-sampled version of . • For each low resolution patch y, we find the sparse representation with respect to . is the L0 norm, denoting the number of non-zero entries. F is a linear feature extractor.

Recover Face Details • L1 norm approximation: • L1 norm approximation: which can be solved through linear programming. • Enforcing spatial consistency: E f i ti l i t • The high-res patch is obtained using

Experiment Results • Training images: FRGC Ver 1.0, high resolution Training images: FRGC Ver 1.0, high resolution faces of size 100-by-100, 540 training image pairs in total. • Dictionary preparation: 10,000 patch pairs (low- y p p , p p ( res patch and high-res patch) are randomly sampled from the training images. • Super-resolution is performed by zooming 4 times in both x and y direction.

Experiment Results • F From left to right: input image, bicubic interpolation, back projection, l ft t i ht i t i bi bi i t l ti b k j ti result from NMF, our final result, and the original.

Conclusion • A novel face hallucination method via sparse A novel face hallucination method via sparse coding is proposed – Part-based sparse basis (NMF) – Patch-based sparse representation • The results reveal the potential successfulness of sparse coding for face hallucination. • Larger face database and more accurate alignment algorithm promise better results .

Affiliations