Face Hallucination via Face Hallucination via Sparse Coding - - PowerPoint PPT Presentation

Face Hallucination via Face Hallucination via Sparse Coding

Jianchao Yang, Hao Tang, Yi Ma, and Thomas Huang

• Dept. of Electrical & Computer Engineering

University of Illinois at Urbana Champaign University of Illinois at Urbana-Champaign

Outline

• What is Face Hallucination?

What is Face Hallucination?

• Related Works

O F k

• Our Framework

– Localized subspace to capture global structure – Local patch recovery from sparse representation

• Experiment Results

Definition

• Super-Resolution (SR)

Super Resolution (SR)

Low resolution images High resolution images

• Face Hallucination

Super-resolution on human faces

• Constraints

– Reconstruction constraints

Close to input image when blurred and down-sampled

– Global structure constraints – Global structure constraints

Recovered image should be like a face

– Sparsity assumption

Sparse coefficients are preserved during down-sampling

Example

CVPR 2008 Wei Liu etc. CVPR 2005

SR Approaches

• Interpolation-Based Approaches

Interpolation Based Approaches

– Bilinear or Bicubic interpolation Interpolation by combining multiple frames – Interpolation by combining multiple frames – Results are blurred with large zoom factor

• Patch-Based Approaches

– Work on image patches – Use priors on patches for regularization – Better priors are needed

Our Approach

• Novel approach from compressed sensing

Novel approach from compressed sensing

– Sparse representation is preserved during down-sampling down sampling

• Two steps

Recover the global structure using localized – Recover the global structure using localized face subspace obtained from Non-negative Matrix Factorization (NMF) Matrix Factorization (NMF) – Recover the local face details using sparse prior for image patches prior for image patches

Non-negative Matrix Non negative Matrix Factorization

• Object function
• Update rules

p

NMF vs. PCA

• PCA bases are holistic, not intuitive and

hard to explain. p

• NMF bases are part-based, psychological

and physiological evidence and physiological evidence.

• Reconstruction results are sharper using

NMF than using PCA NMF than using PCA.

Recover Global Structure

• Super-resolution Model:
• MAP Estimation:
• MAP Estimation:

Recover Global Structure

• Estimation function:

where is the basis obtained from NMF and where is the basis obtained from NMF, and is a prior term on the desired image

• Intermediate high resolution image is
• Intermediate high-resolution image is

given by

Sparse Representation

• Given the over-complete dictionary

Given the over complete dictionary which is composed of K prototype signals. Assume a signal has a sparse representation: g p p

• In practice we may only observe the low
• In practice, we may only observe the low

resolution version of the signal: Try to recover the sparse solution from .

Recover Face Details

• Suppose we have two dictionaries: and ,

where is the down-sampled version of .

• For each low resolution patch y, we find the

sparse representation with respect to . is the L0 norm, denoting the number of non-zero entries. F is a linear feature extractor.

Recover Face Details

• L1 norm approximation:
• L1 norm approximation:

which can be solved through linear programming. E f i ti l i t

• Enforcing spatial consistency:
• The high-res patch is obtained using

Experiment Results

• Training images: FRGC Ver 1.0, high resolution

Training images: FRGC Ver 1.0, high resolution faces of size 100-by-100, 540 training image pairs in total.

• Dictionary preparation: 10,000 patch pairs (low-

y p p , p p ( res patch and high-res patch) are randomly sampled from the training images.

• Super-resolution is performed by zooming 4

times in both x and y direction.

Experiment Results

F l ft t i ht i t i bi bi i t l ti b k j ti

• From left to right: input image, bicubic interpolation, back projection,

result from NMF, our final result, and the original.

Conclusion

• A novel face hallucination method via sparse

A novel face hallucination method via sparse coding is proposed

– Part-based sparse basis (NMF) – Patch-based sparse representation

• The results reveal the potential successfulness of

sparse coding for face hallucination.

• Larger face database and more accurate

alignment algorithm promise better results.

Affiliations