Data-Limited Face Analysis Yibo Hu JD AI Research Previously, - - PowerPoint PPT Presentation

Dual Variational Generation for Data-Limited Face Analysis

Yibo Hu

VALSE Webinar

JD AI Research Previously, CRIPAC, CASIA

https://aberhu.github.io/

09/23/2020

VALSE Webinar

Face Analysis

Face Recognition Face Segmentation Face Anti-spoofing Face 3D Reconstruction Facial Landmark Detection Facial Makeup Transfer Facial Editing

Face analysis contains a wide range of tasks with various applications.

VALSE Webinar

Contents

2 Dual Variational Generation for HFR 3 Application on Face Parsing 1 Variational Autoencoders 4 Summary

VALSE Webinar

Generative Model

Latent: Z Data: X

Generative Models

What are generative models?

Deep Generative Models

GAN VAE

Auto- regressive (PixelRNN, PixelCNN) Invertible Flows (NICE, RealNVP, GLOW)

Others

VAE

VALSE Webinar

Variational Autoencoders

Motivation

For a generative model, we want to perform:

• Sampling: new samples from prior distribution p(x|z)
• Inference: expressive representation from visible data p(z|x)
• Estimation: find Q from a class of possible modes to best describe

an unknown true distribution P

• Point-wise likelihood evaluation: calculate p(x)

𝑞 𝑨|𝑦 = 𝑞 𝑦 𝑨 𝑞(𝑨) 𝑞(𝑦) 𝑞 𝑦 = න 𝑞 𝑦 𝑨 𝑞 𝑨 𝑒𝑨

intractable to compute p(x) and p(z|x)

VALSE Webinar

Variational Autoencoders

Theoretical derivation

Reconstruction Quality 𝐒 Approximation Error 𝐁

VALSE Webinar

Variational Autoencoders

Advantages of VAEs

• There is a clear and recognized way to evaluate the quality of

the model (log-likelihood). ——by Yoshua Bengio

• Stable training compared with GANs while efficient sampling

compared with Auto-regressive models.

• Very straightforward to extend to a wide range of model

architecture.

• Capable to perform sampling, inference, estimation and

likelihood computation with nice theory support.

VALSE Webinar

Contents

2 Dual Variational Generation for HFR 3 Application on Face Parsing 1 Variational Autoencoders 4 Summary

VALSE Webinar

What is Heterogeneous Face Recognition (HFR)?

Face Recognition

Intra-lass distance < inter-class distance Leaning both robust and discriminativefeatures

[SphereFace, CVPR2017]

VALSE Webinar

What is Heterogeneous Face Recognition (HFR)?

Heterogeneous Face Recognition

(a) NIR-VIS (b) Thermal-VIS (c) Sketch-Photo (d) ID-Camera (e) Profile-Frontal Photo

From: CASIA NIR-VIS 2.0 Tufts Face IIITD-Sketch NJU-ID MultiPIE

VALSE Webinar

MS-Celeb-1M:100K IDs with about 10M images

What are the challenges of HFR?

Large Domain Discrepancy

Tufts Face Database:

Insufficient Heterogeneous Data

CASIA NIR-VIS 2.0: 725 IDs with about 18K images + fine-tune w/ heterogeneous data + large-scale VIS data Tufts Face: 113 IDs with about 10K images

How to tackle the challenges ?

Collect as Much Data as Possible Reduce Domain Discrepancy

VALSE Webinar

How to tackle the challenges?

Reduce Domain Discrepancy: Recognition via Generation

Input Synthesized Conditional Synthesis

Generator

Limitations:

Only synthesize one target image with same attributes

• Diversity

Input Synthesized

Generator

VALSE Webinar

How to tackle the challenges?

Reduce Domain Discrepancy: Recognition via Generation

Input Synthesized Conditional Synthesis

Generator

Limitations:

• Diversity
• Identity Preserving

Input Synthesized

Generator Which identity ?

VALSE Webinar

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

𝐹𝑂 𝑦𝑂

𝑨𝑂 𝑨𝑊 𝜈𝑂 𝜏𝑂 𝜈𝑊 𝜏𝑊

𝐹𝑊 𝐸𝑂

𝐺𝑗𝑞

Pairwise Identity Preserving

𝑦𝑊

Reconstructed

𝐸𝑊

Reconstructed

Framework Evolution v0.0.1

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

VALSE Webinar 𝐹𝑂 𝑦𝑂

𝑨𝑂 𝑨𝑊 𝜈𝑂 𝜏𝑂 𝜈𝑊 𝜏𝑊

𝐹𝑊 𝑦𝑊 𝐸𝐽

𝑨𝐽

Concat

𝐺𝑗𝑞

Pairwise Identity Preserving Reconstructed

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Framework Evolution v0.0.2

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

VALSE Webinar

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Framework Evolution v0.0.2

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

𝑨𝑂 𝑨𝑊 𝐸𝐽

Generated

Standard Gaussian Noise

z

𝐺𝑗𝑞

Pairwise Identity Preserving

Variance

VALSE Webinar 𝐹𝑂 𝑦𝑂

𝑨𝑂 𝑨𝑊 𝜈𝑂 𝜏𝑂 𝜈𝑊 𝜏𝑊

𝐹𝑊 𝑦𝑊 𝐸𝐽

𝑨𝐽

Concat

𝐺𝑗𝑞

Pairwise Identity Preserving Reconstructed

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Framework Evolution v0.0.2

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

VALSE Webinar 𝐹𝑂 𝑦𝑂

𝑨𝑂 𝑨𝑊

Distribution Alignment

𝜈𝑂 𝜏𝑂 𝜈𝑊 𝜏𝑊

𝐹𝑊 𝐸𝐽

𝑨𝐽

Concat

𝐺𝑗𝑞

Pairwise Identity Preserving

𝑦𝑊

Reconstructed

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Framework Evolution v0.1.0

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

VALSE Webinar

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Framework Evolution v0.1.0

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

𝐸𝐽

Copy

Standard Gaussian Noise

Ƹ 𝑨𝐽

HFR Net Domain Gap Reduction

z

Generated

VALSE Webinar 𝐸𝐽

Copy

Standard Gaussian Noise

Ƹ 𝑨𝐽

HFR Net Domain Gap Reduction

z

Generated

Dual Variational Generation

Training Stage Testing Stage

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

VALSE Webinar

Dual Variational Generation

Generate Pairs of New Images from Noise

• Abundantdiversity
• Identity consistency of paired images

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

New images Different poses intra-class diversity Paired images with same identity

VALSE Webinar

Dual Variational Generation

Experimental Results

• Significant improvements on the low shot datasets
• Verification rates increase as more generated images are used. 10K-

100K seem enough.

VALSE Webinar

Dual Variational Generation

Experimental Results NIR-VIS NIR-VIS

Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019

Thermal-VIS Sketch-Photo

VALSE Webinar

Dual Variational Generation

New images

Two Challenges of DVG:

Tufts Face:

*MS: Mean Similarity *MIS: Mean Instance Similarity

• Limited inter-class diversity due to the small number of

paired training data

• The way to use the generated data

VALSE Webinar

Dual Variational Generation

How to increase inter-class diversity?

100,000 identities included in the MS-Celeb-1M dataset:

VALSE Webinar

𝐹𝑊

𝜏𝑊 𝜈𝑊

𝐹𝑊 𝐹𝑂 𝐸 Pairwise ID Preserving

𝜈𝑂 𝜏𝑂 𝑔

𝑊

𝐺 𝐻 𝐹𝑂

Domain- specific attribute encoders LightCNN Identity sampler Decoder

𝐸 𝐺 𝐺

𝑡

Dual Variational Generation

Framework Evolution v0.1.1

(a) trainingwith paired heterogeneousdata

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition.

VALSE Webinar

𝜈𝑂

𝐸 Pairwise ID Preserving 𝐻

𝜏𝑊 𝜈𝑊

𝐹𝑊 𝐹𝑂

𝜏𝑂 𝑔

𝑊

𝐺

(b) trainingwith unpaired VIS data

𝐹𝑊 𝐹𝑂

Domain- specific attribute encoders LightCNN Identity sampler Decoder

𝐸 𝐺 𝐺

𝑡

Dual Variational Generation

Framework Evolution v0.1.2

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition.

VALSE Webinar (b) trainingwith unpaired VIS data

Dual Variational Generation

Framework Evolution v0.2.0

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition.

𝐺

𝑡

ID sampling

ሙ 𝑔

𝑊

𝜈𝑂

𝐸 Pairwise ID Preserving 𝐻

𝜏𝑊 𝜈𝑊

𝐹𝑊 𝐹𝑂

𝜏𝑂

𝐺 𝐹𝑊 𝐹𝑂

Domain- specific attribute encoders LightCNN Identity sampler Decoder

𝐸 𝐺 𝐺

𝑡

kl loss

𝑔

𝑊

𝐺

𝑡

𝐹

𝑔

𝑊

reconstruction loss

VALSE Webinar

Dual Variational Generation

Framework Evolution v0.2.0

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition.

𝐺

𝑡

ID sampling

𝐸 𝐻

(c) sampling after training

sampling sampling

VALSE Webinar

Dual Variational Generation

Qualitative Results Diversity Measurement

Tufts Face:

*MS: Mean Similarity *MIS: Mean Instance Similarity

VALSE Webinar

Dual Variational Generation

How to make better use of the generated data?

DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition.

Contrastive Learning: 1) The generated paired heterogeneousimages are regarded as positive pairs 2) The generated images from different samplings are regarded as negative pairs

VALSE Webinar

Quantitative Experiments

Dual Variational Generation

VALSE Webinar

Contents

2 Dual Variational Generation for HFR 3 Application on Face Parsing 1 Variational Autoencoders 4 Summary

VALSE Webinar

Application on Face Parsing

a) Semantic, Instance, panoptic segmentation b) Face parsing c) Human parsing Pixel-level Semantic Understanding Appearance Structure Consistency

VALSE Webinar

Application on Face Parsing

Dual-Structure Disentangling Variational Generation (D2VG)

Dual-Structure Disentangling Variational Generation for Data-Limited Face Parsing. ACM MM 2020 .

Appearance Structure Structure Concatenate Hadamard product

(a) Training pipeline of D2VG

VALSE Webinar

Application on Face Parsing

Dual-Structure Disentangling Variational Generation (D2VG)

Dual-Structure Disentangling Variational Generation for Data-Limited Face Parsing. ACM MM 2020 .

(b) Synthesizing large-scale samples for parsing

VALSE Webinar

Contents

2 Dual Variational Generation for HFR 3 Application on Face Parsing 1 Variational Autoencoders 4 Summary

VALSE Webinar

Summary

• VAE is one of popular generative models that is capable to

perform sampling, inference, estimation and likelihood computation with nice theory support.

• Dual Variational Generation (DVG) is a general framework to

model joint probability distribution of two modalities.

• The generated samples from DVG can be treated as extra data

to boost some data-limited face tasks.

• Future work: other face tasks, more modalities, more diverse,

body or natural image ……

VALSE Webinar

Thanks for Your Attention!!!

Yibo Hu

Email: huyibo871079699@gmail.com