VALSE Webinar Dual Variational Generation for Data-Limited Face Analysis Yibo Hu JD AI Research Previously, CRIPAC, CASIA https://aberhu.github.io/ 09/23/2020
Face Analysis Face analysis contains a wide range of tasks with various applications. Facial Landmark Face Recognition Face Segmentation Facial Editing Detection Face 3D Reconstruction Facial Makeup Transfer Face Anti-spoofing VALSE Webinar
Contents 1 Variational Autoencoders 2 Dual Variational Generation for HFR 3 Application on Face Parsing 4 Summary VALSE Webinar
Generative Models What are generative models? Generative Model Latent: Z Data: X Deep Generative Models Auto- Invertible regressive Flows (NICE, GAN VAE VAE Others (PixelRNN, RealNVP, PixelCNN) GLOW) VALSE Webinar
Variational Autoencoders Motivation For a generative model, we want to perform: • Sampling : new samples from prior distribution p(x|z) • Inference : expressive representation from visible data p(z|x) • Estimation : find Q from a class of possible modes to best describe an unknown true distribution P • Point-wise likelihood evaluation : calculate p(x) 𝑞 𝑨|𝑦 = 𝑞 𝑦 𝑨 𝑞(𝑨) 𝑞(𝑦) 𝑞 𝑦 = න 𝑞 𝑦 𝑨 𝑞 𝑨 𝑒𝑨 intractable to compute p(x) and p(z|x) VALSE Webinar
Variational Autoencoders Theoretical derivation 𝐁 𝐒 Reconstruction Quality Approximation Error VALSE Webinar
Variational Autoencoders Advantages of VAEs - Capable to perform sampling, inference, estimation and likelihood computation with nice theory support. - There is a clear and recognized way to evaluate the quality of the model (log-likelihood). —— by Yoshua Bengio - Stable training compared with GANs while efficient sampling compared with Auto-regressive models. - Very straightforward to extend to a wide range of model architecture. VALSE Webinar
Contents 1 Variational Autoencoders 2 Dual Variational Generation for HFR 3 Application on Face Parsing 4 Summary VALSE Webinar
What is Heterogeneous Face Recognition (HFR)? Face Recognition Intra-lass distance < inter-class distance Leaning both robust and discriminativefeatures [SphereFace, CVPR2017] VALSE Webinar
What is Heterogeneous Face Recognition (HFR)? Heterogeneous Face Recognition (a) NIR-VIS (b) Thermal-VIS (c) Sketch-Photo From: CASIA NIR-VIS 2.0 Tufts Face IIITD-Sketch NJU-ID MultiPIE (d) ID-Camera (e) Profile-Frontal Photo VALSE Webinar
What are the challenges of HFR? Large Domain Discrepancy Tufts Face Database: + fine-tune w / heterogeneous data + large-scale VIS data Insufficient Heterogeneous Data Tufts Face: 113 IDs with about 10K images CASIA NIR-VIS 2.0: 725 IDs with about 18K images MS-Celeb-1M: 100K IDs with about 10M images How to tackle the challenges ? Collect as Much Data as Possible Reduce Domain Discrepancy VALSE Webinar
How to tackle the challenges? Reduce Domain Discrepancy: Recognition via Generation Conditional Synthesis Generator Input Synthesized Limitations: - Diversity Only synthesize one target image with same attributes Generator Input Synthesized VALSE Webinar
How to tackle the challenges? Reduce Domain Discrepancy: Recognition via Generation Conditional Synthesis Generator Input Synthesized Limitations: - Diversity - Identity Preserving Which identity ? Generator Input Synthesized VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.0.1 Reconstructed 𝜈 𝑂 𝑨 𝑂 𝐹 𝑂 𝜏 𝑂 𝐸 𝑂 Pairwise Identity Preserving 𝑦 𝑂 𝐺 𝑗𝑞 𝜈 𝑊 𝑨 𝑊 𝐹 𝑊 𝜏 𝑊 𝐸 𝑊 𝑦 𝑊 Reconstructed Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.0.2 𝜈 𝑂 𝑨 𝑂 Reconstructed 𝑨 𝐽 𝐹 𝑂 𝜏 𝑂 Pairwise Identity Preserving 𝑦 𝑂 𝐸 𝐽 𝐺 𝑗𝑞 𝜈 𝑊 𝑨 𝑊 𝐹 𝑊 𝜏 𝑊 Concat 𝑦 𝑊 Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.0.2 Generated Pairwise Identity 𝑨 𝑂 Standard Gaussian Noise Preserving z 𝐸 𝐽 Variance 𝐺 𝑗𝑞 𝑨 𝑊 Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.0.2 𝜈 𝑂 𝑨 𝑂 Reconstructed 𝑨 𝐽 𝐹 𝑂 𝜏 𝑂 Pairwise Identity Preserving 𝑦 𝑂 𝐸 𝐽 𝐺 𝑗𝑞 𝜈 𝑊 𝑨 𝑊 𝐹 𝑊 𝜏 𝑊 Concat 𝑦 𝑊 Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.1.0 𝜈 𝑂 𝑨 𝑂 Reconstructed 𝑨 𝐽 𝐹 𝑂 𝜏 𝑂 Pairwise Identity 𝑦 𝑂 Preserving Distribution 𝐸 𝐽 Alignment 𝐺 𝑗𝑞 𝜈 𝑊 𝑨 𝑊 𝐹 𝑊 𝜏 𝑊 Concat 𝑦 𝑊 Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Ƹ Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Framework Evolution v0.1.0 Generated 𝑨 𝐽 Domain Gap Reduction Standard Gaussian Noise z 𝐸 𝐽 HFR Net Copy Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Ƹ Dual Variational Generation Training Stage Testing Stage Generated 𝑨 𝐽 Domain Gap Reduction Standard Gaussian Noise z 𝐸 𝐽 HFR Net Copy Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Generate Pairs of New Images from Noise - Abundantdiversity - Identity consistency of paired images Different poses Paired images with intra-class diversity same identity New images Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation Experimental Results Significant improvements on the low shot datasets Verification rates increase as more generated images are used. 10K- 100K seem enough. VALSE Webinar
Dual Variational Generation Experimental Results NIR-VIS NIR-VIS Thermal-VIS Sketch-Photo Dual Variational Generation for Low-Shot Heterogeneous Face Recognition. NeurIPS 2019 VALSE Webinar
Dual Variational Generation New images Tufts Face: *MS: Mean Similarity *MIS: Mean Instance Similarity Two Challenges of DVG: - Limited inter-class diversity due to the small number of paired training data - The way to use the generated data VALSE Webinar
Dual Variational Generation How to increase inter-class diversity? 100,000 identities included in the MS-Celeb-1M dataset: VALSE Webinar
Dual Variational Generation Framework Evolution v0.1.1 Domain- 𝐹 𝑂 𝜈 𝑂 𝐹 𝑂 specific 𝜏 𝑂 attribute 𝐹 𝑊 𝐻 encoders Pairwise ID 𝐸 Decoder 𝑔 𝐺 𝐸 𝑊 Preserving 𝐺 LightCNN 𝜈 𝑊 𝐹 𝑊 Identity 𝐺 𝑡 𝜏 𝑊 sampler (a) trainingwith paired heterogeneousdata DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition. VALSE Webinar
Dual Variational Generation Framework Evolution v0.1.2 𝜈 𝑂 Domain- 𝐹 𝑂 𝐹 𝑂 specific 𝜏 𝑂 attribute 𝐻 𝐹 𝑊 encoders Pairwise ID 𝐸 Decoder 𝐸 𝑔 𝐺 Preserving 𝑊 𝐺 LightCNN 𝜈 𝑊 𝐹 𝑊 Identity 𝐺 𝜏 𝑊 𝑡 sampler (b) trainingwith unpaired VIS data DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition. VALSE Webinar
Dual Variational Generation Framework Evolution v0.2.0 𝜈 𝑂 Domain- 𝐹 𝑂 𝐹 𝑂 specific 𝜏 𝑂 attribute 𝐻 𝐹 𝑊 encoders ID sampling Pairwise ID 𝐸 ሙ Decoder 𝐸 𝐺 𝑔 Preserving 𝑊 𝑡 𝐺 LightCNN 𝜈 𝑊 𝐹 𝑊 reconstruction loss Identity 𝐺 𝜏 𝑊 𝑡 sampler 𝐺 𝐺 𝐹 𝑔 𝑔 𝑡 𝑊 𝑊 kl loss (b) trainingwith unpaired VIS data DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition. VALSE Webinar
Dual Variational Generation Framework Evolution v0.2.0 sampling 𝐻 ID sampling 𝐸 𝐺 𝑡 sampling (c) sampling after training DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition. VALSE Webinar
Dual Variational Generation Qualitative Results Diversity Measurement Tufts Face: *MS: Mean Similarity *MIS: Mean Instance Similarity VALSE Webinar
Dual Variational Generation How to make better use of the generated data? Contrastive Learning : 1) The generated paired heterogeneousimages are regarded as positive pairs 2) The generated images from different samplings are regarded as negative pairs DVG-Face: Dual Variational Generation for Heterogeneous Face Recognition. VALSE Webinar
Recommend
More recommend
