SLIDE 1 Learning a model of facial shape and expression from 4D scans
Tianye Li*, Timo Bolkart*, Michael J. Black, Hao Li, Javier Romero SIGGRAPH Asia 2017
Note: this slide is a static .pdf version (no video) For video, please see: https://youtu.be/36rPTkhiJTM
SLIDE 2 Realistic Virtual Character
Warner Bros. & Paramount Pictures
SLIDE 3 Realistic Virtual Character
Warner Bros. & Paramount Pictures
SLIDE 4 Consumer Application
Apple 2017
SLIDE 5
Spectrum of Face Models
“Low-end” “High-end”
SLIDE 6 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes
SLIDE 7 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes Blanz and Vetter 1999 & Basel Face Model [Paysan et al. 2009]
SLIDE 8 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes Blanz and Vetter 1999 FaceWarehouse [Cao et al. 2014] & Basel Face Model [Paysan et al. 2009]
SLIDE 9 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes Blanz and Vetter 1999 FaceWarehouse [Cao et al. 2014] Wu et al. 2016 & Basel Face Model [Paysan et al. 2009]
SLIDE 10 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes Blanz and Vetter 1999 FaceWarehouse [Cao et al. 2014] Wu et al. 2016 Digital Emily [Alexander et al. 2009] & Basel Face Model [Paysan et al. 2009]
SLIDE 11 Spectrum of Face Models
“Low-end” “High-end”
FACS-based blendshapes Blanz and Vetter 1999 FaceWarehouse [Cao et al. 2014] Wu et al. 2016 Digital Emily [Alexander et al. 2009] & Basel Face Model [Paysan et al. 2009]
SLIDE 12
FLAME Face Model
Issues FLAME
SLIDE 13
FLAME Face Model
Limited identity coverage Learned from ~4000 identities Issues FLAME
SLIDE 14
FLAME Face Model
Limited identity coverage Learned from ~4000 identities Issues FLAME Artist designed expression Learned from high-quality 4D expression scans
SLIDE 15
FLAME Face Model
Limited identity coverage Learned from ~4000 identities Issues FLAME Artist designed expression Learned from high-quality 4D expression scans Over-complete FACS basis Orthogonal expression space
SLIDE 16
FLAME Face Model
Limited identity coverage Learned from ~4000 identities Issues FLAME Artist designed expression Learned from high-quality 4D expression scans Over-complete FACS basis Orthogonal expression space Non-linearity of jaw and neck Modeled as rotatable joints Activated by linear blend skinning Pose blendshapes further capture details
SLIDE 17
FLAME Face Model
Limited identity coverage Learned from ~4000 identities Issues FLAME Artist designed expression Learned from high-quality 4D expression scans Over-complete FACS basis Orthogonal expression space Non-linearity of jaw and neck Modeled as rotatable joints Activated by linear blend skinning Pose blendshapes further capture details Require artist work Fully automatic registration and modeling
SLIDE 18
FLAME Face Model
SLIDE 19
FLAME Face Model
Template
SLIDE 20
FLAME Face Model
Template Shape
SLIDE 21
FLAME Face Model
Template Shape Shape +Pose
SLIDE 22
FLAME Face Model
Template Shape Shape + Pose + Expression Shape +Pose
SLIDE 23 Overview
Co-registration Hirshberg et al. 12
CAESAR dataset Shape Data Registration Shape Model Training MPI FacialMotion dataset Pose Data Registration Pose Model Training Expression Data Registration Expression Model Training D3DFACS dataset Initial Expression Blendshapes
SLIDE 24 Overview
Co-registration Hirshberg et al. 12
CAESAR dataset Shape Data Registration Shape Model Training MPI FacialMotion dataset Pose Data Registration Pose Model Training Expression Data Registration Expression Model Training D3DFACS dataset Initial Expression Blendshapes
SLIDE 25
Shape Model
SLIDE 26
Shape Data
Registration of CAESAR datasets [Robinette et al. 2002]
SLIDE 27
Learned Shape Model
SLIDE 28
Pose Model
SLIDE 29
Pose Data
Registration of MPI FacialMotion datasets for pose training
SLIDE 30
Learned Pose Model
SLIDE 31
Effect of Pose Blendshapes
SLIDE 32
Expression Model
SLIDE 33
Expression Data
SLIDE 34
Expression Data
SLIDE 35
4D Scans into Correspondence
SLIDE 36 Coarse-to-Fine Registration
>1 mm 0 mm
Stage 1: model-only
SLIDE 37 Coarse-to-Fine Registration
>1 mm 0 mm
Stage2: coupled
SLIDE 38 Coarse-to-Fine Registration
>1 mm 0 mm
Stage 3: Texture-based Alignment
SLIDE 39
Effect of Texture-based Alignment
SLIDE 40
Effect of Texture-based Alignment
SLIDE 41 Registration Results
>1 mm 0 mm
SLIDE 42 Registration Results
>1 mm 0 mm
Detail expressions such as eye blinking are also captured
SLIDE 43
Learned Expression Model
SLIDE 44
Results
SLIDE 45 Compare on Identity Space
FaceWarehouse [Cao et al. 2014] 50 components FLAME 49 49 components + 1 for gender
0 mm >1 mm
BU-3DFE scan Fitting Scan-to-Mesh Distance Fitting Scan-to-Mesh Distance
SLIDE 46 Compare on Identity Space
FLAME 198 198 components + 1 for gender
0 mm >1 mm
BU-3DFE scan Fitting Scan-to-Mesh Distance Fitting Scan-to-Mesh Distance Basel Face Model (BFM) [Paysan et al. 2009] 199 components
SLIDE 47 Compare on Identity Space
FLAME 198 198 components + 1 for gender
0 mm >1 mm
BU-3DFE scan Scan-to-Mesh Distance Scan-to-Mesh Distance Basel Face Model (BFM) 199 components BU-3DFE scan
SLIDE 48 Compare on Identity Space
0.5 1 1.5 2
Error [mm]
20 40 60 80 100
Percentage
FLAME 300 FLAME 198 FLAME 90 FLAME 49 FW BFM Full BFM 91 BFM 50
SLIDE 49 Compare on Identity Space
FW 50: 67% BFM 50: 69% FLAME 49: 74% BFM 199: 92% FLAME 198: 94% FLAME 300: 96%
FLAME: our model BFM: Basel Face Model FW FaceWarehouse Note: higher value is better
SLIDE 50 Compare on Expression Space
>3 mm 0 mm
SLIDE 51 Sparse Landmark Fitting
FLAME produces better result in 2D landmark fitting
SLIDE 52 Application: Retargeting
FLAME retargeting pipeline
FLAME Face Model
Source Retargeted
Target Scan Expression & pose Identity
SLIDE 53
Application: Retargeting
SLIDE 54
Conclusion
SLIDE 55 What did we learn
- Large high-quality data
- Separation of identity, pose and expression
- Importance of face prior
- Model and data available for research purpose
SLIDE 56
Future Work
SLIDE 57 Acknowledgement
Tsvetelina Alexiadis, Andrea Keller, Jorge Márquez Data Acquisition Shunsuke Saito & Cassidy Laidlaw Evaluation Yinghao Huang, Ahmed Osman, Naureen Mahmood Discussion Talha Zaman Video Recording Alejandra Quiros-Ramirez Project Website Darren Cosker Advice and D3DFACS Dataset
SLIDE 58
Thank You! http://flame.is.tue.mpg.de/
Registrations for D3DFACS dataset FLAME face model (male / female) with example code
