Learning a model of facial shape and expression from 4D scans - PowerPoint PPT Presentation

Learning a model of facial shape and expression from 4D scans Tianye Li*, Timo Bolkart*, Michael J. Black, Hao Li, Javier Romero SIGGRAPH Asia 2017 Note: this slide is a static .pdf version (no video) For video, please see: https://youtu.be/36rPTkhiJTM

Realistic Virtual Character Warner Bros. & Paramount Pictures

Realistic Virtual Character Warner Bros. & Paramount Pictures

Consumer Application Apple 2017

Spectrum of Face Models “Low-end” “High-end”

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes Blanz and Vetter 1999 & Basel Face Model [Paysan et al. 2009]

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes Blanz and Vetter 1999 & Basel Face Model FaceWarehouse [Paysan et al. 2009] [Cao et al. 2014]

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes Wu et al. 2016 Blanz and Vetter 1999 & Basel Face Model FaceWarehouse [Paysan et al. 2009] [Cao et al. 2014]

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes Wu et al. 2016 Blanz and Vetter 1999 Digital Emily & Basel Face Model FaceWarehouse [Alexander et al. 2009] [Paysan et al. 2009] [Cao et al. 2014]

Spectrum of Face Models “Low-end” “High-end” FACS-based blendshapes Wu et al. 2016 Blanz and Vetter 1999 Digital Emily & Basel Face Model FaceWarehouse [Alexander et al. 2009] [Paysan et al. 2009] [Cao et al. 2014]

FLAME Face Model Issues FLAME

FLAME Face Model Issues FLAME Limited identity coverage Learned from ~4000 identities

FLAME Face Model Issues FLAME Limited identity coverage Learned from ~4000 identities Learned from high-quality 4D expression scans Artist designed expression

FLAME Face Model Issues FLAME Limited identity coverage Learned from ~4000 identities Learned from high-quality 4D expression scans Artist designed expression Orthogonal expression space Over-complete FACS basis

FLAME Face Model Issues FLAME Limited identity coverage Learned from ~4000 identities Learned from high-quality 4D expression scans Artist designed expression Orthogonal expression space Over-complete FACS basis Modeled as rotatable joints Non-linearity of jaw and neck Activated by linear blend skinning Pose blendshapes further capture details

FLAME Face Model Issues FLAME Limited identity coverage Learned from ~4000 identities Learned from high-quality 4D expression scans Artist designed expression Orthogonal expression space Over-complete FACS basis Modeled as rotatable joints Non-linearity of jaw and neck Activated by linear blend skinning Pose blendshapes further capture details Require artist work Fully automatic registration and modeling

FLAME Face Model

FLAME Face Model Template

FLAME Face Model Template Shape

FLAME Face Model Shape Template Shape +Pose

FLAME Face Model Shape Shape Template Shape + Pose +Pose + Expression

Overview CAESAR dataset Shape Data Registration Shape Model Training MPI FacialMotion dataset Pose Data Registration Pose Model Training D3DFACS dataset Co-registration Hirshberg et al. 12 Initial Expression Blendshapes Expression Data Registration Expression Model Training

Overview CAESAR dataset Shape Data Registration Shape Model Training MPI FacialMotion dataset Pose Data Registration Pose Model Training D3DFACS dataset Co-registration Hirshberg et al. 12 Initial Expression Blendshapes Expression Data Registration Expression Model Training

Shape Model

Shape Data Registration of CAESAR datasets [Robinette et al. 2002]

Learned Shape Model

Pose Model

Pose Data Registration of MPI FacialMotion datasets for pose training

Learned Pose Model

Effect of Pose Blendshapes

Expression Model

Expression Data

Expression Data

4D Scans into Correspondence

Coarse-to-Fine Registration >1 mm 0 mm Stage 1: model-only

Coarse-to-Fine Registration >1 mm 0 mm Stage2: coupled

Coarse-to-Fine Registration >1 mm 0 mm Stage 3: Texture-based Alignment

Effect of Texture-based Alignment

Effect of Texture-based Alignment

Registration Results >1 mm 0 mm

Registration Results >1 mm 0 mm Detail expressions such as eye blinking are also captured

Learned Expression Model

Results

Compare on Identity Space >1 mm 0 mm Scan-to-Mesh Scan-to-Mesh Fitting BU-3DFE scan Fitting Distance Distance FaceWarehouse FLAME 49 [Cao et al. 2014] 49 components + 1 for gender 50 components

Compare on Identity Space >1 mm 0 mm Scan-to-Mesh Scan-to-Mesh Fitting BU-3DFE scan Fitting Distance Distance Basel Face Model (BFM) FLAME 198 [Paysan et al. 2009] 198 components + 1 for gender 199 components

Compare on Identity Space >1 mm 0 mm Scan-to-Mesh Scan-to-Mesh BU-3DFE scan Distance Distance Basel Face Model (BFM) FLAME 198 BU-3DFE scan 199 components 198 components + 1 for gender

Compare on Identity Space 100 80 Percentage 60 FLAME 300 FLAME 198 40 FLAME 90 FLAME 49 FW 20 BFM Full BFM 91 BFM 50 0 0 0.5 1 1.5 2 Error [mm]

Compare on Identity Space FLAME 300: 96% FLAME 198: 94% BFM 199: 92% FLAME 49: 74% FLAME: our model BFM 50: 69% BFM: Basel Face Model FW 50: 67% FW FaceWarehouse Note: higher value is better

Compare on Expression Space >3 mm 0 mm

Sparse Landmark Fitting FLAME produces better result in 2D landmark fitting

Application: Retargeting Target Scan Identity Expression & pose FLAME Face Model Source Retargeted FLAME retargeting pipeline

Application: Retargeting

Conclusion

What did we learn • Large high-quality data • Separation of identity, pose and expression • Importance of face prior • Model and data available for research purpose

Future Work

Acknowledgement Tsvetelina Alexiadis, Andrea Keller, Jorge Márquez Data Acquisition Shunsuke Saito & Cassidy Laidlaw Evaluation Yinghao Huang, Ahmed Osman, Naureen Mahmood Discussion Talha Zaman Video Recording Alejandra Quiros-Ramirez Project Website Darren Cosker Advice and D3DFACS Dataset

Thank You! http://flame.is.tue.mpg.de/ Registrations for D3DFACS dataset FLAME face model (male / female) with example code