[PPT] - Face2F ace2Face: ace: Real-time Face Capture and Reenactment of PowerPoint Presentation

SLIDE 1 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Face2F ace2Face: ace:

Real-time Face Capture and Reenactment of RGB-Videos

Justus Thies1, Michael Zollhöfer2, Marc Stamminger1, Christian Theobalt2, Matthias Nießner3

1University of Erlangen-Nuremberg 2Max-Planck-Institute for Informatics 3Stanford University

SLIDE 2 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Related Work

Offline
Online

RGB-D

Real-time Expression Transfer for Facial Reenactment Vdub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track Creating a Photoreal Digital Actor: The Digital Emily Project

RGB

Face2Face: Real-time Face Capture and Reenactment Of RGB-Videos

Special Hardware

SLIDE 3 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Related Work

Offline
Online

RGB-D

Real-time Expression Transfer for Facial Reenactment Vdub: Modifying Face Video of Actors for Plausible Visual Alignment to a Dubbed Audio Track Creating a Photoreal Digital Actor: The Digital Emily Project

RGB

Face2Face: Real-time Face Capture and Reenactment Of RGB-Videos

Special Hardware

SLIDE 4 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Overview

Parametric Face Model

SLIDE 5 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Overview

Parametric Face Model
Face Capture
Energy Formulation
Non-rigid Model-based Bundling

SLIDE 6 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Overview

Parametric Face Model
Face Capture
Energy Formulation
Non-rigid Model-based Bundling
Reenactment
Mouth Retrieval
Comparisons

SLIDE 7 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Overview

Parametric Face Model
Face Capture
Energy Formulation
Non-rigid Model-based Bundling
Reenactment
Mouth Retrieval
Comparisons
Results / Live Demo

SLIDE 8 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Parametric Face Model

SLIDE 9 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸

SLIDE 10 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸 = 6

𝑸 = Φ 𝛽 𝛾 𝜀 𝛿

SLIDE 11 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸 = 6 𝑸 = 6+80

𝑸 = Φ 𝛽 𝛾 𝜀 𝛿

SLIDE 12 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸 = 6+80 𝑸 = 6+80+80

𝑸 = Φ 𝛽 𝛾 𝜀 𝛿

SLIDE 13 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸 = 6+80+80 𝑸 = 6+80+80+76

𝑸 = Φ 𝛽 𝛾 𝜀 𝛿

SLIDE 14 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑸 = Φ 𝛽 𝛾 𝜀 𝛿

𝑸 = 6+80+80+76 𝑸 = 6+80+80+76+27=269

SLIDE 15 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Parametric Face Model

𝑄

SLIDE 16 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Face Capture

SLIDE 17 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Energy Formulation

𝐹 𝑄 =

SLIDE 18 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Energy Formulation

Distance in RGB Color Space

Color Consistency

𝐹 𝑄 = 𝐹𝑑𝑝𝑚 𝑄

𝒎𝟑,𝟐 − 𝒐𝒑𝒔𝒏

SLIDE 19 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Energy Formulation

Distance in Image Space

Color Consistency Feature Similarity

𝐹 𝑄 = 𝐹𝑑𝑝𝑚 𝑄 +𝐹𝑛𝑠𝑙 𝑄

SLIDE 20 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Energy Formulation

Regularization Color Consistency Feature Similarity

𝐹 𝑄 = 𝐹𝑑𝑝𝑚 𝑄 +𝐹𝑛𝑠𝑙 𝑄 +𝐹𝑠𝑓𝑕(𝑄)

−𝟒 𝝉 +𝟒 𝝉 𝟘𝟘, 𝟖%

SLIDE 21 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Non-rigid Model-based Bundling 𝐹𝑢𝑝𝑢𝑏𝑚 𝑸 =

𝑗=0 𝑜

𝐹𝑗 𝑸 → 𝑛𝑗𝑜

SLIDE 22 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Non-rigid Model-based Bundling

Iterative Reweighted Least Squares (IRLS)

 Gauss-Newton:

𝑲𝑼𝑲𝚬𝑸 = −𝑲𝑼𝑮

𝑲(𝑸) =

SLIDE 23 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Non-rigid Model-based Bundling

Input Model

Hierarchy Levels

SLIDE 24 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Tracking

SLIDE 25 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Tracking Comparison

SLIDE 26 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Tracking Comparison

SLIDE 27 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Tracking Comparison

SLIDE 28 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Reenactment

SLIDE 29 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Reenactment

Online RGB-Tracking Preprocessed Video Tracking Identity Expression Illumination Pose Per Frame Identity Expression Illumination Pose Per Frame Reenactment Expression Transfer Mouth Retrieval Compositing Source Actor Target Actor

SLIDE 30 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Reenactment

Online RGB-Tracking Preprocessed Video Tracking Identity Expression Illumination Pose Per Frame Identity Expression Illumination Pose Per Frame Reenactment Expression Transfer Mouth Retrieval Compositing Source Actor Target Actor

SLIDE 31 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Mouth-Retrieval

SLIDE 32 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Mouth-Retrieval

SLIDE 33 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Reenactment Comparison

SLIDE 34 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Live-Demo

SLIDE 35 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Limitations / Future Work

Assumption of Lambertian surface and smooth illumination
No occlusion handling
No person specific details (fine scale details / wrinkles)
Reenactment relies on a training sequence (Mouth retrieval)

SLIDE 36 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

Conclusion

First Real-time Facial Reenactment only based on RGB-videos
Non-Rigid Model-Based Bundling
Sub-Space Deformation Transfer
Image-Based Mouth Synthesis

SLIDE 37 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Thank You!

SLIDE 38 IEEE 2016 Conference on Computer Vision and Pattern Recognition

Results Reenactment Face Capture Face Model

References

O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec.

The Digital Emily Project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses, pages 12:1–12:15. ACM, 2009.

P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt.

Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. In Computer Graphics Forum. Wiley-Blackwell, 2015.

F. Shi, H.-T. Wu, X. Tong, and J. Chai.

Automatic acquisition of high-fidelity facial performances using monocular videos. ACM TOG, 33(6):222, 2014.

C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou.

Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG, 20(3):413–425, 2014.

J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt.

Real-time expression transfer for facial reenactment. ACM Transactions on Graphics (TOG),34(6), 2015.

V. Blanz and T. Vetter.

A morphable model for the synthesis of 3d faces. In Proc. SIGGRAPH, pages 187–194. ACM Press/Addison-Wesley Publishing Co., 1999.