Face2F ace2Face: ace: Real-time Face Capture and Reenactment of - PowerPoint PPT Presentation

IEEE 2016 Conference on Computer Vision and Pattern Recognition Face2F ace2Face: ace: Real-time Face Capture and Reenactment of RGB-Videos Justus Thies 1 , Michael Zollhöfer 2 , Marc Stamminger 1 , Christian Theobalt 2 , Matthias Nießner 3 1 University of Erlangen-Nuremberg 2 Max-Planck-Institute for Informatics 3 Stanford University

IEEE 2016 Conference on Computer Vision and Pattern Recognition Related Work • Offline • Online Creating a Photoreal Digital Actor: Real-time Expression Transfer for Facial Reenactment The Digital Emily Project RGB-D Special Hardware Vdub: Modifying Face Video of Actors for Face2Face: Real-time Face Capture and Reenactment Plausible Visual Alignment to a Dubbed Audio Track Of RGB-Videos RGB

IEEE 2016 Conference on Computer Vision and Pattern Recognition Related Work • Offline • Online Creating a Photoreal Digital Actor: Real-time Expression Transfer for Facial Reenactment The Digital Emily Project RGB-D Special Hardware Vdub: Modifying Face Video of Actors for Face2Face: Real-time Face Capture and Reenactment Plausible Visual Alignment to a Dubbed Audio Track Of RGB-Videos RGB

IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling • Reenactment • Mouth Retrieval • Comparisons Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Overview • Parametric Face Model • Face Capture • Energy Formulation • Non-rigid Model-based Bundling • Reenactment • Mouth Retrieval • Comparisons • Results / Live Demo Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model 𝑸 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 𝑸 = 6 +80 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80 𝑸 = 6 +80+80 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80+80 𝑸 = 6 +80+80+76 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model Φ 𝛽 𝛾 𝑸 = 𝜀 𝛿 𝑸 = 6 +80+80+76 𝑸 = 6 +80+80+76+27= 269 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Parametric Face Model 𝑄 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Face Capture

IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 Distance in RGB Color Space Color Consistency 𝒎 𝟑,𝟐 − 𝒐𝒑𝒔𝒏 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 +𝐹 𝑛𝑠𝑙 𝑄 Distance in Image Space Color Feature Consistency Similarity Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Energy Formulation 𝐹 𝑄 = 𝐹 𝑑𝑝𝑚 𝑄 +𝐹 𝑛𝑠𝑙 𝑄 +𝐹 𝑠𝑓𝑕 (𝑄) Color Feature Regularization Consistency Similarity −𝟒 𝝉 +𝟒 𝝉 𝟘𝟘, 𝟖% Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling 𝑜 𝐹 𝑢𝑝𝑢𝑏𝑚 𝑸 = 𝐹 𝑗 𝑸 → 𝑛𝑗𝑜 𝑗=0 Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling • Iterative Reweighted Least Squares (IRLS) 𝑲 𝑼 𝑲𝚬𝑸 = −𝑲 𝑼 𝑮  Gauss-Newton: 𝑲(𝑸) = Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Non-rigid Model-based Computer Vision and Pattern Recognition Bundling Hierarchy Levels Input Model Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison

IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Tracking Comparison Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment

IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Online RGB-Tracking Reenactment Pose Source Actor Per Frame Expression Transfer Illumination Mouth Retrieval Expression Identity Compositing Preprocessed Video Tracking Target Actor Pose Per Frame Illumination Expression Identity Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Online RGB-Tracking Reenactment Pose Source Actor Per Frame Expression Transfer Illumination Mouth Retrieval Expression Identity Compositing Preprocessed Video Tracking Target Actor Pose Per Frame Illumination Expression Identity Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Mouth-Retrieval Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Mouth-Retrieval Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Reenactment Comparison Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Live-Demo

IEEE 2016 Conference on Computer Vision and Pattern Recognition Limitations / Future Work • Assumption of Lambertian surface and smooth illumination • No occlusion handling • No person specific details (fine scale details / wrinkles) • Reenactment relies on a training sequence (Mouth retrieval) Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Conclusion • First Real-time Facial Reenactment only based on RGB-videos • Non-Rigid Model-Based Bundling • Sub-Space Deformation Transfer • Image-Based Mouth Synthesis Face Model Face Capture Reenactment Results

IEEE 2016 Conference on Computer Vision and Pattern Recognition Thank You!

IEEE 2016 Conference on Computer Vision and Pattern Recognition References • O. Alexander, M. Rogers, W. Lambeth, M. Chiang, and P. Debevec. The Digital Emily Project: photoreal facial modeling and animation. In ACM SIGGRAPH Courses , pages 12:1 – 12:15. ACM, 2009. • P. Garrido, L. Valgaerts, H. Sarmadi, I. Steiner, K. Varanasi, P. Perez, and C. Theobalt. Vdub: Modifying face video of actors for plausible visual alignment to a dubbed audio track. In Computer Graphics Forum . Wiley-Blackwell, 2015. • F. Shi, H.-T. Wu, X. Tong, and J. Chai. Automatic acquisition of high-fidelity facial performances using monocular videos. ACM TOG , 33(6):222, 2014. • C. Cao, Y. Weng, S. Zhou, Y. Tong, and K. Zhou. Facewarehouse: A 3D facial expression database for visual computing. IEEE TVCG , 20(3):413 – 425, 2014. • J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt. Real-time expression transfer for facial reenactment. ACM Transactions on Graphics (TOG) ,34(6), 2015. • V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Proc. SIGGRAPH , pages 187 – 194. ACM Press/Addison-Wesley Publishing Co., 1999. Face Model Face Capture Reenactment Results