Real-time Facial Animation Hao Li Mark Pauly ILM EPFL High-End - - PowerPoint PPT Presentation

▶
real time facial animation
SMART_READER_LITE
LIVE PREVIEW

Real-time Facial Animation Hao Li Mark Pauly ILM EPFL High-End - - PowerPoint PPT Presentation

Mar 14, 2024 157 likes •1.68k views

Real-time Facial Animation Hao Li Mark Pauly ILM EPFL High-End 3D Scanning High-End 3D Scanning Low-Cost Passive Scanning (AGI soft) stereo pair Low-Cost Passive Scanning (AGI soft) stereo pair 3D scan Low-Cost Active Scanning +

slide-1
SLIDE 1

Real-time Facial Animation

Hao Li ILM Mark Pauly EPFL

slide-2
SLIDE 2

High-End 3D Scanning

slide-3
SLIDE 3

High-End 3D Scanning

slide-4
SLIDE 4

Low-Cost Passive Scanning (AGI soft)

stereo pair

slide-5
SLIDE 5

Low-Cost Passive Scanning (AGI soft)

stereo pair 3D scan

slide-6
SLIDE 6

Low-Cost Active Scanning + Temporal Upsampling

Microsoft Kinect & Kinect Fusion

slide-7
SLIDE 7

Rigging & Animation

slide-8
SLIDE 8

Rigging & Animation

slide-9
SLIDE 9

Typical Facial Animation Workflow in Industry

slide-10
SLIDE 10

Typical Facial Animation Workflow in Industry

3D Scanning

slide-11
SLIDE 11

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting

slide-12
SLIDE 12

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Light-weight Rigging

slide-13
SLIDE 13

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Light-weight Rigging Motion Capture

slide-14
SLIDE 14

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Light-weight Rigging Motion Capture Cleanup & Key-Framing

slide-15
SLIDE 15

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Modeling Light-weight Rigging Motion Capture Cleanup & Key-Framing

slide-16
SLIDE 16

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Modeling Light-weight Rigging Motion Capture Cleanup & Key-Framing Complex Rigging

slide-17
SLIDE 17

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Modeling Light-weight Rigging Motion Capture Cleanup & Key-Framing Complex Rigging Retargeting

slide-18
SLIDE 18

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Modeling Light-weight Rigging Motion Capture Cleanup & Key-Framing Complex Rigging Retargeting

slide-19
SLIDE 19

Typical Facial Animation Workflow in Industry

3D Scanning Modeling + Fitting Modeling Light-weight Rigging Motion Capture Cleanup & Key-Framing Complex Rigging Retargeting Key-Framing + Proc.+Sim.

slide-20
SLIDE 20

Markerless Facial Capture

slide-21
SLIDE 21

3D range sensor

slide-22
SLIDE 22

3D range sensor

slide-23
SLIDE 23

Motion can be Captured at the Same Resolution as the Geometry

3D range sensor

slide-24
SLIDE 24

USC ICT Light Stage 5

slide-25
SLIDE 25

USC ICT Light Stage 5

slide-26
SLIDE 26

Goal

slide-27
SLIDE 27

Goal

slide-28
SLIDE 28

Template-Based Tracking

slide-29
SLIDE 29

Template-Based Tracking

slide-30
SLIDE 30

Template-Based Tracking

analyze deformation

slide-31
SLIDE 31

Template-Based Tracking

analyze deformation

slide-32
SLIDE 32

Template-Based Tracking

transfer deformation

slide-33
SLIDE 33

Template-Based Tracking

transfer deformation

slide-34
SLIDE 34

Template-Based Tracking

transfer deformation

slide-35
SLIDE 35

Template-Based Tracking

transfer deformation

slide-36
SLIDE 36

Correspondences Problem

slide-37
SLIDE 37

Correspondences Problem

slide-38
SLIDE 38

Correspondences Problem

slide-39
SLIDE 39

Correspondences Problem

slide-40
SLIDE 40

Non-Rigid Registration

slide-41
SLIDE 41

Pair of 3D Scans

slide-42
SLIDE 42

Pair of 3D Scans

source

slide-43
SLIDE 43

Pair of 3D Scans

target source

slide-44
SLIDE 44

Correspondences are Lost

slide-45
SLIDE 45

Correspondences are Lost

slide-46
SLIDE 46

Correspondences are Lost

?

slide-47
SLIDE 47

Overlapping Regions are Lost

slide-48
SLIDE 48

Overlapping Regions are Lost

  • verlapping regions
slide-49
SLIDE 49

Overlapping Regions are Lost

missing data

  • verlapping regions
slide-50
SLIDE 50

Overlapping Regions are Lost

slide-51
SLIDE 51

Overlapping Regions are Lost

slide-52
SLIDE 52

Non-Rigid Registration

slide-53
SLIDE 53

Non-Rigid Registration

slide-54
SLIDE 54

Three Ingredients

slide-55
SLIDE 55

Three Ingredients

source

slide-56
SLIDE 56

Three Ingredients

source target

slide-57
SLIDE 57

Three Ingredients

source target

slide-58
SLIDE 58

Three Ingredients

source target

detect

  • verlap
slide-59
SLIDE 59

Three Ingredients

source target

detect

  • verlap
slide-60
SLIDE 60

Three Ingredients

source target

detect

  • verlap

correspond

slide-61
SLIDE 61

Three Ingredients

source target

detect

  • verlap

correspond

slide-62
SLIDE 62

Three Ingredients

source target

detect

  • verlap

correspond deform

slide-63
SLIDE 63

Three Ingredients

target registration

detect

  • verlap

correspond deform

slide-64
SLIDE 64

Challenges

detect

  • verlap

correspond deform

slide-65
SLIDE 65

Challenges

detect

  • verlap

correspond deform

slide-66
SLIDE 66

Challenges

detect

  • verlap

correspond deform

slide-67
SLIDE 67

Challenges

detect

  • verlap

correspond deform

deformation

slide-68
SLIDE 68

Challenges

detect

  • verlap

correspond deform

slide-69
SLIDE 69

Challenges

detect

  • verlap

correspond deform

slide-70
SLIDE 70

Challenges

detect

  • verlap

correspond deform

ambiguity

slide-71
SLIDE 71

Challenges

detect

  • verlap

correspond deform

slide-72
SLIDE 72

Challenges

detect

  • verlap

correspond deform

slide-73
SLIDE 73

Challenges

detect

  • verlap

correspond deform

slide-74
SLIDE 74

Challenges

detect

  • verlap

correspond deform

slide-75
SLIDE 75

Challenges

detect

  • verlap

correspond deform

?

slide-76
SLIDE 76

Challenges

detect

  • verlap

correspond deform

slide-77
SLIDE 77

Challenges

detect

  • verlap

correspond deform

slide-78
SLIDE 78

Observation

detect

  • verlap

correspond deform

slide-79
SLIDE 79

Observation

detect

  • verlap

correspond deform

slide-80
SLIDE 80

Observation

detect

  • verlap

correspond deform

helps

slide-81
SLIDE 81

Observation

detect

  • verlap

correspond deform

helps helps

slide-82
SLIDE 82

Observation

detect

  • verlap

correspond deform

slide-83
SLIDE 83

Observation

detect

  • verlap

correspond deform

slide-84
SLIDE 84

Observation

detect

  • verlap

correspond deform

global optimization via local refinement

slide-85
SLIDE 85

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-86
SLIDE 86

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-87
SLIDE 87

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-88
SLIDE 88

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-89
SLIDE 89

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-90
SLIDE 90

Iterative Global Optimization

detect

  • verlap

correspond deform

slide-91
SLIDE 91

Iterative Global Optimization

detect

  • verlap

deform closest point

slide-92
SLIDE 92

Iterative Global Optimization

detect

  • verlap

deform closest point

slide-93
SLIDE 93

Iterative Global Optimization

deform closest point pruning

slide-94
SLIDE 94

Iterative Global Optimization

deform closest point pruning

slide-95
SLIDE 95

Iterative Global Optimization

deform closest point pruning

slide-96
SLIDE 96

Iterative Global Optimization

deform closest point pruning

slide-97
SLIDE 97

Iterative Global Optimization

deform closest point pruning

slide-98
SLIDE 98

Iterative Global Optimization

closest point pruning global

  • ptimization
slide-99
SLIDE 99

Iterative Global Optimization

closest point pruning global

  • ptimization
slide-100
SLIDE 100

Iterative Global Optimization

closest point pruning global

  • ptimization

converges?

slide-101
SLIDE 101

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? no

slide-102
SLIDE 102

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? no

slide-103
SLIDE 103

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? no

slide-104
SLIDE 104

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? yes relax stiffness no

slide-105
SLIDE 105

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? yes relax stiffness no

slide-106
SLIDE 106

Iterative Global Optimization

closest point pruning global

  • ptimization

converges? yes relax stiffness no

slide-107
SLIDE 107

Iterative Global Optimization

closest point pruning global

  • ptimization

Robust Non-Rigid ICP

converges? yes relax stiffness no

slide-108
SLIDE 108

Iterative Global Optimization

closest point pruning global

  • ptimization

Robust Non-Rigid ICP

converges? yes relax stiffness no

slide-109
SLIDE 109

Embedded Deformation Model

detail preservation global consistency

slide-110
SLIDE 110

Embedded Deformation Model

detail preservation global consistency

slide-111
SLIDE 111

Embedded Deformation Model

detail preservation global consistency

slide-112
SLIDE 112

Embedded Deformation Model

detail preservation global consistency

slide-113
SLIDE 113

Embedded Deformation Model

detail preservation global consistency

slide-114
SLIDE 114

Embedded Deformation Model

de-coupled complexity

slide-115
SLIDE 115

Embedded Deformation Model

de-coupled complexity

slide-116
SLIDE 116

Embedded Deformation Model

Erigid

de-coupled complexity

slide-117
SLIDE 117

Embedded Deformation Model

Erigid Esmooth

de-coupled complexity

slide-118
SLIDE 118

Non-Linear Energy Minimization

ci vi

Erigid Esmooth

slide-119
SLIDE 119

Non-Linear Energy Minimization

ci vi

Erigid Esmooth

slide-120
SLIDE 120

Non-Linear Energy Minimization

ci vi

Erigid Esmooth

slide-121
SLIDE 121

Non-Linear Energy Minimization

ci vi

Erigid Esmooth

slide-122
SLIDE 122

Non-Linear Energy Minimization

ci vi

Epoint Erigid Esmooth

slide-123
SLIDE 123

Non-Linear Energy Minimization

ci vi

Eplane Epoint Eplane Erigid Esmooth

[Chen & Medioni ’92]

slide-124
SLIDE 124

Non-Linear Energy Minimization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Erigid Esmooth

[Chen & Medioni ’92]

slide-125
SLIDE 125

Non-Linear Energy Minimization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Esmooth

non-linear least squares minimization

[Chen & Medioni ’92]

slide-126
SLIDE 126

Non-Linear Energy Minimization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Esmooth

non-linear least squares minimization Gauss-Newton method

[Chen & Medioni ’92]

slide-127
SLIDE 127

Non-Linear Energy Minimization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Esmooth

non-linear least squares minimization Gauss-Newton method Jacobian is sparse

[Chen & Medioni ’92]

slide-128
SLIDE 128

Non-Linear Energy Minimization

sparse Cholesky factorization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Esmooth

non-linear least squares minimization Gauss-Newton method Jacobian is sparse

[Chen & Medioni ’92]

slide-129
SLIDE 129

Non-Linear Energy Minimization

sparse Cholesky factorization

ci vi

Eplane Etot

+αpoint +αrigid +αsmooth =

Epoint Eplane Erigid Esmooth

non-linear least squares minimization Gauss-Newton method Jacobian is sparse

that’s it!

[Chen & Medioni ’92]

slide-130
SLIDE 130

Template-Based Tracking

slide-131
SLIDE 131

Not only for Faces!

slide-132
SLIDE 132

template

Not only for Faces!

slide-133
SLIDE 133

template 3D scan

Not only for Faces!

slide-134
SLIDE 134

Not only for Faces!

slide-135
SLIDE 135

Requirements for a Practical System

slide-136
SLIDE 136

Requirements for a Practical System

slide-137
SLIDE 137

Requirements for a Practical System

1.Real-time performance

slide-138
SLIDE 138

Requirements for a Practical System

1.Real-time performance 2.Robustness to noise

slide-139
SLIDE 139

Requirements for a Practical System

1.Real-time performance 2.Robustness to noise 3.High-level semantics

slide-140
SLIDE 140

Real-time Facial Capture

slide-141
SLIDE 141

Objective

slide-142
SLIDE 142

Building Expression Space

tracked template input scan

slide-143
SLIDE 143

Building Expression Space

tracked template input scan

slide-144
SLIDE 144

Expression PCA for Reduced Dimension

slide-145
SLIDE 145

Expression PCA for Reduced Dimension

slide-146
SLIDE 146

Expression PCA for Reduced Dimension

Principal Component Analysis

slide-147
SLIDE 147

Expression PCA for Reduced Dimension

Principal Component Analysis

slide-148
SLIDE 148

Expression PCA for Reduced Dimension

Principal Component Analysis

slide-149
SLIDE 149

Expression PCA for Reduced Dimension

Principal Component Analysis

+w1 +w2 +w3 +w4

slide-150
SLIDE 150

Expression PCA for Reduced Dimension

Principal Component Analysis

+w1 +w2 +w3 +w4 =