Practical Animation Reconstruction Hao Li Local Shape Matching: - - PowerPoint PPT Presentation

▶
practical animation reconstruction
SMART_READER_LITE
LIVE PREVIEW

Practical Animation Reconstruction Hao Li Local Shape Matching: - - PowerPoint PPT Presentation

Aug 23, 2023 212 likes •1.63k views

Practical Animation Reconstruction Hao Li Local Shape Matching: Advanced Animation Reconstruction Correspondence Problem Classification How many meshes? More than two: multi-view registration Initial registration available? Yes: Local

slide-1
SLIDE 1

Local Shape Matching: Advanced Animation Reconstruction

Practical Animation Reconstruction

Hao Li

slide-2
SLIDE 2

Local Shape Matching: Non-Rigid Registration

Correspondence Problem Classification

How many meshes?

More than two: multi-view registration

Initial registration available?

Yes: Local optimization methods

Class of transformations?

Non-rigid deformations

2

slide-3
SLIDE 3

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

animation rigging modeling

slide-4
SLIDE 4

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

animation rigging

surface reconstruction

slide-5
SLIDE 5

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

animation rigging

surface reconstruction

[XYZRGB ’09]

slide-6
SLIDE 6

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

animation

surface reconstruction automatic rigging

[XYZRGB ’09]

slide-7
SLIDE 7

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

animation

surface reconstruction automatic rigging

[Baran & Popović ’07] [XYZRGB ’09]

slide-8
SLIDE 8

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

surface reconstruction automatic rigging motion capture

[Baran & Popović ’07] [XYZRGB ’09]

slide-9
SLIDE 9

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

surface reconstruction automatic rigging motion capture

[Vicon ’08] [Baran & Popović ’07] [XYZRGB ’09]

slide-10
SLIDE 10

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

surface reconstruction automatic rigging motion capture

[Vicon ’08] [Baran & Popović ’07] [XYZRGB ’09]

slide-11
SLIDE 11

Local Shape Matching: Advanced Animation Reconstruction

Performance Capture

3

surface reconstruction automatic rigging motion capture post processing

[Vicon ’08] [Baran & Popović ’07] [XYZRGB ’09]

slide-12
SLIDE 12

Local Shape Matching: Advanced Animation Reconstruction

Performance Retargeting

4

slide-13
SLIDE 13

Local Shape Matching: Advanced Animation Reconstruction

Performance Retargeting

4

slide-14
SLIDE 14

Local Shape Matching: Advanced Animation Reconstruction

Performance Retargeting

4

slide-15
SLIDE 15

Local Shape Matching: Advanced Animation Reconstruction

Markerless Performance Capture

5

[de Aguiar et al. ‘08]

template

slide-16
SLIDE 16

Local Shape Matching: Advanced Animation Reconstruction

Markerless Performance Capture

5

[de Aguiar et al. ‘08]

template cameras

slide-17
SLIDE 17

Local Shape Matching: Advanced Animation Reconstruction

Markerless Performance Capture

5

[de Aguiar et al. ‘08]

template cameras

slide-18
SLIDE 18

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-19
SLIDE 19

Elastic Deformation

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-20
SLIDE 20

Elastic Deformation Sparse Tracking

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-21
SLIDE 21

Elastic Deformation Sparse Tracking Baked-in Details

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-22
SLIDE 22

Jelly Effect

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-23
SLIDE 23

[Vlasic et al. ’08]

Local Shape Matching: Advanced Animation Reconstruction

6

slide-24
SLIDE 24

Local Shape Matching: Advanced Animation Reconstruction

Real-time Dense Acquisition

7

Structured Light Based on Phase-Shift Stereo [Weise et al. ’07]

slide-25
SLIDE 25

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

surface reconstruction automatic rigging motion capture

slide-26
SLIDE 26

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

surface reconstruction automatic rigging motion capture

slide-27
SLIDE 27

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

dynamic shape reconstruction automatic rigging

slide-28
SLIDE 28

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

dynamic shape reconstruction

post processing

automatic rigging

slide-29
SLIDE 29

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

dynamic shape reconstruction automatic rigging

slide-30
SLIDE 30

Local Shape Matching: Advanced Animation Reconstruction

Animation Reconstruction

8

dynamic shape reconstruction automatic rigging

slide-31
SLIDE 31

Local Shape Matching: Advanced Animation Reconstruction

Objective

9

Input 3D Scan Sequence Space-time Reconstruction

[Li et al. ’09]

slide-32
SLIDE 32

Local Shape Matching: Advanced Animation Reconstruction

Challenges

10

slide-33
SLIDE 33

Local Shape Matching: Advanced Animation Reconstruction

Challenges

10

slide-34
SLIDE 34

Local Shape Matching: Advanced Animation Reconstruction

Challenges

10

[Süssmuth et al. ’08]

Correspondence Issues

slide-35
SLIDE 35

Local Shape Matching: Advanced Animation Reconstruction

Challenges

10

[Süssmuth et al. ’08]

Correspondence Issues

[Wand et al. ’09]

Topology Issues

slide-36
SLIDE 36

Local Shape Matching: Advanced Animation Reconstruction

Deforming Physical Subject

11

slide-37
SLIDE 37

Local Shape Matching: Advanced Animation Reconstruction

Partial Scans

12

slide-38
SLIDE 38

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

13

slide-39
SLIDE 39

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

13

slide-40
SLIDE 40

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

13

Large-Scale Motion

slide-41
SLIDE 41

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

14

slide-42
SLIDE 42

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

14

slide-43
SLIDE 43

Local Shape Matching: Advanced Animation Reconstruction

Partial and Non-Rigid Registration

14

Small-Scale Dynamics

slide-44
SLIDE 44

Local Shape Matching: Advanced Animation Reconstruction

Bi-Resolution Approach

15

Warping a coarse template

slide-45
SLIDE 45

Local Shape Matching: Advanced Animation Reconstruction

Bi-Resolution Approach

15

Warping a coarse template

slide-46
SLIDE 46

Local Shape Matching: Advanced Animation Reconstruction

Bi-Resolution Approach

15

Warping a coarse template

slide-47
SLIDE 47

Local Shape Matching: Advanced Animation Reconstruction

Bi-Resolution Approach

15

Warping a coarse template

slide-48
SLIDE 48

Local Shape Matching: Advanced Animation Reconstruction

Bi-Resolution Approach

16

Synthesizing small scale details

slide-49
SLIDE 49

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

slide-50
SLIDE 50

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

slide-51
SLIDE 51

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

detail estimation

slide-52
SLIDE 52

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

slide-53
SLIDE 53

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

non-rigid registration

slide-54
SLIDE 54

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

slide-55
SLIDE 55

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

17

slide-56
SLIDE 56

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

18

slide-57
SLIDE 57

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

18

non-rigid registration detail estimation Large-Scale Motion

slide-58
SLIDE 58

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

18

detail estimation Large-Scale Motion

slide-59
SLIDE 59

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

18

detail estimation detail estimation Large-Scale Motion

slide-60
SLIDE 60

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction Framework

18

detail estimation detail estimation detail aggregation Large-Scale Motion Fine-Scale Dynamics

slide-61
SLIDE 61

Local Shape Matching: Advanced Animation Reconstruction

Sequential Non-Rigid Registration

19

input data

data provided by Stanford and MPI Saarbrücken

slide-62
SLIDE 62

Local Shape Matching: Advanced Animation Reconstruction

Sequential Non-Rigid Registration

19

input data template fitting

data provided by Stanford and MPI Saarbrücken

slide-63
SLIDE 63

Local Shape Matching: Advanced Animation Reconstruction

Adaptive Deformation Model

20

Input Scans Warped Template with Graph

slide-64
SLIDE 64

Local Shape Matching: Advanced Animation Reconstruction

Detail Aggregation

21

slide-65
SLIDE 65

Local Shape Matching: Advanced Animation Reconstruction

Detail Aggregation

21

slide-66
SLIDE 66

Local Shape Matching: Advanced Animation Reconstruction

Detail Aggregation

21

Single Frame Synthesis

slide-67
SLIDE 67

Local Shape Matching: Advanced Animation Reconstruction

Detail Aggregation

21

Single Frame Synthesis Multi-Frame Aggregation

slide-68
SLIDE 68

Local Shape Matching: Advanced Animation Reconstruction

Forward-Backward Pass

22

slide-69
SLIDE 69

Local Shape Matching: Advanced Animation Reconstruction

Forward-Backward Pass

22

slide-70
SLIDE 70

Local Shape Matching: Advanced Animation Reconstruction

Forward-Backward Pass

22

slide-71
SLIDE 71

Local Shape Matching: Advanced Animation Reconstruction

Forward-Backward Pass

22

slide-72
SLIDE 72

Local Shape Matching: Advanced Animation Reconstruction

Super Resolution

23

Input Scan

slide-73
SLIDE 73

Local Shape Matching: Advanced Animation Reconstruction

Super Resolution

23

Input Scan Warped Template

slide-74
SLIDE 74

Local Shape Matching: Advanced Animation Reconstruction

Super Resolution

23

Input Scan Warped Template Reconstruction

slide-75
SLIDE 75

Local Shape Matching: Advanced Animation Reconstruction

Super Resolution

23

Input Scan Warped Template Reconstruction Overlaid Scan

slide-76
SLIDE 76

Local Shape Matching: Advanced Animation Reconstruction

Final Reconstruction – 100 Frames

24

Reconstruction Input Scans Textured Reconstruction

slide-77
SLIDE 77

Local Shape Matching: Advanced Animation Reconstruction

Crumpling Paper Bag – 85 Frames

25

Reconstruction Input Scans Textured Reconstruction

slide-78
SLIDE 78

Local Shape Matching: Advanced Animation Reconstruction

Facial Expressions – 200 Frames

26

Reconstruction Input Scans Overlaid Scans

slide-79
SLIDE 79

Local Shape Matching: Advanced Animation Reconstruction

Limitations: Drift

27

dynamic shape reconstruction automatic rigging

slide-80
SLIDE 80

Local Shape Matching: Advanced Animation Reconstruction

Limitations: Drift

27

dynamic shape reconstruction automatic rigging training

slide-81
SLIDE 81

Local Shape Matching: Advanced Animation Reconstruction

Limitations: Drift

27

dynamic shape reconstruction automatic rigging training

slide-82
SLIDE 82

Local Shape Matching: Advanced Animation Reconstruction

Facial Animation

28

[Weise et al. ’09]

slide-83
SLIDE 83

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

slide-84
SLIDE 84

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

scans textures

slide-85
SLIDE 85

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction scans textures

slide-86
SLIDE 86

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction texture mapping scans textures

slide-87
SLIDE 87

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking

texture mapping scans textures

slide-88
SLIDE 88

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking

texture mapping scans textures

slide-89
SLIDE 89

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking

texture mapping dimension reduction scans textures

slide-90
SLIDE 90

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking
  • nline tracking

texture mapping dimension reduction scans textures

slide-91
SLIDE 91

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking
  • nline tracking

texture mapping dimension reduction scans textures

slide-92
SLIDE 92

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking
  • nline tracking

texture mapping dimension reduction retargeting scans textures

slide-93
SLIDE 93

Local Shape Matching: Advanced Animation Reconstruction

System Overview

29

template construction

  • ffline tracking
  • nline tracking

texture mapping dimension reduction retargeting scans textures

slide-94
SLIDE 94

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

slide-95
SLIDE 95

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

slide-96
SLIDE 96

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

Principal Component Analysis

slide-97
SLIDE 97

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

Principal Component Analysis

slide-98
SLIDE 98

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

Principal Component Analysis

slide-99
SLIDE 99

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

Principal Component Analysis

+w1 +w2 +w3 +w4

slide-100
SLIDE 100

Local Shape Matching: Advanced Animation Reconstruction

Dimension Reduction

30

Principal Component Analysis

+w1 +w2 +w3 +w4 =

slide-101
SLIDE 101

Local Shape Matching: Advanced Animation Reconstruction

Limitations: Varying Topology

31

slide-102
SLIDE 102

Local Shape Matching: Advanced Animation Reconstruction

Temporally Coherent Shape Completion

32

[Li et al. ’11]

partial data reconstruction reconstruction partial data

slide-103
SLIDE 103

Local Shape Matching: Advanced Animation Reconstruction

Temporally Coherent Shape Completion

32

[Li et al. ’11]

partial data reconstruction reconstruction partial data

slide-104
SLIDE 104

Local Shape Matching: Advanced Animation Reconstruction

Temporally Coherent Shape Completion

32

[Li et al. ’11]

partial data reconstruction reconstruction partial data

slide-105
SLIDE 105

Local Shape Matching: Advanced Animation Reconstruction

Light Stage 6 Capture

33

Lightstage 6 (USC-ICT) [Vlasic et al. ‘09]

slide-106
SLIDE 106

Local Shape Matching: Advanced Animation Reconstruction

Light Stage 6 Capture

33

Lightstage 6 (USC-ICT) [Vlasic et al. ‘09] 8 Normal Maps / Frame

slide-107
SLIDE 107

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

34 per view reconstruction

slide-108
SLIDE 108

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

34 per view reconstruction non-rigid registration

slide-109
SLIDE 109

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

34 per view reconstruction non-rigid registration poisson reconstruction

slide-110
SLIDE 110

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

35

Capture

slide-111
SLIDE 111

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

35

Integration Capture

slide-112
SLIDE 112

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

35

Integration Capture Matching

slide-113
SLIDE 113

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

35

Integration Capture Matching Merging

slide-114
SLIDE 114

Local Shape Matching: Advanced Animation Reconstruction

Reconstruction

35

Integration Capture Matching Merging

slide-115
SLIDE 115

Local Shape Matching: Advanced Animation Reconstruction

Naive Shape Completion

36

slide-116
SLIDE 116

Local Shape Matching: Advanced Animation Reconstruction

Dynamic Shape Completion

37 input scan

slide-117
SLIDE 117

Local Shape Matching: Advanced Animation Reconstruction

Dynamic Shape Completion

37 input scan visual hull hole filling

slide-118
SLIDE 118

Local Shape Matching: Advanced Animation Reconstruction

Dynamic Shape Completion

37 input scan visual hull hole filling space-time filtering

slide-119
SLIDE 119

Local Shape Matching: Advanced Animation Reconstruction

Dynamic Shape Completion

37 input scan visual hull hole filling space-time filtering detail re-synthesis

slide-120
SLIDE 120

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1

slide-121
SLIDE 121

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 current frame

slide-122
SLIDE 122

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 current frame

slide-123
SLIDE 123

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 current frame

slide-124
SLIDE 124

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 current frame

slide-125
SLIDE 125

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 current frame

slide-126
SLIDE 126

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 w=100 current frame

slide-127
SLIDE 127

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 w=100 w=10 w=10 current frame

slide-128
SLIDE 128

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 w=100 w=10 w=10 current frame

slide-129
SLIDE 129

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 w=100 w=10 w=10 w=2 current frame

slide-130
SLIDE 130

Local Shape Matching: Advanced Animation Reconstruction

Temporal Filtering

38

frame

n-1 n n+1 w=100 w=10 w=10 w=1 w=1 w=2 current frame

slide-131
SLIDE 131

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39

slide-132
SLIDE 132

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow

slide-133
SLIDE 133

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow fine-scale registration

slide-134
SLIDE 134

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow fine-scale registration

slide-135
SLIDE 135

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow fine-scale registration

Coarse Registration Fine Registration

slide-136
SLIDE 136

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow fine-scale registration

Coarse Registration Fine Registration Merging

slide-137
SLIDE 137

Local Shape Matching: Advanced Animation Reconstruction

Short Range Correspondences

39 coarse-scale alignment + graph prediction + surf/optical flow fine-scale registration

Coarse Registration Fine Registration Merging

slide-138
SLIDE 138

Local Shape Matching: Advanced Animation Reconstruction

Free-Viewpoint Video

40

[Li et al. ’11]

slide-139
SLIDE 139

Local Shape Matching: Advanced Animation Reconstruction

Free-Viewpoint Video

40

[Li et al. ’11]

slide-140
SLIDE 140

Local Shape Matching: Non-Rigid Registration

41

www.hao-li.com

slide-141
SLIDE 141

Local Shape Matching: Non-Rigid Registration

41

?

www.hao-li.com