SLIDE 1
Alec Jacobson
Columbia University Skinning: Real-time Shape Deformation
Automatic Skinning via Constrained Energy Optimization
1
Traditional skinning pipeline is labor intensive
2
Automatic Skinning via Constrained Energy Optimization Alec Jacobson - - PowerPoint PPT Presentation
Automatic Skinning via Constrained Energy Optimization Alec Jacobson - - PowerPoint PPT Presentation
Skinning: Real-time Shape Deformation Automatic Skinning via Constrained Energy Optimization Alec Jacobson Columbia University 1 Traditional skinning pipeline is labor intensive v v 0 = X w j ( v ) T j 1 j 2 H 2
SLIDE 2
SLIDE 3
Traditional skinning pipeline is labor intensive
H
3
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 4
Traditional skinning pipeline is labor intensive
H
4
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 5
Traditional skinning pipeline is labor intensive
H
5
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 6
Traditional skinning pipeline is labor intensive
H
6
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 7
Traditional skinning pipeline is labor intensive
H
7
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 8
Traditional skinning pipeline is labor intensive
H
8
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆ wj
SLIDE 9
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
Traditional skinning pipeline is labor intensive
H
9
Tj wj
SLIDE 10
Recent research automates each step
construct handles compute weights choose transforms
10
v0 = X
j2H
wj(v)Tj ✓ v 1 ◆
SLIDE 11
Recent research automates each step
construct handles compute weights choose transforms
11
SLIDE 12
Skeleton implies shape…
[Blair 1994] 12
SLIDE 13
…and shape implies skeleton
[Au et al. 2008] 13
SLIDE 14
State-of-the-art skeleton extraction techniques split into two camps
[Au et al. 2008] [Wang & Lee 2008] [Tagliasacchi et al. 2012] [Cao et al. 2010; Huang et al. 2013]
Surface flow Shape segmentation
Segmentation Inner skeleton
[Katz & Tal 2003] [Huang et al. 2009] [Bharaj et al. 2012]
14
SLIDE 15
Surface flow degenerates to skeleton approximating medial axis
15
SLIDE 16
Surface flow degenerates to skeleton approximating medial axis
16
SLIDE 17
Adjacency graph of rigid segmentation encodes topology of skeleton
[Huang et al. 2009]
Assume we know which parts will deform rigidly
17
SLIDE 18
But which parts will deform rigidly?
[Huang et al. 2009]
?
18
SLIDE 19
Eigenmodes of as-rigid-as-possible energy form a reasonable prior
[Huang et al. 2009] 19
SLIDE 20
Eigenmodes of as-rigid-as-possible energy form a reasonable prior
[Huang et al. 2009] 20
SLIDE 21
Eigenmodes of as-rigid-as-possible energy form a reasonable prior
[Huang et al. 2009] 21
SLIDE 22
Eigenmodes of as-rigid-as-possible energy form a reasonable prior
[Huang et al. 2009] 22
SLIDE 23
Skeleton extraction problems remain open
- Joint placement (center of rotation)?
- Guarantee bones are inside?
- Respect (intrinsic) symmetries?
23
SLIDE 24
+
Skeleton embedding assumes skeletal topology is known
[Baran & Popovic 2007] 24
SLIDE 25
+
Categorization of desirable qualities implies energy minimization
[Baran & Popovic 2007] 25
1. avoid short bone chains, 2. maintain angles between bones, 3. keep lengths of symmetric bones proportional, 4. avoid overlapping bones, 5. place special “feet” bones below others, 6. avoid zero-length bones, 7. maintain bone orientations, 8. place extremities close to surface, 9. avoid disparities in Euclidean and bone-path distances
- 10. …
SLIDE 26
+
Categorization of desirable qualities implies energy minimization
[Baran & Popovic 2007] 26
1. avoid short bone chains, 2. maintain angles between bones, 3. keep lengths of symmetric bones proportional, 4. avoid overlapping bones, 5. place special “feet” bones below others, 6. avoid zero-length bones, 7. maintain bone orientations, 8. place extremities close to surface, 9. avoid disparities in Euclidean and bone-path distances
- 10. …
E(V) =
9
X
i=1
λiEi(V)
SLIDE 27
Skeleton embedding problems remain open
- Directly measure
implied deformation?
- Partial skeletons?
27
Input partial skeleton
SLIDE 28
Similar problems arise for automatic cages
- Fully exterior?
- Tight fitting?
- Symmetric?
- “Well-placed”?
28
Interpenetrating mesh simplification Fully enclosing cage
SLIDE 29
Recent research automates each step
construct handles compute weights choose transforms
29
SLIDE 30
Weights should obtain a few basic qualities
30
SLIDE 31
Weights should obtain a few basic qualities
31
SLIDE 32
[Shepard 1968], [Schaefer et al. 2006], etc.
Inverse Euclidean distance weights are too crude, show obvious artifacts
32
weights optimized inside shape
wj(v) = 1 kci vk2
SLIDE 33
[Shepard 1968], [Schaefer et al. 2006], etc.
Inverse Euclidean distance weights are too crude, show obvious artifacts
33
weights optimized inside shape
wj(v) = 1 kci vk2
SLIDE 34
Generalize barycentric coordinates capture shape-awareness via low-resolution “cage”
34
v0 =
m
X
j=1
wj(v)c0
j
SLIDE 35
“Coordinate” or reproduction property defines any point in cage as weighted sum of vertices
35
“coordinates”
v0 =
m
X
j=1
wj(v)c0
j
v =
m
X
j=1
wj(v)cj
SLIDE 36
Coordinate-based deformation is a special case of linear blend skinning
36
“coordinates” restricted to translation
v0 =
m
X
j=1
wj(v)c0
j
v0 =
m
X
j=1
wj(v)Tj ✓ v 1 ◆
SLIDE 37
Coordinate-based deformation is a special case of linear blend skinning
37
“coordinates”
v0 =
m
X
j=1
wj(v)c0
j
v0 =
m
X
j=1
wj(v)
- I | c0
j − cj
✓ v 1 ◆
SLIDE 38
Coordinate-based deformation is a special case of linear blend skinning
38
“coordinates”
v0 =
m
X
j=1
wj(v)
- c0
j − cj + v
- v0 =
m
X
j=1
wj(v)c0
j
SLIDE 39
Coordinate-based deformation is a special case of linear blend skinning
39
“coordinates”
v0 =
m
X
j=1
wj(v)c0
j
v =
m
X
j=1
wj(v)cj v0 =
m
X
j=1
wj(v)
- c0
j − cj + v
SLIDE 40
Coordinate-based deformation is a special case of linear blend skinning
40
“coordinates”
v0 =
m
X
j=1
wj(v)c0
j
v =
m
X
j=1
wj(v)cj v0 =
m
X
j=1
wj(v)c0
j
SLIDE 41
Coordinate-based deformation is a special case of linear blend skinning
41
“coordinates”
“Higher order GBC” [Langer et al. 2008] are identical to linear blend skinning
v0 =
m
X
j=1
wj(v)c0
j
v =
m
X
j=1
wj(v)cj v0 =
m
X
j=1
wj(v)c0
j
SLIDE 42
Not all coordinates are created equal
42
SLIDE 43
Not all coordinates are created equal
43
SLIDE 44
Not all coordinates are created equal
44
SLIDE 45
Not all coordinates are created equal
No free lunch? Are some properties mutually exclusive?
45
SLIDE 46
Working only with cages is too restrictive…
46
SLIDE 47
Working only with cages is too restrictive…
47
SLIDE 48
Working only with cages is too restrictive…
48
SLIDE 49
Working only with cages is too restrictive…
49
SLIDE 50
Discontinuous projection onto surface can be smoothed out ˆ w
[Baran & Popović 2007]
Closest visible bone
SLIDE 51
Discontinuous projection onto surface can be smoothed out w ˆ w
[Baran & Popović 2007] 51
argmin
wj
Z
Ω
krwjk2 + hj(wj ˆ wj)2 dA
SLIDE 52
smoothness
Discontinuous projection onto surface can be smoothed out w ˆ w
[Baran & Popović 2007] 52
argmin
wj
Z
Ω
krwjk2 + hj(wj ˆ wj)2 dA
SLIDE 53
argmin
wj
Z
Ω
krwjk2 + hj(wj ˆ wj)2 dA
Discontinuous projection onto surface can be smoothed out w ˆ w
[Baran & Popović 2007]
“data”
53
SLIDE 54
Discontinuous projection onto surface can be smoothed out w ˆ w
[Baran & Popović 2007] 54
argmin
wj
Z
Ω
krwjk2 + hj(wj ˆ wj)2 dA
SLIDE 55
Gradient energy weights not smooth at handles
55
argmin
wj
Z
Ω
krwjk2 dA argmin
wj
Z
Ω
(∆wj)2 dA
SLIDE 56
argmin
wj
Z
Ω
krwjk2 dA
Gradient energy weights not smooth at handles
56
argmin
wj
Z
Ω
(∆wj)2 dA
SLIDE 57
Gradient energy weights not smooth at handles
57
∆wj = 0 ∆2wj = 0
SLIDE 58
Any single handle type is too restrictive…
58
SLIDE 59
Any single handle type is too restrictive…
59
SLIDE 60
Each handle type has a specific task, more than just a different modeling metaphor
60
SLIDE 61
Each handle type has a specific task, more than just a different modeling metaphor
61
SLIDE 62
Non-negative, local weights are mandatory
62
[Botsch & Kobbelt 2004]
0 ≤ wj ≤ 1 argmin
wj
Z
Ω
(∆wj)2 dA
SLIDE 63
Non-negative, local weights are mandatory
63
[Botsch & Kobbelt 2004]
0 ≤ wj ≤ 1 argmin
wj
Z
Ω
(∆wj)2 dA
SLIDE 64
64
Spurious extrema cause distracting artifacts
local max local min
0 ≤ wj ≤ 1
SLIDE 65
65
Must explicitly prohibit spurious extrema
local max local min
wj is “monotonic”
SLIDE 66
66
Previous methods fail in one way or another
No free lunch? Are some properties mutually exclusive?
Euclidean ∆wj ∆2wj
smooth ✓
- ✓
non-negative ✓ ✓
- shape-aware
- ✓
✓ local
- /✓
- monotonic
- ✓
- arbitrary handles
- ✓
✓
[Shepard 1968, Sibson 1980, Schaefer et al. 2006] [Baran & Popovic 2007, Joshi et al. 2007] [Botsch & Kobbelt 2004, Sorkine et al. 2004, Finch et al. 2011]
SLIDE 67
67
Euclidean ∆wj ∆2wj
smooth ✓
- ✓
non-negative ✓ ✓
- shape-aware
- ✓
✓ local
- /✓
- monotonic
- ✓
- arbitrary handles
- ✓
✓
[Shepard 1968, Sibson 1980, Schaefer et al. 2006] [Baran & Popovic 2007, Joshi et al. 2007] [Botsch & Kobbelt 2004, Sorkine et al. 2004, Finch et al. 2011]
Previous methods fail in one way or another
No free lunch? Are some properties mutually exclusive?
SLIDE 68
Constrained optimization ensures satisfaction of all properties
68
+ shape-aware + smoothness
[Botsch & Kobbelt 2004, Sorkine et al. 2004, Joshi & Carr 2008, Jacobson et al. 2010, Finch et al. 2011, Andrews et al. 2011]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV
SLIDE 69
Constrained optimization ensures satisfaction of all properties
69
+ shape-aware + smoothness + arbitrary handles
[Botsch & Kobbelt 2004, Sorkine et al. 2004, Joshi & Carr 2008, Jacobson et al. 2010, Finch et al. 2011, Andrews et al. 2011]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV wj(v) = 1 v ∈ hj, v ∈ hk linear on cage facets
SLIDE 70
Constrained optimization ensures satisfaction of all properties
70
+ shape-aware + smoothness + arbitrary handles + non-negativity
[Jacobson et al. 2011]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV 0 ≤ wj ≤ 1,
m
X
j=1
wj = 1
SLIDE 71
Constrained optimization ensures satisfaction of all properties
71
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality
[Jacobson et al. 2011]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV 0 ≤ wj ≤ 1,
m
X
j=1
wj = 1
SLIDE 72
[Rustamov 2011]
Constrained optimization ensures satisfaction of all properties
72
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV kwk1 = 1
SLIDE 73
[Rustamov 2011]
Constrained optimization ensures satisfaction of all properties
73
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV kwk1 = 1 !
m
X
j=1
|wj| = 1
SLIDE 74
[Rustamov 2011]
Constrained optimization ensures satisfaction of all properties
74
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV kwk1 = 1 !
m
X
j=1
|wj| = 1 !
m
X
j=1
wj = 1, 0 wj 1
SLIDE 75
Constrained optimization ensures satisfaction of all properties
75
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality + monotonicity
[Weinkauf et al. 2011, Jacobson et al. 2012, Günther et al. 2014]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV rwj · ruj > 0
SLIDE 76
76
Previous methods fail in one way or another
Euclidean ∆wj = u ∆2wj
smooth ✓
- ✓
non-negative ✓ ✓
- shape-aware
- ✓
✓ local
- /✓
- monotonic
- ✓
- arbitrary handles
- ✓
✓
[Shepard 1968, Sibson 1980, Schaefer et al. 2006] [Baran & Popovic 2007, Joshi et al. 2007] [Botsch & Kobbelt 2004, Sorkine et al. 2004, Finch et al. 2011]
SLIDE 77
Constrained optimization ensures satisfaction of all properties
77
+ shape-aware + smoothness + arbitrary handles + non-negativity + locality + monotonicity
[Weinkauf et al. 2011, Jacobson et al. 2012, Günther Günther et al. 2014 et al. 2014]
argmin
wj,j=1,...,m m
X
j=1
Z
Ω
(∆wj)2 dV rwj · ruj > 0
SLIDE 78
78
Weights retain nice properties in 3D
SLIDE 79
79
Weights retain nice properties in 3D
SLIDE 80
Variational formulation allows additional, problem-specific constraints
80
SLIDE 81
Variational formulation allows additional, problem-specific constraints
81
SLIDE 82
Energy encodes prior on deformation, could swap smoothness for rigidity
82 Right: [Kavan & Sorkine 2012] Left, middle: smoothness energy
SLIDE 83
Energy encodes prior on deformation, could swap smoothness for rigidity
83 Right: [Kavan & Sorkine 2012] Left, middle: smoothness energy
SLIDE 84
FEM struggles with meshes in the wild
84
SLIDE 85
FEM struggles with meshes in the wild
85 [Jacobson et al. 2013]
SLIDE 86
FEM struggles with meshes in the wild
86 [Jacobson et al. 2013] [Dionne & de Lasa 2013]
SLIDE 87
FEM struggles with meshes in the wild
87 [Jacobson et al. 2013] [Bharaj et al. 2012] [Dionne & de Lasa 2013]
SLIDE 88
Recent research automates each step
construct handles compute weights choose transforms
88
SLIDE 89
Good weights are not enough
89
Rotation chosen ignorant of shape’s deformation
[Jacobson et al. 2011], cf. traditional IK
Rotation optimized for best deformation
[Jacobson et al. 2011]
SLIDE 90
Good weights are not enough
90
Rotation chosen ignorant of shape’s deformation
[Jacobson et al. 2011], cf. traditional IK
Rotation optimized for best deformation
[Jacobson et al. 2011]
SLIDE 91
Good weights are not enough
91
Rotation chosen ignorant of shape’s deformation
[Jacobson et al. 2011], cf. traditional IK
Rotation optimized for best deformation
[Jacobson et al. 2011]
SLIDE 92
Shape-aware IK finds best transformations as measured by shape’s deformation
92
SLIDE 93
Shape-aware IK finds best transformations as measured by shape’s deformation
93
SLIDE 94
Skinning is a coherent subspace of deformation
94
full space: #V x 3 DOFs
[traditional physics, modeling]
SLIDE 95
skinning space: #B x 12 DOFs
[Der et al. 2006, Huang et al. 2006, Au et al. 2007, Jacobson et al. 2012]
Skinning is a coherent subspace of deformation
SLIDE 96
Skinning is a coherent subspace of deformation
96
modal space: 50 linear modes
[Hildebrandt et al. 2011, Barbic et al. 2012]
SLIDE 97
Skinning is a coherent subspace of deformation
97
inherits semantics from rig
SLIDE 98
Regularity and parsimony allows very fast non-linear energy optimization at runtime
98 [Jacobson et al. 2012]
SLIDE 99
Same idea applies for real-time physically based dynamics
99 [Faure et al. 2011, Jacobson 2013, Liu et al. 2013, Bouaziz et al. 2014]
SLIDE 100
Posing still requires more research
- Safe contacts, collisions in real time
- Leverage amassed data: semantics
- Better virtual and physical interfaces
100
SLIDE 101
Recent research automates each step
construct handles compute weights choose transforms
101
SLIDE 102
Acknowledgements…
- Olga Sorkine-Hornung
- Eitan Grinspun
- Ilya Baran, David Günther, Qi-Xang Huang, Jan
Reininghaus, Jovan Popovic, Tino Weinkauf
- Intel Doctoral Fellowship
102
SLIDE 103
Alec Jacobson
Columbia University Skinning: Real-time Shape Deformation
Automatic Skinning via Constrained Energy Optimization
103