Computing and Processing Correspondences with Functional Maps Maks - - PowerPoint PPT Presentation
Computing and Processing Correspondences with Functional Maps Maks Ovsjanikov 1 Etienne Corman 1 Michael Bronstein 2 , 3 , 4 a 2 Mirela Ben-Chen 5 Leonidas Guibas 6 Emanuele Rodol` eric Chazal 7 Alexander Bronstein 5 , 2 , 3 Fr ed 1 Ecole
1/66
Maks Ovsjanikov1 Etienne Corman1 Michael Bronstein2,3,4 Emanuele Rodol` a2 Mirela Ben-Chen5 Leonidas Guibas6 Fr´ ed´ eric Chazal7 Alexander Bronstein5,2,3
1Ecole Polytechnique 2USI Lugano 3Tel Aviv University 4Intel 5Technion 4Stanford University 7INRIA
SIGGRAPH Asia Tutorial, Macau, 6 December 2016
2/66
3/66
Functional correspondence matrix C expressed in the Laplacian eigenbases
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
3/66
Functional correspondence matrix C expressed in the Laplacian eigenbases
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
4/66
M N φN
2
φN
3
φN
4
φN
5
φN
6
φM
2
φM
3
φM
4
φM
5
φM
6
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
4/66
M N φN
2
φN
3
φN
4
φN
5
φN
6
φM
2
φM
3
φM
4
φM
5
φM
6
Isometric manifolds with simple spectrum: sign ambiguity TF φM
i
= ±φN
i
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
4/66
M N φN
2
φN
3
φN
4
φN
5
φN
6
φM
2
φM
3
φM
4
φM
5
φM
6
Isometric manifolds with simple spectrum: sign ambiguity TF φM
i
= ±φN
i
General spectrum: ambiguous rotation of eigenspace
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
4/66
M N φN
2
φN
3
φN
4
φN
5
φN
6
φM
2
φM
3
φM
4
φM
5
φM
6
Isometric manifolds with simple spectrum: sign ambiguity TF φM
i
= ±φN
i
General spectrum: ambiguous rotation of eigenspace Non-isometric manifolds: eigenvectors can differ dramatically in
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
4/66
M N φN
2
φN
3
φN
4
φN
5
φN
6
φM
2
φM
3
φM
4
φM
5
φM
6
Isometric manifolds with simple spectrum: sign ambiguity TF φM
i
= ±φN
i
General spectrum: ambiguous rotation of eigenspace Non-isometric manifolds: eigenvectors can differ dramatically in
Incompatibilities tend to increase with frequency
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
5/66
M N f g ⇡ a1 + a2 + · · · + ak ⇡ b1 + b2 + · · · + bk # TF # φM
1
φM
2
φM
k
φN
1
φN
2
φN
k
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
5/66
M N f g ⇡ a1 + a2 + · · · + ak ⇡ b1 + b2 + · · · + bk # TF # φM
1
φM
2
φM
k
φN
1
φN
2
φN
k
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
5/66
M N f g ⇡ ˆ a1 + ˆ a2 + · · · + ˆ ak ⇡ ˆ b1 + ˆ b2 + · · · + ˆ bk # TF # ˆ φM
1
ˆ φM
2
ˆ φM
k
ˆ φN
1
ˆ φN
2
ˆ φN
k
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
5/66
M N f g ⇡ ˆ a1 + ˆ a2 + · · · + ˆ ak ⇡ ˆ b1 + ˆ b2 + · · · + ˆ bk # TF # ˆ φM
1
ˆ φM
2
ˆ φM
k
ˆ φN
1
ˆ φN
2
ˆ φN
k
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij)
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: δij = hˆ φM
i , ˆ
φM
j iL2(M) = k0
X
l,m=1
plipmjhφM
l , φM m iL2(M)
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: δij = hˆ φM
i , ˆ
φM
j iL2(M) = k0
X
l=1
pliplj
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: δij = hˆ φM
i , ˆ
φM
j iL2(M) = k0
X
l=1
pliplj = (P>P)ij
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I Approximate eigenbasis: approximately diagonalizes the Laplacian hˆ φM
i , ∆ˆ
φM
j iL2(M) = k0
X
l,m=1
plipmjhφM
l , ∆φM m iL2(M)
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I Approximate eigenbasis: approximately diagonalizes the Laplacian hˆ φM
i , ∆ˆ
φM
j iL2(M) = k0
X
l,m=1
plipmjλmhφM
l , φM m iL2(M)
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I Approximate eigenbasis: approximately diagonalizes the Laplacian hˆ φM
i , ∆ˆ
φM
j iL2(M) = k0
X
l=1
plipljλl
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I Approximate eigenbasis: approximately diagonalizes the Laplacian hˆ φM
i , ∆ˆ
φM
j iL2(M) = k0
X
l=1
plipljλl = (P>ΛM,k0P)ij
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
6/66
Find a new pair of approximate orthonormal eigenbases ˆ φM
i
=
k0
X
j=1
pjiφM
j
ˆ φN
i
=
k0
X
j=1
qjiφN
j
i = 1, . . . , k parametrized by k0 ⇥ k matrices P = (pij) and Q = (qij) Coupling: P>A ⇡ Q>B Orthonormality: P>P = I and Q>Q = I Approximate eigenbasis: approximately diagonalizes the Laplacian hˆ φM
i , ∆ˆ
φM
j iL2(M) = k0
X
l=1
pliplj = (P>ΛM,k0P)ij ⇡ 0, i 6= j
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
7/66
min
P,Q
s.t. P>P = I Q>Q = I
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
7/66
min
P,Q
s.t. P>P = I Q>Q = I Off-diagonal elements penalty off(X) = P
i6=j x2 ij
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
7/66
min
P,Q
s.t. P>P = I Q>Q = I Off-diagonal elements penalty off(X) = P
i6=j x2 ij
Dirichlet energy off(X) = trace(X) for k0 > k
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
7/66
min
P,Q
F
s.t. P>P = I Q>Q = I Off-diagonal elements penalty off(X) = P
i6=j x2 ij
Dirichlet energy off(X) = trace(X) for k0 > k If Frobenius norm is used and k0 = k, due to rotation invariance C = QP> is the functional correspondence matrix
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
7/66
min
P,Q
s.t. P>P = I Q>Q = I Off-diagonal elements penalty off(X) = P
i6=j x2 ij
Dirichlet energy off(X) = trace(X) for k0 > k If Frobenius norm is used and k0 = k, due to rotation invariance C = QP> is the functional correspondence matrix Robust norm kXk2,1 = P
j kxjk2 allows coping with outliers
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
8/66
Isometric Elements of P>ΛM,k0P and Q>ΛN ,k0Q
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
8/66
Isometric Non-isometric Elements of P>ΛM,k0P and Q>ΛN ,k0Q
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
9/66
Mesh with 8.5K vertices Mesh with 850 vertices
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
10/66
Mesh with 8.5K vertices Point cloud with 850 vertices
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
11/66
Functional correspondence matrix C expressed in standard Laplacian eigenbases
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
11/66
Functional correspondence matrix C expressed in coupled approximate eigenbases
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013
12/66
M1 M2 Mp CijAi ⇡ Aj Mi Mj Kovnatsky, Bronstein2, Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016
12/66
M1 M2 Mp P>
i Ai ⇡ P> j Aj
Mi Mj Kovnatsky, Bronstein2, Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016
12/66
P1 P2 Pp Pi Pj M1 M2 Mp P>
i Ai ⇡ P> j Aj
Mi Mj Kovnatsky, Bronstein2, Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016
13/66
min
P1,...,Pp p
X
i=1
trace(P>
i ΛMiPi) + µ
X
i6=j
kP>
i Ai P> j Ajk
s.t. P>
i Pi = I
‘Synchronization problem’ Matrices P1, . . . , Pp orthogonally align the p eigenbases
Kovnatsky, Bronstein2, Glashoff, Kimmel 2013; Kovnatsky, Glashoff, Bronstein 2016
14/66
15/66
P
15/66
P
16/66
min
x2R3 x>Ax
s.t. x>x = 1
Minimization of a quadratic function on the sphere
16/66
min
x2S(3,1) x>Ax
Minimization of a quadratic function on the sphere
17/66
X(k) X(k+1) S Absil et al. 2009
17/66
X(k) rf(X(k)) PX(k) rSf(X(k)) TX(k)S S Absil et al. 2009
17/66
X(k) rf(X(k)) PX(k) α(k)rSf(X(k)) TX(k)S S Absil et al. 2009
17/66
X(k) rf(X(k)) PX(k) α(k)rSf(X(k)) RX(k) X(k+1) TX(k)S S Absil et al. 2009
18/66
repeat Compute extrinsic gradient rf(X(k)) Projection: rSf(X(k)) = PX(k)(rf(X(k))) Compute step size α(k) along the descent direction rSf(X(k)) Retraction: X(k+1) = RX(k)(α(k)rSf(X(k))) k k + 1 until convergence;
Absil et al. 2009; Boumal et al. 2014
18/66
repeat Compute extrinsic gradient rf(X(k)) Projection: rSf(X(k)) = PX(k)(rf(X(k))) Compute step size α(k) along the descent direction rSf(X(k)) Retraction: X(k+1) = RX(k)(α(k)rSf(X(k))) k k + 1 until convergence; Projection P and retraction R operators are manifold-dependent
Absil et al. 2009; Boumal et al. 2014
18/66
repeat Compute extrinsic gradient rf(X(k)) Projection: rSf(X(k)) = PX(k)(rf(X(k))) Compute step size α(k) along the descent direction rSf(X(k)) Retraction: X(k+1) = RX(k)(α(k)rSf(X(k))) k k + 1 until convergence; Projection P and retraction R operators are manifold-dependent Typically expressed in closed form
Absil et al. 2009; Boumal et al. 2014
18/66
repeat Compute extrinsic gradient rf(X(k)) Projection: rSf(X(k)) = PX(k)(rf(X(k))) Compute step size α(k) along the descent direction rSf(X(k)) Retraction: X(k+1) = RX(k)(α(k)rSf(X(k))) k k + 1 until convergence; Projection P and retraction R operators are manifold-dependent Typically expressed in closed form “Black box”: need to provide only f(X) and gradient rf(X)
Absil et al. 2009; Boumal et al. 2014
19/66
P
2
19/66
P
smooth
non-smooth
20/66
min
X2S(n,k)
f(X) | {z }
smooth
+ g(X) | {z }
non-smooth
Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, Bronstein 2016
20/66
min
X2S(n,k) Z2Rn⇥k
f(X) | {z }
smooth
+ g(Z) |{z}
non-smooth
s.t. Z = X
Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, Bronstein 2016
20/66
min
X2S(n,k) Z2Rn⇥k
f(X) | {z }
smooth
+ g(Z) |{z}
non-smooth
s.t. Z = X Apply the method of multipliers only to the constraint Z = X min
X2S(n,k) Z2Rn⇥k
f(X) + g(Z) + ρ
2kX Z + Uk2 F
Solve alternating w.r.t. X and Z and updating U U + X Z
Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, Bronstein 2016
20/66
min
X2S(n,k) Z2Rn⇥k
f(X) | {z }
smooth
+ g(Z) |{z}
non-smooth
s.t. Z = X Apply the method of multipliers only to the constraint Z = X min
X2S(n,k) Z2Rn⇥k
f(X) + g(Z) + ρ
2kX Z + Uk2 F
Solve alternating w.r.t. X and Z and updating U U + X Z Problem breaks into Smooth manifold optimization sub-problem w.r.t. X, and Non-smooth unconstrained sub-problem w.r.t. Z
Hestenes 1969; Powell 1969; Kovnatsky, Glashoff, Bronstein 2016
21/66
Initialize k 1, Z(1) = X(1), U(1) = 0. repeat X-step: X(k+1) = argmin
X2S
f(X) + ρ
2kX Z(k) + U(k)k2 F
Z-step: Z(k+1) = argmin
Z
g(Z) + ρ
2kX(k+1) Z + U(k)k2 F
Update U(k+1) = U(k) + X(k+1) Z(k+1) k k + 1 until convergence;
Kovnatsky, Glashoff, Bronstein 2016
21/66
Initialize k 1, Z(1) = X(1), U(1) = 0. repeat X-step: X(k+1) = argmin
X2S
f(X) + ρ
2kX Z(k) + U(k)k2 F
Z-step: Z(k+1) = argmin
Z
g(Z) + ρ
2kX(k+1) Z + U(k)k2 F
Update U(k+1) = U(k) + X(k+1) Z(k+1) k k + 1 until convergence; Solver/number of optimization iterations in X- and Z-steps
Kovnatsky, Glashoff, Bronstein 2016
21/66
Initialize k 1, Z(1) = X(1), U(1) = 0. repeat X-step: X(k+1) = argmin
X2S
f(X) + ρ
2kX Z(k) + U(k)k2 F
Z-step: Z(k+1) = argmin
Z
g(Z) + ρ
2kX(k+1) Z + U(k)k2 F
Update U(k+1) = U(k) + X(k+1) Z(k+1) k k + 1 until convergence; Solver/number of optimization iterations in X- and Z-steps X-step and X-step in some problems have a closed form
Kovnatsky, Glashoff, Bronstein 2016
21/66
Initialize k 1, Z(1) = X(1), U(1) = 0. repeat X-step: X(k+1) = argmin
X2S
f(X) + ρ
2kX Z(k) + U(k)k2 F
Z-step: Z(k+1) = argmin
Z
g(Z) + ρ
2kX(k+1) Z + U(k)k2 F
Update U(k+1) = U(k) + X(k+1) Z(k+1) k k + 1 until convergence; Solver/number of optimization iterations in X- and Z-steps X-step and X-step in some problems have a closed form Parameter ρ > 0 can be chosen fixed or adapted
Kovnatsky, Glashoff, Bronstein 2016
22/66
Robust (MADMM) Least squares
Correspondence computed with data containing 10% outliers
Kovnatsky, Glashoff, Bronstein 2016
23/66
24/66
φ2 φ3 φ4 φ5 φ6 φ7 φ8 φ9 ψ2 ψ3 ψ4 ψ5 ψ6 ψ7 ψ8 ψ9 ζ2 ζ3 ζ4 ζ5 ζ6 ζ7 ζ8 ζ9
Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions)
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
25/66
Functional correspondence matrix C
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
26/66
∆M ∆ ¯
M
∆M ∆ ¯
M
φ1 φ2 φ3 φ1 φ2 φ3 ¯ φ1 ¯ φ2 ¯ φ3 ¯ φ1 ¯ φ2 ¯ φ3 M ¯ M λ1 λ2 λ3 ¯ λ1 ¯ λ2 ¯ λ3 λ1 λ2 λ3 ¯ λ1 ¯ λ2 ¯ λ3
Ignoring boundary interaction: disjoint parts (block-diagonal matrix) Eigenvectors = Mixture of eigenvectors of the parts
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
27/66
10 20 30 40 50 0.00 2.00 4.00 6.00 8.00 ·102 eigenvalue number r k N M
Slope r
k ≈ |M| |N | (depends on the area of the cut)
Consistent with Weyl’s law
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
28/66
∆M ∆ ¯
M
∆M+tDM ∆ ¯ M+tD ¯ M
tE tE> M ¯ M Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
29/66
Value of f
10 20
Eigenvector perturbation depends on length and position of the boundary Perturbation strength ≤ c R
∂M f(m)dm, where
f(m) =
n
X
i,j=1 j6=i
✓φi(m)φj(m) λi λj ◆2
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
30/66
Model shape M Query shape N Part M ✓ M ⇡ isometric to N Data f1, . . . , fq 2 L2(N) g1, . . . , gq 2 L2(M) Partial functional map (TF fi)(m) ⇡ gi(m), m 2 M
Model M Query N Part M TF Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
30/66
Model shape M Query shape N Part M ✓ M ⇡ isometric to N Data f1, . . . , fq 2 L2(N) g1, . . . , gq 2 L2(M) Partial functional map TF fi ⇡ gi · v, v : M ! [0, 1]
Model M Query N Part v TF Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
30/66
Model shape M Query shape N Part M ✓ M ⇡ isometric to N Data f1, . . . , fq 2 L2(N) g1, . . . , gq 2 L2(M) Partial functional map CA ⇡ B(v), v : M ! [0, 1] A =
i , fjiL2(N )
=
i , gj · viL2(M)
Query N Part v C Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
30/66
Model shape M Query shape N Part M ✓ M ⇡ isometric to N Data f1, . . . , fq 2 L2(N) g1, . . . , gq 2 L2(M) Partial functional map CA ⇡ B(v), v : M ! [0, 1] A =
i , fjiL2(N )
=
i , gj · viL2(M)
Query N Part v C
Optimization problem w.r.t. correspondence C and part v min
C,v kCA B(v)k2,1 + ρcorr(C) + ρpart(v)
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
31/66
min
C,v kCA B(v)k2,1 + ρcorr(C) + ρpart(v)
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
31/66
min
C,v kCA B(v)k2,1 + ρcorr(C) + ρpart(v)
Part regularization
Area preservation R
M v(m)dx ≈ |N|
Spatial regularity = small boundary length (Mumford-Shah)
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein2 2008
31/66
min
C,v kCA B(v)k2,1 + ρcorr(C) + ρpart(v)
Part regularization
Area preservation R
M v(m)dx ≈ |N|
Spatial regularity = small boundary length (Mumford-Shah)
Correspondence regularization
Slanted diagonal structure Approximate ortho-projection (C>C)i6=j ≈ 0 rank(C) ≈ r
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein2 2008
32/66
C W C>C
20 40 60 80 100 2 4
singular values Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
33/66
C-step: fix v⇤, solve for correspondence C min
C kCA B(v⇤)k2,1 + ρcorr(C)
v-step: fix C⇤, solve for part v min
v
kC⇤A B(v)k2,1 + ρpart(v)
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
33/66
C-step: fix v⇤, solve for correspondence C min
C kCA B(v⇤)k2,1 + ρcorr(C)
v-step: fix C⇤, solve for part v min
v
kC⇤A B(v)k2,1 + ρpart(v)
Iteration 1 2 3 4 Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
34/66
20 40 60 80 100 104 105 106 107 108 109 1010 Iteration Energy C-step v-step 5 10 15 20 25 Time (sec.) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
35/66
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
35/66
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
35/66
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
35/66
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
36/66
0.05 0.1 0.15 0.2 0.25 20 40 60 80 100 50 100 150 50 100 150 Geodesic error % Correspondences
PFM
Correspondence performance for different basis size k
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
37/66
0.05 0.1 0.15 0.2 0.25 20 40 60 80 100
Geodesic Error % Correspondences
Cuts
0.05 0.1 0.15 0.2 0.25
Geodesic Error
Holes PFM RF IM EN GT
SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 (PFM); Sahillio˘ glu, Yemez 2012 (IM); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF)
38/66
20 40 60 80 0.2 0.4 0.6 0.8 1
Partiality (%) Mean geodesic error
Cuts
20 40 60 80
Partiality (%)
Holes PFM RF IM EN GT
SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 (PFM); Sahillio˘ glu, Yemez 2012 (IM); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF)
39/66
Correspondence Correspondence error
10%
Boscaini, Masci, Rodol` a, Bronstein 2016
40/66
Correspondence Correspondence error
10%
Boscaini, Masci, Rodol` a, Bronstein 2016
41/66
7.5%
Pointwise geodesic error (in % of geodesic diameter) Monti, Boscaini, Masci, Rodol` a, Svoboda, Bronstein 2016
42/66
Reference Correspondence visualization (similar colors encode corresponding points) Training: FAUST / Testing: FAUST Monti, Boscaini, Masci, Rodol` a, Svoboda, Bronstein 2016
42/66
Reference Correspondence visualization (similar colors encode corresponding points) Training: FAUST / Testing: SCAPE+TOSCA Monti, Boscaini, Masci, Rodol` a, Svoboda, Bronstein 2016
43/66
Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016
43/66
Litany, Rodol` a, Bronstein2, Cremers 2016
44/66
N N M M
CNN slant / |N| |N| CMM slant / |M| |M|
Litany, Rodol` a, Bronstein2, Cremers 2016
44/66
N N M M
CNM = CMMCN MCNN slant / |N| |N| |M| |M|
Litany, Rodol` a, Bronstein2, Cremers 2016
44/66
N N M M
CNM = CMMCN MCNN slant / |N| |N| |M| |M| = |M| |N|
Litany, Rodol` a, Bronstein2, Cremers 2016
45/66
Litany, Rodol` a, Bronstein2, Cremers 2016
45/66
Litany, Rodol` a, Bronstein2, Cremers 2016
Litany, Bronstein2 2012
47/66
Input Model M Parts N1, . . . , Np Output Segmentation Mi ✓ M Located parts Ni ✓ Ni Clutter N c
i
Missing parts M0 Correspondences TFi M1 M2 TF1 TF2 N2 N1 N c
2
N c
1
M0
Model M Part N2 Part N1 Litany, Rodol` a, Bronstein2, Cremers 2016
47/66
Input Model M Parts N1, . . . , Np Output Segmentation ui :M![0, 1] Located parts vi :Ni![0, 1] Clutter 1 vi Missing parts u0 Correspondences Ci u1 u2 C1 C2 v2 v1 u0
Model M Part N2 Part N1 Litany, Rodol` a, Bronstein2, Cremers 2016
48/66
min
Ci,ui,vi p
X
i=1
kCiAi(vi) B(ui)k2,1 +
p
X
i=0
ρpart(ui, vi) +
p
X
i=1
ρcorr(Ci) s.t.
p
X
i=0
ui = 1
Litany, Rodol` a, Bronstein2, Cremers 2016
49/66
Outer iteration 1
Litany, Rodol` a, Bronstein2, Cremers 2016
49/66
Outer iteration 2
Litany, Rodol` a, Bronstein2, Cremers 2016
49/66
Outer iteration 3
Litany, Rodol` a, Bronstein2, Cremers 2016
50/66
80 90 100 110 120 130 140 150 160
Iteration number Time (sec)
30 32 34 36 38 40 42 44 46 48
Litany, Rodol` a, Bronstein2, Cremers 2016
51/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT)
Litany, Rodol` a, Bronstein2, Cremers 2016
51/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT) Segmentation
Litany, Rodol` a, Bronstein2, Cremers 2016
51/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter No Missing part No Data term Dense (SHOT) Correspondence
Litany, Rodol` a, Bronstein2, Cremers 2016
52/66
Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Segmentation
Litany, Rodol` a, Bronstein2, Cremers 2016
52/66
Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Correspondence
Litany, Rodol` a, Bronstein2, Cremers 2016
53/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT)
Litany, Rodol` a, Bronstein2, Cremers 2016
53/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Segmentation
Litany, Rodol` a, Bronstein2, Cremers 2016
53/66
Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Correspondence
Litany, Rodol` a, Bronstein2, Cremers 2016
54/66
Cosmo, Rodol` a, Torsello, Bronstein 2016
55/66
M S1 C1 C>
1 C1
S2 C2 C>
2 C2
S3 C3 C>
3 C3
S4 C4 C>
4 C4
θ = 0.6 θ = 0.8 θ = 1.1 θ = 1.6
Slanted diagonal structure (angle θ has to be estimated) C>C has sparse diagonal Good descriptor + initialization is crucial! (learned descriptor)
Cosmo, Rodol` a, Torsello, Bronstein 2016
56/66
Cosmo, Rodol` a, Torsello, Bronstein 2016
56/66
Cosmo, Rodol` a, Torsello, Bronstein 2016
56/66
Cosmo, Rodol` a, Torsello, Bronstein 2016
57/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , φN j iL2(N) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Litany, Rodol` a, Bronstein2 2016
57/66
M N N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , φN j iL2(N) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Litany, Rodol` a, Bronstein2 2016
57/66
M N N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , v · φN j iL2(N ) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Litany, Rodol` a, Bronstein2 2016
57/66
M N N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , v · φN j iL2(N ) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Complicated alternating optimization w.r.t. v and C
Litany, Rodol` a, Bronstein2 2016
57/66
M N N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , v · φN j iL2(N ) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Complicated alternating optimization w.r.t. v and C Explicit spatial model v of the part
Litany, Rodol` a, Bronstein2 2016
57/66
M N N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
φN
1
φN
2
φN
3
φN
4
φN
5
φN
6
Slanted diagonal: hTF φM
i , v · φN j iL2(N ) ⇡ ±δi,πj
πj ⇡ j |N |
|M|
Complicated alternating optimization w.r.t. v and C Explicit spatial model v of the part ) O(n) complexity!
Litany, Rodol` a, Bronstein2 2016
58/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
ˆ φN
1
ˆ φN
2
ˆ φN
3
ˆ φN
4
ˆ φN
5
ˆ φN
6
Find a new basis {ˆ φN
i }k i=1 such that
hTF φM
i , ˆ
φN
j iL2(N ) ⇡ δij
Litany, Rodol` a, Bronstein2 2016
58/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
ˆ φN
1
ˆ φN
2
ˆ φN
3
ˆ φN
4
ˆ φN
5
ˆ φN
6
Find a new basis {ˆ φN
i }k i=1 such that
hTF φM
i , Pk l=1 qljφN l iL2(N ) ⇡ δij
Litany, Rodol` a, Bronstein2 2016
58/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
ˆ φN
1
ˆ φN
2
ˆ φN
3
ˆ φN
4
ˆ φN
5
ˆ φN
6
Find a new basis {ˆ φN
i }k i=1 such that
hTF φM
i , Pk l=1 qljφN l iL2(N ) ⇡ δij
New basis functions {ˆ φN
i }k i=1 are localized on N
Litany, Rodol` a, Bronstein2 2016
58/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
ˆ φN
1
ˆ φN
2
ˆ φN
3
ˆ φN
4
ˆ φN
5
ˆ φN
6
Find a new basis {ˆ φN
i }k i=1 such that
hTF φM
i , Pk l=1 qljφN l iL2(N ) ⇡ δij
New basis functions {ˆ φN
i }k i=1 are localized on N
Optimization over coefficients Q = (qij)
Litany, Rodol` a, Bronstein2 2016
58/66
M N TF φM
1
φM
2
φM
3
φM
4
φM
5
φM
6
ˆ φN
1
ˆ φN
2
ˆ φN
3
ˆ φN
4
ˆ φN
5
ˆ φN
6
Find a new basis {ˆ φN
i }k i=1 such that
hTF φM
i , Pk l=1 qljφN l iL2(N ) ⇡ δij
New basis functions {ˆ φN
i }k i=1 are localized on N
Optimization over coefficients Q = (qij) ) O(k2) complexity!
Litany, Rodol` a, Bronstein2 2016
59/66
Ak⇥q Πk⇥k Bk⇥q Litany, Rodol` a, Bronstein2 2016
59/66
Ar⇥q Πr⇥k Bk⇥q
Π is k ⇥ r partial permutation with elements (πi, i) = ±1 and r ⇡ k |M|
|N |
Litany, Rodol` a, Bronstein2 2016
59/66
Ar⇥q Πr⇥k Bk⇥q
Π is k ⇥ r partial permutation with elements (πi, i) = ±1 and r ⇡ k |M|
|N |
Relax Π ⇡ Q> s.t. Q>Q = I (k ⇥ r ortho-projection) min
Q
trace(Q>ΛN ,kQ) + µkAr Q>Bkk2,1 s.t. Q>Q = I
Litany, Rodol` a, Bronstein2 2016; Kovnatsky, Glashoff, Bronstein2, Kimmel 2013 (Joint diag)
59/66
Ar⇥q Πr⇥k Bk⇥q
Π is k ⇥ r partial permutation with elements (πi, i) = ±1 and r ⇡ k |M|
|N |
Relax Π ⇡ Q> s.t. Q>Q = I (k ⇥ r ortho-projection) min
Q
trace(Q>ΛN ,kQ) + µkAr Q>Bkk2,1 s.t. Q>Q = I
Optimization on the Stiefel manifold with k2 variables
Litany, Rodol` a, Bronstein2 2016; Kovnatsky, Glashoff, Bronstein2, Kimmel 2013 (Joint diag)
59/66
Ar⇥q Πr⇥k Bk⇥q
Π is k ⇥ r partial permutation with elements (πi, i) = ±1 and r ⇡ k |M|
|N |
Relax Π ⇡ Q> s.t. Q>Q = I (k ⇥ r ortho-projection) min
Q
trace(Q>ΛN ,kQ) + µkAr Q>Bkk2,1 s.t. Q>Q = I
Non-smooth optimization on the Stiefel manifold with k2 variables
Litany, Rodol` a, Bronstein2 2016; Kovnatsky, Glashoff, Bronstein2, Kimmel 2013 (Joint diag); Kovnatsky, Glashoff, Bronstein 2016 (MADMM)
59/66
Ar⇥q Πr⇥k Bk⇥q
Π is k ⇥ r partial permutation with elements (πi, i) = ±1 and r ⇡ k |M|
|N |
Relax Π ⇡ Q> s.t. Q>Q = I (k ⇥ r ortho-projection) min
Q
trace(Q>ΛN ,kQ) + µkAr Q>Bkk2,1 s.t. Q>Q = I
Non-smooth optimization on the Stiefel manifold with k2 variables Non-rigid alignment of eigenfunctions
Litany, Rodol` a, Bronstein2 2016; Kovnatsky, Glashoff, Bronstein2, Kimmel 2013 (Joint diag); Kovnatsky, Glashoff, Bronstein 2016 (MADMM)
60/66
Full shape N φM
2 , φM 3
and φN
2 , φN 3
Laplacian eigenbasis Part M φM
2 , φM 3
and ˆ φN
2 , ˆ
φN
3
New basis Litany, Rodol` a, Bronstein2 2016
62/66
Initialization 75 150 700 4000 Litany, Rodol` a, Bronstein2 2016
63/66
SPFM PFM rank = 36 rank = 23 rank = 7 Litany, Rodol` a, Bronstein2 2016
64/66
0.05 0.1 0.15 0.2 0.25 20 40 60 80 100
Geodesic Error % Correspondences
cuts
0.05 0.1 0.15 0.2 0.25
Geodesic Error
holes
SPFM JAD RF PFM GT IM EN SHREC’16 Partial Matching benchmark: Rodol` a et al. 2016; Methods: Unpublished work (SPFM); Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 (PFM); Sahillio˘ glu, Yemez 2012 (IM); Rodol` a, Bronstein, Albarelli, Bergamasco, Torsello 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF)
65/66
1 10 20 30 40 50 100 150 200
Number of vertices (⇥104) Mean time per iteration (sec) SPFM PFM Litany, Rodol` a, Bronstein2 2016
66/66
Litany, Rodol` a, Bronstein2 2016; data: Bogo et al. 2014 (FAUST)
66/66
Litany, Rodol` a, Bronstein2 2016; data: Bogo et al. 2014 (FAUST)
66/66
Litany, Rodol` a, Bronstein2 2016; data: Rodola et al. 2016 (SHREC)
66/66
Litany, Rodol` a, Bronstein2 2016; data: Rodola et al. 2016 (SHREC)