Partial Functional Correspondence Emanuele Rodol` a USI Lugano - - PowerPoint PPT Presentation

1/26

Partial Functional Correspondence

Emanuele Rodol` a

USI Lugano Joint work with

D. Cremers M.M. Bronstein L. Cosmo A. T

• rsello

SGP, Berlin, 21 June 2016

2/26

Shape correspondence problem

Isometric

2/26

Shape correspondence problem

Isometric Partial

2/26

Shape correspondence problem

Isometric Partial Different representation

2/26

Shape correspondence problem

Isometric Partial Different representation Non-isometric

3/26

Point-wise maps

xi X yj Y t

Point-wise maps t: X → Y

3/26

Functional maps

f F(X) g F(Y ) T

Functional maps T: F(X) → F(Y )

Ovsjanikov et al., 2012

4/26

Functional correspondence

f g ↓ T ↓ Ovsjanikov et al., 2012

4/26

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ Ovsjanikov et al., 2012

4/26

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ ↓ C⊤ ↓ Translates Fourier coefficients from Φ to Ψ Ovsjanikov et al., 2012

4/26

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ ↓ C⊤ ↓ Translates Fourier coefficients from Φ to Ψ ≈ Ψk Φ⊤

k

g⊤Ψk = f ⊤ΦkC where Φk = (φ1, . . . , φk), Ψk = (ψ1, . . . , ψk) are Laplace-Beltrami eigenbases

Ovsjanikov et al., 2012

4/26

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ ↓ C⊤ ↓ Translates Fourier coefficients from Φ to Ψ ≈ Ψk Φ⊤

k

G⊤Ψk = F⊤ΦkC where Φk = (φ1, . . . , φk), Ψk = (ψ1, . . . , ψk) are Laplace-Beltrami eigenbases

Ovsjanikov et al., 2012

5/26

Functional correspondence in Laplacian eigenbases

For isometric simple spectrum shapes C is diagonal since ψi = ±Tφi

6/26

Our setting

Full model Partial query

7/26

Partial Laplacian eigenvectors

φ2 φ3 φ4 φ5 φ6 φ7 φ8 φ9 ψ2 ψ3 ψ4 ψ5 ψ6 ψ7 ψ8 ψ9 ζ2 ζ3 ζ4 ζ5 ζ6 ζ7 ζ8 ζ9

Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions)

8/26

Partial Laplacian eigenvectors

Functional correspondence matrix C

9/26

Perturbation analysis: intuition

∆X ∆X ∆ ¯

X

φ1 φ2 φ3 φ1 φ2 φ3 ¯ φ1 ¯ φ2 ¯ φ3 X ¯ X

Ignoring boundary interaction: disjoint parts (block-diagonal matrix) Eigenvectors = Mixture of eigenvectors of the parts

10/26

Perturbation analysis: eigenvalues

10 20 30 40 50 0.00 2.00 4.00 6.00 8.00 ·10−2 eigenvalue number r k Y X

Slope r

k ≈ area(X) area(Y ) (depends on the area of the cut)

Consistent with Weyl’s law

11/26

Perturbation analysis: details

∆X ∆ ¯

X

∆X+tDX ∆ ¯ X+tD ¯ X

tE tE⊤ X ¯ X

11/26

Perturbation analysis: details

∆X ∆ ¯

X

∆X+tDX ∆ ¯ X+tD ¯ X

tE tE⊤ X ¯ X

“How would the Laplacian eigenvalues and eigenvectors of the red part change if we attached a blue part to it?”

12/26

Perturbation analysis: boundary interaction strength

Value of f

10 20

Eigenvector perturbation depends on length and position of the boundary Perturbation strength ≤ c

• ∂X f(x)dx, where

f(x) =

n

• i,j=1

j=i

φi(x)φj(x) λi − λj 2

13/26

Partial functional maps

Given full model shape Y and query shape X corresponding to an unknown approximately isometric part Y ′ ⊂ Y , the partial functional map T : F(X) → F(Y ) is given by Tf = diag(v)g where v ∈ F(Y ) is an indicator function of the part

solve

⇒

14/26

Partial functional maps

Given full model shape Y and query shape X corresponding to an unknown approximately isometric part Y ′ ⊂ Y , the partial functional map T : F(X) → F(Y ) is given by Tf = diag(v)g where v ∈ F(Y ) is an indicator function of the part Optimization problem w.r.t. correspondence and part min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

where η(t) = 1

2(tanh(2t − 1) + 1) saturates the part membership

function

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

η(v)dx 2 + µ2

• Y

∇Y η(v)dx

Bronstein and Bronstein 2008

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

η(v)dx 2

• area preservation

+ µ2

• Y

∇Y η(v)dx

• Mumford−Shah

Bronstein and Bronstein 2008

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

η(v)dx 2

• area preservation

+ µ2

• Y

∇Y η(v)dx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F + µ4

• i=j

(C⊤C)2

ij + µ5

• i

((C⊤C)ii − di)2

Bronstein and Bronstein 2008

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

η(v)dx 2

• area preservation

+ µ2

• Y

∇Y η(v)dx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F

• slant

+ µ4

• i=j

(C⊤C)2

ij

• ≈ orthogonality

+ µ5

• i

((C⊤C)ii − di)2

• rank≈r

Bronstein and Bronstein 2008

15/26

Partial functional maps

min

C,v F⊤ΦC − G⊤diag(η(v))Ψ2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

η(v)dx 2

• area preservation

+ µ2

• Y

∇Y η(v)dx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F

• slant

+ µ4

• i=j

(C⊤C)2

ij

• ≈ orthogonality

+ µ5

• i

((C⊤C)ii − di)2

• rank≈r

F, G = dense SHOT descriptor in all our experiments

Bronstein and Bronstein 2008; Tombari et al. 2010

16/26

Structure of partial functional correspondence

C W C⊤C

20 40 60 80 100 2 4

singular values

17/26

Alternating minimization

C-step: fix v∗, solve for correspondence C min

C F⊤ΦC − G⊤diag(η(v∗))Ψ2,1 + ρcorr(C)

v-step: fix C∗, solve for part v min

v

F⊤ΦC∗ − G⊤diag(η(v))Ψ2,1 + ρpart(v)

17/26

Alternating minimization

C-step: fix v∗, solve for correspondence C min

C F⊤ΦC − G⊤diag(η(v∗))Ψ2,1 + ρcorr(C)

v-step: fix C∗, solve for part v min

v

F⊤ΦC∗ − G⊤diag(η(v))Ψ2,1 + ρpart(v)

Iteration 1 2 3 4

18/26

Example of convergence

20 40 60 80 100 104 105 106 107 108 109 1010 Iteration Energy C-step ν-step 5 10 15 20 25 Time (sec.)

19/26

Examples of partial functional maps

19/26

Examples of partial functional maps

19/26

Examples of partial functional maps

19/26

Examples of partial functional maps

20/26

Partial functional maps vs Functional maps

0.05 0.1 0.15 0.2 0.25 20 40 60 80 100 50 100 150 50 100 150 Geodesic error % Correspondences

PFM

• Func. maps

Correspondence performance for different rank values k

21/26

Partial correspondence performance (SHREC’16)

0.05 0.1 0.15 0.2 0.25 20 40 60 80 100

Geodesic Error % Correspondences

Cuts

0.05 0.1 0.15 0.2 0.25

Geodesic Error

Holes PFM RF IM EN GT

SHREC’16 Partial Matching benchmark Rodol` a et al. 2016; Methods: Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 (PFM); Sahillio˘ glu and Yemez 2012 (IM); Rodol` a et al. 2012 (GT); Rodol` a et al. 2013 (EN); Rodol` a et al. 2014 (RF)

22/26

Conclusions

Partial deformable shape matching is a challenging problem, much less investigated than the full case

22/26

Conclusions

Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio

22/26

Conclusions

Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio Spectral shape analysis still yields good results!

22/26

Conclusions

Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio Spectral shape analysis still yields good results! Code available at http://sites.google.com/site/erodola/publications

22/26

Conclusions

Partial deformable shape matching is a challenging problem, much less investigated than the full case Functional correspondence has a slanted diagonal structure, with slant equal to area ratio Spectral shape analysis still yields good results! Code available at http://sites.google.com/site/erodola/publications

Thank you!

23/26

24/26

Perturbation analysis: details

∆X ∆ ¯

X

∆X+tDX ∆ ¯ X+tD ¯ X

tE tE⊤ X ¯ X

24/26

Perturbation analysis: details

∆X ∆ ¯

X

∆X+tDX ∆ ¯ X+tD ¯ X

tE tE⊤ X ¯ X

“How would the Laplacian eigenvalues and eigenvectors of the red part change if we attached a blue part to it?”

24/26

Perturbation analysis: details

PX P E DX n × n n × ¯ n X ¯ X

“How would the Laplacian eigenvalues and eigenvectors of the red part change if we attached a blue part to it?”

25/26

Perturbation analysis: details

Denote ∆X + tPX = Φ(t)Λ(t)Φ(t)⊤, ∆ ¯

X = ¯

Φ ¯ Λ ¯ Φ⊤, Φ = Φ(0), and Λ = Λ(0). Theorem 1 (eigenvalues) The derivative of the non-trivial eigenvalues is given by d dtλi = φ⊤

i PXφi

PX =

• DX

25/26

Perturbation analysis: details

Denote ∆X + tPX = Φ(t)Λ(t)Φ(t)⊤, ∆ ¯

X = ¯

Φ ¯ Λ ¯ Φ⊤, Φ = Φ(0), and Λ = Λ(0). Theorem 1 (eigenvalues) The derivative of the non-trivial eigenvalues is given by d dtλi = φ⊤

i PXφi

PX =

• DX
• Theorem 2 (eigenvectors) Assuming λi = λj for i = j and λi = ¯

λj for all i, j, the derivative of the non-trivial eigenvectors is given by d dtφi =

n

• j=1

j=i

φ⊤

i PXφj

λi − λj φj +

¯ n

• j=1

φ⊤

i P ¯

φj λi − ¯ λj ¯ φj P = E

26/26

(bi-)Laplacian perturbation: typical picture

Plate Punctured plate Figure: Filoche, Mayboroda 2009