Non-Rigid Puzzles Or Litany Tel Aviv University Joint work with - - PowerPoint PPT Presentation

1/36

Non-Rigid Puzzles

Or Litany

Tel Aviv University Joint work with

1/36

Non-Rigid Puzzles

and other stories

Or Litany

Tel Aviv University Joint work with

2/36

Shape correspondence problem

Isometric

2/36

Shape correspondence problem

Isometric Partial

2/36

Shape correspondence problem

Isometric Partial Different representation

2/36

Shape correspondence problem

Isometric Partial Different representation Non-isometric

3/36

Point-wise maps

xi X yj Y t

Point-wise maps t: X → Y

3/36

Functional maps

f F(X) g F(Y ) T

Functional maps T: F(X) → F(Y )

Ovsjanikov et al. 2012

4/36

Functional correspondence

f g ↓ T ↓ Ovsjanikov et al. 2012

4/36

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ Ovsjanikov et al. 2012

4/36

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ ↓ C ↓ Translates Fourier coefficients from Φ to Ψ Ovsjanikov et al. 2012

4/36

Functional correspondence

f g ≈ a1 + a2 + · · · + ak ≈ b1 + b2 + · · · + bk ↓ T ↓ ↓ C ↓ Translates Fourier coefficients from Φ to Ψ ≈ Ψk Φ⊤

k

Ψ⊤

k g = CΦ⊤ k f

where Φk = (φ1, . . . , φk), Ψk = (ψ1, . . . , ψk) are Laplace-Beltrami eigenbases

Ovsjanikov et al. 2012

5/36

Fourier analysis (non-Euclidean spaces)

The Laplacian is invariant to isometries

φ1 φ2 φ3 φ4 ψ1 ψ2 ψ3 ψ4

6/36

Functional correspondence in Laplacian eigenbases

C = Ψ⊤

k TΦk ⇒ cij = ψi, Tϕj

For isometric simple spectrum shapes, C is diagonal since ψi = ±Tφi

7/36

Part-to-full correspondence

Full model Partial query

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

8/36

Partial Laplacian eigenvectors

φ2 φ3 φ4 φ5 φ6 φ7 φ8 φ9 ψ2 ψ3 ψ4 ψ5 ψ6 ψ7 ψ8 ψ9 ζ2 ζ3 ζ4 ζ5 ζ6 ζ7 ζ8 ζ9

Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions)

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

8/36

Partial Laplacian eigenvectors

φ2 φ3 φ4 φ5 φ6 φ7 φ8 φ9 ψ2 ψ3 ψ4 ψ5 ψ6 ψ7 ψ8 ψ9 ζ2 ζ3 ζ4 ζ5 ζ6 ζ7 ζ8 ζ9

Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions)

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

9/36

Partial Laplacian eigenvectors

Functional correspondence matrix C Slope ≈ ratio of the two surface areas

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

10/36

Partial functional maps

Tf = diag(v)g

solve

⇒

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

10/36

Partial functional maps

Tf = diag(v)g Optimization problem w.r.t. correspondence and part min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v) solve

⇒

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

v dx 2 + µ2

• Y

∇Y vdx

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

v dx 2

• area preservation

+ µ2

• Y

∇Y vdx

• Mumford−Shah

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

v dx 2

• area preservation

+ µ2

• Y

∇Y vdx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F + µ4

• i=j

(C⊤C)2

ij + µ5

• i

((C⊤C)ii − di)2

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

v dx 2

• area preservation

+ µ2

• Y

∇Y vdx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F

• slant

+ µ4

• i=j

(C⊤C)2

ij

• ≈ orthogonality

+ µ5

• i

((C⊤C)ii − di)2

• rank≈r

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008

11/36

Partial functional maps

min

C,v CΦ⊤F − Ψ⊤diag(v)G2,1 + ρcorr(C) + ρpart(v)

Part regularization ρpart(v) = µ1

• area(X) −
• Y

v dx 2

• area preservation

+ µ2

• Y

∇Y vdx

• Mumford−Shah

Correspondence regularization ρcorr(C) = µ3C ◦ W2

F

• slant

+ µ4

• i=j

(C⊤C)2

ij

• ≈ orthogonality

+ µ5

• i

((C⊤C)ii − di)2

• rank≈r

F, G = dense SHOT descriptor fields

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008; Tombari et al. 2010

12/36

Structure of partial functional correspondence

C W C⊤C

20 40 60 80 100 2 4

singular values

13/36

Non-rigid puzzles

Given a full template model, find correspondence to multiple parts

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

13/36

Non-rigid puzzles

Given a full template model, find correspondence to multiple parts

Segmentation

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

13/36

Non-rigid puzzles

Given a full template model, find correspondence to multiple parts

Correspondence

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

14/36

Non-rigid puzzles

We allow for overlapping, missing, or extra parts

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

15/36

Partial Laplacian eigenvectors

Functional correspondence matrix C Slope ≈ ratio of the two surface areas

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016

16/36

Slant estimation

Key observation: the slope of C is given by ≈ area(N )

area(M) even with clutter!

φj unknown ψi1 M ξi2 N Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

16/36

Slant estimation

Key observation: the slope of C is given by ≈ area(N )

area(M) even with clutter!

φj unknown ψi1 M ξi2 N

Upper body to scene (j, i1) Upper body to human (j, i2) Scene to human (i1, i2)

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

16/36

Slant estimation

Key observation: the slope of C is given by ≈ area(N )

area(M) even with clutter!

φj unknown ψi1 M ξi2 N

Upper body to scene (j, i1) Upper body to human (j, i2) Scene to human (i1, i2) i1 ≈ j|M|/|M| i2 ≈ j|N|/|N|

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

16/36

Slant estimation

Key observation: the slope of C is given by ≈ area(N )

area(M) even with clutter!

φj unknown ψi1 M ξi2 N

Upper body to scene (j, i1) Upper body to human (j, i2) Scene to human (i1, i2) i1 ≈ j|M|/|M| i2 ≈ j|N|/|N| |M| = |N|

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

16/36

Slant estimation

Key observation: the slope of C is given by ≈ area(N )

area(M) even with clutter!

φj unknown ψi1 M ξi2 N

Upper body to scene (j, i1) Upper body to human (j, i2) Scene to human (i1, i2) i1 ≈ j|M|/|M| i2 ≈ j|N|/|N| |M| = |N| θ = i2

i1 = |N | |M|

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

17/36

Functional puzzles

min

Ci,Mi⊆M,Ni⊆Ni p

• i=1

CiΦi(Mi)⊤Fi − Ψ(Ni)⊤Gi2,1 +

p

• i=1

ρcorr(Ci) +

p

• i=0

ρpart(Mi) +

p

• i=1

ρpart(Ni) s.t. Mi ∩ Mj = ∅ ∀i = j M0 ∪ M1 ∪ · · · = M |Mi| = |Ni| ≥ α|Ni|

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016; Litany, Bronstein, Bronstein 2012

17/36

Functional puzzles

min

Ci,Mi⊆M,Ni⊆Ni p

• i=1

CiΦi(Mi)⊤Fi − Ψ(Ni)⊤Gi2,1 +

p

• i=1

ρcorr(Ci) +

p

• i=0

ρpart(Mi) +

p

• i=1

ρpart(Ni) s.t. Mi ∩ Mj = ∅ ∀i = j M0 ∪ M1 ∪ · · · = M |Mi| = |Ni| ≥ α|Ni| 3-way alternating optimization w.r.t. blocks {Ci}, {Mi}, {Ni} p matching problems are solved simultaneously

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016; Litany, Bronstein, Bronstein 2012

18/36

Example of convergence

80 90 100 110 120 130 140 150 160

Iteration number Time (sec)

30 32 34 36 38 40 42 44 46 48

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

19/36

Example of convergence

19/36

Example of convergence

19/36

Example of convergence

20/36

Overlapping parts example

Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Segmentation

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

20/36

Overlapping parts example

Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT) Correspondence

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

20/36

Overlapping parts example

Model/Part Synthetic (FAUST) Transformation Near-isometric Clutter Yes (overlap) Missing part No Data term Dense (SHOT)

0.0 0.1

Correspondence error

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

21/36

Missing parts example

Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT)

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

21/36

Missing parts example

Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Segmentation

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

21/36

Missing parts example

Model/Part Synthetic (TOSCA) Transformation Isometric Clutter Yes (extra part) Missing part Yes Data term Dense (SHOT) Correspondence

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

22/36

Scanned data example

Model/Part Synthetic (TOSCA) / Scan Transformation Non-Isometric Clutter No Missing part No Data term Sparse deltas

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

22/36

Scanned data example

Model/Part Synthetic (TOSCA) / Scan Transformation Non-Isometric Clutter No Missing part No Data term Sparse deltas Segmentation

Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

23/36

Non-rigid puzzle vs Partial functional map

Partial functional map (pair-wise) Non-rigid puzzle

Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016

24/36

Summary

Practical scenarios: Multiple parts, missing parts, overlap, non-isometric, etc.

24/36

Summary

Practical scenarios: Multiple parts, missing parts, overlap, non-isometric, etc. Slanted diagonal structure holds in the presence of clutter

24/36

Summary

Practical scenarios: Multiple parts, missing parts, overlap, non-isometric, etc. Slanted diagonal structure holds in the presence of clutter Joint matching and segmentation ≫ pair-wise matching

24/36

Summary

Practical scenarios: Multiple parts, missing parts, overlap, non-isometric, etc. Slanted diagonal structure holds in the presence of clutter Joint matching and segmentation ≫ pair-wise matching Works with dense descriptors, but better descriptors are welcome...

24/36

Summary

Practical scenarios: Multiple parts, missing parts, overlap, non-isometric, etc. Slanted diagonal structure holds in the presence of clutter Joint matching and segmentation ≫ pair-wise matching Works with dense descriptors, but better descriptors are welcome... Code available at https://github.com/orlitany/NonRigidPuzzles

25/36

Bonus∗: Going fully spectral

PFM has two major drawbacks: Runtime is O(n)

∗Unpublished work

25/36

Bonus∗: Going fully spectral

PFM has two major drawbacks: Runtime is O(n) Slanted diagonal as prior requires heavy engineering

∗Unpublished work

25/36

Bonus∗: Going fully spectral

PFM has two major drawbacks: Runtime is O(n) Slanted diagonal as prior requires heavy engineering Our idea: “reorder” the eigenfunctions and get rid of the indicator

∗Unpublished work

25/36

Bonus∗: Going fully spectral

PFM has two major drawbacks: Runtime is O(n) Slanted diagonal as prior requires heavy engineering Our idea: “reorder” the eigenfunctions and get rid of the indicator Runtime is O(1)

∗Unpublished work

25/36

Bonus∗: Going fully spectral

PFM has two major drawbacks: Runtime is O(n) Slanted diagonal as prior requires heavy engineering Our idea: “reorder” the eigenfunctions and get rid of the indicator Runtime is O(1) State-of-the-art in Partial and Topology SHREC contests

∗Unpublished work

26/36

Localized basis functions

φ1 φ2 φ3 φ4 φ5 φ6 φ7 φ8 φ9 φ10 ψ1 ψ2 ψ3 ψ4 ψ5 ψ6 ψ7 ψ8 ψ9 ψ10 ˆ ψ1 ˆ ψ2 ˆ ψ3 ˆ ψ4 ˆ ψ5 ˆ ψ6 ˆ ψ7 ˆ ψ8 ˆ ψ9 ˆ ψ10

∗Unpublished work

27/36

Fully spectral partial shape matching

min

Q

• ff(QTΛQ) + WrA − WrQB2,1

s.t. QTQ = I Compute new basis functions as linear combinations of Laplacian eigenfunctions

∗Unpublished work

27/36

Fully spectral partial shape matching

min

Q

• ff(QTΛQ) + WrA − WrQB2,1

s.t. QTQ = I Compute new basis functions as linear combinations of Laplacian eigenfunctions “Straighten” the slant

∗Unpublished work

28/36

Geometric interpretation

Laplacian eigenbasis Ours

29/36

Geometric interpretation

Video

∗Unpublished work

30/36

Fully spectral partial shape matching

Init. 5 25 75 150 700 1400 4000

Localization is manifested as a shrinking of spectral coefficients

∗Unpublished work

31/36

Increasing partiality

Ours PFM rank = 36 rank = 23 rank = 7

∗Unpublished work

32/36

SHREC’16 Partiality

0.05 0.1 0.15 0.2 0.25 20 40 60 80 100

Geodesic Error % Correspondences

cuts

0.05 0.1 0.15 0.2 0.25

Geodesic Error

holes Ours JAD RF PFM GT IM EN

∗Unpublished work

33/36

SHREC’16 Topology

0.05 0.1 0.15 0.2 0.25 20 40 60 80 100

Geodesic Error % Correspondences Ours PFM RF GE EM CO

∗Unpublished work

34/36

Runtime

1 10 20 30 40 50 100 150 200

Number of vertices (×104) Mean time per iteration (sec) Ours PFM

∗Unpublished work

35/36

Examples

∗Unpublished work

35/36

Examples

∗Unpublished work

35/36

Examples

∗Unpublished work

36/36

Summary

Simpler: localization is attained in the spectral domain

36/36

Summary

Simpler: localization is attained in the spectral domain Faster: constant complexity (does not depend on shape size)

36/36

Summary

Simpler: localization is attained in the spectral domain Faster: constant complexity (does not depend on shape size) Better: state of the art results on challenging benchmarks

36/36

Summary

Simpler: localization is attained in the spectral domain Faster: constant complexity (does not depend on shape size) Better: state of the art results on challenging benchmarks

Thank you!