Fully Spectral Partial Shape Matching Or Litany 1 , 2 a 3 Emanuele - PowerPoint PPT Presentation

Fully Spectral Partial Shape Matching Or Litany 1 , 2 a 3 Emanuele Rodol` Alexander Bronstein 1 , 2 , 4 Michael Bronstein 1 , 2 , 3 1 Tel Aviv University 2 Intel 3 USI Lugano 4 Technion Eurographics, 27 April 2017 1/30

3D sensing applications LIDAR Velodyne HDL-64E (as in the Google Car); Intel RealSense R200 3D camera; FaceShift Inc. ; Me ; A cute baby 2/30

3D sensing applications Non-rigid deformations Limited view points LIDAR Velodyne HDL-64E (as in the Google Car); Intel RealSense R200 3D camera; FaceShift Inc. ; Me ; A cute baby 2/30

Shape correspondence problem Isometric 3/30

Shape correspondence problem Isometric Partial 3/30

Shape correspondence problem Isometric Partial Topological noise 3/30

Shape correspondence problem Isometric Partial Topological noise Different representation 3/30

Shape correspondence problem Isometric Partial Topological noise Different representation Non-isometric 3/30

Point-wise maps t y j x i X Y Point-wise maps t : X → Y 4/30

Functional maps g f F ( X ) F ( Y ) T Functional maps T : F ( X ) → F ( Y ) Ovsjanikov et al. 2012 4/30

Functional correspondence f ↓ T ↓ g Ovsjanikov et al. 2012 5/30

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ T ↓ g ≈ b 1 + b 2 + · · · + b k Ovsjanikov et al. 2012 5/30

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ ↓ Translates Fourier coefficients from Φ to Ψ T C ↓ ↓ g ≈ b 1 + b 2 + · · · + b k Ovsjanikov et al. 2012 5/30

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ ↓ Φ ⊤ ≈ Translates Fourier coefficients from Φ to Ψ T Ψ k C k ↓ ↓ g ≈ b 1 + b 2 + · · · + b k Ψ ⊤ k g = CΦ ⊤ k f where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al. 2012 5/30

Fourier analysis (non-Euclidean spaces) The Laplacian is invariant to isometries φ 1 φ 2 φ 3 φ 4 ψ 1 ψ 2 ψ 3 ψ 4 6/30

Functional correspondence in Laplacian eigenbases C = Ψ ⊤ k TΦ k ⇒ c ij = � ψ i , Tϕ j � For isometric simple spectrum shapes, C is diagonal since ψ i = ± T φ i 7/30

Part-to-full correspondence Full model Partial query 8/30

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 9/30

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 9/30

Partial Laplacian eigenvectors Functional correspondence matrix C Slope ≈ ratio of the two surface areas Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 10/30

Going fully spectral PFM has two major drawbacks: Explicit spatial indicator → runtime is O ( n ) solve ⇒ 11/30

Going fully spectral PFM has two major drawbacks: Explicit spatial indicator → runtime is O ( n ) The partiality prior requires heavy engineering � 2 � � � ρ part ( v ) = µ 1 area( X ) − η ( v ) dx + µ 2 ξ ( v ) �∇ Y η ( v ) � dx Y Y � � η ( v ) − 1 ξ ( v ) = δ 2 η ( v ) = 1 2(tanh(2 v − 1) + 1) ρ corr ( C ) = µ 3 � C ◦ W � 2 � ( C ⊤ C ) 2 � (( C ⊤ C ) ii − d i ) 2 F + µ 4 ij + µ 5 i � = j i 11/30

Going fully spectral PFM has two major drawbacks: Explicit spatial indicator → runtime is O ( n ) The partiality prior requires heavy engineering Our idea: “reorder” and spatially localize the eigenfunctions 11/30

Going fully spectral PFM has two major drawbacks: Explicit spatial indicator → runtime is O ( n ) The partiality prior requires heavy engineering Our idea: “reorder” and spatially localize the eigenfunctions No indicator → Runtime is O ( k 2 ) 11/30

Going fully spectral PFM has two major drawbacks: Explicit spatial indicator → runtime is O ( n ) The partiality prior requires heavy engineering Our idea: “reorder” and spatially localize the eigenfunctions No indicator → Runtime is O ( k 2 ) One-to-one correspondence yields a simple prior 11/30

Localized basis functions φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 φ 10 ψ 1 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 ψ 10 ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ψ 1 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 ψ 10 12/30

Localization Energy minimized in PFM C ,v � CA − B ( v ) � + ρ corr ( C ) + ρ part ( v ) min v : N → [0 , 1] A = ( � φ i , f j � M ) B ( v ) = ( � ψ i , v · g j � N ) 13/30

Localization Energy minimized in PFM min C ,v � CA − B ( v ) � + ρ corr ( C ) + ρ part ( v ) Satisfying the data-term induces a localizing map C 13/30

Localization 14/30

Localization 14/30

Localization 14/30

Localization 14/30

Localization 15/30

Localization W r ×� Ψ , F � = Q ⊤ ×� Φ , G � 15/30

Localization W r ×� Ψ , F � = Q ⊤ ×� Φ , G � = � Q ⊤ Φ , G � = � ˆ Φ , G � 15/30

Localized basis functions φ 1 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 φ 10 ψ 1 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 ψ 10 ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ˆ ψ 1 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 ψ 10 16/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min 17/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min Compute new basis functions as linear combinations of Laplace-Beltrami eigenfunctions 17/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min Compute new basis functions as linear combinations of Laplace-Beltrami eigenfunctions Non-smooth optimization on the Stiefel manifold with k × r variables 17/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min Compute new basis functions as linear combinations of Laplace-Beltrami eigenfunctions Non-smooth optimization on the Stiefel manifold with k × r variables Rank r controls amount of partiality 17/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min Compute new basis functions as linear combinations of Laplace-Beltrami eigenfunctions Non-smooth optimization on the Stiefel manifold with k × r variables Rank r controls amount of partiality Descriptors control location of partiality 17/30

Fully spectral partial correspondence Our problem Q ∈ S ( k,r ) off( Q ⊤ Λ N Q ) + µ � W r A − Q ⊤ B � 2 , 1 min Compute new basis functions as linear combinations of Laplace-Beltrami eigenfunctions Non-smooth optimization on the Stiefel manifold with k × r variables Rank r controls amount of partiality Descriptors control location of partiality Two-sided partiality ( P , Q ) ∈ S 2 ( k,r ) off( P ⊤ Λ M P ) + off( Q ⊤ Λ N Q ) + µ � P ⊤ A − Q ⊤ B � 2 , 1 min 17/30

Importance of descriptors and rank (a) (b) (c) 18/30

Importance of descriptors and rank (a) (b) (c) r/k = 0 . 1 r/k = 0 . 5 r/k = 1 . 0 (full) 18/30

Geometric interpretation Full shape N Part M 19/30

Geometric interpretation φ M 2 , φ M and φ N 3 , φ N Full shape N Laplacian eigenbasis 3 5 Part M 19/30

Geometric interpretation φ M 2 , φ M and φ N 3 , φ N Full shape N Laplacian eigenbasis 3 5 and ˆ 2 , ˆ φ M 2 , φ M φ N φ N Part M New basis 3 3 19/30

Animation 20/30

Convergence example Initialization 75 150 700 4000 21/30

Increasing partiality SPFM PFM rank = 36 rank = 23 rank = 7 22/30

Robustness Ours PFM 23/30

Runtime 200 SPFM Mean time per iteration (sec) PFM 150 100 50 0 1 10 20 30 40 Number of vertices ( × 10 4 ) 24/30

SHREC’16 Partiality cuts holes 100 % Correspondences 80 60 40 20 0 0 0.05 0.1 0.15 0.2 0.25 0 0.05 0.1 0.15 0.2 0.25 Geodesic Error Geodesic Error FSPM JAD RF PFM IM EN GT 25/30

SHREC’16 Topology 100 FSPM 80 % Correspondences PFM RF 60 GE EM 40 CO 20 0 0 0.05 0.1 0.15 0.2 0.25 Geodesic Error 26/30

Correspondence examples: topological noise data: Bogo et al. 2014 (FAUST) 27/30

Correspondence examples: topological noise data: Bogo et al. 2014 (FAUST) 27/30

Correspondence examples: topological noise data: L¨ ahner et al. 2016 (SHREC) 27/30

Correspondence examples: topological noise data: L¨ ahner et al. 2016 (SHREC) 27/30

Partiality data: Cosmo et al. 2016 (SHREC) 28/30

Failure cases 29/30

Summary Simpler: localization is attained in the spectral domain 30/30