Non-Rigid Puzzles Or Litany Tel Aviv University Joint work with - PowerPoint PPT Presentation

Non-Rigid Puzzles Or Litany Tel Aviv University Joint work with 1/36

Non-Rigid Puzzles and other stories Or Litany Tel Aviv University Joint work with 1/36

Shape correspondence problem Isometric 2/36

Shape correspondence problem Isometric Partial 2/36

Shape correspondence problem Isometric Partial Different representation 2/36

Shape correspondence problem Isometric Partial Different representation Non-isometric 2/36

Point-wise maps t y j x i X Y Point-wise maps t : X → Y 3/36

Functional maps g f F ( X ) F ( Y ) T Functional maps T : F ( X ) → F ( Y ) Ovsjanikov et al. 2012 3/36

Functional correspondence f ↓ T ↓ g Ovsjanikov et al. 2012 4/36

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ T ↓ g ≈ b 1 + b 2 + · · · + b k Ovsjanikov et al. 2012 4/36

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ ↓ Translates Fourier coefficients from Φ to Ψ T C ↓ ↓ g ≈ b 1 + b 2 + · · · + b k Ovsjanikov et al. 2012 4/36

Functional correspondence ≈ a 1 + a 2 + · · · + a k f ↓ ↓ Φ ⊤ ≈ Translates Fourier coefficients from Φ to Ψ T Ψ k C k ↓ ↓ g ≈ b 1 + b 2 + · · · + b k Ψ ⊤ k g = CΦ ⊤ k f where Φ k = ( φ 1 , . . . , φ k ) , Ψ k = ( ψ 1 , . . . , ψ k ) are Laplace-Beltrami eigenbases Ovsjanikov et al. 2012 4/36

Fourier analysis (non-Euclidean spaces) The Laplacian is invariant to isometries φ 1 φ 2 φ 3 φ 4 ψ 1 ψ 2 ψ 3 ψ 4 5/36

Functional correspondence in Laplacian eigenbases C = Ψ ⊤ k TΦ k ⇒ c ij = � ψ i , Tϕ j � For isometric simple spectrum shapes, C is diagonal since ψ i = ± T φ i 6/36

Part-to-full correspondence Full model Partial query Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 7/36

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 8/36

Partial Laplacian eigenvectors ζ 2 ζ 3 ζ 4 ζ 5 ζ 6 ζ 7 ζ 8 ζ 9 ψ 2 ψ 3 ψ 4 ψ 5 ψ 6 ψ 7 ψ 8 ψ 9 φ 2 φ 3 φ 4 φ 5 φ 6 φ 7 φ 8 φ 9 Laplacian eigenvectors of a shape with missing parts (Neumann boundary conditions) Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 8/36

Partial Laplacian eigenvectors Functional correspondence matrix C Slope ≈ ratio of the two surface areas Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 9/36

Partial functional maps Tf = diag( v ) g solve ⇒ Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 10/36

Partial functional maps Tf = diag( v ) g Optimization problem w.r.t. correspondence and part C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min solve ⇒ Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 10/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 11/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − + µ 2 �∇ Y v � dx v dx Y Y Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008 11/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − + µ 2 �∇ Y v � dx v dx Y Y � �� � � �� � area preservation Mumford − Shah Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008 11/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − + µ 2 �∇ Y v � dx v dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 F + µ 4 ij + µ 5 i � = j i Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008 11/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − + µ 2 �∇ Y v � dx v dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 + µ 4 + µ 5 F ij � �� � i � = j i slant � �� � � �� � rank ≈ r ≈ orthogonality Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008 11/36

Partial functional maps C ,v � CΦ ⊤ F − Ψ ⊤ diag( v ) G � 2 , 1 + ρ corr ( C ) + ρ part ( v ) min Part regularization � � 2 � � ρ part ( v ) = µ 1 area( X ) − + µ 2 �∇ Y v � dx v dx Y Y � �� � � �� � area preservation Mumford − Shah Correspondence regularization � � ρ corr ( C ) = µ 3 � C ◦ W � 2 ( C ⊤ C ) 2 (( C ⊤ C ) ii − d i ) 2 + µ 4 + µ 5 F ij � �� � i � = j i slant � �� � � �� � rank ≈ r ≈ orthogonality F , G = dense SHOT descriptor fields Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016; Bronstein and Bronstein 2008; Tombari et al. 2010 11/36

Structure of partial functional correspondence 4 2 0 0 20 40 60 80 100 C ⊤ C C W singular values 12/36

Non-rigid puzzles Given a full template model, find correspondence to multiple parts Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 13/36

Non-rigid puzzles Given a full template model, find correspondence to multiple parts Segmentation Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 13/36

Non-rigid puzzles Given a full template model, find correspondence to multiple parts Correspondence Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 13/36

Non-rigid puzzles We allow for overlapping, missing, or extra parts Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 14/36

Partial Laplacian eigenvectors Functional correspondence matrix C Slope ≈ ratio of the two surface areas Rodol` a, Cosmo, Bronstein, Torsello, Cremers 2016 15/36

Slant estimation Key observation: the slope of C is given by ≈ area( N ) area( M ) even with clutter! unknown M N φ j ψ i 1 ξ i 2 Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 16/36

Slant estimation Key observation: the slope of C is given by ≈ area( N ) area( M ) even with clutter! unknown M Upper body to scene ( j, i 1 ) N Upper body to human ( j, i 2 ) Scene to human ( i 1 , i 2 ) φ j ψ i 1 ξ i 2 Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 16/36

Slant estimation Key observation: the slope of C is given by ≈ area( N ) area( M ) even with clutter! unknown M Upper body to scene ( j, i 1 ) N Upper body to human ( j, i 2 ) Scene to human ( i 1 , i 2 ) i 1 ≈ j |M| / | M | i 2 ≈ j |N| / | N | φ j ψ i 1 ξ i 2 Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 16/36

Slant estimation Key observation: the slope of C is given by ≈ area( N ) area( M ) even with clutter! unknown M Upper body to scene ( j, i 1 ) N Upper body to human ( j, i 2 ) Scene to human ( i 1 , i 2 ) i 1 ≈ j |M| / | M | i 2 ≈ j |N| / | N | | M | = | N | φ j ψ i 1 ξ i 2 Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 16/36

Slant estimation Key observation: the slope of C is given by ≈ area( N ) area( M ) even with clutter! unknown M Upper body to scene ( j, i 1 ) N Upper body to human ( j, i 2 ) Scene to human ( i 1 , i 2 ) i 1 ≈ j |M| / | M | i 2 ≈ j |N| / | N | | M | = | N | i 1 = |N | θ = i 2 |M| φ j ψ i 1 ξ i 2 Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 16/36

Functional puzzles p � � C i Φ i ( M i ) ⊤ F i − Ψ ( N i ) ⊤ G i � 2 , 1 min C i ,M i ⊆M ,N i ⊆N i i =1 p p p � � � + ρ corr ( C i ) + ρ part ( M i ) + ρ part ( N i ) i =1 i =0 i =1 M i ∩ M j = ∅ ∀ i � = j s . t . M 0 ∪ M 1 ∪ · · · = M | M i | = | N i | ≥ α |N i | Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016; Litany, Bronstein, Bronstein 2012 17/36

Functional puzzles p � � C i Φ i ( M i ) ⊤ F i − Ψ ( N i ) ⊤ G i � 2 , 1 min C i ,M i ⊆M ,N i ⊆N i i =1 p p p � � � + ρ corr ( C i ) + ρ part ( M i ) + ρ part ( N i ) i =1 i =0 i =1 M i ∩ M j = ∅ ∀ i � = j s . t . M 0 ∪ M 1 ∪ · · · = M | M i | = | N i | ≥ α |N i | 3-way alternating optimization w.r.t. blocks { C i } , { M i } , { N i } p matching problems are solved simultaneously Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016; Litany, Bronstein, Bronstein 2012 17/36

Example of convergence Time (sec) 30 32 34 36 38 40 42 44 46 48 80 90 100 110 120 130 140 150 160 Iteration number Litany, Rodol` a, Bronstein, Bronstein, Cremers 2016 18/36

Example of convergence 19/36

Example of convergence 19/36

Example of convergence 19/36