Improving Shape retrieval by Spectral Matching and Meta Similarity - PowerPoint PPT Presentation

Improving Shape retrieval by Spectral Matching and Meta Similarity Amir Egozi (BGU), Yosi Keller (BIU) and Hugo Guterman (BGU) Department of Electrical and Computer Engineering, Ben-Gurion University of the Negev 1 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 2 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 2 / 21

Problem statement S = { s i } n i = 1 , and Q = { q i } m i = 1 , s i , q i ∈ R 2 , are two shapes Ψ ( S , Q ) ∈ [ 0, 1 ] quantifies the ”shape” similarity. Invariant to similarity transformation (translation, rotation, and isotropic scaling) Robust to articulation Resilient to boundary noise and non-linear deformation 3 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 3 / 21

Related work: Feature extraction 1 Curvature [Fischler and Wolf TPAMI’94] Shape-contexts (SC) [Belongie et al . TPAMI’02] Inner-distance SC (ID-SC) [Ling and Jacobs TPAMI’07] Correspondences 2 Hungarian algorithm [Munkres 1957] Dynamic Programming [Ling and Jacobs TPAMI’07] Agglomerate local similarities into a global similarity 3 measure 4 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 4 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 4 / 21

Local Shape Descriptors Shape contexts [Belongie et al. TPAMI’03 ] For s i ∈ S , a 2D histogram of the relative distances and orientations to the other ( n − 1 ) points: � � h i ( k ) = # s i � = s j | ( s i − s j ) ∈ bin ( k ) 5 / 21

Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI’07] Properties: Robust to articulation Capturing part structures ID-SC - An extension to shape contexts Euclidean distance is replaced by the inner-distance 6 / 21

Local Shape Descriptors Utilize the ID-SC to obtain a set of candidate assignments E = { ( s i , q i ′ ) } , s.t. d ( Φ ( s i ) , Φ ( q i ′ )) < T Retain the k NN For each point s i ∈ S . 7 / 21

Local Shape Descriptors Utilize the ID-SC to obtain a set of candidate assignments E = { ( s i , q i ′ ) } , s.t. d ( Φ ( s i ) , Φ ( q i ′ )) < T Retain the k NN For each point s i ∈ S . 7 / 21

Local Shape Descriptors Utilize the ID-SC to obtain a set of candidate assignments E = { ( s i , q i ′ ) } , s.t. d ( Φ ( s i ) , Φ ( q i ′ )) < T Retain the k NN For each point s i ∈ S . Example Let |S| = |Q| = 100 , | E | = |S| × |Q| = 10, 000 , for k = 5 , | E | = |S| × k = 500 . 7 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 7 / 21

Assignment problems Quadratic assignment problem (QAP) Definition A pair-wise affinity function, Ω : E × E → R + , measures the cost of a pair of individual assignments. � � −| D S ( s i , s j ) − D Q ( q i ′ , q j ′ ) | 2 Ω ( e i , e j ) = exp σ > 0 , σ 8 / 21

Quadratic Assignment Problem Affinity matrix structure This affinity measure is: Purely (intrinsic) geometrical measure Invariant to: translation, rotation and reflection. Not invariant to: scaling (uniform and affine) 9 / 21

Assignment problems Quadratic assignment problem (QAP) � � −| D S ( s i , s j ) − D Q ( q i ′ , q j ′ ) | 2 a ij = Ω ( e i , e j ) = exp σ > 0 , σ Definition The affinity matrix consists of all pairwise affinities, A = ( a ij ) ∈ R N × N . 10 / 21

Quadratic Assignment Problem Affinity matrix structure (cont.) Serialization constraint - keeps only affinities such that: � � � − ∆ � q i ′ , q j ′ �� � ∆ � < ∆ max s i , s j 11 / 21

Quadratic assignment problem (QAP) Goal - find M ⊂ E which maximizes Ω ( e i , e j ) = x T A x , ∑ e i ∈ M , e j ∈ M and obeys the matching constraints. The assignment set M can be represented by a binary vector.   0 1 0 0   X = 1 0 0 0 0 0 0 1 x = ( 0 1 0 0 | 1 0 0 0 | 0 0 0 1 ) T 12 / 21

QAP solutions Spectral matching [Leordeanu and Hebert, ICCV’05] The optimization problem x T A x C x = b , x ∈ { 0, 1 } N x = arg max � s.t. x This optimization problem is NP-complete! more 13 / 21

QAP solutions Spectral matching [Leordeanu and Hebert, ICCV’05] The optimization problem x T A x C x = b , x ∈ { 0, 1 } N x = arg max � s.t. x This optimization problem is NP-complete! So, relax the binary constraint and the matching constraints [Leordeanu and Hebert, ICCV’05] z T A z z T z = 1, z ∈ R N � z = arg max s.t. z Since A is symmetric, the maximum is achieved by the leading eigenvector of A . The matching constraints are enforced at the discretization stage. more 13 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 13 / 21

Similarity measure - Conclusion x ∈ { 0, 1 } N , the similarity Given the estimated matching vector � measure is given by: x T A � Ψ ( S , Q ) = � x Properties: Ψ ( S , Q ) measure the intrinsic geometrical distortion between the two shapes. Invariance to translation, rotation and isotropic scaling (by normalization). Robust to articulation and boundary noise. 14 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 14 / 21

Graph based recognition Basic idea My friend’s friend is my friend 0.6 4 0.8 0.5 . 0 0 . 8 0 0.6 . X 0.7 4 0.5 0.1 0.5 0.5 15 / 21

Shape Meta-Similarity Related works: Graph Transduction [Xiang et al . TPAMI’10] x n + 1 = M x n x n + 1 ( i 0 ) = 1; Contextual dissimilarity measure [Jegou et al . TPAMI’10] Improves bag-of-features based image retrieval 16 / 21

Shape Meta-Similarity Our approach Instead of node similarity use structural similarity Given Ψ ij = Ψ ( S i , S j ) , for all i and j , Shape Meta-descriptor Λ i ∈ R N + is defined as,  Ψ ij  if S j ∈ N i ∑ S j ∈N i Ψ ij Λ i =  0 otherwise . The Meta-Similarity is Ψ M ij = || Λ i − Λ j || 1 17 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 17 / 21

Experimental results MPEG7 Shape-CE-1 MPEG7 Shape-CE-1: 1400 images from 70 categories, with 20 images per category. 18 / 21

Experimental results 1 MPEG7 Shape-CE-1 Retrival rates - one-to-one similarity Algorithm Performance Visual Parts [Latecki et al . CVPR’00] 76.45% Shape Contexts [Belongie et al . TPAMI’02] 76.51% MDS+SC+DP [Ling and Jacobs TPAMI’07] 84.35% Planar Graph cuts [Schmidt et al . CVPR’09] 85% IDSC+DP [Ling and Jacobs TPAMI’07] 85.40% IDSC+DP+EMD- L 1 [Ling and Okada TPAMI’07] 86.56% GM+IDSC 87.47 % GM+SC 88.11 % Contour Flexibility [Xu et al . TPAMI’09] 89.31% 19 / 21

Experimental results 2 MPEG7 Shape-CE-1 Retrival rates - Graph-based similarity Algorithm Performance Graph Transduction [Yang et al . ECCV’08] 91% GM+IDSC+Meta Descriptor 91.46% GM+SC+Meta Descriptor 92.51% Locally Constrained Diffusion [Yang et al . CVPR’09] 93.32% 20 / 21

Talk Outline Problem Statement Related works The proposed scheme Local shape descriptors Matching algorithm Similarity measure Meta-similarity Experimental results Conclusions 20 / 21

Shape similarity measure - Conclusions We presented a new approach for measuring the similarity between shapes. It measure the intrinsic geometrical distortion between the two shapes. Better for articulated objects. We present an efficient meta-descriptor and meta-similarity. 21 / 21

Thank you! 22 / 21

Related work: Intermediate between local and global Bending invariant signatures [Elad and Kimmel TPAMI’03] Idea: Embed geodesic distance into Euclidean space via Multi-dimensional scaling (MDS) 23 / 21

Related work: Intermediate between local and global Bending invariant signatures [Elad and Kimmel TPAMI’03] Introduced for 2D manifolds, later to 2D shapes [Ling and Jacobs, TPAMI’07] and [Bronstein et al, IJCV’08]. Back 23 / 21

Background: Local Shape Descriptors Inner-distance shape contexts [Ling and Jacobs TPAMI’07] An extension to shape contexts Euclidean distance is replaced directly with the inner-distance Shape Contexts (SC) 10 2 10 1 0 9 Distance 10 0 13 5 10 − 1 3 3 π 0 π/ 2 3 π/ 2 2 π 1 1 0 0 0 0 0 5 0 0 24 / 21