3D Deep Clustering a clustering framework for unsupervised learning of 3D object feature descriptors LAMBDA Workshop on Retrieval and Shape analysis Alexandros Georgiou M.Sc. in “Algorithms, Logic and Discrete Mathematics” 1/12
Problem Construct feature descriptors for 3D objects . 2/12
Problem Construct feature descriptors for 3D objects . Application dependent. 2/12
Problem Construct feature descriptors for 3D objects . Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point. ⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval. 2/12
Problem Construct feature descriptors for 3D objects . Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point. ⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval. Global descriptors describe the whole shape and are often produced by aggregating local descriptors, e.g. bag-of-features paradigm. 2/12
Problem Construct feature descriptors for 3D objects . Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point. ⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval. Global descriptors describe the whole shape and are often produced by aggregating local descriptors, e.g. bag-of-features paradigm. ⋆ Use Convolutional Neural Networks (defined on non-Euclidean domains) and learn task-specific features. 2/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . 3/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . ⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds. 3/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . ⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds. Define convolution between functions on manifolds. 3/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . ⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds. Define convolution between functions on manifolds. ⊲ Spectral methods. 3/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . ⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds. Define convolution between functions on manifolds. ⊲ Spectral methods. ⊲ Spatial methods. 3/12
non-Euclidean CNNs 3D objects are modeled as compact Riemannian manifolds . ⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds. Define convolution between functions on manifolds. ⊲ Spectral methods. ⊲ Spatial methods. Other non-Euclidean domains, e.g. graphs. 3/12
non-Euclidean CNNs MoNet [Monti et al. ] 4/12
non-Euclidean CNNs MoNet [Monti et al. ] ⊲ x point on a manifold, y ∈ N ( x ) points in the neighborhood of x . 4/12
non-Euclidean CNNs MoNet [Monti et al. ] ⊲ x point on a manifold, y ∈ N ( x ) points in the neighborhood of x . ⊲ u ( x , y ) is a d -dimensional vector of pseudo-coordinates . 4/12
non-Euclidean CNNs MoNet [Monti et al. ] ⊲ x point on a manifold, y ∈ N ( x ) points in the neighborhood of x . ⊲ u ( x , y ) is a d -dimensional vector of pseudo-coordinates . ⊲ Weighting function (kernel) � � w Θ ( u ) = w 1 ( u ) , . . . , w n ( u ) 4/12
non-Euclidean CNNs MoNet [Monti et al. ] ⊲ x point on a manifold, y ∈ N ( x ) points in the neighborhood of x . ⊲ u ( x , y ) is a d -dimensional vector of pseudo-coordinates . ⊲ Weighting function (kernel) � � w Θ ( u ) = w 1 ( u ) , . . . , w n ( u ) ⊲ Patch operator � � � D i ( x ) f = w i u ( x , y ) f ( y ) y ∈N ( x ) 4/12
non-Euclidean CNNs MoNet [Monti et al. ] ⊲ x point on a manifold, y ∈ N ( x ) points in the neighborhood of x . ⊲ u ( x , y ) is a d -dimensional vector of pseudo-coordinates . ⊲ Weighting function (kernel) � � w Θ ( u ) = w 1 ( u ) , . . . , w n ( u ) ⊲ Patch operator � � � D i ( x ) f = w i u ( x , y ) f ( y ) y ∈N ( x ) ⊲ Define convolution as n � � � ( x ) = g i D i ( x ) f f ⋆ g i =1 4/12
How do we learn the parameters ? 5/12
How do we learn the parameters ? Supervised learning (classification) 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . ⊲ f Θ is the convnet mapping and f Θ ( x ) is the vector of features. 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . ⊲ f Θ is the convnet mapping and f Θ ( x ) is the vector of features. ⊲ g Θ is a classifier that predicts the correct labels on top of the features f Θ ( x ). 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . ⊲ f Θ is the convnet mapping and f Θ ( x ) is the vector of features. ⊲ g Θ is a classifier that predicts the correct labels on top of the features f Θ ( x ). ⊲ Training set X = { x 1 , . . . , x n } with associated labels y 1 , . . . , y n ∈ Y . 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . ⊲ f Θ is the convnet mapping and f Θ ( x ) is the vector of features. ⊲ g Θ is a classifier that predicts the correct labels on top of the features f Θ ( x ). ⊲ Training set X = { x 1 , . . . , x n } with associated labels y 1 , . . . , y n ∈ Y . ⊲ Learn Θ by optimizing the following problem: n � min ℓ ( g Θ ◦ f Θ ( x i ) , y i ) Θ i =1 for some loss function ℓ . 5/12
How do we learn the parameters ? Supervised learning (classification) ⊲ F Θ ( x ) = g Θ ◦ f Θ ( x ) = y ∈ Y on input x ∈ X . ⊲ f Θ is the convnet mapping and f Θ ( x ) is the vector of features. ⊲ g Θ is a classifier that predicts the correct labels on top of the features f Θ ( x ). ⊲ Training set X = { x 1 , . . . , x n } with associated labels y 1 , . . . , y n ∈ Y . ⊲ Learn Θ by optimizing the following problem: n � min ℓ ( g Θ ◦ f Θ ( x i ) , y i ) Θ i =1 for some loss function ℓ . What about unsupervised learning ? 5/12
6/12
How it works 7/12
How it works 7/12
How it works Produce the vector of features f Θ ( x i ) through the convnet. 7/12
How it works Produce the vector of features f Θ ( x i ) through the convnet. Assign a pseudo-label y i to each f Θ ( x i ) using a clustering method, e.g. k -means. 7/12
How it works Produce the vector of features f Θ ( x i ) through the convnet. Assign a pseudo-label y i to each f Θ ( x i ) using a clustering method, e.g. k -means. Update the parameters of the convet, just as in the case of supervised learning, by predicting these pseudo-labels. 7/12
How it works: Implementation details This method is prone to trivial solutions. 8/12
How it works: Implementation details This method is prone to trivial solutions. ⊲ Empty clusters. 8/12
How it works: Implementation details This method is prone to trivial solutions. ⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. 8/12
How it works: Implementation details This method is prone to trivial solutions. ⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k -means optimization. 8/12
How it works: Implementation details This method is prone to trivial solutions. ⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k -means optimization. ⊲ Trivial parameterization. 8/12
Recommend
More recommend
