3D Deep Clustering a clustering framework for unsupervised learning - - PowerPoint PPT Presentation

1/12

3D Deep Clustering

a clustering framework for unsupervised learning of 3D object feature descriptors LAMBDA Workshop on Retrieval and Shape analysis

Alexandros Georgiou

M.Sc. in “Algorithms, Logic and Discrete Mathematics”

2/12

Problem

Construct feature descriptors for 3D objects.

2/12

Problem

Construct feature descriptors for 3D objects. Application dependent.

2/12

Problem

Construct feature descriptors for 3D objects. Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point.

⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval.

2/12

Problem

Construct feature descriptors for 3D objects. Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point.

⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval.

Global descriptors describe the whole shape and are often produced by aggregating local descriptors, e.g. bag-of-features paradigm.

2/12

Problem

Construct feature descriptors for 3D objects. Application dependent. Local feature descriptors assign to each point on the shape a vec- tor in some multi-dimensional descriptor space representing the local structure of the shape around that point.

⊲ Used in higher-level tasks such as establishing correspondence between shapes or shape retrieval.

Global descriptors describe the whole shape and are often produced by aggregating local descriptors, e.g. bag-of-features paradigm. ⋆ Use Convolutional Neural Networks (defined on non-Euclidean domains) and learn task-specific features.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds.

Define convolution between functions on manifolds.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds.

Define convolution between functions on manifolds.

⊲ Spectral methods.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds.

Define convolution between functions on manifolds.

⊲ Spectral methods. ⊲ Spatial methods.

3/12

non-Euclidean CNNs

3D objects are modeled as compact Riemannian manifolds.

⊲ The operation of convolution between functions on Euclidean spaces is taken for granted. ⊲ But it is not well defined for manifolds.

Define convolution between functions on manifolds.

⊲ Spectral methods. ⊲ Spatial methods.

Other non-Euclidean domains, e.g. graphs.

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

⊲ x point on a manifold, y ∈ N(x) points in the neighborhood of x.

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

⊲ x point on a manifold, y ∈ N(x) points in the neighborhood of x. ⊲ u(x, y) is a d-dimensional vector of pseudo-coordinates.

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

⊲ x point on a manifold, y ∈ N(x) points in the neighborhood of x. ⊲ u(x, y) is a d-dimensional vector of pseudo-coordinates. ⊲ Weighting function (kernel) wΘ(u) =

• w1(u), . . . , wn(u)

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

⊲ x point on a manifold, y ∈ N(x) points in the neighborhood of x. ⊲ u(x, y) is a d-dimensional vector of pseudo-coordinates. ⊲ Weighting function (kernel) wΘ(u) =

• w1(u), . . . , wn(u)
• ⊲ Patch operator

Di(x)f =

• y∈N (x)

wi

• u(x, y)
• f (y)

4/12

non-Euclidean CNNs

MoNet [Monti et al. ]

⊲ x point on a manifold, y ∈ N(x) points in the neighborhood of x. ⊲ u(x, y) is a d-dimensional vector of pseudo-coordinates. ⊲ Weighting function (kernel) wΘ(u) =

• w1(u), . . . , wn(u)
• ⊲ Patch operator

Di(x)f =

• y∈N (x)

wi

• u(x, y)
• f (y)

⊲ Define convolution as

• f ⋆ g
• (x) =

n

• i=1

giDi(x)f

5/12

How do we learn the parameters ?

5/12

How do we learn the parameters ?

Supervised learning (classification)

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X.

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X. ⊲ fΘ is the convnet mapping and fΘ(x) is the vector of features.

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X. ⊲ fΘ is the convnet mapping and fΘ(x) is the vector of features. ⊲ gΘ is a classifier that predicts the correct labels on top of the features fΘ(x).

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X. ⊲ fΘ is the convnet mapping and fΘ(x) is the vector of features. ⊲ gΘ is a classifier that predicts the correct labels on top of the features fΘ(x). ⊲ Training set X = {x1, . . . , xn} with associated labels y1, . . . , yn ∈ Y.

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X. ⊲ fΘ is the convnet mapping and fΘ(x) is the vector of features. ⊲ gΘ is a classifier that predicts the correct labels on top of the features fΘ(x). ⊲ Training set X = {x1, . . . , xn} with associated labels y1, . . . , yn ∈ Y. ⊲ Learn Θ by optimizing the following problem: min

Θ n

• i=1

ℓ(gΘ ◦ fΘ(xi), yi) for some loss function ℓ.

5/12

How do we learn the parameters ?

Supervised learning (classification)

⊲ FΘ(x) = gΘ ◦ fΘ(x) = y ∈ Y on input x ∈ X. ⊲ fΘ is the convnet mapping and fΘ(x) is the vector of features. ⊲ gΘ is a classifier that predicts the correct labels on top of the features fΘ(x). ⊲ Training set X = {x1, . . . , xn} with associated labels y1, . . . , yn ∈ Y. ⊲ Learn Θ by optimizing the following problem: min

Θ n

• i=1

ℓ(gΘ ◦ fΘ(xi), yi) for some loss function ℓ.

What about unsupervised learning ?

6/12

7/12

How it works

7/12

How it works

7/12

How it works

Produce the vector of features fΘ(xi) through the convnet.

7/12

How it works

Produce the vector of features fΘ(xi) through the convnet. Assign a pseudo-label yi to each fΘ(xi) using a clustering method, e.g. k-means.

7/12

How it works

Produce the vector of features fΘ(xi) through the convnet. Assign a pseudo-label yi to each fΘ(xi) using a clustering method, e.g. k-means. Update the parameters of the convet, just as in the case of supervised learning, by predicting these pseudo-labels.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k-means

• ptimization.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k-means

• ptimization.

⊲ Trivial parameterization.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k-means

• ptimization.

⊲ Trivial parameterization. If the vast majority of inputs is assigned to a few clusters, the param- eters will exclusively discriminate between them. This, for example, is caused when the number of inputs per class is highly unbalanced in supervised classigication.

8/12

How it works: Implementation details

This method is prone to trivial solutions.

⊲ Empty clusters. An optimal decision is to assign all of the inputs to a single cluster. This is caused by absence of mechanisms to prevent empty clusters. ⋆ Automatically reassign empty clusters during the k-means

• ptimization.

⊲ Trivial parameterization. If the vast majority of inputs is assigned to a few clusters, the param- eters will exclusively discriminate between them. This, for example, is caused when the number of inputs per class is highly unbalanced in supervised classigication. ⋆ Sample inputs based on a uniform distribution over the classes, or pseudo-labels.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

⊲ The structure of fΘ is crucial.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

⊲ The structure of fΘ is crucial. ⊲ Other possible clustering methods besides k-means.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

⊲ The structure of fΘ is crucial. ⊲ Other possible clustering methods besides k-means. ⊲ Find the right tasks/datasets.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

⊲ The structure of fΘ is crucial. ⊲ Other possible clustering methods besides k-means. ⊲ Find the right tasks/datasets. ⊲ Evaluate the method.

9/12

Our goals

Problem

Construct task-specific feature descriptors for 3D objects. ⋆ Construct a non-Euclidean CNN, FΘ = gΘ ◦ fΘ, that classifies 3D

• bjects so that fΘ produces good task-specific features and learn the

hyper-parameter Θ in an unsupervised setting as in DeepCluster.

⊲ The structure of fΘ is crucial. ⊲ Other possible clustering methods besides k-means. ⊲ Find the right tasks/datasets. ⊲ Evaluate the method. ⊲ What about other non-Euclidean domains such as graphs ?

10/12

Our goals

Applications

10/12

Our goals

Applications

⊲ Invariant descriptors

10/12

Our goals

Applications

⊲ Invariant descriptors ⊲ Shape correspondence

10/12

Our goals

Applications

⊲ Invariant descriptors ⊲ Shape correspondence ⊲ Shape retrieval

10/12

Our goals

Applications

⊲ Invariant descriptors ⊲ Shape correspondence ⊲ Shape retrieval

Datasets

⊲ ShapeNet ⊲ FAUST ⊲ ABC

11/12

The end

Thank you !

12/12

Bibliography

Caron, Mathilde & Bojanowski, Piotr & Joulin, Armand & Douze, Matthijs, Deep Clustering for Unsupervised Learning of Visual Fea- tures. Monti, Federico & Boscaini, Davide & Masci, Jonathan & Rodol` a, Emanuele & Svoboda, Jan & Bronstein, Michael, Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. Bronstein, Michael & Bruna, Joan & Lecun, Yann & Szlam, Arthur & Vandergheynst, Pierre, Geometric Deep Learning: Going beyond Euclidean data.