estimating metrics suitable to an empirical manifold of
play

Estimating metrics suitable to an empirical manifold of shapes, - PowerPoint PPT Presentation

Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Estimating metrics suitable to an empirical manifold of shapes, using transport against the curse of dimensionality Guillaume Charpiat


  1. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  2. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  3. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  4. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric L 2 inner product vs. rigidifying inner product Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  5. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Motivation Introduction Motivation ◮ Shape spaces : which metric ? (to define similarity/distance between shapes) ◮ Hausdorff distance ◮ Symmetric difference area ◮ Quotients by transformation groups (rotation, translation, scaling, affine...) ◮ Shape evolution, morphing : priors on probable deformations ? = ⇒ Which local metric on deformations ? (metric on the manifold of shapes) ◮ L 2 norm of instantaneous deformations ◮ L 2 + curvature, H 1 ◮ rigid motion more probable = ⇒ associated metric ◮ = ⇒ learn the suitable metric from examples (datasets of shapes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  6. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  7. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  8. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  9. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  10. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Issues Issues ◮ Sparse sets of highly varying shapes ◮ e.g. human silhouettes ◮ high intrinsic dimension ( � 30) ◮ = ⇒ no dense training set ◮ to compare quantities defined on different shapes : need for correspondences ◮ match shape with different topologies ? ◮ very frequent topological changes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  11. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Searching for solutions Main existing approaches and their limitations Approach 1 : mean + modes model Approach 2 : distance-based approaches, such as kernel methods Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  12. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  13. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  14. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  15. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i ◮ PCA : diagonalize correlation matrix C : C ij = ˙ ˛ ¸ W M → S i ˛ W M → S j = ⇒ eigenmodes e k with eigenvalues λ k : best coordinate system Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  16. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) ◮ Mean M , shapes S i , warpings W M → S i ◮ PCA : diagonalize correlation matrix C : C ij = ˙ ˛ ¸ W M → S i ˛ W M → S j = ⇒ eigenmodes e k with eigenvalues λ k : best coordinate system ◮ any new deformation W of M : X W = α k e k + noise k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  17. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  18. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k ◮ associated inner product on deformations, in the tangent space of M : 1 X � W 1 | W 2 � = α 1 , k α 2 , k λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  19. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ Mahalanobis distance : d ( M + W , ( S )) = k λ 2 k k ◮ associated inner product on deformations, in the tangent space of M : 1 X � W 1 | W 2 � = α 1 , k α 2 , k λ 2 k k ◮ defines a deformation cost � W � 2 = � W | W � Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  20. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ probability p ( W ) ∝ exp ( − k ) : Gaussian distribution 2 λ 2 k k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  21. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) α 2 X ◮ probability p ( W ) ∝ exp ( − k ) : Gaussian distribution 2 λ 2 k k ◮ defines a Gaussian shape prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  22. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  23. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) ◮ Any inner product < | > P in tangent space of the mean ⇒ Gaussian distribution D P ( W ) ∝ exp( −� W � 2 = P ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  24. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Mean + modes model : PCA in tangent space (Gaussian distribution) X ◮ Empirical distribution : D emp = δ W M → Si i (possibly smoothed by a kernel) ◮ Any inner product < | > P in tangent space of the mean ⇒ Gaussian distribution D P ( W ) ∝ exp( −� W � 2 = P ) ◮ Best P for Kullback-Leibler( D P |D emp ) : PCA! Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  25. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 1 : mean + modes model → example from my PhD thesis ֒ Automatic alignment − → and average shape computation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  26. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Statistics (PCA) on deformation fields − → between the mean shape and each sample modes of deformation = deformation prior = Gaussian probabilistic model Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  27. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  28. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior ◮ requires a mean shape (does not always make sense, e.g. person walking) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  29. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Example of application : image segmentation with shape prior without shape prior with shape prior ◮ requires a mean shape (does not always make sense, e.g. person walking) ◮ requires all deformations between the mean and samples : ⇒ relatively similar sample shapes (otherwise, not reliable) = Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  30. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  31. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  32. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  33. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ Kernel : symmetric definite positive function k ( x , y ) ◮ Expresses the similarity between x and y ◮ Typically, the Gaussian kernel : k ( x , y ) = exp ( − d ( x , y ) 2 ) ◮ For each point x i : k i ( y ) := k ( x i , y ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  34. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ choice of a distance, of a kernel ? ◮ distance between 2 shapes : not much informative (wrt deformations) ◮ rebuild geometry of space of shapes from distances ? ◮ distances are not reliable/meaningful for far shapes ◮ = ⇒ needs for a representative neighborhood, i.e. a high dataset density (not affordable) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  35. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion State of the art Approach 2 : distance-based methods (e.g. kernel methods) ◮ choice of a distance, of a kernel ? ◮ distance between 2 shapes : not much informative (wrt deformations) ◮ rebuild geometry of space of shapes from distances ? ◮ distances are not reliable/meaningful for far shapes ◮ = ⇒ needs for a representative neighborhood, i.e. a high dataset density ◮ in a high-dimensional manifold, all distances are similar, and all points are on the boundary of the manifold ◮ = ⇒ cannot work, need for more information than distances Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  36. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  37. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  38. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  39. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences ◮ but shape matching reliable only for close shapes Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  40. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Main idea ◮ consider deformations (not just distances) ◮ should not require high density of training set ◮ no magic (to handle/interpolate sparse sets) : add a prior ◮ prior chosen : transported deformations make sense, i.e. a deformation observed on one shape can be applied to other shapes ◮ transport requires correspondences ◮ but shape matching reliable only for close shapes ◮ = ⇒ compute correspondences between close shapes only, and combine small steps of reliable correspondences to build longer-distance correspondences Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  41. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Main idea Map ◮ Close shape matching ◮ Transport ◮ Metric estimation (statistics on transported deformations) ◮ Theoretical justifications Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  42. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Shape matching Simple case : two shapes, A and B, with one connected component Z � f � 2 + α �∇ f � 2 dA inf f : A → B A - shape sampling - dynamic time warping - theory & experiments : higher sampling rate on target A A (s+ds) (s) A f f (s) (s+ds) B B m(s) B m(s+ds) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  43. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Shape matching Simple case : two shapes, A and B, with one connected component Z � f � 2 + α �∇ f � 2 dA inf f : A → B A - shape sampling - dynamic time warping - theory & experiments : higher sampling rate on target Usual case : random topologies Usual cases = more complex (more than 10 connected components in this silhouette) but [ one connected component → connected components i = the same Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  44. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Further possible improvements ◮ as such, allows appearing points (mismatches) ◮ allows disappearing points : matching to ∅ with a fixed high cost ◮ pb : better matchings, but energy value loses meaning Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  45. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Close shape matching Further possible improvements ◮ as such, allows appearing points (mismatches) ◮ allows disappearing points : matching to ∅ with a fixed high cost ◮ pb : better matchings, but energy value loses meaning Drawbacks ◮ specific to planar curves ◮ not symmetric : m A → B � = m − 1 B → A Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  46. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  47. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  48. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j ◮ Transport (translation, naive) : T L ∀ h : S j → X , j → i ( h ) : → X S i ` T L ´ j → i ( h ) ( s ) = h ( m i → j ( s )) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  49. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Local transport Transport Local transport ◮ Set of shapes ( S i ) i ∈ I (e.g. from a video segmentation) ◮ Two shapes S i and S j = ⇒ their correspondence field m i → j ◮ Transport (translation, naive) : T L ∀ h : S j → X , j → i ( h ) : S i → X T L ` ´ j → i ( h ) ( s ) = h ( m i → j ( s )) ⇒ reliability w L ◮ Associated cost : E ( m i → j ) = ` ´ i → j ∝ exp − α E ( m i → j ) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  50. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Global transport ⇒ reliability w L ◮ Associated cost : E ( m i → j ) = ` ´ i → j ∝ exp − α E ( m i → j ) ◮ close shapes : reliable; distant shapes : not reliable ◮ = ⇒ search for paths of small steps in the training set ( S i ) ◮ graph : nodes = shapes, edges = transport, weights = transport cost ◮ shortest path between pairs of shapes : global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  51. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Global transport ◮ Associated cost : E ( m i → j ) = ⇒ reliability w L i → j ∝ exp ` − α E ( m i → j ) ´ ◮ close shapes : reliable; distant shapes : not reliable ⇒ search for paths of small steps in the training set ( S i ) ◮ = ◮ graph : nodes = shapes, edges = transport, weights = transport cost ◮ shortest path between pairs of shapes : global transport ◮ compose : T G i → j = T L i n → j o T L o T L i 1 → i 2 o T L i n − 1 → i n o ... i → i 1 ◮ reliability : w G Y w L i → j = i k → i k +1 k ◮ use transport to propagate information Example : colored walker Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  52. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  53. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  54. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Correspondence field Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  55. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  56. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  57. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Forearm rotation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  58. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  59. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  60. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport to another shape Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  61. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  62. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported forearm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  63. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (translation) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  64. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transported arm rotation (better) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  65. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product > 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  66. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product = 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  67. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport L 2 inner-product < 0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  68. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Remarks about transport Why ? ◮ transport : of deformations : needed to increase training set density Which ? ◮ “translation” : ok for short pathes ◮ transport : not obvious (muscles + T-shirt artifacts) ◮ criterion to assess transport quality / suitability ? ◮ transport : should be learned (from video sequences ?) ◮ path could depend on deformation transported What properties ? ◮ probability of a deformation transported : can differ ◮ inner product : no reason to be transport-invariant Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  69. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  70. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent ⇒ connection Transport = Given transport, under few hypotheses (e.g. smoothness), it is possible to recover the associated infinitesimal connection : T h → 0 V γ ( h ) − V γ (0) ˛ = d γ dt T t → 0 ˛ ∇ X V = lim V γ ( t ) . γ ˛ h h → 0 ˛ t =0 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  71. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Global transport Transport in differential geometry A connection ∇ is Riemannian if the parallel transport it defines preserves the metric g . Metric connection : ∇ X g ( · , · ) = 0 for all vector fields X on M ◮ not satisfied (probability of a deformation depends on the shape) = ⇒ not Riemannian : transport and metric are independent ⇒ connection Transport = Given transport, under few hypotheses (e.g. smoothness), it is possible to recover the associated infinitesimal connection : T h → 0 V γ ( h ) − V γ (0) ˛ = d γ dt T t → 0 ˛ ∇ X V = lim V γ ( t ) . γ ˛ h h → 0 ˛ t =0 Connection = ⇒ transport : Given a covariant derivative ∇ , the transport along a curve γ is obtained by integrating the condition ∇ ˙ γ = 0 . Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  72. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  73. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  74. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  75. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  76. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  77. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  78. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Statistics on deformations Metric estimation (statistics on deformations) ◮ set of shapes ( S i ), local deformations m i → j , transport T G i → k ◮ = ⇒ transport deformations to a particular shape S k : f i → k i → j = T G i → k ( m i → j ) are, ∀ i , j , deformations defined on the same shape S k with reliability weights w k ij = w G i → k w L i → j ◮ statistics, for k fixed : PCA ◮ PCA with weights, and with H 1 -norm ⇒ eigenmodes e n (= principal deformations) with eigenvalues λ n ◮ = ⇒ defines an inner product P k = metric in the tangent space of the ◮ = shape S k ◮ P k varies smoothly as a function of k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  79. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : dancing sequence (9s, 24Hz), shape 1 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  80. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : shape 2 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  81. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Example of results Example of results : shape 3 Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  82. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  83. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  84. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Best metric for a given empirical manifold (all shapes together) ? ◮ needs a smoothness criterion ( = ⇒ transport) Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  85. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Theoretical justifications Theoretical justifications The best metric ? Searching for principal modes of deformations which vary smoothly (as a function of the shape S k ) ? ◮ vain quest : hairy ball theorem = ⇒ no best smooth direction field (or then it has to vanish sometimes) Best metric for a given distribution (on one shape) ? ◮ = ⇒ PCA gives the best metric for a criterion based on Kullback-Leibler divergence between distributions Best metric for a given empirical manifold (all shapes together) ? ◮ needs a smoothness criterion ( = ⇒ transport) ◮ = ⇒ best metric for a criterion involving transport & K-L divergence. ⇒ best metric for another criterion involving transport & L 2 -norm of ◮ = distributions. Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  86. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  87. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  88. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  89. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  90. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k j w L ◮ Empirical distributions : D emp i = P i → j δ f i → j ◮ Transported distribution : via T i → k ( δ f ) = δ T i → k ( f ) . X w G ◮ Criterion : best ( P k ) for KL ( D P k | T i → k ( D emp i )) ik i , k Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  91. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold ◮ set of shapes ( S i ), local deformations f i → j , transport T G i → k ◮ Empirical distributions : D emp i = P j w L i → j δ f i → j ◮ Transported distribution : via T i → k ( δ f ) = δ T i → k ( f ) . X w G ◮ Criterion : best ( P k ) for KL ( D P k | T i → k ( D emp i )) ik i , k X KL ( D P k |D T ◮ = best ( P k ) for emp k ) k where D T X w k emp k = i → j δ f k i → j i , j ◮ Transported deformations to any shape S k : f k i → j = T G i → k ( f i → j ) with reliability weights w k i → j = w G i → k w L i → j ◮ = the one obtained by weighted PCA on transported deformations Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

  92. Introduction Searching for solutions Shape matching Transport Metric estimation Theory Conclusion Best metric Best metric for a given empirical manifold (again!) ◮ empirical distributions : D emp i ◮ kernel-smoothed empirical distributions : D K emp i = g 0 i d µ ◮ g 0 i : density functions in the tangent space of the shape S i Guillaume Charpiat Pulsar project - INRIA Metrics that suit an empirical manifold of shapes

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend