Direct Fitting of Gaussian Mixture Models Leonid Keselman , Martial - PowerPoint PPT Presentation

Direct Fitting of Gaussian Mixture Models Leonid Keselman , Martial Hebert Robotics Institute Carnegie Mellon University May 29, 2019 https://github.com/leonidk/direct_gmm 1

Representations of 3D data Point Cloud Point Cloud Triangular + Normals Mesh Nearest Neighbor Plane Fit Screened Poisson Surface Reconstruction 2

Gaussian Mixture Models for 3D Shapes GMM fit to object surface Benefits • Closed-form expression • Can represent contiguous surfaces • Easy to build from noisy data • Sparse 3

Gaussian Mixture Model (GMM) ( Σ * is symmetric, $ ) * = 1 ! " = $ ) * +("; . * , Σ * ) positive-semidefinite * ) * ≥ 0 %&' 4

Gaussian Mixtures as a shape representation Efficient Representation Mesh Registration Frame Registration B. Eckart, K. Kim, J. Kautz. W. Tabib, C. O’Meadhra, N. Michael ECCV ( 2018) IEEE R-AL ( 2018) B. Eckart, K. Kim, A. Troccoli, A. Kelly, J. Kautz. 5 CVPR ( 2016)

Fitting a Gaussian Mixture Model 1. Obtain 3D Point Cloud 2. Select Initial Parameters 3. Iterate Expectation & Maximization i. E-Step: Each point gets a likelihood ii. M-Step: Each mixture gets parameters 6

7

The E-Step (Given GMM parameters) ! "# = 1 ' " ((* # ; , " , Σ " ) & # Affiliation between point j & mixture i & # = 0 ' 1 ((* # ; , 1 , Σ 1 ) 1 Normalization constant for point j 8

The M-Step (Given point-mixture weights) mixtures points ( * !" = $ $ + )% log / ) 0(2 % ; 4 ) , Σ ) ) lower-bound loss %&' )&' To get new parameters: takes derivatives, set equal to zero, and solve < )% = + )% 8!" 8!" 8!" = 0 = 0 = 0 ; ) = $ < )% 8Σ ) 8/ ) 84 ) % 4 ) = 1 Σ ) = 1 / ) = ; ) < )% (2 % −4 ) )(2 % −4 ) ) ? $ < )% 2 % $ ; ; = 9 % % ) )

10

Geometric Objects in a Probability Distribution Known curve in a given 2D probability distribution 11

Geometric Objects in a Probability Distribution , ℓ curve ≅ ( -(/ ) ) )*+ Consider sampling N points from this curve 12

Geometric Objects in a Probability Distribution + , , ℓ curve ≅ ( -(/ ) ) )*+ Take a geometric mean to account for sample number 13

Geometric Objects in a Probability Distribution + , , ℓ curve ≅ ( -(/ ) ) )*+ + , , ℓ curve = lim ,→6 ( -(/ ) ) )*+ + , , = lim ,→6 exp log ( -(/ ) ) )*+ , The curve will be the value in the limit 1 = lim ,→6 exp < = log(-(/ ) )) )*+ 14

Geometric Objects in a Probability Distribution + , , ℓ curve ≅ ( -(/ ) ) )*+ + , , ℓ curve = lim ,→6 ( -(/ ) ) )*+ + , , = lim ,→6 exp log ( -(/ ) ) )*+ , 1 = lim ,→6 exp < = log(-(/ ) )) = exp > log(-(/)) ?/ )*+ 15

Geometric Objects in a Probability Distribution ! = exp & log(+(,)) ., 1. I f +(,) = 0 on curve, then L = 0 2. Invariant to reparameterization 16

! " Area of each triangle # " Centroid of each triangle $ " , & " , ' " Triangle vertices 17

The E-Step (Given GMM parameters) ! "# = 1 ' " ((* # ; , " , Σ " ) & # Affiliation between point j & mixture i & # = 0 ' 1 ((* # ; , 1 , Σ 1 ) 1 Normalization constant for point j 2 # Area of each triangle , # Centroid of each triangle 3 # , 4 # , & # Triangle vertices 18

The New E-Step (Given GMM parameters) ! "# = 1 Taylor Approximation ' " ( # )(+ # ; + " , Σ " ) & (2 terms) # Affiliation between object j & mixture i & # = 0 ' 1 ( 1 )(+ # ; + 1 , Σ 1 ) 1 Normalization constant for object j ( # Area of each triangle + # Centroid of each triangle 2 # , 3 # , & # Triangle vertices 19

The M-Step (Given point-mixture weights) * " = % ( "' = 1 "' " + % " = & ( "' ! " = 1 ' & ( "' ) ' % ' " Σ " = 1 ( "' () ' −! " )() ' −! " ) 0 & % ' " 20

The New M-Step (Given point-mixture weights) * " = % ( "' = 2 ' 3 "' " + % " = & ( "' ! " = 1 ' & ( "' ) ' % ' " Σ " = 1 ( "' () ' −! " )() ' −! " ) 0 + Σ ' & % ' " 2 ' Area of each triangle Σ ' = 1 0 + 6 0 + 7 0 − 3 ! ' ! ' 0 12 4 ' 4 ' ' 6 ' 7 ! ' Centroid of each triangle ' ' 4 ' , 6 ' , 7 ' Triangle vertices 21

What is Σ " ? 22

23

Results Did all that math actually help us fit better/faster GMMs? 24

Evaluate across a wide Using different inputs Measure the likelihood range of mixtures of a high-density point (6 to 300) classic algorithm cloud (higher is better) • Vertices of the mesh • Triangle centroids our method • Approximate (E only) • Exact (E + M steps) 25

Full E+M method works in all cases 26

Stable under even random initialization! 27

Applications Are these models actually more useful? 28

Mesh Registration (P2D) Method 1. Apply a random rotation + translation to the point cloud 2. Find transformation to maximize the likelihood of the points • Perform P2D with GMMs fit to i. mesh vertices ii. mesh triangles Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” 29 ECCV (2018)

Mesh-based GMMs are more accurate 30

Across multiple models Rotation Error Translation Error (% of ICP) (% of ICP) 150 200 100 100 50 0 0 Armadillo Bunny Dragon Happy Lucy Armadillo Bunny Dragon Happy Lucy points mesh points mesh 31

Frame Registration (D2D) Method 1. Use a sequence from an RGBD Sensor • 2,500 frame TUM sequence from a Microsoft Kinect 2. Pairwise registration between t & t-1 frames • Optimize the D2D L2 distance • Build GMMs using square pixels as the geometric object W. Tabib, C. O’Meadhra, N. Michael . “On-Manifold GMM Registration” 32 IEEE R-AL (2018)

Representing points using pixel squares 33

D2D Registration Results Compared to standard GMM • 2.4% improvement in RMSE • 22% faster D2D convergence 34

Que Questions ns? ! = exp & log(+(,)) ., 35

The End! 36

Extra Slides 37

How to fit a Gaussian Mixture Model? 1. Obtain any collection of objects 2. Perform Expectation + Maximization i. E-Step: Each point gets a likelihood ii. M-Step: Each mixture gets new parameters 38

Extension to arbitrary primitives Vasconcelos, Lippman. "Learning mixture hierarchies.” 39 Advances in Neural Information Processing Systems ( 1999)

Approximation area-weighted geometric mean using the primitive’s centroids 40

Product Integral Formulation • Product integrals provide a resampling-invariant loss function • Given S samples, of M primitives, with N mixture components • This can be evaluated in the limit of samples (with a geometric mean) 41

42

43

44

For GMMs we will use the lower bound 45

P2D Registration Results Model Rotation Error Translation Error (% of ICP) (% of ICP) points mesh points mesh 127 37 37 161 33 33 Armadillo 50 28 28 41 17 17 Bunny 68 25 25 40 19 19 Dragon 101 27 27 85 27 27 Happy 95 23 23 122 35 35 Lucy 46

Mesh Registration with P2D Method 1. Apply a random rotation + translation to the point cloud 2. Point-to-Distribution (P2D) registration of point cloud to GMM • Perform tests with GMMs fit to i. mesh vertices ii. mesh triangles • Optimize the GMM likelihood with rigid body transformation (q & t) • BFGS Optimization using numerical gradients, starting from identity Eckart, Kim, Kautz. “HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration.” 47 ECCV (2018)

Acknowledgements • Martial Hebert • Robotics Institute • Reviewers • Funding? 48

Representing points using pixel squares 49