Shape Representation Jin Xie Department of Computer Science and - PowerPoint PPT Presentation

Deep Learning Based 3D Shape Representation Jin Xie Department of Computer Science and Engineering Nanjing University of Science and Technology, China 1

Outline • Overview of 3D deep learning • Deep learned 3D shape feature for retrieval • Learned 3D shape spectral feature for correspondence • Learned barycentric representation of 3D shape for cross-domain retrieval 2

Overview of 3D deep learning • 3D data representation format:  RGB-D image  Mesh  Point cloud Mesh RGB-D Point cloud 3

Overview of 3D deep learning • Academic community: very active from 2015  Large 3D dataset: ShapeNet (Stanford), ModelNet (Princeton)  Intersection of three areas: computer graphics/computer vision/machine learning  Industry community: broad applications  Robotics  Autonomous driving  Virtual reality  3D print/smart manufacturing  … 4

Overview of 3D deep learning • Challenges in 3D deep learning:  3D model: geometric structure information ; 2D image: pixel value  3D model: irregular data structure; 2D image: regular data structure (From Wikipedia) 5

Overview of 3D deep learning • Challenges in 3D deep learning:  Large deformations of 3D shapes  Large structure variations of 3Dshapes  Partial models of 3D shapes 6

Deep learned 3D shape feature • Deep learning based 3D shape feature (Global):  Diffusion geometry [1]  Voxelization [2]  Projection [3] 7

Deep learned 3D shape feature [1] J. Xie, Y. Fang, F. Zhu and E. K. Wong, Deepshape:deep learned shape descriptor for 3D shape matching and retrieval, CVPR 2015. [2] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D shapenets: A deep representation for volumetric shapes, CVPR 2015. [3] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3D shape recognition, ICCV 2015. [4] S. Bai, X. Bai, Z. Zhou, Z. Zhang, and L. Jan Latecki. Gift: A real-time and scalable 3D shape search engine, CVPR 2016. [5] J. Xie, M. Wang, Y. Fang. Learned Binary Spectral Shape Descriptor for 3D Shape Correspondence, CVPR 2016. [6] L. Wei, Q. Huang, D. Ceylan, E. Vouga and H. Li. Dense human body correspondences using convolutional networks, CVPR 2016. [7] Y. Li, H. Su, X. Guo, L. J. Guibas. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation, CVPR 2017. [8] R. Qi, H. Su, K. Mo, L. J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, CVPR 2017. [9] G. Riegler, A. O. Ulusoy, A. Geiger. OctNet: Learning Deep 3D Representations at High Resolutions, CVPR 2017. [10] R. Klokov, V. S. Lempitsky. Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models, ICCV 2017. [11] D. Litany, T. Remez, E. Rodola, A.M. Bronstein, M.M. Bronstein. Deep Functional Maps: Structured Prediction for Dense Shape Correspondence, ICCV 2017. [12] R. Qi, Y. Li, H. Su, L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, NIPS 2017. 8

Deep learned 3D shape feature for retrieval • Heat diffusion based 3D shape feature J. Xie, Y. Fang, F. Zhu and E. K. Wong, Deepshape:deep learned shape descriptor for 3D shape matching and retrieval, CVPR 2015. 9

Deep learned 3D shape feature for retrieval • Heat diffusion based 3D shape feature(Global)  Employ heat kernel signature (HKS) to form shape distribution  Develop discriminative auto-encoder to learn global 3D shape feature 10

Deep learned 3D shape feature for retrieval • Heat kernel signature  Heat diffusion equation on a shape:  K   t LK  t t is the heat kernel, is the Laplace-Beltrami operator. K L t  n  v  w , if i j  i     i j , cot cot   i 1  i j , i j , , if i ~ j        1 W w , if i ~ j w L A W 2  i j , i j ,    0, otherwise ij 0, otherwise    ij v j 11

Deep learned 3D shape feature for retrieval • Heat kernel signature (HKS)  Given an initial Dirac delta distribution, the solution of heat diffusion equation:   K exp( tL ) t  Based on the spectral decomposition theorem:      v t k x x ( , ) e ( x ) ( x ) i t j m i j i m i  Heat kernel signature: diagonal value of heat kernel     v t 2 k x ( ) e ( x ) i t j i j i 12

Deep learned 3D shape feature for retrieval • Heat kernel signature  Heat kernel describes the quantity of heat passing from one vertex to another vertex after time interval t. J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on heat diffusion, Proceedings of the Symposium on Geometry Processing, 2009. 13

Deep learned 3D shape feature for retrieval • Heat diffusion in SIFT (Scale invariant feature transform) D. Lowe, Distinctive features from scale-invariant keypoints, IJCV 2004. 14

Deep learned 3D shape feature for retrieval • Multiscale shape distribution:  Use the histogram to estimate the probabilistic distribution of HKSs of vertices at each scale: 15

Deep learned 3D shape feature for retrieval • Learn deep feature with discriminative auto-encoder C 1 1 1    2   2    t t t t t t t J W b ( , ) x G F x ( ( )) W ( ( tr S ( )) z tr S z ( ( ))) i i w b 2 F 2 F 2  i 1 16

Deep learned 3D shape feature for retrieval • Learned shape descriptor: 17

• Comparison evaluations: Shrec’14 Human dataset Shec’14 LSCRTB dataset 18

Learned binary spectral shape descriptor for correspondence • Learn spectral shape descriptor (local): J. Xie, M. Wang, Y. Fang. Learned Binary Spectral Shape Descriptor for 3D Shape Correspondence, CVPR 2016. 19

Learned binary spectral shape descriptor for correspondence • Learn spectral shape descriptor :  Construct spectral representation of 3D shapes: geometry vector 2 (𝑦 𝑘 ), 𝜚 2 2 (𝑦 𝑘 ),⋅⋅⋅, 𝜚 𝑡 2 (𝑦 𝑘 ) 𝑕(𝑦 𝑘 ) = (𝑐(𝑤 1 ), 𝑐(𝑤 2 ),⋅⋅⋅, 𝑐(𝑤 𝑡 )))𝜚(𝑦 𝑘 ), 𝜚(𝑦 𝑘 ) = [𝜚 1 is the cubic B-spine basis function. b v ( ) s 20

Learned binary spectral shape descriptor for correspondence • Binary spectral shape descriptor with metric learning:     N N  1 1  1 1 2 2       2   2 K K K K K J W b ( , ) min h h h h b h W w b , i j i j i i F M 2 2 M 2 2 N 2 2     i 1 j x i 1 j x   i i  K b sgn( h ) i i are the positive/negative point pairs on a pair of shapes. x , x   i i 21

Learned binary spectral shape descriptor for correspondence • Evaluation: Tosca dataset: 16 bit 32 bit 64 bit 22

Scape dataset: 23

Learn barycentric representation of 3D shapes • Learn barycentric representation of 3D shapes for sketch-based 3D shape retrieval: J. Xie, G. Dai, F. Zhu and Y. Fang, Learning barycentric representations of 3D shapes for sketch-based 3D shape retrieval, CVPR 2017. 24

Learn barycentric representation of 3D shapes • Multi-view CNN based 3D shape representation H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3D shape recognition, ICCV 2015. 25

Learn barycentric representation of 3D shapes • Barycentric representation of 3D shapes:  Max-view pooling does not exploit information from all views  Wasserstein barycenters as a nonlinear pooling operation  Optimal transportation: The set of transportation plans between probability distributions p and q: 𝑠×𝑡 ; 𝑈1 = 𝑞, 𝑈 𝑈 1 = 𝑟 𝑆(𝑞, 𝑟) = {𝑈 ∈ ℝ + The distance can be defined: D p q ( , )    D p q ( , ) min M T ,  T R p q ( , ) Regularized optimal transportation:             D p q ( , ) min M T , T ,log T M / T diag u Kdiag v K ( ) ( ), e  T R p q ( , ) M.Cuturi. Sinkhorn distances: lightspeed computation of optimal transport, NIPS 2013. 26

Learn barycentric representation of 3D shapes • Isotropic Wasserstein barycenters of 3D shapes: n    , is the Wasserstein distance. argmin D p p ( , ) D p p ( , ) p i b i b i b  i 1  It can be solved with the Sinkhorn fixed-point algorithm. N. Bonneel, G. Peyre, and M. Cuturi, Wasserstein barycentric coordinates: histogram regression using optimal transport, ACM Trans. Graphics, 2016. 27

Learn barycentric representation of 3D shapes • Cross-domain matching with learned Wasserstein barycenters: n n 1   1   2 2 2 2          2 1 2 1 argmin z z max(0, z z ) L L     , j i j i 1 1 2 2 1 2 n 2 m 2     j 1 i c j ( ) j 1 i c j ( ) j j 28

Learn barycentric representation of 3D shapes • Sketch-based 3D shape retrieval: 29

• Comparison evaluations: Shrec’13 dataset Shrec’14 dataset 30