3D Descriptor Design and Learning for Robust Non-rigid Shape Matching Jianwei Guo 郭建伟 NLPR, Institute of Automation, Chinese Academy of Sciences Sept.17, 2020 GAMES Webinar 1
Team--3D Visual Computing [ACM TOG (SIGGRAPH Asia) 2012] [GMP 2006, CAD 2012, JCAD 2018] [ACM TOG (SigAsia) 2016] [CAGD 2016] [TVCG 2017] [ CAGD 2018 ] [ IEEE TIFS 2018 ] [CAD 2020] [CGF 2018] [CAD 2019] [TVCG 2018] [CGF 2019] [TVCG 2020] [TVCG 2020] [ACM TOG 2020] [ECCV 2018] [CVPR 2019] [ ACM TOG (SIGGRAPH) 2020] Mode delin ing & Mesh sh Optimi imizatio zation Shape Analysis alysis 2
3D Shape Descriptor Local descriptor Global descriptor 3
3D Shape Descriptor Descriptor design goals: Discriminative: able to determine if a pair • of vertices is similar or different Robust: work with different discretizations • of a surface Local descriptor 4
Applications Shape matching/correspondence Shape retrieval [Ovsjanikov et al. 2017;Wang et al. 2018] [Ovsjanikov et al. 2009] Shape segmentation Surface registration [Rustamov 2007] 5 [Lui et al. 2010]
Shape matching/correspondence 6
Shape matching/correspondence Van Kaick et al.: A survey on shape correspondence . CGF, 2011. 7 Ovsjanikov et al.: Computing and processing correspondences with functional maps . SIGGRAPH ASIA 2016 Courses.
Related Work Spatial domain approaches ❖ SI [Johnson and Hebert 3DSC [Frome et al. 2004] RoPS [Guo et al. 2013] SHOT [Tombari et al. 2010] 1999] PFH [Rusu et al. 2008] TriSI [Guo et al. 2015] FPFH [Rusu et al. 2009] Mesh-HOG [Zaharescu et al. 2009] Guo et al.: A comprehensive performance evaluation of 3D local feature descriptors . IJCV, 2016. 8
Related Work Spectral domain approaches ❖ Shape-DNA [1999Reuter et al. GPS [Rustamov 2007 ] HKS [Sun et al. 2010] 2006] Scale-invariant HKS [Bronstein WKS [Aubry et al. 2011] DTEP [Melziet al. 2018] and Kokkinos 2010] 9
Related Work Deep learning approaches ❖ [Zeng et al. 2017] [Huang et al. 2018] CGF [Khoury et al. 2017] PPFNet [Deng et al. 2018] • Mesh-aware OSD [Litman and Bronstein 2014] [Boscaini et al. 2015, 2016] [Masci et al. 2018] 10
Related Work Non-learned descriptors ❖ Supervised descriptors ❖ • + pros : : WKS and HKS are robust • + pros: GNNs compute discriminative descriptors • - cons : not as discriminative as • - cons : less robust to different surface supervised descriptors discretizations 11
Datasets SCAPE [Anguelov et al. 2005] TOSCA [Bronstein et al. 2008] SPRING [Yang et al. 2014] FAUST [Bogo et al. 2014] 12
Our Work Contributions ❖ — Two new non-learned features, namely, Local Point Signature (LPS) and Wavelet Energy Decomposition Signature (WEDS). They exhibit high resilience to changes in mesh resolution, triangulation, scale, and rotation. — Two supervised frameworks to transform the non-learned features to more discriminative descriptors [ECCV 2018] [CVPR 2019] [ CVMJ 2020 ] [ ACM TOG (SIGGRAPH) 2020] 13
1-Descriptor Learning using Geometry Images Motivation ❖ Geometry images [Gu et al. 2002] 14
1-Descriptor Learning using Geometry Images Motivation ❖ Desbrun et al.: Intrinsic parameterizations of surface meshes . CGF, 2002 15
1-Descriptor Learning using Geometry Images Triplet Neural Network ❖ 16
1-Descriptor Learning using Geometry Images Triplet Loss ❖ • • Classic triplet loss: Min-Coefficient of Variation (Min-CV) loss: 17
1-Descriptor Learning using Geometry Images Results: shape matching ❖ 18
1-Descriptor Learning using Geometry Images Discrete Geodesic Polar Coordinates (DGPC) ❖ ill-shaped triangles degenerate triangles DGPC [ Melvær and Reimers 2012 ] Melvær and Reimers. Geodesic polar coordinates on polygonal meshes . CGF, 2012 19
1-Descriptor Learning using Geometry Images Results ： shape correspondence ❖ • Cumulative match characteristic (CMC) • Cumulative geodesic error （ CGE ） FAUST SPRING 20
1-Descriptor Learning using Geometry Images Results ： shape correspondence ❖ SCAPE 21
1-Descriptor Learning using Geometry Images Results ： shape correspondence ❖ FAUST 22
2-Robust Local Spectral Descriptor Motivation ❖ Approach? Dataset? Resolutions, triangulations, scales, rotations 23
2-Robust Local Spectral Descriptor Motivation ❖ Dirichlet energy → : S f R ( ) ( ) ( ) ( ) 2 = = E f f v dv f v f v dv 24 S S
2-Robust Local Spectral Descriptor Local Point Signature ❖ ( ) ( ) ( ) ( ) 2 = = ( ) E f f v dv f v f v dv = − div f f S S Laplace – Beltrami operator + − cot cot ij ij if , i j areadjacent 2 a i + cot cot = = ik ik L if i j ij 2 a k i 0 otherwise + ( ) cot cot ( ) 1 ( ) ( ) = ij ij = = − - = − Mesh f v f v f v L , 0,1,..., 1 | 0,1,..., 1 M i i k i i k i i j i i 2 a ( ) → j N v i : f V R i 25
2-Robust Local Spectral Descriptor Local Point Signature ❖ = = − = , 0,1,..., 1 T T A i i k , A i i i j i j A Spectral coefficients − ( ) N 1 = = = = T 2 AL T E f f f , f f A j j j j j A j =0 − − ( ) 1 1 d N N d = → = = d 2 2 ( , ,..., ) : E F F f f f V R ij j j ij 1 2 d = = i 1 j =0 j =0 i 1 d d d = 2 2 2 , ,..., sf − − 1 1 2 2 1 1 i i N iN = = = i 1 i 1 i 1 26
2-Robust Local Spectral Descriptor Local Point Signature ❖ • Build a local patch mesh around a vertex • Compute Laplacian eigenvectors and eigenvalues • Compute spectral coefficients = = = → T , 3 f f A ( , , ): X x x x V R j j j 1 2 3 A • LPS is derived from Dirichlet energy and expressed as follows 3 3 3 = 2 2 2 , ,.., LPS 1 1 2 2 16 16 i i i = = = 1 1 1 i i i 27
2-Robust Local Spectral Descriptor Descriptor learning ❖ 28
2-Robust Local Spectral Descriptor Descriptor learning ❖ 29
2-Robust Local Spectral Descriptor Dataset ❖ Discrete optimization method ( Wang et al. 2019) 30
2-Robust Local Spectral Descriptor Results: Robust to resolution and triangulation ❖ 31
2-Robust Local Spectral Descriptor Results: Comparison ❖ 32
3-Descriptor Learning using Multiscale GCNs Motivation ❖ – Time-consuming – Miss global information cut a geodesic disk 34
3-Descriptor Learning using Multiscale GCNs Graph wavelets ❖ Wavelet filter functions Scaling functions Wavelet functions • Multiscale property of wavelet functions 35
3-Descriptor Learning using Multiscale GCNs Graph wavelets ❖ • Reconstruction property of multiscale wavelet functions • Robustness to the change of resolution and triangulation 36
3-Descriptor Learning using Multiscale GCNs Wavelet Energy Decomposition Signature (WEDS) ❖ Reconstruct discrete Dirichlet energy Restructure into a sum per vertex Collect the local energy using multiscale wavelets 37
3-Descriptor Learning using Multiscale GCNs Wavelet Energy Decomposition Signature (WEDS) ❖ 38
3-Descriptor Learning using Multiscale GCNs Multiscale Graph Convolution Network ❖ • Convolutions on graphs are defined as: • ChebyNet approximates using 𝑛 -order polynomials: • MGCN approximates using multiscale wavelet filter basis: 39
3-Descriptor Learning using Multiscale GCNs Multiscale Graph Convolution Network ❖ • The multiscale convolution can be simplified as follows: Robustness to discretization Convolve local and global information 40
3-Descriptor Learning using Multiscale GCNs Results: performance of WEDS ❖ 41
3-Descriptor Learning using Multiscale GCNs Results: shape correspondence ❖ FAUST 42
3-Descriptor Learning using Multiscale GCNs Results: robust to resolution and triangulation ❖ 43
3-Descriptor Learning using Multiscale GCNs Results: robust to resolution and triangulation ❖ Extended FAUST 44
3-Descriptor Learning using Multiscale GCNs Results: non-isometric shapes ❖ 45
Conclusions Contributions ❖ — Two non-learned features: Local Point Signature (LPS) and Wavelet Energy Decomposition Signature (WEDS). — Two supervised frameworks to transform the non-learned features to more discriminative descriptors Future work ❖ — Extend to point clouds and triangle soups — Industrial applications 46
Acknowledgements Co-authors: CCF- 腾讯犀牛鸟科研基金 Dong-Ming Yan , CASIA Yiqun Wang , CASIA Hanyu Wang , University of Maryland-College Park Jing Ren, Peter Wonka ， KAUST Code and Data: LDGI: https://github.com/jianweiguo/local3Ddescriptorlearning LPS: https://github.com/yiqun-wang/LPS MGCN: https://github.com/yiqun-wang/MGCN Thank you! 47
Recommend
More recommend
