3D Descriptor Design and Learning for Robust Non-rigid Shape - - PowerPoint PPT Presentation

▶
3d descriptor design and learning for robust non rigid
SMART_READER_LITE
LIVE PREVIEW

3D Descriptor Design and Learning for Robust Non-rigid Shape - - PowerPoint PPT Presentation

May 23, 2023 137 likes •612 views

3D Descriptor Design and Learning for Robust Non-rigid Shape Matching Jianwei Guo NLPR, Institute of Automation, Chinese Academy of Sciences Sept.17, 2020 GAMES Webinar 1 Team--3D Visual Computing [ACM TOG (SIGGRAPH Asia) 2012]

slide-1
SLIDE 1

3D Descriptor Design and Learning for Robust Non-rigid Shape Matching

Jianwei Guo 郭建伟

1

Sept.17, 2020

NLPR, Institute of Automation, Chinese Academy of Sciences

GAMES Webinar

slide-2
SLIDE 2

2

Team--3D Visual Computing

Mode delin ing & Mesh sh Optimi imizatio zation Shape Analysis alysis

[CGF 2018] [TVCG 2017] [ACM TOG (SigAsia) 2016] [TVCG 2018] [TVCG 2020] [TVCG 2020] [ACM TOG 2020] [CAGD 2016] [CAD 2019] [IEEE TIFS 2018] [ACM TOG (SIGGRAPH) 2020] [CVPR 2019] [ECCV 2018] [ACM TOG (SIGGRAPH Asia) 2012] [GMP 2006, CAD 2012, JCAD 2018] [CGF 2019] [CAD 2020] [CAGD 2018]

slide-3
SLIDE 3

3

3D Shape Descriptor

Local descriptor Global descriptor

slide-4
SLIDE 4

4

3D Shape Descriptor

Local descriptor Descriptor design goals:

  • Discriminative: able to determine if a pair
  • f vertices is similar or different
  • Robust: work with different discretizations
  • f a surface
slide-5
SLIDE 5

5

Applications

Shape matching/correspondence Shape segmentation Shape retrieval

[Ovsjanikov et al. 2017;Wang et al. 2018] [Rustamov 2007] [Ovsjanikov et al. 2009] [Lui et al. 2010]

Surface registration

slide-6
SLIDE 6

6

Shape matching/correspondence

slide-7
SLIDE 7

7

Shape matching/correspondence

Van Kaick et al.: A survey on shape correspondence. CGF, 2011. Ovsjanikov et al.: Computing and processing correspondences with functional maps. SIGGRAPH ASIA 2016 Courses.

slide-8
SLIDE 8

8

Related Work

❖ Spatial domain approaches

SI [Johnson and Hebert 1999] 3DSC [Frome et al. 2004] TriSI [Guo et al. 2015] RoPS [Guo et al. 2013] PFH [Rusu et al. 2008] FPFH [Rusu et al. 2009] SHOT [Tombari et al. 2010] Mesh-HOG [Zaharescu et al. 2009] Guo et al.: A comprehensive performance evaluation of 3D local feature descriptors. IJCV, 2016.

slide-9
SLIDE 9

9

Related Work

❖ Spectral domain approaches

Shape-DNA [1999Reuter et al. 2006] GPS [Rustamov 2007 ] HKS [Sun et al. 2010] Scale-invariant HKS [Bronstein and Kokkinos 2010] WKS [Aubry et al. 2011] DTEP [Melziet al. 2018]

slide-10
SLIDE 10

10

Related Work

❖ Deep learning approaches

[Huang et al. 2018] [Zeng et al. 2017] CGF [Khoury et al. 2017] OSD [Litman and Bronstein 2014] [Boscaini et al. 2015, 2016] [Masci et al. 2018]

  • Mesh-aware

PPFNet [Deng et al. 2018]

slide-11
SLIDE 11

11

Related Work

  • + pros: : WKS and HKS are robust
  • + pros: GNNs compute discriminative

descriptors

  • - cons : not as discriminative as

supervised descriptors

  • - cons : less robust to different surface

discretizations

❖ Non-learned descriptors ❖ Supervised descriptors

slide-12
SLIDE 12

12

Datasets

SPRING [Yang et al. 2014] FAUST [Bogo et al. 2014] SCAPE [Anguelov et al. 2005] TOSCA [Bronstein et al. 2008]

slide-13
SLIDE 13

13

Our Work

❖ Contributions

— Two new non-learned features, namely, Local Point Signature (LPS) and Wavelet Energy Decomposition Signature (WEDS). They exhibit high resilience to changes in mesh resolution, triangulation, scale, and rotation. — Two supervised frameworks to transform the non-learned features to more discriminative descriptors

[ACM TOG (SIGGRAPH) 2020] [CVPR 2019] [ECCV 2018] [CVMJ 2020]

slide-14
SLIDE 14

14

❖ Motivation

1-Descriptor Learning using Geometry Images

Geometry images [Gu et al. 2002]

slide-15
SLIDE 15

15

Desbrun et al.: Intrinsic parameterizations of surface meshes. CGF, 2002

1-Descriptor Learning using Geometry Images

❖ Motivation

slide-16
SLIDE 16

16

❖ Triplet Neural Network

1-Descriptor Learning using Geometry Images

slide-17
SLIDE 17

17

❖ Triplet Loss

  • Classic triplet loss:
  • Min-Coefficient of Variation (Min-CV) loss:

1-Descriptor Learning using Geometry Images

slide-18
SLIDE 18

18

❖ Results: shape matching

1-Descriptor Learning using Geometry Images

slide-19
SLIDE 19

19

❖ Discrete Geodesic Polar Coordinates (DGPC)

Melvær and Reimers. Geodesic polar coordinates on polygonal meshes. CGF, 2012

DGPC [Melvær and Reimers 2012] ill-shaped triangles degenerate triangles

1-Descriptor Learning using Geometry Images

slide-20
SLIDE 20

20

❖ Results：shape correspondence FAUST SPRING

  • Cumulative match characteristic (CMC)
  • Cumulative geodesic error （CGE）

1-Descriptor Learning using Geometry Images

slide-21
SLIDE 21

21

❖ Results：shape correspondence SCAPE

1-Descriptor Learning using Geometry Images

slide-22
SLIDE 22

22

❖ Results：shape correspondence FAUST

1-Descriptor Learning using Geometry Images

slide-23
SLIDE 23

23

Resolutions, triangulations, scales, rotations

Approach? Dataset?

❖ Motivation

2-Robust Local Spectral Descriptor

slide-24
SLIDE 24

24

( ) ( ) ( ) ( )

2 S S

E f f v dv f v f v dv =  = 

 

: f R → S

❖ Motivation

2-Robust Local Spectral Descriptor

Dirichlet energy

slide-25
SLIDE 25

25

2-Robust Local Spectral Descriptor

( ) ( ) ( ) ( )

2 S S

E f f v dv f v f v dv =  = 

 

( )

div f f  = − 

( )

( )

( )

( )

( )

cot cot 1

  • 2

i

ij ij i i j j N v i

f v f v f v a  

+  =

M Mesh

: f V R →

cot cot if , 2 cot cot if 2

ij ij i ik ik ij k i

i j areadjacent a i j a

  • therwise

    +  −    + = =     

L , 0,1,..., 1

i i i i

k   =  = − L  

| 0,1,..., 1

i i

k  = −

❖ Local Point Signature

Laplace–Beltrami operator

slide-26
SLIDE 26

26

2-Robust Local Spectral Descriptor

( )

1 T 2 =0 N j j j

E f f f  

−

= = AL

,

T j j j

f f  =  = 

A

A

( )

1 1 2 2 1 =0 =0 1 d N N d ij j j ij i j j i

E F    

− − = =

= =

  

1 2

( , ,..., ) :

d d

F f f f R V = →

, 0,1,..., 1

i i i i

k   =  = − T A

,

T i j i j

  =  

A

A

2 2 2 1 1 2 2 1 1 1 1 1

, ,...,

d d d i i N iN i i i

sf      

− − = = =

  =    

  

❖ Local Point Signature

Spectral coefficients

slide-27
SLIDE 27

27

2-Robust Local Spectral Descriptor

❖ Local Point Signature

3 1 2 3

( , , ): X x x x V R = →

3 3 3 2 2 2 1 1 2 2 16 16 1 1 1

, ,..,

i i i i i i

LPS      

= = =

  =      

  

,

T j j j

f f  =  = 

A

A

  • Compute Laplacian eigenvectors and eigenvalues
  • Compute spectral coefficients
  • LPS is derived from Dirichlet energy and expressed as follows
  • Build a local patch mesh around a vertex
slide-28
SLIDE 28

28

2-Robust Local Spectral Descriptor

❖ Descriptor learning

slide-29
SLIDE 29

29

2-Robust Local Spectral Descriptor

❖ Descriptor learning

slide-30
SLIDE 30

30

2-Robust Local Spectral Descriptor

❖ Dataset

Discrete optimization method (Wang et al. 2019)

slide-31
SLIDE 31

31

2-Robust Local Spectral Descriptor

❖ Results: Robust to resolution and triangulation

slide-32
SLIDE 32

32

2-Robust Local Spectral Descriptor

❖ Results: Comparison

slide-33
SLIDE 33

34

3-Descriptor Learning using Multiscale GCNs

❖ Motivation cut a geodesic disk

– Time-consuming – Miss global information

slide-34
SLIDE 34

35

3-Descriptor Learning using Multiscale GCNs

❖ Graph wavelets

Scaling functions Wavelet functions Wavelet filter functions

  • Multiscale property of wavelet functions
slide-35
SLIDE 35

36

3-Descriptor Learning using Multiscale GCNs

❖ Graph wavelets

  • Reconstruction property of multiscale wavelet functions
  • Robustness to the change of resolution and triangulation
slide-36
SLIDE 36

37

3-Descriptor Learning using Multiscale GCNs

❖ Wavelet Energy Decomposition Signature (WEDS) Reconstruct discrete Dirichlet energy Restructure into a sum per vertex Collect the local energy using multiscale wavelets

slide-37
SLIDE 37

38

3-Descriptor Learning using Multiscale GCNs

❖ Wavelet Energy Decomposition Signature (WEDS)

slide-38
SLIDE 38

39

3-Descriptor Learning using Multiscale GCNs

❖ Multiscale Graph Convolution Network

  • Convolutions on graphs are defined as:
  • ChebyNet approximates using 𝑛-order polynomials:
  • MGCN approximates using multiscale wavelet filter basis:
slide-39
SLIDE 39

40

3-Descriptor Learning using Multiscale GCNs

❖ Multiscale Graph Convolution Network

  • The multiscale convolution can be simplified as follows:

Convolve local and global information Robustness to discretization

slide-40
SLIDE 40

41

3-Descriptor Learning using Multiscale GCNs

❖ Results: performance of WEDS

slide-41
SLIDE 41

42

3-Descriptor Learning using Multiscale GCNs

❖ Results: shape correspondence FAUST

slide-42
SLIDE 42

43

3-Descriptor Learning using Multiscale GCNs

❖ Results: robust to resolution and triangulation

slide-43
SLIDE 43

44

3-Descriptor Learning using Multiscale GCNs

❖ Results: robust to resolution and triangulation Extended FAUST

slide-44
SLIDE 44

45

3-Descriptor Learning using Multiscale GCNs

❖ Results: non-isometric shapes

slide-45
SLIDE 45

46

Conclusions

❖ Contributions

— Two non-learned features: Local Point Signature (LPS) and Wavelet Energy Decomposition Signature (WEDS). — Two supervised frameworks to transform the non-learned features to more discriminative descriptors

❖ Future work

— Extend to point clouds and triangle soups — Industrial applications

slide-46
SLIDE 46

47

Acknowledgements

Thank you!

Dong-Ming Yan, CASIA Yiqun Wang, CASIA Hanyu Wang, University of Maryland-College Park Jing Ren, Peter Wonka， KAUST CCF-腾讯犀牛鸟科研基金 Co-authors:

LDGI: https://github.com/jianweiguo/local3Ddescriptorlearning LPS: https://github.com/yiqun-wang/LPS MGCN: https://github.com/yiqun-wang/MGCN

Code and Data: