Shape Representation Jin Xie Department of Computer Science and - - PowerPoint PPT Presentation

Deep Learning Based 3D Shape Representation

Jin Xie Department of Computer Science and Engineering Nanjing University of Science and Technology, China

1

Outline

• Overview of 3D deep learning
• Deep learned 3D shape feature for retrieval
• Learned 3D shape spectral feature for correspondence
• Learned barycentric representation of 3D shape for cross-domain

retrieval

2

Overview of 3D deep learning

• 3D data representation format:
• RGB-D image
• Mesh
• Point cloud

RGB-D Mesh Point cloud

3

Overview of 3D deep learning

• Academic community: very active from 2015
• Large 3D dataset: ShapeNet (Stanford), ModelNet (Princeton)
• Intersection of three areas: computer graphics/computer vision/machine

learning

 Industry community: broad applications

• Robotics
• Autonomous driving
• Virtual reality
• 3D print/smart manufacturing
• …

4

Overview of 3D deep learning

• Challenges in 3D deep learning:
• 3D model: geometric structure information ;

2D image: pixel value

• 3D model: irregular data structure;

2D image: regular data structure

(From Wikipedia)

5

Overview of 3D deep learning

• Challenges in 3D deep learning:
• Large deformations of 3D shapes
• Large structure variations of 3Dshapes
• Partial models of 3D shapes

6

Deep learned 3D shape feature

• Deep learning based 3D shape feature (Global):
• Diffusion geometry [1]
• Voxelization [2]
• Projection [3]

7

Deep learned 3D shape feature

[1] J. Xie, Y. Fang, F. Zhu and E. K. Wong, Deepshape:deep learned shape descriptor for 3D shape matching and retrieval, CVPR 2015. [2] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D shapenets: A deep representation for volumetric shapes, CVPR 2015. [3] H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for 3D shape recognition, ICCV 2015. [4] S. Bai, X. Bai, Z. Zhou, Z. Zhang, and L. Jan Latecki. Gift: A real-time and scalable 3D shape search engine, CVPR 2016. [5] J. Xie, M. Wang, Y. Fang. Learned Binary Spectral Shape Descriptor for 3D Shape Correspondence, CVPR 2016. [6] L. Wei, Q. Huang, D. Ceylan, E. Vouga and H. Li. Dense human body correspondences using convolutional networks, CVPR 2016. [7] Y. Li, H. Su, X. Guo, L. J. Guibas. SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation, CVPR 2017. [8] R. Qi, H. Su, K. Mo, L. J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, CVPR 2017. [9] G. Riegler, A. O. Ulusoy, A. Geiger. OctNet: Learning Deep 3D Representations at High Resolutions, CVPR 2017. [10] R. Klokov, V. S. Lempitsky. Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models, ICCV 2017. [11] D. Litany, T. Remez, E. Rodola, A.M. Bronstein, M.M. Bronstein. Deep Functional Maps: Structured Prediction for Dense Shape Correspondence, ICCV 2017. [12] R. Qi, Y. Li, H. Su, L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space, NIPS 2017.

8

Deep learned 3D shape feature for retrieval

• Heat diffusion based 3D shape feature
• J. Xie, Y. Fang, F. Zhu and E. K. Wong, Deepshape:deep learned shape descriptor for 3D shape

matching and retrieval, CVPR 2015.

9

Deep learned 3D shape feature for retrieval

• Heat diffusion based 3D shape feature(Global)
• Employ heat kernel signature (HKS) to form shape distribution
• Develop discriminative auto-encoder to learn global 3D shape feature

10

Deep learned 3D shape feature for retrieval

• Heat kernel signature
• Heat diffusion equation on a shape:

is the heat kernel, is the Laplace-Beltrami operator.

t

K t t

LK

 

 

t

K

L

i

v

j

v

ij



ij



, 1 , , 1 , ,

, cot cot , ~ , ~ 2 0, 0,

n i j i i j i j i j i j

w if i j if i j W w if i j w L A W

• therwise
• therwise

 

 

                 



11

Deep learned 3D shape feature for retrieval

• Heat kernel signature (HKS)
• Given an initial Dirac delta distribution, the solution of heat diffusion

equation:

• Based on the spectral decomposition theorem:
• Heat kernel signature: diagonal value of heat kernel

exp( )

t

K tL  

( , ) ( ) ( )

i

v t t j m i j i m i

k x x e x x  





2

( ) ( )

i

v t t j i j i

k x e x 





12

Deep learned 3D shape feature for retrieval

• Heat kernel signature
• Heat kernel describes the quantity of heat passing from one vertex to

another vertex after time interval t.

• J. Sun, M. Ovsjanikov, and L. Guibas. A concise and provably informative multi-scale signature based on

heat diffusion, Proceedings of the Symposium on Geometry Processing, 2009.

13

Deep learned 3D shape feature for retrieval

• Heat diffusion in SIFT (Scale invariant feature transform)
• D. Lowe, Distinctive features from scale-invariant keypoints, IJCV 2004.

14

Deep learned 3D shape feature for retrieval

• Multiscale shape distribution:
• Use the histogram to estimate the probabilistic distribution of HKSs of

vertices at each scale:

15

Deep learned 3D shape feature for retrieval

• Learn deep feature with discriminative auto-encoder

2 2 1

1 1 1 ( , ) ( ( )) ( ( ( )) ( ( ))) 2 2 2

C t t t t t t t i i w b F F i

J W b x G F x W tr S z tr S z  



    



16

Deep learned 3D shape feature for retrieval

• Learned shape descriptor:

17

• Comparison evaluations:

Shrec’14 Human dataset Shec’14 LSCRTB dataset

18

Learned binary spectral shape descriptor for correspondence

• Learn spectral shape descriptor (local):
• J. Xie, M. Wang, Y. Fang. Learned Binary Spectral Shape Descriptor for 3D Shape Correspondence,

CVPR 2016.

19

Learned binary spectral shape descriptor for correspondence

• Learn spectral shape descriptor :
• Construct spectral representation of 3D shapes:

is the cubic B-spine basis function.

𝑕(𝑦𝑘) = (𝑐(𝑤1), 𝑐(𝑤2),⋅⋅⋅, 𝑐(𝑤𝑡)))𝜚(𝑦𝑘), 𝜚(𝑦𝑘) = [𝜚1

2(𝑦𝑘), 𝜚2 2(𝑦𝑘),⋅⋅⋅, 𝜚𝑡 2(𝑦𝑘)

( )

s

b v

20

geometry vector

Learned binary spectral shape descriptor for correspondence

• Binary spectral shape descriptor with metric learning:

are the positive/negative point pairs on a pair of shapes.

2 2 2 2 , 2 2 2 1 1

1 1 1 1 ( , ) min 2 2 2

i i

N N K K K K K w b i j i j i i F i j x i j x

J W b h h h h b h W M M N    

 

   

       

 

sgn( )

K i i

b h 

,

i i

x x

 

21

Learned binary spectral shape descriptor for correspondence

• Evaluation:

Tosca dataset:

16 bit 32 bit 64 bit

22

Scape dataset:

23

Learn barycentric representation of 3D shapes

• Learn barycentric representation of 3D shapes for sketch-based 3D

shape retrieval:

• J. Xie, G. Dai, F. Zhu and Y. Fang, Learning barycentric representations of 3D shapes for sketch-based

3D shape retrieval, CVPR 2017.

24

Learn barycentric representation of 3D shapes

• Multi-view CNN based 3D shape representation
• H. Su, S. Maji, E. Kalogerakis, and E. G. Learned-Miller. Multi-view convolutional neural networks for

3D shape recognition, ICCV 2015.

25

Learn barycentric representation of 3D shapes

• Barycentric representation of 3D shapes:
• Max-view pooling does not exploit information from all views
• Wasserstein barycenters as a nonlinear pooling operation
• Optimal transportation:

The set of transportation plans between probability distributions p and q: The distance can be defined: Regularized optimal transportation:

M.Cuturi. Sinkhorn distances: lightspeed computation of optimal transport, NIPS 2013.

𝑆(𝑞, 𝑟) = {𝑈 ∈ ℝ+

𝑠×𝑡; 𝑈1 = 𝑞, 𝑈𝑈1 = 𝑟

( , ) D p q

( , )

( , ) min ,

T R p q

D p q M T



  

( , )

( , ) min , ,log

T R p q

D p q M T T T 



     

/

( ) ( ),

M

T diag u Kdiag v K e

  

 

26

Learn barycentric representation of 3D shapes

• Isotropic Wasserstein barycenters of 3D shapes:
• , is the Wasserstein distance.
• It can be solved with the Sinkhorn fixed-point algorithm.
• N. Bonneel, G. Peyre, and M. Cuturi, Wasserstein barycentric coordinates: histogram regression using
• ptimal transport, ACM Trans. Graphics, 2016.

1

argmin ( , )

b

n p i b i i

D p p 





( , )

b i

D p p

27

Learn barycentric representation of 3D shapes

• Cross-domain matching with learned Wasserstein barycenters:

2 2 1 2

2 2 2 1 2 1 , 1 1 2 2 2 2 1 ( ) 1 ( )

1 1 argmin max(0, )

n n j i j i j i c j j i c j j j

z z z z L L n m

 

  

   

     

     

28

Learn barycentric representation of 3D shapes

• Sketch-based 3D shape retrieval:

29

• Comparison evaluations:

Shrec’13 dataset Shrec’14 dataset

30

3D shape analysis

• 3D shape analysis:

[1] [2] [4] [1] Fan Zhu, Jin Xie and Yi Fang, Heat diffusion long-short term memory learning for 3D shape analysis, ECCV 2016. [2] Guoxian Dai, Jin Xie and Yi Fang, Metric-based generative adversarial network, ACM MM 2017. [3] Guoxian Dai, Jin Xie, Fan Zhu and Yi Fang, Deep correlation learning for sketch based 3D shape retrieval, AAAI 2017. [4] Jing Zhu, Jin Xie and Yi Fang, Learning adversarial 3D model generation with 2D image enhancer, AAAI 2018.

31

Thank you for your attention

32