SymmetryNet: Learning to Predict Reflectional and Rotational - - PowerPoint PPT Presentation

▶
symmetrynet learning to predict reflectional and
SMART_READER_LITE
LIVE PREVIEW

SymmetryNet: Learning to Predict Reflectional and Rotational - - PowerPoint PPT Presentation

Oct 22, 2022 546 likes •920 views

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images Yifei Shi, Junwen Huang, Hongjia Zhang, Xin Xu, Szymon Rusinkiewicz and Kai Xu National University of Defense Technology

slide-1
SLIDE 1

SymmetryNet: Learning to Predict Reflectional and Rotational Symmetries of 3D Shapes from Single-View RGB-D Images

Yifei Shi, Junwen Huang, Hongjia Zhang, Xin Xu, Szymon Rusinkiewicz and Kai Xu Princeton University National University of Defense Technology

slide-2
SLIDE 2

Motiv ivatio ion

  • Symmetry is omnipresent in both nature and the synthetic world
slide-3
SLIDE 3

Motiv ivatio ion

  • Purely geometric symmetry detection approach

➢ Based on symmetry correspondences (counterparts) detection

[Mitra et al. 2006] [Ecins et al. 2018]

slide-4
SLIDE 4

Motiv ivatio ion

  • Detect object symmetries from a single-view RGB-D image

➢ Partial observation (self-occlusion) ➢ Mutual occlusion ➢ Noise

  • No sufficient symmetry correspondence
  • Learning-based approach?
slide-5
SLIDE 5

Motiv ivatio ion

  • Naïve learning-based approach

➢ Training: memorize the symmetry axes of the category ➢ Testing: perform object classification & pose estimation

Symmetry

slide-6
SLIDE 6

Motiv ivatio ion

  • Symmetry is supported by local geometric cues

➢ Find local correspondence ➢ Aggregate and output symmetry

  • The searching of local shape correspondence

benefits from the global feature

slide-7
SLIDE 7

Our approach

  • Multi-task learning: predict not only the symmetry axis (global), but

also the symmetry correspondences (local)

  • Could detect multiple symmetries
  • Could handle both reflectional symmetry and rotational symmetry
  • End-to-end trainable
slide-8
SLIDE 8

Rela lated work

  • 3D symmetry detection

Partial symmetry detection [Mitra et al. 2006] Intrinsic symmetry detection [Xu et al. 2009]

slide-9
SLIDE 9

Rela lated work

  • 3D symmetry detection

Geometric fitting [Ecins et al. 2018] Slippage analysis [Gelfand et al. 2004] View-based symmetry detection [Li et al. 2004]

slide-10
SLIDE 10

Problem settin ing

  • Input: an RGB-D image of an 3D object

➢ Object segmentation is known

  • Output: the extrinsic reflectional and rotational symmetries
slide-11
SLIDE 11

Method

…

weights

Poin int-wis ise feature ext xtraction

…

CNN PointNet global feature point-wise features

…

ground-truth Multi-task learning

… … … … …

MLP point-wise prediction

Poin int-wis ise symmetry ry prediction

  • Symmetry prediction network

point feature

slide-12
SLIDE 12

Method

  • Loss function

Point-wise loss:

  • ref. sym. or
  • rot. sym. or

no sym.

  • sym. parameters &

counterpart(s) location Dense point loss:

slide-13
SLIDE 13

Method

  • Loss function of reflectional symmetry

The probability of point j being the counterparts of point i

slide-14
SLIDE 14

Method

  • Loss function of rotational symmetry

The probability of point j being the counterparts of point i

slide-15
SLIDE 15

Method

Point-wise features

…

Binary classification Optimal assignment Multiple symmetry

  • utputs
  • Handle multiple symmetries

ground-truth

slide-16
SLIDE 16

Method

  • Inference

➢ Step 1: Prediction aggregation (Density-Based Spatial Clustering) ➢ Step 2: Visibility-based verification

… Prediction aggregation Verification

slide-17
SLIDE 17

Benchmark

  • We construct the 3D symmetry detection benchmark on …

➢ ShapeNet (synthetic) ➢ YCB (real) ➢ ScanNet (real)

slide-18
SLIDE 18

Experiments

  • Qualitative results

RGB-D image Predicted symmetry

slide-19
SLIDE 19

Experiments

  • Qualitative results

RGB-D image Predicted symmetry

slide-20
SLIDE 20

Experiments

  • Qualitative results

RGB-D image Predicted symmetry

slide-21
SLIDE 21

Experiments

  • Qualitative results

RGB-D image Predicted symmetry

slide-22
SLIDE 22

Experiments

  • Compare to baselines

➢ Geometric Fitting [Ecins et al. 2018] ➢ RGB-D Retrieval [Yang et al. 2018] ➢ Shape Completion [Liu et al. 2020]

  • Evaluation metric

➢ Precision-recall curve [Funk et al. 2017]

slide-23
SLIDE 23

Experiments

  • Compare to baselines on ShapeNet

ShapeNet holdout view ShapeNet holdout instance ShapeNet holdout category

slide-24
SLIDE 24

Experiments

  • Qualitative comparison

Ground-truth RGB-D image Geometric Fitting Shape Completion RGB-D Retrieval Ours

slide-25
SLIDE 25

Experiments

  • Qualitative comparison

Ground-truth RGB-D image Geometric Fitting Shape Completion RGB-D Retrieval Ours

slide-26
SLIDE 26

Experiments

  • Ablation study

ShapeNet holdout view ShapeNet holdout instance ShapeNet holdout category

slide-27
SLIDE 27

Experiments

  • Sensitively to occlusion
slide-28
SLIDE 28

Experiments

light (50-60%) occlusion medium (60-70%) occlusion heavy (70-80%) occlusion

  • Sensitively to occlusion
slide-29
SLIDE 29

Experiments

  • Visualization of the predicted counterparts

large error small error

slide-30
SLIDE 30

Experiments

  • Runtime analysis
slide-31
SLIDE 31

Failure cases

  • Spherical symmetry
  • Completely missing data
slide-32
SLIDE 32

Applications

  • 6D pose estimation

DenseFusion + Symmetry prediction DenseFusion [Wang et al. 2019]

slide-33
SLIDE 33

Applications

  • Symmetry-based segmentation
slide-34
SLIDE 34

Future work

  • Hierarchical symmetries
  • Self-supervised approach
  • Integrate symmetry prediction into X
slide-35
SLIDE 35

Thank you