3DMatch Learning Local Geometric Descriptors from RGB-D - - PowerPoint PPT Presentation

▶
3dmatch
SMART_READER_LITE
LIVE PREVIEW

3DMatch Learning Local Geometric Descriptors from RGB-D - - PowerPoint PPT Presentation

Sep 15, 2023 150 likes •354 views

3DMatch Learning Local Geometric Descriptors from RGB-D Reconstructions Andy Zeng, Shuran Song, Matthias Niener, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser Matching Features in 3D Data with Local 3D Descriptors match descriptor

slide-1
SLIDE 1

3DMatch

Learning Local Geometric Descriptors from RGB-D Reconstructions

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser

slide-2
SLIDE 2

Matching Features in 3D Data

with Local 3D Descriptors

descriptor

0.58 0.21 0.92 0.67 0.04 0.53

match descriptor

0.58 0.21 0.92 0.67 0.04 0.53

slide-3
SLIDE 3

commodity depth cameras

Matching Features in 3D Data

with Local 3D Descriptors Applications of Matching 3D Features:

  • Scan registration and loop closures for 3D reconstruction
  • Model registration for pose estimation
  • 3D mesh correspondence
slide-4
SLIDE 4

Matching Local Features in 3D Scans is Hard

Sensor Limitations Partial Surfaces and Occlusion Point Density Changes Other Anomalies

?

dense sparse noisy

slide-5
SLIDE 5

Matching Local Features in 3D Scans is Hard

Sensor Limitations Partial Surfaces and Occlusion Point Density Changes Other Anomalies

Previous local 3D descriptors only address part of the problem. FPFH [Rusu et al.] Spin-Images [Johnson et al.] SHOT [Salti et al.] Goal: train a data-driven local 3D descriptor that learns from example correspondences on real 3D scans.

slide-6
SLIDE 6

3DMatch

3D ConvNet

0.58 0.21 0.92 0.67 0.04 0.53 … 0.58 0.21 0.92 0.67 0.04 0.53 …

match!

TDF TDF

3D patch

3DMatch

3D ConvNet

3DMatch: Data-Driven Local 3D Descriptor

slide-7
SLIDE 7

matches non-matches

3DMatch: Data-Driven Local 3D Descriptor

slide-8
SLIDE 8

Training Data?

Problem: how do we get the training data?

Manually label correspondences?

  • time consuming
  • prone to errors

Sensor Limitations Partial Surfaces and Occlusion Point Density Changes Other Anomalies

Extremely Challenging

Is there a way to obtain training data automatically?

slide-9
SLIDE 9

Self-Supervised Learning from RGB-D Reconstructions

frame #7459 frame #4903

long-range correspondence

slide-10
SLIDE 10

Self-Supervised Learning from RGB-D Reconstructions

50+ RGB-D reconstructions > 8 million correspondences 3 applications

slide-11
SLIDE 11

Application #1: 3D Reconstruction

slide-12
SLIDE 12

Application #1: 3D Reconstruction

3DMatch + RANSAC geometric registration

slide-13
SLIDE 13

Application #1: 3D Reconstruction

Rusu et al. 3DMatch + RANSAC

slide-14
SLIDE 14

Application #1: 3D Reconstruction

generalization requirement: low

slide-15
SLIDE 15

Application #2: 6D Object Pose Estimation

full partial smaller scale

slide-16
SLIDE 16

Application #3: Mesh Correspondences

slide-17
SLIDE 17

Application #3: Mesh Correspondences

slide-18
SLIDE 18

Takeaways

3DMatch: Data-Driven Local 3D Descriptor RGB-D Reconstructions as Training Data Code & Benchmarks: http://3dmatch.cs.princeton.edu

slide-19
SLIDE 19

3DMatch

Learning Local Geometric Descriptors from RGB-D Reconstructions

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, Thomas Funkhouser

Code & Benchmarks: http://3dmatch.cs.princeton.edu