PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D - - PowerPoint PPT Presentation

▶
planematch patch coplanarity prediction
SMART_READER_LITE
LIVE PREVIEW

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D - - PowerPoint PPT Presentation

Feb 05, 2024 37 likes •418 views

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz, Thomas Funkhouser Princeton University National University of Defense Technology Technical University of Munich

slide-1
SLIDE 1

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction

Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz, Thomas Funkhouser

Princeton University National University of Defense Technology Technical University of Munich Google

slide-2
SLIDE 2

RGB-D Reconstruction

Microsoft Kinect Structure Sensor Xtion

slide-3
SLIDE 3

RGB-D Reconstruction

Bundle Fusion [Dai et al. 17]

slide-4
SLIDE 4

RGB-D Reconstruction

BundleFusion [Dai et al. 2017] ElasticFusion [Whelan et al. 2016] VoxelHashing [Niessner et al. 2013] KinectFusion [Newcombe/Izadi et al. 2011] Robust Recon. [Choi et al. 2015]

slide-5
SLIDE 5

VoxelHashing BundleFusion

Loop Closure

slide-6
SLIDE 6

Loop Closure -> Feature Descriptor

RGB Features:

  • SIFT, SURF, ORB, Freak, …
  • LIFT, MatchNet, …

Geometric Features:

  • SHOT, FPFH, SpinImages, …
  • 3DMatch, …

Are there additional primitives? Keypoint-based

slide-7
SLIDE 7

Our Id Idea: Planar Feature Descriptors

Coplanar Surface Patches

slide-8
SLIDE 8

Existing Planar Matching is Local

Online Structure Analysis [Zhang et al. 2015] Fine-to-Coarse Registration [Halber and Funkhouser 2017]

Point-to-Plane ICP [Chen & Medioni 91]

slide-9
SLIDE 9

Long-Range Constraints for SLAM

Coplanar Surface Patches

slide-10
SLIDE 10

Task: Co-planarity Matching?

… …

slide-11
SLIDE 11

PlaneMatch: Learning Co-planarity Features

➢Color ➢Depth ➢Normals ➢Plane Segmentation (Mask) ➢…

… Learn from 3D data!

slide-12
SLIDE 12

Siamese Network Architecture

256D 256D 256D

slide-13
SLIDE 13

Siamese Network Architecture

256D 256D 256D

slide-14
SLIDE 14

Siamese Network Architecture

256D 256D 256D

slide-15
SLIDE 15

Siamese Network Architecture

256D 256D 256D

slide-16
SLIDE 16

Step 1: : Ext xtract Planar Patches

RGB Depth Planar Patches

slide-17
SLIDE 17

Step 2: : Ext xtract Global Rep. / Patch

RGB Patch Mask Depth Normals

slide-18
SLIDE 18

Step 3: : Ext xtract Local Rep. / Patch

RGB Patch Mask Depth Normals

slide-19
SLIDE 19

Local / Global Representations

Local Representation Global Representation

slide-20
SLIDE 20

Siamese Network Architecture

256D 256D 256D

slide-21
SLIDE 21

Training: Self-Supervised Learning

ScanNet [Dai et al. 2017]

Anchor Positive Negative Anchor Positive Negative

…

10 million triplets

slide-22
SLIDE 22

Triplets for Training

Positive Negative Anchor

slide-23
SLIDE 23

Benchmark for Task of f Co-planarity Matching

Positive pair (6k) Negative pair (6k)

By patch size By pair distance

slide-24
SLIDE 24

PlaneMatch Evaluation

slide-25
SLIDE 25

PlaneMatch Ablation Study

slide-26
SLIDE 26

PlaneMatch Registration

: transformation matrix : indicator variables ( ∈ [0,1] )

slide-27
SLIDE 27

PlaneMatch Registration

: transformation matrix : indicator variables ( ∈ [0,1] ) : plane pair set : plane-to-plane distance : confidence weight Pairs predicted by coplanarity network

cop cop cop cop

π

slide-28
SLIDE 28

PlaneMatch Registration

: transformation matrix : indicator variables ( ∈ [0,1] ) : point pair set : point-to-point distance : confidence weight

kp kp kp kp

π Pairs predicted by SIFT keypoints

slide-29
SLIDE 29

PlaneMatch Registration

: transformation matrix : indicator variables ( ∈ [0,1] ) : threshold for error (0.01 m)

If > , = 0 If < , = 1

Robust optimization following [Choi et al. 15] / [Zollhoefer et al. 14] / [Zach et al. 14]

slide-30
SLIDE 30

PlaneMatch Registration Results

.

slide-31
SLIDE 31

PlaneMatch Registration Results

PlaneMatch (Ours) BundleFusion [Dai et al.17]

slide-32
SLIDE 32

PlaneMatch Registration Results

PlaneMatch (Ours) BundleFusion [Dai et al.17]

slide-33
SLIDE 33

Evaluation on TUM-RGBD

RMSE in cm (lower is better)

slide-34
SLIDE 34

Ablation on TUM-RGBD

RMSE in cm (lower is better)

slide-35
SLIDE 35

Effect of f Long-range Co-planar Pairs

50% deduction 100% deduction 0% deduction

1-5m 1-5m 1-5m 1-5m

slide-36
SLIDE 36

Effect of f Long-range Co-planar Pairs

50% deduction 100% deduction 0% deduction

1-5m 1-5m 1-5m 1-5m

slide-37
SLIDE 37

Effect of f Long-range Co-planar Pairs

50% deduction 100% deduction 0% deduction

1-5m 1-5m 1-5m 1-5m

slide-38
SLIDE 38

Conclusion

  • 1. New task: co-planarity matching
  • 2. Feature learning using self-supervision
  • 3. Integration with robust optimization into SLAM

Yifei Shi Kai Xu Tom Funkhouser Szymon Rusinkiewicz

Thank You!