SLIDE 1
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Problem Statement
2
Input 3D Point Cloud Output Semantic Segmentation
Ground Chair Table Wall
…
Ceiling
Exploring Spatial Context for 3D Semantic Segmentation of Point - - PowerPoint PPT Presentation
Exploring Spatial Context for 3D Semantic Segmentation of Point - - PowerPoint PPT Presentation
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Francis Engelmann* Theodora Kontogianni* Alexander Hermans Bastian Leibe Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Problem Statement Input
SLIDE 2
SLIDE 3
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds [Charles R. Qi et al. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, CVPR 2017]
Previous Work
3
- Most existing approaches: first convert into another representation
- Voxel-grid (3D CNN), Projection (2D CNN), …
- Pioneering work: PointNet operates directly on point clouds [CVPR’17]
SLIDE 4
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Previous Work: PointNet
4
N x D+D’ MLP M Global Feature S C MLP N x D N x D’ 1 x D’ N x M Point Features
Idea: Given a point cloud, learn feature descriptor using max-pooling.
(simplified PointNet model)
Local Context Global Context
SLIDE 5
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Our method
5
Multi-Scale Block Feature
MLP (64,128) MLP (64,128) MLP (64,128) M
: max pool
M M M
Output Score
S
: stack : concatenate
Multi-Scale Blocks same position different scales MLP (O) N x O M S 1 x O N x 2O S O=256 O=128 MLP (M) N x M 1 x 384 N x 128+384 N x D N x D N x D
Consolidation Unit (CU)
C
Input-Level Context
C C C
:block feature
Grid Blocks different positions same scale
Input-Level Context
N x D N x D N x D N x D MLP (128,64) M 1 x 64 S 1 x 64 GRU-RNN S C MLP (M)
Output Score
N x M MLP (128,64) shared MLP (128,64) MLP (128,64) M M M S 1 x 64 C MLP (M) shared S 1 x 64 C MLP (M) shared S 1 x 64 C MLP (M) shared
GRU RNN (unrolled)
1 x 64 1 x 64 1 x 64
Recurrent Consolidation Unit (RCU)
shared shared
… … … … … …
: max pool
M S
: stack : concatenate
C
:block feature
Two explorative models …
Details at the poster
SLIDE 6
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Consolidation Units:
Share and reinforce context between points within the same subset.
Our method: Consolidation Units
6
GRU RNN (unrolled) Recurrent Consolidation Unit (RCU)
MLP (O) N x O M S 1 x O N x 2O
Consolidation Unit (CU)
C
Recurrent Consolidation Units:
Share context between neighboring subsets of points.
SLIDE 7
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Qualitative Results [S3DIS dataset, Armeni et al. CVPR’16]
7
Input PointNet Ours Ground Truth Example 1 Example 2
SLIDE 8
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Qualitative Results [virtual KITTI dataset, Gaidon et al. CVPR16]
8
Our prediction Input XYZ-RGB Ground Truth
SLIDE 9
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Quantitative Results
9
Geometry Only Geometry & Appearance XYZ-RGB input features XYZ input features
SLIDE 10
- f 10
Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds
Conclusion
10
Project page: https://www.vision.rwth-aachen.de/page/3dsemseg
Input-Level Context Output-Level Context
See you at our poster!
We present novel mechanisms (Consolidation Units) to:
- share local context globally across the scene
- reinforce/consolidate local context