Exploring Spatial Context for 3D Semantic Segmentation of Point - PowerPoint PPT Presentation

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Francis Engelmann* Theodora Kontogianni* Alexander Hermans Bastian Leibe

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Problem Statement Input Output Chair Table Wall Ground … Ceiling 3D Point Cloud Semantic Segmentation 2 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Previous Work • Most existing approaches: first convert into another representation • Voxel-grid (3D CNN), Projection (2D CNN), … • Pioneering work: PointNet operates directly on point clouds [CVPR’17] [Charles R. Qi et al. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, CVPR 2017] 3 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Previous Work: PointNet Idea : Given a point cloud, learn feature descriptor using max-pooling. 1 x D’ N x D’ M N x D M S C MLP MLP x N Point Global N x D+D’ Features Feature (simplified PointNet model) Local Context Global Context 4 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Our method Two explorative models … Input-Level Context Output Score Multi-Scale N x D M MLP (64,128) Block Feature N x M N x 128+384 1 x 384 N x D MLP (64,128) M C S C O=256 O=128 MLP (M) Details at the poster N x D M MLP (64,128) :block feature Consolidation Unit (CU) : max pool N x O N x 2 � O M Output Score 1 x O Input-Level Context Multi-Scale Blocks : stack M S C N x M MLP (O) S S same position 1 x 64 1 x 64 different scales N x D : concatenate MLP (128,64) M S C C MLP (M) … … shared shared 1 x 64 1 x 64 N x D M S C MLP (128,64) MLP (M) … … shared shared GRU-RNN 1 x 64 1 x 64 N x D MLP (128,64) M S C MLP (M) … … shared shared 1 x 64 1 x 64 N x D M S C MLP (128,64) MLP (M) Grid Blocks Recurrent Consolidation Unit (RCU) :block feature different positions : max pool same scale M : stack S : concatenate GRU RNN (unrolled) C 5 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Our method: Consolidation Units Recurrent Consolidation Units: Share context between neighboring subsets of points. Recurrent Consolidation Unit (RCU) GRU RNN (unrolled) Consolidation Units: Share and reinforce context between points within the same subset. Consolidation Unit (CU) N x O N x 2 � O 1 x O C MLP (O) M S 6 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Qualitative Results [S3DIS dataset, Armeni et al. CVPR’16] Input PointNet Ours Ground Truth Example 1 Example 2 7 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Qualitative Results [virtual KITTI dataset, Gaidon et al. CVPR16] Input XYZ-RGB Our prediction Ground Truth 8 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Quantitative Results Geometry & Appearance Geometry Only XYZ-RGB input features XYZ input features 9 of 10

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds Conclusion We present novel mechanisms (Consolidation Units) to: - share local context globally across the scene - reinforce/consolidate local context Input-Level Context Output-Level Context See you at our poster! Project page: https://www.vision.rwth-aachen.de/page/3dsemseg 10 of 10