3D Deep Learning: An Overview based on My Work Hao Su Feb 23, 2018 - PowerPoint PPT Presentation

Progressive Voxel Refinement 61 1/30/2018 Hao Su

Fundamental Challenges of 3D Deep Learning 3D has many representations: Rasterized form multi-view RGB(D) images (regular grids) volumetric polygonal mesh Geometric form (irregular) point cloud Cannot directly apply CNN primitive-based models Hao Su 02/23/2018 62

Deep Learning on Polygonal Meshes

Mesh as 3D Input ▪ Deep Learning on Graphs Hao Su 64 02/23/2018

Geometry-aware Convolution can be Important image credit: D. Boscaini, et al. image credit: D. Boscaini, et al. convolutional convolutional considering along spatial underlying geometry coordinates Hao Su 65 02/23/2018

Meshes can be represented as graphs 3D shape graph social network molecules Hao Su 66 02/23/2018

How to define convolution kernel on graphs? • Desired properties: • locally supported (w.r.t graph metric) • allowing weight sharing across different coordinates from Shuman et al. 2013 Hao Su 67 02/23/2018

Issues of Geodesic CNN • The local charting method relies on a fast marching-like procedure requiring a triangular mesh. • The radius of the geodesic patches must be sufficiently small to acquire a topological disk. • No effective pooling, purely relying on convolutions to increase receptive field. Hao Su 68 02/23/2018

Spectral construction: Spectral CNN Fourier analysis Convert convolution to multiplication in spectral domain Hao Su 69 02/23/2018

Bases on meshes: eigenfunction of Laplacian- Bertrami operator Hao Su 70 02/23/2018

Synchronization of functional space across meshes Functional map Li Yi, Hao Su, Xingwen Guo, Leonidas Guibas “SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation” CVPR2017 (spotlight) Hao Su 71 02/23/2018

Mesh as 3D Output ▪ At the heart a surface parameterization problem Hao Su 72 02/23/2018

Deep learning on surface parameterization Use CNN to predict the parameterization, then convert to 3D mesh Step 1 Step 2 Ayan Sinha, Asim Unmesh, Qixing Huang, Karthik Ramani “SurfNet: Generating 3D shape surfaces using deep residual networks” CVPR2017 Hao Su 73 02/23/2018

Deep Learning on Point Cloud Representation

Point Cloud: the Most Common Sensor Output Figure from the recent VoxelNet paper from Apple. 75 1/30/2018 Hao Su

Point Cloud as 3D Input ▪ Deep Learning on Sets (orderless) Hao Su 76 02/23/2018

Properties of a desired neural network on point clouds D N 2D array representation Point cloud: N orderless points, each represented by a D dim coordinate Hao Su*, Charles Qi*, Kaichun Mo, Leonidas Guibas “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation” CVPR2017 (oral) 77 1/30/2018 Hao Su

Properties of a desired neural network on point clouds D N 2D array representation Point cloud: N orderless points, each represented by a D dim coordinate 78 1/30/2018 Hao Su

Properties of a desired neural network on point clouds D D represents the same set as N N 2D array representation Point cloud: N orderless points, each represented by a D dim coordinate 79 1/30/2018 Hao Su

Permutation invariance: Symmetric function f ( x 1 , x 2 , … , x n ) ≡ f ( x π 1 , x π 2 , … , x π n ) x i ∈ ! D , Examples: f ( x 1 , x 2 , … , x n ) = max{ x 1 , x 2 , … , x n } f ( x 1 , x 2 , … , x n ) = x 1 + x 2 +…+ x n … 80 1/30/2018 Hao Su

Construct symmetric function family Observe: f ( x 1 , x 2 , … , x n ) = γ ! g ( h ( x 1 ), … , h ( x n )) is symmetric if is symmetric g 81 1/30/2018 Hao Su

Construct symmetric function family Observe: f ( x 1 , x 2 , … , x n ) = γ ! g ( h ( x 1 ), … , h ( x n )) is symmetric if is symmetric g h (1,2,3) (1,1,1) (2,3,2) (2,3,4) 82 1/30/2018 Hao Su

Construct symmetric function family Observe: f ( x 1 , x 2 , … , x n ) = γ ! g ( h ( x 1 ), … , h ( x n )) is symmetric if is symmetric g h (1,2,3) simple symmetric function g (1,1,1) (2,3,2) (2,3,4) 83 1/30/2018 Hao Su

Construct symmetric function family Observe: f ( x 1 , x 2 , … , x n ) = γ ! g ( h ( x 1 ), … , h ( x n )) is symmetric if is symmetric g h (1,2,3) simple symmetric function γ g (1,1,1) (2,3,2) (2,3,4) PointNet (vanilla) 84 1/30/2018 Hao Su

Q: What symmetric functions can be constructed by PointNet? Symmetric functions PointNet (vanilla) 85 1/30/2018 Hao Su

A: Universal approximation to continuous symmetric functions Theorem: f :2 X → ! A Hausdorff continuous symmetric function can be arbitrarily approximated by PointNet. S ⊆ ! d , PointNet (vanilla) 86 1/30/2018 Hao Su

PointNet is Light-weight Space complexity (#params) 100M 100000K multi-view Saves 95% GPU memory volumetric ⎩ ⎨ ⎧ 10000K 10M point cloud 1000K 1M MVCNN Subvolume VRN PointNet [Su et al. 2015] [Su et al. 2016] [Su et al. 2016] [Su et al. 2017] 87 1/30/2018 Hao Su

Robustness to data corruption 88 1/30/2018 Hao Su

Robustness to data corruption Segmentation from partial scans 89 1/30/2018 Hao Su

Visualize what is learned by reconstruction Salient points are discovered! 90 1/30/2018 Hao Su

PointNet v2.0: Multi-Scale PointNet N points in N 1 points in N 2 points in (x,y) (x,y, f ) (x,y, f’ ) 1. Larger receptive field in higher layers 2. Less points in higher layers (more scalable) 3. Weight sharing 4. Translation invariance (local coordinates in local regions) Charles Qi, Hao Su, Li Yi, Leonidas Guibas “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space” NIPS 2017 1/30/2018 Hao Su 91

Fuse 2D and 3D: Frustum PointNets for 3D Object Detection + Leveraging mature 2D detectors for region proposal and 3D search space reduction + Solving 3D detection problem with 3D data and 3D deep learning architectures My latest paper accepted at CVPR 2018 92 1/30/2018 Hao Su

Our method ranks No. 1 on KITTI 3D Object Detection Benchmark We get 5% higher AP than Apple’s recent CVPR submission and more than 10% higher AP than previous SOTA in easy category ... 93 1/30/2018 Hao Su

Our method ranks No. 1 on KITTI 3D Object Detection Benchmark We are also 1 st place for smaller objects (ped. and cyclist) winning with even bigger margins. ... ... 94 1/30/2018 Hao Su

Remarkable box estimation accuracy even with a dozen of points or with very partial point cloud 95 1/30/2018 Hao Su

96 1/30/2018 Hao Su

Point Cloud as 3D Output ▪ Deep Learning to Generate Combinatorial Objects Hao Su 97 02/23/2018

Supervision from “Synthesize for Learning” ShapeNet Renderer 98 02/23/2018

3D Representation: Point Cloud Describe shape for the whole object Usable as network output? No prior works in the deep learning community! Hao Su 99 02/23/2018

3D Prediction by Point Clouds Input Reconstructed 3D point cloud Hao Su, Haoqiang Fan, Leonidas Guibas “A Point Set Generation Network for 3D Object Reconstruction from a Single Image” CVPR2017 (oral) 100 02/23/2018