Reconstruction and Repair of 3D Surfaces TalkID 23152 This session - PowerPoint PPT Presentation

government-funded by supervised by Deep 3D – Machine Learning for Reconstruction and Repair of 3D Surfaces TalkID 23152 This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep learning techniques and an insight into our approach for 3D surface repair . GTC 2017, Munich, 11.10.2017

government-funded by supervised by Pascal Laube • PhD Student at the Institute for Optical Systems at the HTWG Konstanz supervised by government-funded by • Main focus: Machine Learning for… • … Surface Reconstruction pascal.laube@gmail.com • … Defect Detection and Repair (Inpainting) • … Medical Imaging GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representation: The 2D case Output Neural Network Grid in euclidean space (in this case CNN) GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representation: In 3D? Point Cloud ? ? Mesh ? Neural Network Any manifold (NURBS, impl . surf., …) GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Voxels [Vishakh Hegde et al., NIPS (2016)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Voxels [Zhirong Wu et al., CVPR (2015)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Multi-View [Hang Su et al., ICCV (2015)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Multi-View [Liuhao Ge et al., CVPR (2016)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Graph Signal Processing • Graph Laplacian or Laplace Beltrami Operator as ∆𝑔 = −𝑒𝑗𝑤(𝛼𝑔) • Laplacian Eigenfunctions generalize to Fourier bases. Convolution in the spectral domain is defined… [M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)] …but filters are base dependent. GTC 2017, Munich, 11.10.2017

government-funded by supervised by Representations: Graph Signal Processing • Train filters in geodesic polar coordinates. • Pool rotation angles • Many other methods using [J. Masci et al., ICCV (2015)] different kernels (heat diffusion, gauss …) GTC 2017, Munich, 11.10.2017

government-funded by supervised by Data Sets • 127,915 CAD Models • 662 Object Categories • Different Subsets • 51,300 Models • 270 Object Categories in 12.000 Model Subsets Many smaller specialized Data Sets GTC 2017, Munich, 11.10.2017

government-funded by supervised by Problem: Defect on Surface with Detail- and Base-Geometry Fraunhofer IPT GTC 2017, Munich, 11.10.2017

government-funded by supervised by Problem: Defect on Surface with Detail- and Base-Geometry Werkzeugbau Siegfried Hofmann GmbH GTC 2017, Munich, 11.10.2017

government-funded by supervised by Problem: Defect on Surface with Detail- and Base-Geometry (3) • High resolution meshes with > 1m vertices • Base Geometry and Relief GTC 2017, Munich, 11.10.2017

government-funded by supervised by Our Approach B-Spline Approx. Base Geometry or Approx. by Geometric Primitive or Multiresolut. Surfaces Detail Geometry Heightmap Multiresolution Neural Nets Seperation Novelty Detection using Surface with Defect 1 3 2 for Inpainting Base Geo. – Detail Geo. Autoencoders GTC 2017, Munich, 11.10.2017

government-funded by supervised by Novelty Detection using Autoencoders 2 • Defect unknown • Healthy state unknown What do we know? • Textures have to be ergodic: Statistical properties are constant for single sample and whole collection GTC 2017, Munich, 11.10.2017

government-funded by supervised by Novelty Detection using Autoencoders 2 Autoencoder should be unable to sufficiently reconstruct Defects Train Autoencoder on Ergodic Set of Textures GTC 2017, Munich, 11.10.2017

government-funded by supervised by Novelty Detection using Autoencoders 2 Loss Parallelizable to multiple GPUs Samples GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Texture Synthesis 3 Activation Network Synth. Network [L. Gatys et al., NIPS (2015)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Style Transfer 3 [L. Gatys et al., arxiv.org (2015)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Example 3 Defect Closeup 2048x2048 GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Patches 3 • Inpainting a Region with arbitrary size? • Inpaint Patch by Patch Local Style Global Style 2048x2048 GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Results 3 … … 1. Start 3. Inpaint Patches: • Apply Detail • Large Child Weight 2. Inpaint Patches: • Small Parent Weight • Large Parent Weight GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Results 3 Result Result Closeup GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Results Heightmap 3 Parallelizable to multiple GPUs GTC 2017, Munich, 11.10.2017

government-funded by supervised by Multiresolution Neural Nets for Inpainting: Results Surface 3 GTC 2017, Munich, 11.10.2017

government-funded by supervised by Outlook • Neural Nets in high dimensional irregular domains • Michael M. Bronstein et al., “Geometric deep [M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)] learning: going beyond Euclidean data” (2017) • Michaël Defferrard , “Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering” (2016) [J. Masci et al., ICCV (2015)] GTC 2017, Munich, 11.10.2017

government-funded by supervised by Thank You GTC 2017, Munich, 11.10.2017