Reconstruction and Repair of 3D Surfaces TalkID 23152 This session - - PowerPoint PPT Presentation

▶
reconstruction and repair
SMART_READER_LITE
LIVE PREVIEW

Reconstruction and Repair of 3D Surfaces TalkID 23152 This session - - PowerPoint PPT Presentation

Aug 05, 2023 227 likes •524 views

government-funded by supervised by Deep 3D Machine Learning for Reconstruction and Repair of 3D Surfaces TalkID 23152 This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep

slide-1
SLIDE 1 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Deep 3D – Machine Learning for Reconstruction and Repair

  • f 3D Surfaces

TalkID 23152

This session will give the audience a quick overview of recent developments in the field of 3D surface analysis with deep learning techniques and an insight into our approach for 3D surface repair.

slide-2
SLIDE 2 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

  • PhD Student at the Institute for Optical Systems

at the HTWG Konstanz

  • Main focus: Machine Learning for…
  • … Surface Reconstruction
  • … Defect Detection and Repair (Inpainting)
  • … Medical Imaging

pascal.laube@gmail.com

Pascal Laube

government-funded by supervised by

slide-3
SLIDE 3 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representation: The 2D case

Output Grid in euclidean space Neural Network

(in this case CNN)

slide-4
SLIDE 4 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representation: In 3D?

Neural Network Point Cloud Mesh Any manifold (NURBS,

  • impl. surf., …)

? ? ?

slide-5
SLIDE 5 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Voxels

[Vishakh Hegde et al., NIPS (2016)]

slide-6
SLIDE 6 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Voxels

[Zhirong Wu et al., CVPR (2015)]

slide-7
SLIDE 7 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Multi-View

[Hang Su et al., ICCV (2015)]

slide-8
SLIDE 8 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Multi-View

[Liuhao Ge et al., CVPR (2016)]

slide-9
SLIDE 9 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Graph Signal Processing

[M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]

  • Graph Laplacian or Laplace

Beltrami Operator as ∆𝑔 = −𝑒𝑗𝑤(𝛼𝑔)

  • Laplacian Eigenfunctions

generalize to Fourier bases. Convolution in the spectral domain is defined… …but filters are base dependent.

slide-10
SLIDE 10 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Representations: Graph Signal Processing

  • Train filters in geodesic polar

coordinates.

  • Pool rotation angles

[J. Masci et al., ICCV (2015)]

  • Many other methods using

different kernels (heat diffusion, gauss…)

slide-11
SLIDE 11 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Data Sets

  • 127,915 CAD Models
  • 662 Object Categories
  • Different Subsets
  • 51,300 Models
  • 270 Object Categories in 12.000 Model Subsets

Many smaller specialized Data Sets

slide-12
SLIDE 12 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry

Fraunhofer IPT

slide-13
SLIDE 13 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry

Werkzeugbau Siegfried Hofmann GmbH

slide-14
SLIDE 14 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Problem: Defect on Surface with Detail- and Base-Geometry (3)

  • High resolution meshes with

> 1m vertices

  • Base Geometry and Relief
slide-15
SLIDE 15 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Our Approach

B-Spline Approx.

  • Approx. by

Geometric Primitive Multiresolut. Surfaces

Seperation Base Geo. – Detail Geo. Surface with Defect Novelty Detection using Autoencoders Multiresolution Neural Nets for Inpainting

Detail Geometry Heightmap Base Geometry

1 2 3

  • r
  • r
slide-16
SLIDE 16 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

  • Defect unknown
  • Healthy state unknown

What do we know?

  • Textures have to be ergodic:

Statistical properties are constant for single sample and whole collection

2

slide-17
SLIDE 17 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

Train Autoencoder on Ergodic Set of Textures Autoencoder should be unable to sufficiently reconstruct Defects

2

slide-18
SLIDE 18 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Novelty Detection using Autoencoders

Loss Samples

2

Parallelizable to multiple GPUs

slide-19
SLIDE 19 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Texture Synthesis 3

[L. Gatys et al., NIPS (2015)]

Activation Network

  • Synth. Network
slide-20
SLIDE 20 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3

[L. Gatys et al., arxiv.org (2015)]

Multiresolution Neural Nets for Inpainting: Style Transfer

slide-21
SLIDE 21 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Example 3

2048x2048

Defect Closeup

slide-22
SLIDE 22 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Patches 3

  • Inpainting a Region with arbitrary size?
  • Inpaint Patch by Patch

Local Style Global Style

2048x2048

slide-23
SLIDE 23 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Results 3

  • 1. Start
  • 2. Inpaint Patches:
  • Large Parent Weight

…

  • 3. Inpaint Patches:
  • Apply Detail
  • Large Child Weight
  • Small Parent Weight

…

slide-24
SLIDE 24 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Multiresolution Neural Nets for Inpainting: Results 3

Result Result Closeup

slide-25
SLIDE 25 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3 Multiresolution Neural Nets for Inpainting: Results Heightmap

Parallelizable to multiple GPUs

slide-26
SLIDE 26 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

3 Multiresolution Neural Nets for Inpainting: Results Surface

slide-27
SLIDE 27 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Outlook

[J. Masci et al., ICCV (2015)] [M. Bronstein et al., Sig. Proc. Mag. 34.4 (2017)]

  • Neural Nets in high dimensional irregular domains
  • Michael M. Bronstein et al., “Geometric deep

learning: going beyond Euclidean data” (2017)

  • Michaël Defferrard, “Convolutional Neural

Networks on Graphs with Fast Localized Spectral Filtering” (2016)

slide-28
SLIDE 28 government-funded by supervised by

GTC 2017, Munich, 11.10.2017

Thank You