CreativeAI 3D (Geometric) Domain Niloy Mitra Iasonas Kokkinos - PowerPoint PPT Presentation

<latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="63ak468AXCor9nBfBLOpfhlLeqM=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZKQVdFt24s4K9QDuWTJpYzPJkGSEMvQd3LhQxK3v4863MW1noa0/BD7+cw45w9izrRx3W8nt7a+sbmV3y7s7O7tHxQPj1paJorQJpFcqk6ANeVM0KZhtNOrCiOAk7bwfh6Vm8/UaWZFPdmElM/wkPBQkawsVbrtiwequf9YsmtuHOhVfAyKEGmRr/41RtIkRUGMKx1l3PjY2fYmUY4XRa6CWaxpiM8ZB2LQocUe2n82n6Mw6AxRKZ8waO7+nkhxpPUkCmxnhM1IL9dm5n+1bmLCSz9lIk4MFWTxUZhwZCSanY4GTFi+MQCJorZXREZYWJsQEVbAje8smr0KpWPMt3tVL9KosjDydwCmXw4ALqcAMNaAKBR3iGV3hzpPivDsfi9ack80cwx85nz9g645U</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> <latexit sha1_base64="pPG/+UG2ht75Yz5Ajs7KOd+Azk=">AB7XicbZDLSgMxFIbP1Fut6pLN8Ei1E2ZsYIui27cWcFeoB1LJs20sZlkSDJCGfoOblwo4tb3cefbmLaz0NYfAh/OYec8wcxZ9q47reTW1ldW9/Ibxa2tnd294r7B0tE0Vog0guVTvAmnImaMw2k7VhRHAaetYHQ9rbeqNJMinszjqkf4YFgISPYWKt5WxYP1dNeseRW3JnQMngZlCBTvVf86vYlSIqDOFY647nxsZPsTKMcDopdBNY0xGeEA7FgWOqPbT2bYTdGKdPgqlsk8YNHN/T6Q40nocBbYzwmaoF2tT879aJzHhpZ8yESeGCjL/KEw4MhJNT0d9pigxfGwBE8XsrogMscLE2IAKNgRv8eRlaJ5VPMt356XaVRZHo7gGMrgwQXU4Abq0ACj/AMr/DmSOfFeXc+5q05J5s5hD9yPn8AYnCOVQ=</latexit> What’s Different in 3D? • Number of Voxels grows as versus occupied surface O ( n 3 ) O ( n 2 ) �7

Data Representation .. Many Possibilities! points �8

Data Representation .. Many Possibilities! points voxels cells patches �8

Challenges �9

Challenges 1. Representation   �9

Challenges 1. Representation   2. Neighborhood information • who are the neighbouring elements • how are the elements ordered   �9

Challenges 1. Representation   2. Neighborhood information • who are the neighbouring elements • how are the elements ordered   3. Extrinsic versus intrinsic representation   �9

Challenges 1. Representation   2. Neighborhood information • who are the neighbouring elements • how are the elements ordered   3. Extrinsic versus intrinsic representation   4. Simplicity versus memory/runtime tradeoff �9

Representation for 3D • Image-based   • Volumetric   • Surface-based   • Point-based �10

Representation for 3D: Multi-view CNN [Kalogerakis et al. 2015] �12

Representation for 3D: Multi-view CNN regular image analysis networks [Kalogerakis et al. 2015] �12

3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation �13

Integrating View Information �14

Representation for 3D: Local Multi-view CNN [Huang et al. 2018] �15

Representation for 3D: Local Multi-view CNN Segmentation Correspondence Feature matching Predicting semantic functions [Huang et al. 2018] �15

Representation for 3D: Local Multi-view CNN Segmentation Correspondence Feature matching Predicting semantic functions [Huang et al. 2018] localized renderings for point-wise features �15

Tangent Convolutions loses information due to occlusion [Tatarchenko et al. 2018] �16

Tangent Convolutions loses information due to occlusion project to local patches   (contrast with PCPNet construction) [Tatarchenko et al. 2018] �16

Dealing with Sparse Points �17

Dealing with Sparse Points �18

Improved Performance �19

Representation for 3D • Image-based • PROS: directly use image networks, good performance • CONS: rendering is slow and memory-heavy, not very geometric • Volumetric   • Point-based   • Surface-based �20

3D CNNs : Direct Approach [Xiao et al. 2014] �22

VoxNet [Maturana et al. 15] ▸ Binary occupancy, density grid, etc. rotational invariance �23

Visualization of First Level Filters �24

Representation for 3D: Volumetric Deformation [Yumer and Mitra 2016] �25

Efficient Volumetric Datastructures [Wang et al. 2017] �26

Data Structure and CNN Operations shuffled keys labels downsampling example (encode position in space) (parent label → child indices) ( “where there is an octant, there is CNN computation” ) faster neighbor access �27

Efficient Volumetric Datastructures Encoder Decoder/generator Wang et al. 2017 only generate non-empty voxels [Hane et al. 2018] �28

Efficient Volumetric Datastructures [Hane et al. 2018] �29

Lower Memory Footprint �30

Adaptive O-CNN [Wang et al. 2018] image to planar patch-based shapes

First-order Patches OCNN Adaptive OCNN �32

Field Probing Neural Networks for 3D Data [Li et al. 2016] �33

Spatial Probes �34

Method Details �35

Representation for 3D • Image-based   • Volumetric • PROS: adaptations of image networks • CONS: special layers for hierarchical datastructures, still too coarse   • Surface-based   • Point-based �36

Local/Global Parameterizations �38

Local/Global Parameterizations Geometry Image [Sinha et al. 2016] �38

Local/Global Parameterizations Geometry Image Metric Alignment (GWCNN) [Sinha et al. 2016] [Ezuz et al. 2017] �38

Shape Surfaces using Geometry Images �39

Using Geodesic Patches: GCNN [Masci et al. 2015] �40

<latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> <latexit sha1_base64="TquvtCTwYBxPgA56TL/7lpYdxY0=">ACO3icbZDLSxBEMZ7NA8z5rHRYy6FizBDRGYkEAkIknjwaIKrws6y1PTWuI09D7prxGXY/yuX/BO5eIlh0jINf07s4hPgoafnxfFdX1pZVWlqPoh7e0/Ojxk6crz/zV5y9evuq8XjuxZW0k9WSpS3OWoiWtCuqxYk1nlSHMU02n6cWnmX96ScaqsjmSUWDHM8LlSmJ7KRh50uQWIZDWAIwVUIH/bAT2ydD5uEx8S4ZaAwYLhLSQHpB20HpjQDw7cWBYGZmshsNON9qO5gX3IW6hK9o6Gna+J6NS1jkVLDVa24+jigcNGlZS09RPaksVygs8p7DAnOyg2Z+xQ2nTKCrDTuFQxz9f+JBnNrJ3nqOnPksb3rzcSHvH7N2e6gUVMxVysSirNXAJsyBhpAxJ1hMHKI1yfwU5RoOSXdy+CyG+e/J9ONnZjh1/ftfd/9jGsSLeiA0RiFi8F/viUByJnpDiq7gWv8SN98376f32/ixal7x2Zl3cKu/vP2AxqYM=</latexit> Using Geodesic Patches: GCNN X ( f ? a )( x ) := a ( ✓ + ∆ ✓ , r )( D ( x ) f )( r, ✓ ) [Masci et al. 2015] θ ,r �40

GCNN Architecture �41

Handling Rotational Ambiguity �42

Parameterization for Surface Analysis map 3D surface to 2D domain [Maron et al. 2017] �43

Parameterization for Surface Analysis [Maron et al. 2017] �44

Parameterization for Surface Analysis • Map 3D surface to 2D domain   • One such mapping: flat torus (seamless => translation-invariant)   • Many mappings exists: sample a few and average result   • Which functions to map?   XYZ, normals, curvature, … [Maron et al. 2017] �45

Parameterization for Surface Analysis [Maron et al. 2017] �46

Texture Transfer (Parameterization + Alignment) [Wang et al. 2016] �47

AtlasNet for Surface Generation condition decoded points on 2D patches [Groueix et al. 2018] �48

AtlasNet for Surface Generation Latent representation can be inferred from images or point clouds [Groueix et al. 2018] �49

AtlasNet for Surface Generation Quad Mesh is generated by mapping a regular grid in 2D domain to 3D points Latent representation can be inferred from images or point clouds [Groueix et al. 2018] �50

AtlasNet for Surface Generation texture coordinates come for free!! Latent representation can be inferred from images or point clouds �51

Representation for 3D • Image-based   • Volumetric   • Surface-based • PROS: parameterize + image networks (instrinsic representation) • CONS: suffers from parameterisation artefacts (local versus global distortion),   requires good quality mesh   • Point-based �52

Representation for 3D: Point-based • Common representation: native representation   • Easy to obtain from meshes, depth scans, laser scans �54

In Original Representation • Common representation   • Easy to obtain from meshes, depth scans, laser scans   • Unstructured (e.g., any permutation of points gives same shape!) [Qi et al. 2017] �55

PointNet for Point Cloud Analysis permutation-invariant functions [Qi et al. 2017] �56

PointNet for Point Cloud Analysis Use MLPs (h) and max-pooling (g) as simple symmetric functions [Qi et al. 2017] �57

PointNet Architecture [Qi et al. 2017] �58

PointNet for Point Cloud Analysis �59

PointNet++ [Qi et al. 2018] �60

PCPNet for Local Point Cloud Analysis [Guerrero et al. 2018] �61

CreativeAI 3D (Geometric) Domain Niloy Mitra Iasonas Kokkinos - PowerPoint PPT Presentation

CreativeAI 3D (Geometric) Domain Niloy Mitra Iasonas Kokkinos Paul Guerrero Nils Thuerey Tobias Ritschel UCL UCL UCL TUM UCL Timetable Niloy Paul Nils Introduction 2:15 pm X X X 2:25 pm X Machine Learning Basics Theory

Chapter 8 Binomial and Geometric Distribu7ons 8.2 Geometric

Common Architecture Elements SIGGRAPH Asia Course CreativeAI: Deep Learning for Graphics 1

Geometric Optimization Piotr Indyk April 26, 2005 Lecture 19: Geometric Optimization Geometric

Geometric Representations 3D Graphics Motivation Geometric representation What do we want

2D Geometric Transformations Question : How do we represent a geometric object in the plane?

Geometric Modeling An Example 2 Geomertic Modeling I - Center for Graphics and Geometric

PDE-based Geometric Modeling and Interactive Sculpting for Graphics Hong Qin Center for Visual

y TtEnheints orientation note from Htt on 2T outward normals Def a geometric ideal

Geometric Algebra A powerful tool for solving geometric problems in visual computing Leandro A.

EXAMPLES OF FOUR-DIMENSIONAL GEOMETRIC TRANSITION Joint with S. Riolo Fribourg, 8th May 2019 W

Yowzer! Theyre electrifying! I hate my job Geometric Algorithms 1 Basic Geometric Objects

Geometric Graphs Sathish Govindarajan Indian Institute of Science, Bangalore Workshop on

Lecture 8 : The Geometric Distribution 0/ 24 The geometric distribution is a special case of

Subdivision Surfaces 1 Geometric Modeling Geometric Modeling Sometimes need more than

Subdivision Surfaces 1 Geometric Modeling Geometric Modeling Sometimes need more than

The Geometric Burnsides Problem Brandon Seward University of Michigan Geometric and

Intro to Image Understanding (CSC420) Projects Proposal Deadline : Nov 2 (Sunday), 11.59pm, 2014

CSC420: Intro to Image Understanding Introduction Sanja Fidler September 11, 2014 Sanja Fidler

CSC420: Intro to Image Understanding Introduction Sanja Fidler January 8, 2018 Sanja Fidler

Present bufer fips in Xwayland GSOC 2017, landed in Xserver 1.20 Roman Gilg 1 / 9 Problem

3D Pattern Recognition Using Deep Neural Networks for Liquid Argon Time Projection Chambers

WITH DEEP NEURAL NETWORKS INTELLIGENT ROBOTICS SEMINAR PIA UK 25.11.2019 OUTLINE 1.

Structure of Vision Problems Alan Yuille (UCLA). Machine Learning Theory of Machine Learning

Lecture: Motion Juan Carlos Niebles and Ranjay Krishna Stanford Vision and Learning Lab