Deep Learning for Geometry Processing 3D Representations - PowerPoint PPT Presentation

Deep Learning for Geometry Processing

3D Representations View-Based and Volumetric CNNs

3D Representations for Object Classification

Multi-View CNNs Su et al. 2015

Multi-View CNNs Su et al. 2015

Multi-View CNNs Su et al. 2015

Multi-View CNNs Su et al. 2015

Multi-View CNNs Su et al. 2015

Volumetric CNNs Wu et al. 2015

Volumetric CNNs Wu et al. 2015

Volumetric CNNs Wu et al. 2015

Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015

Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015

Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015

3DMatch Zeng et al. 2016

3DMatch Zeng et al. 2016

3DMatch Zeng et al. 2016

Training Data Zeng et al. 2016

3DMatch Embedding Zeng et al. 2016

3DMatch Results Zeng et al. 2016

Shape Classification Results Qi et al. 2016

Cause 1: Architecture and Engineering

Cause 2: Resolution Qi et al. 2016

Compatible Representation Qi et al. 2016

Investigating Architectures Qi et al. 2016

Different Architecture and Same Resolution Qi et al. 2016

3D CNN with Micro-Neural Network Qi et al. 2016

3D CNN with Micro-Neural Network Qi et al. 2016

Investigating Resolution Qi et al. 2016

Investigating Resolution Qi et al. 2016

Application Dense Correspondences of Clothed Humans

3D Human Capture Microsoft 2013

3D Human Capture Microsoft 2015

3D Human Capture [Dou et al. ’16]

Analysis & Reasoning

Correspondences on Clothed Human Bodies

Shape Analysis model human body body pose gender BMI … raw scan SCAPE model of Lee from Hirshberg et al. 2012

Motion Understanding t raw scans

Motion Understanding raw scans “grasping”

Correspondences?

Non-Rigid Registration [Li et al. 2008] correspondences target source overlap

Large Pose Changes source & target [Li et al. 09] [Huang et al. 08]

Descriptors designed descriptor learned descriptor [Hebert 99] [Taylor et al. 12] [Bronstein et al. 06] [Pons-Moll et al. 15] … … partial scans complete model [Litman & Bronstein 14] [Jain & Zhang 06] [Rodola et al. 14] [Bronstein et al. 10] (or small holes) [Windheuser et al. 14] [Kim et al. 11] [Macsi et al. 15] [Windheuser et al. 14] … [Chen & Koltun 15] …

Clothed and Partial Data immense space of variations

Classification Networks

Deep Convolutional Neural Network input feature “puppy” DNN image descriptor descriptor ≃ x 1 = f 1 ( x 0 ) x 2 = f 2 ( x 1 ) y = x k = f k ( x k − 1 ) x 0 classification network, e.g. AlexNet [Krizhevsky et al. 2012]

Deep Convolutional Neural Network depth feature DNN 3D model image descriptor

Deep Convolutional Neural Network depth feature “butt” DNN 3D model image descriptor descriptor ≃ “butt”

Loss Function Training Data DNN Loss Function Classification?

Classification Task descriptors are far apart

How to preserve distances?

? Deep Convolutional Neural Network Training Data DNN Loss Function

Loss Function Training Data Loss Function Triplet Loss (Anchor,Positive,Negative)

Multi-Segmentation A B A A B B C D C C D D A B A B + C D C D

Multiple Segmentation

Buffon-Laplace Needle Problem (18th Century) P ( x ) 1.0 0.8 0.6 0.4 0.2 x 0.2 0.4 0.6 0.8 1.0

Distance Preserving Learning 500 classes 100 random segmentations

Distance Preserving Learning 100 segmentations AlexNet 500 classes DNN Classification image descriptor 1x512x512 16x512x512 Classification Classification

Variation on Clothing DNN DNN Classification image descriptor Classification Classification 2100 meshes 33 landmarks Landmark DNN image descriptor Classification SCAPE MIT Yobi3D Yobi3D Yobi3D

Training Data Shape & Pose Clothing SCAPE MIT Yobi3D Yobi3D Yobi3D

Evaluation 100 100 100 98 98 98 CNN-S 96 96 96 CNN CNN % correspondences % correspondences % correspondences CO CO CO 94 94 94 BIM BIM BIM Möbius Möbius Möbius 92 92 92 RF RF RF 90 90 90 ENC ENC ENC C2FSym C2FSym C2FSym 88 88 88 EM EM EM C2F C2F C2F 86 86 86 GMDS GMDS GMDS SM SM SM 84 84 84 82 82 82 FAUST dataset 80 80 80 0 0 0 10 10 10 20 20 20 30 30 30 40 40 40 50 50 50 60 60 60 70 70 70 80 80 80 90 100 90 100 90 100 centimeters centimeters centimeters

Results

Results: Static Shapes Microsoft 2015

Results: Static Shapes Microsoft 2015

Results: Dynamic Shapes

Results: Dynamic Shape Reconstruction Microsoft 2015

4 Stationary Kinects Microsoft 2015

Dense Correspondences Microsoft 2015

Applications

Low Cost Capture & Moving Target ECCV 2016 Microsoft 2015

Registration and Reconstruction ECCV 2016 Microsoft 2015

Filtering and Texture Reconstruction ECCV 2016 Microsoft 2015

Application Photorealistic Texture Synthesis

Photo-Realistic Faces Using Deep Learning

Inspiration: Style Transfer(Gatys et al. 2016)

Deep CNN-based Synthesis Approach

Feature Correlations (Gatys et al. 2015) Feature correlation Feature response

Texture Analysis

Texture Synthesis (Gatys et al. 2015) loss function: total loss

Texture Synthesis (Saito et al. 2016)

Different Number of Mid-Layers

Detail Preservation via Convex Combination

Consistent Reconstruction from Different Views

Comparison

SIGGRAPH Asia 2016 CVPR 2016

Thanks!