SLIDE 1
Deep Learning for Geometry Processing
3D Representations
View-Based and Volumetric CNNs
Deep Learning for Geometry Processing 3D Representations - - PowerPoint PPT Presentation
Deep Learning for Geometry Processing 3D Representations - - PowerPoint PPT Presentation
Deep Learning for Geometry Processing 3D Representations View-Based and Volumetric CNNs 3D Representations for Object Classification Multi-View CNNs Su et al. 2015 Multi-View CNNs Su et al. 2015 Multi-View CNNs Su et al. 2015 Multi-View
SLIDE 2
SLIDE 3
3D Representations for Object Classification
SLIDE 4
Multi-View CNNs
Su et al. 2015
SLIDE 5
Multi-View CNNs
Su et al. 2015
SLIDE 6
Multi-View CNNs
Su et al. 2015
SLIDE 7
Multi-View CNNs
Su et al. 2015
SLIDE 8
Multi-View CNNs
Su et al. 2015
SLIDE 9
Volumetric CNNs
Wu et al. 2015
SLIDE 10
Volumetric CNNs
Wu et al. 2015
SLIDE 11
Volumetric CNNs
Wu et al. 2015
SLIDE 12
Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015
SLIDE 13
Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015
SLIDE 14
Learned Features: 3D Primitives / Filter Visualization Wu et al. 2015
SLIDE 15
3DMatch
Zeng et al. 2016
SLIDE 16
3DMatch
Zeng et al. 2016
SLIDE 17
Zeng et al. 2016
3DMatch
SLIDE 18
Training Data
Zeng et al. 2016
SLIDE 19
3DMatch Embedding
Zeng et al. 2016
SLIDE 20
3DMatch Results
Zeng et al. 2016
SLIDE 21
Shape Classification Results
Qi et al. 2016
SLIDE 22
Cause 1: Architecture and Engineering
SLIDE 23
Cause 2: Resolution
Qi et al. 2016
SLIDE 24
Compatible Representation
Qi et al. 2016
SLIDE 25
Investigating Architectures
Qi et al. 2016
SLIDE 26
Different Architecture and Same Resolution
Qi et al. 2016
SLIDE 27
3D CNN with Micro-Neural Network
Qi et al. 2016
SLIDE 28
3D CNN with Micro-Neural Network
Qi et al. 2016
SLIDE 29
Investigating Resolution
Qi et al. 2016
SLIDE 30
Investigating Resolution
Qi et al. 2016
SLIDE 31
Application
Dense Correspondences of Clothed Humans
SLIDE 32
Microsoft 2013
3D Human Capture
SLIDE 33
3D Human Capture
Microsoft 2015
SLIDE 34
[Dou et al. ’16]
3D Human Capture
SLIDE 35
Analysis & Reasoning
SLIDE 36
Correspondences on Clothed Human Bodies
SLIDE 37
Shape Analysis
model raw scan human body body pose gender BMI …
SCAPE model of Lee from Hirshberg et al. 2012
SLIDE 38
Motion Understanding
raw scans t
SLIDE 39
Motion Understanding
raw scans “grasping”
SLIDE 40
Correspondences?
SLIDE 41
Non-Rigid Registration [Li et al. 2008]
target source
- verlap
correspondences
SLIDE 42
Large Pose Changes
source & target [Li et al. 09] [Huang et al. 08]
SLIDE 43
Descriptors
partial scans complete model (or small holes) designed descriptor learned descriptor
[Hebert 99] [Bronstein et al. 06] … [Jain & Zhang 06] [Bronstein et al. 10] [Kim et al. 11] [Windheuser et al. 14] [Chen & Koltun 15] … [Taylor et al. 12] [Pons-Moll et al. 15] … [Litman & Bronstein 14] [Rodola et al. 14] [Windheuser et al. 14] [Macsi et al. 15] …
SLIDE 44
Clothed and Partial Data
immense space of variations
SLIDE 45
Classification Networks
SLIDE 46
Deep Convolutional Neural Network
classification network, e.g. AlexNet [Krizhevsky et al. 2012]
feature descriptor input image
DNN
x1 = f1(x0) x2 = f2(x1)
x0
y = xk = fk(xk−1)
≃
“puppy” descriptor
SLIDE 47
Deep Convolutional Neural Network
feature descriptor
DNN
3D model depth image
SLIDE 48
Deep Convolutional Neural Network
feature descriptor
DNN
3D model depth image
“butt”
≃
“butt” descriptor
SLIDE 49
Loss Function
DNN Training Data Loss Function
Classification?
SLIDE 50
Classification Task
descriptors are far apart
SLIDE 51
How to preserve distances?
SLIDE 52
Deep Convolutional Neural Network
DNN Training Data Loss Function
?
SLIDE 53
Loss Function
Training Data Triplet Loss
(Anchor,Positive,Negative)
Loss Function
SLIDE 54
Multi-Segmentation
A B C D A B C D A B C D A B C D A B C D +
SLIDE 55
Multiple Segmentation
SLIDE 56
Buffon-Laplace Needle Problem (18th Century)
0.2 0.4 0.6 0.8 1.0 x 0.2 0.4 0.6 0.8 1.0 P(x)
SLIDE 57
Distance Preserving Learning
500 classes 100 random segmentations
SLIDE 58
Distance Preserving Learning
AlexNet 100 segmentations 500 classes
Classification Classification DNN Classification
descriptor image
16x512x512 1x512x512
SLIDE 59
Variation on Clothing
SCAPE MIT Yobi3D Yobi3D Yobi3D
2100 meshes 33 landmarks
Classification Classification DNN Classification
descriptor image
DNN Landmark Classification
descriptor image
DNN
SLIDE 60
Training Data
SCAPE MIT Yobi3D Yobi3D Yobi3D
Shape & Pose Clothing
SLIDE 61
FAUST dataset
Evaluation
centimeters 10 20 30 40 50 60 70 80 90 100 % correspondences 80 82 84 86 88 90 92 94 96 98 100
CO BIM Möbius RF ENC C2FSym EM C2F GMDS SM
centimeters 10 20 30 40 50 60 70 80 90 100 % correspondences 80 82 84 86 88 90 92 94 96 98 100
CNN CO BIM Möbius RF ENC C2FSym EM C2F GMDS SM
centimeters 10 20 30 40 50 60 70 80 90 100 % correspondences 80 82 84 86 88 90 92 94 96 98 100
CNN-S CNN CO BIM Möbius RF ENC C2FSym EM C2F GMDS SM
SLIDE 62
Results
SLIDE 63
Microsoft 2015
Results: Static Shapes
SLIDE 64
Microsoft 2015
Results: Static Shapes
SLIDE 65
Results: Dynamic Shapes
SLIDE 66
Microsoft 2015
Results: Dynamic Shape Reconstruction
SLIDE 67
Microsoft 2015
4 Stationary Kinects
SLIDE 68
Microsoft 2015
Dense Correspondences
SLIDE 69
Applications
SLIDE 70
Microsoft 2015
Low Cost Capture & Moving Target
ECCV 2016
SLIDE 71
Microsoft 2015
Registration and Reconstruction
ECCV 2016
SLIDE 72
Microsoft 2015
Filtering and Texture Reconstruction
ECCV 2016
SLIDE 73
Application
Photorealistic Texture Synthesis
SLIDE 74
Photo-Realistic Faces Using Deep Learning
SLIDE 75
Inspiration: Style Transfer(Gatys et al. 2016)
SLIDE 76
Deep CNN-based Synthesis Approach
SLIDE 77
Feature Correlations (Gatys et al. 2015)
Feature correlation Feature response
SLIDE 78
Texture Analysis
SLIDE 79
Texture Synthesis (Gatys et al. 2015)
loss function: total loss
SLIDE 80
Texture Synthesis (Saito et al. 2016)
SLIDE 81
Different Number of Mid-Layers
SLIDE 82
Detail Preservation via Convex Combination
SLIDE 83
Consistent Reconstruction from Different Views
SLIDE 84
Comparison
SLIDE 85
SIGGRAPH Asia 2016 CVPR 2016
SLIDE 86
Thanks!