Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild Shangzhe Wu Christian Rupprecht Andrea Vedaldi V ISUAL G EOMETRY G ROUP , U NIVERSITY OF O XFORD
Agenda v Problem Introduction v Method Overview v Results v Discussions v Conclusions 29
What is 3D Reconstruction? Vision Reconstruction Rendering Graphics 2D Observations 3D Representation 30
Multi-view 3D Reconstruction Building Rome in a Day. Agarwal et al. ICCV’09 static scene but.. the world is dynamic 33
Multi-view 3D Reconstruction Building Rome in a Day. The Relightables: Volumetric Performance Capture of Humans Agarwal et al. ICCV’09 with Realistic Relighting. Guo et al. SIGGRAPH Asia’19 static scene 100 cameras too expensive for me :( 34
Learning-based Single-view 3D Reconstruction Neural Network Need supervision ! 3D prior learned during training 35
Supervision during Training 3D ground truth or multi-views depth maps shape models silhouettes keypoints camera viewpoint 36
Unsupervised Learning of 3D Objects 3D ground truth or multi-views depth maps shape models silhouettes keypoints camera viewpoint 37
Unsupervised Learning of 3D Objects Training Data Output Unsup3D single-view images of a category instance-specific 3D shapes NO other supervision! 38
input reconstruction input reconstruction 39
Unsupervised Learning of 3D Objects Training Data Output Unsup3D single-view images of a category instance-specific 3D shapes NO other supervision! 40
Symmetries in the World 41
Training Pipeline: Photo-Geometric Autoencoding 42
Photo-Geometric Autoencoding input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 43
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 44
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry input # encoder encoder encoder decoder decoder view ! depth " texture Reconstruction Loss Renderer reconstruction $ # 45
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped 46
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss ? Renderer reconstruction $ # 47
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss Renderer reconstruction $ # 48
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder ? ? view ! depth " depth "′ texture flipped flip switch Reconstruction Loss ? Renderer reconstruction $ # 49
Photo-Geometric Autoencoding Q1 : How to avoid degenerate solutions? A1 : Enforce symmetry by flipping : horizontal flip input # encoder encoder encoder decoder decoder view ! depth " depth "′ texture flipped flip switch Reconstruction Loss Renderer reconstruction $ # 50
Photo-Geometric Autoencoding Q2 : What about non-symmetric lighting? : horizontal flip input # encoder encoder encoder decoder decoder view ! depth " depth "′ texture flipped 51
Photo-Geometric Autoencoding Q2 : What about non-symmetric lighting? A2 : Enforce symmetry on albedo : horizontal flip input # encoder encoder encoder encoder decoder decoder view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 52
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? : horizontal flip input # encoder encoder encoder encoder decoder decoder view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 53
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 54
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 55
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 56
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 57
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 58
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 59
Photo-Geometric Autoencoding Q3 : Non-symmetric albedo, deformation, etc? A3 : Predict uncertainty : horizontal flip input # encoder encoder encoder encoder encoder decoder decoder decoder conf. ) conf. )′ view ! depth " depth "′ light ' albedo ( albedo (′ flip switch Reconstruction Loss shading Renderer reconstruction $ canonical view & # 60
Results on human faces Images taken from CelebA, 3DFAW 61
input reconstruction input reconstruction 62
Results on face paintings Images taken from [1] 63 [1] Elliot J. Crowley, Omkar M. Parkhi, and Andrew Zisserman. Face painting: querying art with photos. In Proc. BMVC, 2015.
input reconstruction input reconstruction 64
Results on abstract faces Images taken from [1] and the Internet 65 [1] Elliot J. Crowley, Omkar M. Parkhi, and Andrew Zisserman. Face painting: querying art with photos. In Proc. BMVC, 2015.
input reconstruction input reconstruction 66
Results on video frames Video clips taken from VoxCeleb2 We do not use videos for training or fine-tuning. These results are obtained by applying our model trained on CelebA frame by frame . 67
68 recon. new view rotated recon. new view rotated input input recon. new view rotated recon. new view rotated input input
Relighting effects Images taken from CelebA 69
input reconstruction input reconstruction 70
Results on cat faces Images taken from [2] and [3] [2] Weiwei Zhang, Jian Sun, and Xiaoou Tang. Cat head detection - how to effectively exploit shape and texture features. In Proc. ECCV, 2008. [3] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In Proc. CVPR, 2012. 71
input reconstruction input reconstruction 72
Results on synthetic cars Images rendered using ShapeNet 73
input reconstruction input reconstruction 74
Symmetry Plane Visualization 75
Asymmetry Visualization 76
Discussion: Ablation Studies 77
Ablation – Symmetry full Input w/o albedo flip w/o depth flip Depth Normal Shading Albedo Shaded Recon. Insight #1 : Symmetry avoids degeneracy 78
Ablation – Lighting (Shape from Shading) full Input w/o lighting Depth Normal Shading Albedo Shaded Recon. Insight #2 : Lighting avoids bumpy shapes and provides cues for shape 79
Recommend
More recommend
