Neural l Renderin ing Prof. Leal-Taix and Prof. Niessner 1 - - PowerPoint PPT Presentation
Neural l Renderin ing Prof. Leal-Taix and Prof. Niessner 1 Renderin ing Camera Def. - Intrinsics - Often: 3D Scene: - focal length - Material - principal point) - Lighting - Geometry (incl. animation) Camera View Point -
1
2
3D Scene:
(incl. animation) Camera View Point
Camera Def.
The Rendering Equation [Kajiya 86]
3
Textures Material & Lighting Geometry
4
ICCV’09 [Agarwal et al.]: Building Rome in a Day
5
Computer Graphics
Computer Vision
6
3D Model + Textures + Shading -> Synthetic Image Star Wars Rogue One
Discriminator loss Generator loss
[Karras et al. 18]
Generative Adversarial Networks
7
Novel View point synthesis:
8
Neural Network
scene description, lighting, materials, etc. 6 DoF Camera Pose / View Point
Ground truth for training
Testing
9
10
11
12
[Sitzmann et al. CVPR’19] Deep Voxels
13
– Why learn 3D operations with 2D Convs !?!? – We know how 3D transformations work
[ R | t | t ]
– Incorporate these into the architectures
– Example application: novel view point synthesis
[Sitzmann et al. CVPR’19] Deep Voxels
14
2D U-Net
Renderin g Lifting Layer 2D 3D
2D U-Net
2D Feature Extraction
Source View R, t
Projection Layer 3D 2D
Output Source
Target View R, t
3D U-Net
3D Features
Simplified overview for novel view synthesis [Sitzmann et al. CVPR’19] Deep Voxels
15
[Sitzmann et al. CVPR’19] Deep Voxels
16
Issue: we don’t know the depth for the target!
Occlusion Network:
[Sitzmann et al. CVPR’19] Deep Voxels
17
[Sitzmann et al. ’18] Deep Voxels
18
[Sitzmann et al. ’18] Deep Voxels
19
– No need to take specific care for temp. coherency!
– I.e., cGAN for new pose in a given scene
[Sitzmann et al. ’18] Deep Voxels
20
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric
Both Scene Representation and Differentiable Renderer often adapted from traditional computer graphics.
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
21
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
22
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
23
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
24
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
25
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3)
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
26
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3)
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
27
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3) High quality Requires good SFM No compact representation
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
28
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3) High quality Requires good SFM No compact representation
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
29
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3) High quality Requires good SFM No compact representation High quality Requires good SFM
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
30
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Traced Volumetric Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3) High quality Requires good SFM No compact representation High quality Requires good SFM
Pros Pros Cons Cons Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
31
Scene Scene Representa tion tion Renderer
ℝ3՜ ℝ𝑜 ReLU MLP Generalized (learned) sphere-tracing
Generalizati
Hypernetwork
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
32
Full 3D Reconstruction from single image!
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
33
Scene Scene Representa tion tion Renderer
ℝ6՜ ℝ3 ReLU MLP + Positional Encoding View Direction Volumetric, stratified sampling
Generalizati
None.
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
34
Photorealistic, including view-dependence! (~100 images)
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
35
Scene Scene Representa tion tion
Multi-Plane Images Voxelgrids Image-based Point Clouds Implicit Function
Rendere rer
(Alpha) compositing Volumetric Ray-based Rasterization Splatting Sphere-Tracing Volumetric
Pros Pros Cons Cons
Fast rendering High quality Generalizes Only 2.5D Size “True 3D” High quality No reconstruction priors Memory O(n3) High quality Requires good SFM No compact representation High quality Requires good SFM High quality May generalize! Expensive rendering, training
Slides: Vincent Sitzmann (Eurographics State-of-the-art on Neural Rendering)
36
37
Siggraph’19 [Thies et al.]: Neural Textures 3D Geometry Neural Texture
38
UV-Map Sampled Texture Siggraph’19 [Thies et al.]: Neural Textures 3D Geometry
Rendering 3D 2D
View R, t
Neural Texture
39
UV-Map Renderer Output Image Sampled Texture Siggraph’19 [Thies et al.]: Neural Textures 3D Geometry
Rendering 3D 2D
View R, t
Neural Texture
40
Albedo Depth Normal Lighting
Deferred Renderer Handcrafted ”Feature Maps“
Siggraph’19 [Thies et al.]: Neural Textures
41
Albedo Depth Normal Lighting
Deferred Renderer Handcrafted ”Feature Maps“
Learned
Neural
Siggraph’19 [Thies et al.]: Neural Textures
42
UV-Map Renderer Output Image Sampled Texture 3D Geometry
Rendering 3D 2D
View R, t
Neural Texture Siggraph’19 [Thies et al.]: Neural Textures
43
Siggraph’19 [Thies et al.]: Neural Textures
44
Input UV-Map Ours
Siggraph’19 [Thies et al.]: Neural Textures
45
Ground Truth Ours
Siggraph’19 [Thies et al.]: Neural Textures
46
Geometry Editing Input Sequence
Siggraph’19 [Thies et al.]: Neural Textures
47
Siggraph’19 [Thies et al.]: Neural Textures
48
Siggraph’19 [Thies et al.]: Neural Textures
49
Siggraph’19 [Thies et al.]: Neural Textures
50
Siggraph’19 [Thies et al.]: Neural Textures
51
Siggraph’19 [Thies et al.]: Neural Textures
52
Siggraph’19 [Thies et al.]: Neural Textures
53
Siggraph’19 [Thies et al.]: Neural Textures
54
Siggraph’19 [Thies et al.]: Neural Textures
56
Siggraph’19 [Thies et al.]: Neural Textures
57
58
59
60
[Hannun et al.] DeepSpeech RNN
Output of the RNN of DeepSpeech:
We use a time window (n=16)
61
62
63
Person-specific Blendshape Expression Model
64
Audio2Expression Training
65
Hundreds of commentator videos available
66
Flame Model Basel Model
with pose
67
68
69
70
71
72
Photo-realistic Reconstruction
73
Siggraph’19 [Thies et al.]: Neural Textures
74
Why learn 3D operations, such as transformations?
Capsule networks are motivated by inverse graphics [Sabour et al. 17]
75
77
78
79
80
81
82
Dense Sparse
[Liu et al. 19] Neural Rendering and Reenactment of Human Actor Videos
83
84
Dense Sparse
85