[PPT] - Neural Rendering Chuan Li Lambda Labs Collaborators: Thu PowerPoint Presentation

SLIDE 1

Neural Rendering

Chuan Li Lambda Labs

Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang

SLIDE 2

Model Pictures

Forward (Computer Graphics)

SLIDE 3

Model Pictures

Forward (Computer Graphics) Inverse (Computer Vision)

SLIDE 4

SLIDE 5

SLIDE 6

Integral of the incident radians

SLIDE 7

BRDF

SLIDE 8

SLIDE 9

SLIDE 10

32K SPP Ray Tracing (90 mins 12 CPU Cores) The Tungsten Renderer

SLIDE 11

P0 P1

SLIDE 12

P0 P1

SLIDE 13

R01 | T01

P0 P1

SLIDE 14

Inverse (Computer Vision)

R01 | T01

P0 P1

SLIDE 15

P2

R12 | T12

Inverse (Computer Vision)

R01 | T01

P0 P1

SLIDE 16

Building Rome in a Day

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski

SLIDE 17

Model Pictures

Sub-module End-2-End Differentiable Rendering

SLIDE 18

1 SPP 2048 SPP

SLIDE 19

Sub-modules

Mastering the game of Go with deep neural networks and tree search

David Silver et al.

SLIDE 20

Sub-modules

Value Network

Mastering the game of Go with deep neural networks and tree search

David Silver et al.

SLIDE 21

Sub-modules

Value Network Policy Network

Mastering the game of Go with deep neural networks and tree search

David Silver et al.

SLIDE 22

2^15 SPP 4 SPP

Value Networks Denoising

SLIDE 23

2^15 SPP

Value Networks Denoising Policy Networks Same SPP

4 SPP

SLIDE 24

2^15 SPP

Value Networks Denoising Policy Networks Same SPP

4 SPP

SLIDE 25

4 SPP Denoised 1 sec 2080 Ti 32K SPP Ray Tracing 90 mins 12 cores CPU Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019

SLIDE 26

loss

Decoder Encoder Input

x

Ref Output

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019

SLIDE 27

L1 VGG Loss L1 VGG Loss + GAN

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019

SLIDE 28

loss

Decoder Encoder Diffuse Input

x

Diffuse Output Decoder Encoder Specular Input

x

Specular Output

Ref Output

Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019

SLIDE 29

Auxiliary

loss

Decoder Encoder Diffuse Input

x

Diffuse Output Decoder Encoder Specular Input

x

Specular Output

Ref Output

Albedo, normal, depth

SLIDE 30

Auxiliary

Conv LeakyReLU Conv

x

Element-wise Biasing

SLIDE 31

Conv

Auxiliary

LeakyReLU Conv Conv LeakyReLU Conv

x

Element-wise Biasing Element-wise Scaling

SLIDE 32

Conv

Auxiliary

LeakyReLU Conv Conv LeakyReLU Conv

x

Element-wise Biasing (OR) Element-wise Scaling (AND)

SLIDE 33

Denoise comparison 4 SPP Adversarial Monte Carlo denoising with conditioned auxiliary feature modulation

B Xu et al. Siggraph Asia 2019

SLIDE 34

2^15 SPP

Value Networks Denoising Policy Networks Same SPP

4 SPP

SLIDE 35

Neural Importance Sampling

Thomas Müller et al. ACM Transactions on Graphics 2019

SLIDE 36

incidence radiance map

Neural Importance Sampling

Thomas Müller et al. ACM Transactions on Graphics 2019

SLIDE 37

SLIDE 38

Neural Importance Sampling

Thomas Müller et al. ACM Transactions on Graphics 2019

SLIDE 39

Neural Importance Sampling

Thomas Müller et al. ACM Transactions on Graphics 2019

SLIDE 40

Model Pictures

Sub-module End-2-End Differentiable Rendering

SLIDE 41

Ray Tracing Image Centric Rasterization Object Centric

SLIDE 42

Ray Tracing Image Centric Rasterization Object Centric Visibility

SLIDE 43

Ray Tracing Image Centric Rasterization Object Centric Shading

SLIDE 44

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good

NN friendly

Great Yes No Enemy

SLIDE 45

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good

NN friendly

Great Yes No Enemy

SLIDE 46

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good

NN friendly

Great Yes No Enemy

SLIDE 47

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good

NN friendly

Great Yes No Enemy

SLIDE 48

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 49

Neural Voxels 32 x 32 x 32 x 16

3D Encoder

Neural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 50

Neural Voxels 32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512 Neural Pixels

Visibility Neural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 51

Neural Voxels 32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512 Neural Pixels

Visibility Neural Voxels

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 52

Neural Voxels 32 x 32 x 32 x 16

3D Encoder

3D-2D

32 x 32 x 512 Neural Pixels

2D Decoder

Shading Neural Voxels Visibility MSE pixel loss

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 53

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 54

Contour Toon Ambient Occlusion

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 55

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 56

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 57

3D Encoder

3D-2D

Neural Pixels

2D Decoder Texture Network

Neural Texture Voxels

r

Neural Voxels

Channel-wise Concatenation

64 x 64 x 64 x 4 64 x 64 x 64 x 1

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 58

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 59

Same shape, different textures Same texture, different shapes

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Thu Nguyen-Phuoc et al. NeurIPS 2018

SLIDE 60

Depth Map Voxel Point Cloud Mesh Memory Good Very Poor Poor Very Good

NN friendly

Great Yes No Enemy

SLIDE 61

Rasterization a RGB point cloud

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019

SLIDE 62

Rasterization a neural point cloud (First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019

SLIDE 63

Rasterization a neural point cloud (First three PCA dimensions of the neural descriptor)

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019

SLIDE 64

Neural Point-Based Graphics

KA Aliev et al, arxiv 2019

RBG rasterization Neural rasterization

SLIDE 65

Neural 3D Mesh Renderer

H Kato et al, CVPR 2018

Deferred Neural Rendering: Image Synthesis using Neural Textures

J Thies et al, Siggraph 2019

SLIDE 66

Model Pictures

Sub-module End-2-End

SLIDE 67

?

SLIDE 68

Target Approximation

SLIDE 69

Target Approximation Rendered Approximation

SLIDE 70

Loss

Back-propagate

Target Rendered Approximation Approximation

SLIDE 71

Loss Target Rendered Approximation Updated Approximation

Back-propagate

SLIDE 72

Loss Target Rendered Approximation Updated Approximation

Back-propagate For Free

SLIDE 73

Loss Target Rendered Approximation Updated Approximation

Back-propagate Expensive

SLIDE 74

Loss Target Rendered Approximation Decoder Encoder Reconstruction Rendering

SLIDE 75