Differentiable Rendering Theory and Applications Cheng Zhang - - PowerPoint PPT Presentation

β–Ά
differentiable rendering theory and applications
SMART_READER_LITE
LIVE PREVIEW

Differentiable Rendering Theory and Applications Cheng Zhang - - PowerPoint PPT Presentation

Jan 29, 2024 617 likes β€’1.13k views

Differentiable Rendering Theory and Applications Cheng Zhang Department of Computer Science University of California, Irvine Outline Introduction Definition Motivations Related work Our work A Differential Theory of

slide-1
SLIDE 1

Differentiable Rendering Theory and Applications

Cheng Zhang Department of Computer Science University of California, Irvine

slide-2
SLIDE 2

Outline

  • Introduction
  • Definition
  • Motivations
  • Related work
  • Our work
  • A Differential Theory of Radiative Transfer (SIGGRAPH ASIA 2019)
  • Future work
slide-3
SLIDE 3

What is diff. rendering?

Rendering Image I

𝐺 𝝆

Geometry Camera Material Light

Scene Parameter 𝝆

slide-4
SLIDE 4

What is diff. rendering?

Derivative Image 𝑱$

πœ–&𝐺(𝝆)

Geometry Camera Material Light

Scene Parameter 𝝆

slide-5
SLIDE 5

Why is diff. rendering important?

Rendering Image I

Geometry Camera Material Light

Scene Parameter 𝝆 Inverse Rendering

slide-6
SLIDE 6

Why is diff. rendering important?

  • Inverse rendering
  • Enable gradient-based optimization
  • Backpropagation through rendering (machine learning)

Scene Param. Error Derivative Img. Current Img. Target Img.

slide-7
SLIDE 7

Why is diff. rendering important?

  • Inverse rendering
  • Enable gradient-based optimization
  • Backpropagation through rendering (machine learning)
slide-8
SLIDE 8

Related work

  • Rasterization rendering
  • Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning (ICCV 2019)
  • Neural 3D Mesh Renderer (CVPR 2018)
  • TensorFlow, pytorch3D etc.

Soft Rasterizer Neural 3D Mesh Renderer

slide-9
SLIDE 9

Related work

Volume Scattering Gkioulekas et al. 2013, 2016 Human Teeth Velinov et al. 2018 NLOS 3D Reconstruction Tsai et al. 2019 Fabrication Sumin et al. 2019 Reflectance & Lighting Estimation Azinovic et al. 2019 Cloth Rendering Khungurn et al. 2015

Not General

slide-10
SLIDE 10

Related work

  • Inverse transport networks, Che et at. [2018]
  • Volumetric scattering βœ“
  • Geometry X
  • Differentiable Monte Carlo ray tracing through edge sampling, Li et at. [2018]
  • Volumetric scattering X
  • Geometry βœ“
slide-11
SLIDE 11

Our work

  • A Differential Theory of Radiative Transfer (SIGGRAPH ASIA 2019)
  • Differential theory of radiative transfer
  • Captures all surface and volumetric light transport effects
  • Supports derivative computation with respect to any parameters
  • Monte Carlo estimator
  • Unbiased estimation
  • Analogous to volumetric path tracing
slide-12
SLIDE 12

Radiative Transfer

  • Ra

Radiative Transfer

a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin)

Kutz et al. 2017

Gkioulekas et al. 2013

slide-13
SLIDE 13

Radiative Transfer

  • Ra

Radiative Transfer

a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin)

  • Now used in many areas
  • Astrophysics (light transport in space)
  • Biomedicine (light transport in human tissue)
  • Nuclear science & engineering (neutron transport)
  • Remote sensing
  • …
slide-14
SLIDE 14

Radiative Transfer

  • Ra

Radiative Transfer

a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin)

π’š 𝛛

slide-15
SLIDE 15

Radiative Transfer Equation (RTE)

𝑀 = 𝐿,𝐿-𝑀 + 𝑅

Collision

  • perator

Transport

  • perator

Source

Radiative Transfer Equation

(Operator Form)

slide-16
SLIDE 16

Transport Operator 𝐿,

𝑀 π’š, 𝝏 = 2

3 4

π‘ˆ π’š$, π’š (𝐿-𝑀) π’š$, 𝝏 π‘’πœ + 𝑅

Tr Transmittance

π‘ˆ 𝑦$, 𝑦 = exp βˆ’ 2

3 ?

𝜏A π’š βˆ’ 𝜐$𝝏 𝑒 𝜐$

Ex Extinc nction n coefficien ent 𝜏A π’š

controls how frequently light scatters and is also known as optical density

π’š$= π’š-πœπ›› π’š 𝛛 𝜐 𝐸

slide-17
SLIDE 17

Collision Operator 𝐿-

  • pha

phase e func unction n 𝑔

D π’š, 𝝏𝒋, 𝝏

A probability density over π•₯G given π’š and 𝝏𝒋

  • sc

scatteri ring coefficient 𝜏H π’š

𝑀 π’š, 𝝏 = 𝐿,𝜏H π’š 2

π•₯I𝑔 D π’š$, 𝝏𝒋, 𝝏 𝑀 π’š$, 𝝏𝒋 𝒆𝝏𝒋 + 𝑅

=: 𝑀LMN(𝑦, πœ•) in-scattered radiance

π’š$ π’š 𝛛 𝝏𝒋

slide-18
SLIDE 18

radiant emission

π’š$ π’š 𝛛 π’šπŸ 𝑴𝒇 𝐸 𝑴𝒕

Source 𝑅

  • Ab

Absorption coeffici cient 𝜏T π’š

  • In

Inter erfacial r radiance e 𝑀H

Boundary condition Interfacial radiance Attenuation

𝑀 π’š, 𝝏 = 𝐿,𝐿-𝑀 + 2

3 4

π‘ˆ π’š$, π’š 𝜏T π’š 𝑀U π’š$, 𝝏 π‘’πœ + π‘ˆ(π’šπŸ, π’š) 𝑀H(π’šπŸ, 𝝏)

slide-19
SLIDE 19

𝑳𝑼 Transport Operator 𝑹 Source 𝑳𝒅 Collision Operator Radiative Transfer Equation

(Integral Form)

𝑀 π’š, 𝝏 = 2

3 4

π‘ˆ π’š$, π’š 𝜏H π’š 2

π•₯I𝑔 D π’š$, 𝝏𝒋, 𝝏 𝑀 π’š$, 𝝏𝒋 𝒆𝝏𝒋 π‘’πœ

+ 2

3 4

π‘ˆ π’š$, π’š 𝜏T π’š$ 𝑀U π’š$, 𝝏 π‘’πœ + π‘ˆ(π’šπŸ, π’š)𝑀H(π’šπŸ, 𝝏)

slide-20
SLIDE 20

Differentiating the RTE

Differentiating individual operators

𝑀 = 𝐿,𝐿-𝑀 + 𝑅 πœ–&𝑀 = πœ–&(𝐿,𝐿-𝑀) + πœ–&𝑅

Di Differentiating bo both h side des

slide-21
SLIDE 21

𝐿𝑑𝑀 𝝏 = 𝜏H 2

π•₯I 𝑔 D 𝝏𝒋, 𝝏 𝑀 𝝏𝒋 d𝝏𝒋

𝑔 𝝏𝒋

(π’š omitted for notational simplicity)

πœ–& 2

π•₯I𝑔 𝝏𝒋 d𝝏𝒋 = ?

Differentiating the Collision Operator

Scattering coefficient Phase function

𝑀 = 𝐿,𝐿-𝑀 + 𝑅 RTE:

Requires differentiating a (spherical) integral

slide-22
SLIDE 22

Differentiating the Collision Operator

when 𝑔 has discontinuities that depend 𝜌

β‰ 

+

2

π•₯Iπœ–&𝑔 𝝏𝒋 d𝝏𝒋

πœ–& 2

π•₯I𝑔 𝝏𝒋 d𝝏𝒋 =

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ βˆ†π‘”(𝝏𝒋)d𝝏𝒋

Bo Boundary y term 𝐿-𝑀 𝝏 = 𝜏H 2

π•₯I 𝑔 D 𝝏𝒋, 𝝏 𝑀 𝝏𝒋 d𝝏𝒋

𝑔 𝝏𝒋

πœ–& 2

π•₯I𝑔 𝝏𝒋 d𝝏𝒋

(according to Reynolds transport theorem)

slide-23
SLIDE 23

Boundary term

change rate of discontinuity (in the normal direction) βˆ†π‘” is the difference of integrand 𝑔 across the discontinuity 𝝏𝒋 discontinuities of integrand f

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ βˆ†π‘”(𝝏𝒋)d𝝏𝒋

βˆ†π‘”(𝝏𝒋) = 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋

when 𝑔

D is continuous

𝐿-𝑀 𝝏 = 𝜏H 2

π•₯I 𝑔 D 𝝏𝒋, 𝝏 𝑀 𝝏𝒋 d𝝏𝒋

𝑔 𝝏𝒋

slide-24
SLIDE 24

Sources of Discontinuity

Visibility πœ•` Normal

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

The boundary term:

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

The boundary term:

𝜌

πœ•`

slide-25
SLIDE 25

Sources of Discontinuity

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

The boundary term:

𝜌

πœ•`

Reduces to the change rate of 𝝏𝒋 (as an angle)

slide-26
SLIDE 26

Sources of Discontinuity

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

The boundary term:

𝜌

πœ•` βˆ†π‘€ πœ•` = 𝑀 βˆ’ 𝑀( )

(with the absence of attenuation)

slide-27
SLIDE 27

visualization of 𝑀 visualization of discontinuity curves π•₯ line integral

Discontinuities in 3D

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

slide-28
SLIDE 28

Discontinuities curves:

Projection of moving geometric edges onto the sphere

Discontinuities in 3D

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

slide-29
SLIDE 29

Edge normal

  • in the tangent space of the sphere
  • perpendicular to the discontinuity curve at direction 𝝏𝒋

Discontinuities in 3D

2

π•₯

𝒐, πœ–ππ’‹ πœ–πœŒ 𝑔

D 𝝏𝒋, 𝝏 βˆ†π‘€ 𝝏𝒋 d𝝏𝒋

slide-30
SLIDE 30

Other Terms in the RTE

(𝐿,𝐿-𝑀) 𝑦, πœ• = 2

3 4

π‘ˆ 𝑦$, 𝑦 (𝐿-𝑀)(𝑦$, πœ•)π‘’πœ Transport operator

𝑀 = 𝐿,𝐿-𝑀 + 𝑅

Transmittance

𝑅 = π‘ˆ(π’šπŸ, π’š)𝑀H(π’šπŸ, 𝝏) Source

slide-31
SLIDE 31

Full Radiance Derivative

This is Eq. (32) of the paper

slide-32
SLIDE 32

Significance of the Boundary Terms

Or

  • Orig. Image

𝑄bcde 𝑄fLghi 𝑄fLghi = 𝑄3 + 𝜌 𝑧 𝑄bcde = 𝑄

l +

𝜌 Initial position (constant)

slide-33
SLIDE 33

Significance of the Boundary Terms

Or

  • Orig. Image

De

  • Deriv. Image

De

  • Deriv. Image

(no boundary term)

slide-34
SLIDE 34

𝑀 = 𝐿,𝐿-𝑀 + 𝑅 πœ–&𝑀 = πœ–&(𝐿,𝐿-𝑀) + πœ–&𝑅 Differentiating the RTE: Summary

Key:

  • Tracking discontinuities of integrands
  • Establishing boundary terms accordingly
slide-35
SLIDE 35

𝑀 = 𝐿,𝐿-𝑀 + 𝑅 Differential RTE

πœ–&𝑀 𝑀 = 𝐿,𝐿- πΏβˆ— 𝐿,𝐿- πœ–&𝑀 𝑀 + πœ–&𝑅 𝑅

Boundary terms are included

πœ–&𝑀 = πœ–&(𝐿,𝐿-𝑀) + πœ–&𝑅

slide-36
SLIDE 36

Differentiable Volumetric Path Tracing

π’šπŸ π››πŸ π’šπŸ π››πŸ π’šπŸ‘ π››πŸ‘ π’šπŸ’

Side Path 1 Side Path 2

βˆ†π‘€ βˆ†π‘€ βˆ†π‘€

Component 2:

Side paths (for estimating βˆ†π‘€)

Component 1:

Derivative of path throughput

slide-37
SLIDE 37

Results

slide-38
SLIDE 38

Or

  • Orig. Image

𝑄bcde 𝑄fLghi 𝑄fLghi = 𝑄3 + 𝜌 𝑧 𝑄bcde = 𝑄

l +

𝜌 Initial position (constant)

Results: Validation

slide-39
SLIDE 39

Results: Validation

(e (equal-ti time c e com

  • mparison

son)

Or

  • Orig. Image

Ou Ours lar large spac acin ing sm small sp spacing Fi Finite Diff. Ab Absol

  • lute

di differ erenc ence

slide-40
SLIDE 40

Results: Inverse Rendering

  • Scene configurations
  • participating media
  • changing geometry
  • Optimization
  • L2 loss for its simplicity
  • Any differentiable metric can be used with our method
slide-41
SLIDE 41

Apple in a Box

#iterations Time (CPU core minute per iteration) 80 12.2

Target Optimization process Parameters

Cube roughness Apple position

slide-42
SLIDE 42

Camera Pose

#iterations Time (CPU core minute per iteration) 220 9.3

Target Optimization Process

slide-43
SLIDE 43

None-Line-of-Sight

#iterations Time (CPU core minute per iteration) 100 9

Target Optimization Process

  • Diff. View

(n (not

  • t used in op
  • ptimization
  • n)
slide-44
SLIDE 44

None-Line-of-Sight

Heterogeneous medium Density scaling (param) Medium orientation (param)

slide-45
SLIDE 45

None-Line-of-Sight

#iterations Time (CPU core minute per iteration) 60 7.6

Target Optimization process

  • Diff. View
slide-46
SLIDE 46

Design-Inspired

Spotlight A Light direction Light color Light falloff angle Spotlight B Parameters

slide-47
SLIDE 47

Design-Inspired

Target Optimization process

#iterations Time (CPU core minute per iteration) 110 11.2

slide-48
SLIDE 48

Design-inspired

#iterations Time (CPU core minute per iteration) 100 27.2

Target Optimization Process

  • Diff. View
slide-49
SLIDE 49

Future work

  • Differentiable rendering is slow due to
  • Main term
  • Needs better sampling methods
  • Boundary term
  • Detecting visibility changes (e.g., object silhouettes)
  • Tracing side paths
slide-50
SLIDE 50

Thank you