Differentiable Rendering Theory and Applications Cheng Zhang - PowerPoint PPT Presentation

Differentiable Rendering Theory and Applications Cheng Zhang Department of Computer Science University of California, Irvine

Outline • Introduction • Definition • Motivations • Related work • Our work • A Differential Theory of Radiative Transfer (SIGGRAPH ASIA 2019) • Future work

What is diff. rendering? Geometry 𝐺 𝝆 Camera Material Light Scene Parameter 𝝆 Rendering Image I

What is diff. rendering? Geometry 𝜖 & 𝐺 ( 𝝆 ) Camera Material Light Derivative Image 𝑱 $ Scene Parameter 𝝆

Why is diff. rendering important? Geometry Inverse Camera Material Rendering Light Scene Parameter 𝝆 Rendering Image I

Why is diff. rendering important? • Inverse rendering • Enable gradient-based optimization • Backpropagation through rendering (machine learning) Error Current Img. Target Img. Derivative Img. Scene Param.

Why is diff. rendering important? • Inverse rendering • Enable gradient-based optimization • Backpropagation through rendering (machine learning)

Related work • Rasterization rendering • Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning (ICCV 2019) • Neural 3D Mesh Renderer (CVPR 2018) • TensorFlow, pytorch3D etc. Neural 3D Mesh Renderer Soft Rasterizer

Related work Not General Volume Scattering Cloth Rendering Human Teeth Gkioulekas et al. 2013, 2016 Khungurn et al. 2015 Velinov et al. 2018 NLOS 3D Reconstruction Fabrication Reflectance & Lighting Estimation Tsai et al. 2019 Sumin et al. 2019 Azinovic et al. 2019

Related work • Inverse transport networks , Che et at. [2018] • Volumetric scattering ✓ • Geometry X • Differentiable Monte Carlo ray tracing through edge sampling , Li et at. [2018] • Volumetric scattering X • Geometry ✓

Our work • A Differential Theory of Radiative Transfer (SIGGRAPH ASIA 2019) • Differential theory of radiative transfer • Captures all surface and volumetric light transport effects • Supports derivative computation with respect to any parameters • Monte Carlo estimator • Unbiased estimation • Analogous to volumetric path tracing

Radiative Transfer • Ra Radiative Transfer a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin) Kutz et al. 2017 Gkioulekas et al. 2013

Radiative Transfer • Ra Radiative Transfer a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin) • Now used in many areas • Astrophysics (light transport in space) • Biomedicine (light transport in human tissue) • Nuclear science & engineering (neutron transport) • Remote sensing • …

Radiative Transfer • Ra Radiative Transfer a mathematical model describing how light interacts within participating media (e.g. smoke) and translucent materials (e.g. marble and skin) 𝛛 𝒚

Radiative Transfer Equation (RTE) Collision Transport operator operator Source 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 Radiative Transfer Equation (Operator Form)

Transport Operator 𝐿 , Transmittance Tr ? 𝑈 𝑦 $ , 𝑦 = exp − 2 𝜏 A 𝒚 − 𝜐 $ 𝝏 𝑒 𝜐 $ 𝐸 3 𝛛 Extinc Ex nction n coefficien ent 𝜏 A 𝒚 𝜐 𝒚 𝒚 $ = 𝒚 - 𝜐𝛛 controls how frequently light scatters and is also known as optical density 4 𝑈 𝒚 $ , 𝒚 (𝐿 - 𝑀) 𝒚 $ , 𝝏 𝑒𝜐 + 𝑅 𝑀 𝒚, 𝝏 = 2 3

Collision Operator 𝐿 - • pha phase e func unction n 𝑔 D 𝒚, 𝝏 𝒋 , 𝝏 A probability density over 𝕥 G given 𝒚 𝝏 𝒋 and 𝝏 𝒋 • sc scatteri ring coefficient 𝜏 H 𝒚 𝛛 𝒚 𝒚 $ =: 𝑀 LMN (𝑦, 𝜕) in-scattered radiance D 𝒚 $ , 𝝏 𝒋 , 𝝏 𝑀 𝒚 $ , 𝝏 𝒋 𝒆𝝏 𝒋 + 𝑅 𝑀 𝒚, 𝝏 = 𝐿 , 𝜏 H 𝒚 2 𝕥 I 𝑔

Source 𝑅 • Ab Absorption coeffici cient 𝜏 T 𝒚 • In Inter erfacial r radiance e 𝑀 H 𝐸 Boundary condition 𝛛 𝒚 𝟏 𝑴 𝒕 𝒚 $ 𝑴 𝒇 𝒚 Attenuation 4 𝑈 𝒚 $ , 𝒚 𝜏 T 𝒚 𝑀 U 𝒚 $ , 𝝏 𝑒𝜐 + 𝑈(𝒚 𝟏 , 𝒚) 𝑀 H (𝒚 𝟏 , 𝝏) 𝑀 𝒚, 𝝏 = 𝐿 , 𝐿 - 𝑀 + 2 3 radiant emission Interfacial radiance

𝑳 𝑼 𝑳 𝒅 Transport Operator Collision Operator 4 𝑈 𝒚 $ , 𝒚 𝜏 H 𝒚 2 D 𝒚 $ , 𝝏 𝒋 , 𝝏 𝑀 𝒚 $ , 𝝏 𝒋 𝒆𝝏 𝒋 𝑒𝜐 𝑀 𝒚, 𝝏 = 2 𝕥 I 𝑔 3 4 𝑈 𝒚 $ , 𝒚 𝜏 T 𝒚 $ 𝑀 U 𝒚 $ , 𝝏 𝑒𝜐 + 𝑈(𝒚 𝟏 , 𝒚)𝑀 H (𝒚 𝟏 , 𝝏) + 2 3 𝑹 Radiative Transfer Equation Source (Integral Form)

Differentiating the RTE 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 Di Differentiating bo both h side des 𝜖 & 𝑀 = 𝜖 & (𝐿 , 𝐿 - 𝑀) + 𝜖 & 𝑅 Differentiating individual operators

Differentiating the Collision Operator RTE: 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 𝑔 𝝏 𝒋 ( 𝒚 omitted for notational simplicity) 𝐿𝑑𝑀 𝝏 = 𝜏 H 2 𝕥 I 𝑔 D 𝝏 𝒋 , 𝝏 𝑀 𝝏 𝒋 d𝝏 𝒋 Scattering Phase coefficient function 𝕥 I 𝑔 𝝏 𝒋 d𝝏 𝒋 = ? Requires differentiating 𝜖 & 2 a (spherical) integral

Differentiating the Collision Operator 𝑔 𝝏 𝒋 𝐿 - 𝑀 𝝏 = 𝜏 H 2 𝕥 I 𝑔 D 𝝏 𝒋 , 𝝏 𝑀 𝝏 𝒋 d𝝏 𝒋 𝒐, 𝜖𝝏 𝒋 when 𝑔 has discontinuities 𝕥 I 𝑔 𝝏 𝒋 d𝝏 𝒋 = ≠ 𝜖 & 2 2 𝕥 I 𝜖 & 𝑔 𝝏 𝒋 d𝝏 𝒋 + 2 ∆𝑔(𝝏 𝒋 )d𝝏 𝒋 that depend 𝜌 𝜖𝜌 𝕥 Boundary Bo y term 𝜖 & 2 𝕥 I 𝑔 𝝏 𝒋 d𝝏 𝒋 (according to Reynolds transport theorem)

Boundary term 𝑔 𝝏 𝒋 ∆𝑔(𝝏 𝒋 ) = 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 𝐿 - 𝑀 𝝏 = 𝜏 H 2 𝕥 I 𝑔 D 𝝏 𝒋 , 𝝏 𝑀 𝝏 𝒋 d𝝏 𝒋 when 𝑔 D is continuous 𝒐, 𝜖𝝏 𝒋 2 ∆𝑔(𝝏 𝒋 )d𝝏 𝒋 𝜖𝜌 𝕥 ∆𝑔 is the difference of discontinuities of change rate of discontinuity integrand 𝑔 across the integrand f (in the normal direction) discontinuity 𝝏 𝒋

Sources of Discontinuity The boundary term: 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜖𝜌 𝕥 The boundary term: 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜖𝜌 𝜕 ` 𝜕 ` 𝕥 𝜌 Visibility Normal

Sources of Discontinuity The boundary term: 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜕 ` 𝜖𝜌 𝕥 Reduces to the change 𝜌 rate of 𝝏 𝒋 (as an angle)

Sources of Discontinuity The boundary term: 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜕 ` 𝜖𝜌 𝕥 𝜌 ∆𝑀 𝜕 ` = 𝑀 − 𝑀( ) (with the absence of attenuation)

Discontinuities in 3D visualization of 𝑀 visualization of discontinuity curves 𝕥 line integral 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜖𝜌 𝕥

Discontinuities in 3D Discontinuities curves: Projection of moving geometric edges onto the sphere 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜖𝜌 𝕥

Discontinuities in 3D Edge normal in the tangent space of the sphere • perpendicular to the discontinuity curve at direction 𝝏 𝒋 • 𝒐, 𝜖𝝏 𝒋 2 𝑔 D 𝝏 𝒋 , 𝝏 ∆𝑀 𝝏 𝒋 d𝝏 𝒋 𝜖𝜌 𝕥

Other Terms in the RTE 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 Transport operator 4 𝑈 𝑦 $ , 𝑦 (𝐿 - 𝑀)(𝑦 $ , 𝜕)𝑒𝜐 (𝐿 , 𝐿 - 𝑀) 𝑦, 𝜕 = 2 3 Transmittance Source 𝑅 = 𝑈(𝒚 𝟏 , 𝒚)𝑀 H (𝒚 𝟏 , 𝝏)

Full Radiance Derivative This is Eq. (32) of the paper

Significance of the Boundary Terms 𝑧 0 𝑄 fLghi 𝑄 fLghi = 𝑄 3 + 𝜌 0 𝑄 bcde 0 𝑄 bcde = 𝑄 l + 𝜌 0 Or Orig. Image Initial position (constant)

Significance of the Boundary Terms De Deriv. Image Or Orig. Image De Deriv. Image (no boundary term)

Differentiating the RTE: Summary 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 Key: • Tracking discontinuities of integrands • Establishing boundary terms accordingly 𝜖 & 𝑀 = 𝜖 & (𝐿 , 𝐿 - 𝑀) + 𝜖 & 𝑅

Differential RTE 𝑀 = 𝐿 , 𝐿 - 𝑀 + 𝑅 𝜖 & 𝑀 = 𝜖 & (𝐿 , 𝐿 - 𝑀) + 𝜖 & 𝑅 Boundary terms are included 𝜖 & 𝑀 𝜖 & 𝑀 + 𝜖 & 𝑅 = 𝐿 , 𝐿 - 𝐿 ∗ 0 𝐿 , 𝐿 - 𝑀 𝑀 𝑅

Differentiable Volumetric Path Tracing Side Path 1 Side Path 2 ∆𝑀 𝒚 𝟏 𝛛 𝟏 𝒚 𝟐 ∆𝑀 Component 1: 𝛛 𝟐 Derivative of path throughput ∆𝑀 𝒚 𝟑 Component 2: 𝛛 𝟑 Side paths (for estimating ∆𝑀 ) 𝒚 𝟒

Results

Results: Validation 𝑧 0 𝑄 fLghi 𝑄 fLghi = 𝑄 3 + 𝜌 0 𝑄 bcde 0 𝑄 bcde = 𝑄 l + 𝜌 0 Orig. Image Or Initial position (constant)

Results: Validation Absol Ab olute Fi Finite Diff. differ di erenc ence lar large spac acin ing sm small sp spacing Orig. Image Or Ours Ou (e (equal-ti time c e com omparison son)