Estimation from an Arbitrary Number of Images Duan Gao 1,3 , Xiao Li - PowerPoint PPT Presentation

Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images Duan Gao 1,3 , Xiao Li 2,3 , Yue Dong 3 , Pieter Peers 4 , Kun Xu 1 , Xin Tong 3 1 Tsinghua University 2 University of Science and Technology of China 3 Microsoft Research Asia 4 College of William & Mary

Rendering 1

Materials Geometry Material 2

Appearance Estimation 3

Related Work • Classic Inverse Rendering • Multi-Image Heuristics-based Appearance Modeling. • Single/Few Image Reflectance [Aittala et al. 2015] [Dong et al. 2014] Modeling • Learning-based Appearance Modeling [Li et al. 2018] [Deschaintre et al. 2018] 4

Our goal Unified framework 5

Three lighting conditions Challenges Reference Non-trivial to combine current solutions • Learning-based methods: Render hard to extend to arbitrary number of inputs 6

Three lighting conditions Challenges Reference Non-trivial to combine current solutions Render • Classic Inverse Rendering: failed when input number is insufficient. 7

Our goal Single Plausible … Multiple Accurate 8

Assumptions • Planar object • Point light source collocated with the camera • Fix distance between object plane and camera 9

Overview Key Idea: Deep Inverse Rendering SVBRDF auto-encoder SVBRDFs Multiple measurements 10

Overview Key Idea: Deep Inverse Rendering SVBRDF auto-encoder SVBRDFs Multiple measurements 11

Overview Key Idea: Deep Inverse Rendering SVBRDFs Multiple measurements 12

SVBRDF auto-encoder 13

Training SVBRDF auto-encoder Training Loss: 14

Training SVBRDF auto-encoder Normal Diffuse Roughness Specular Render Training Loss: Reference Ours 15

Training SVBRDF auto-encoder Training Loss: Latent space smoothness: 16

Training SVBRDF auto-encoder Normal Diffuse Roughness Specular Render Closeup Training Loss: Ours Reference Latent space smoothness: smoothness Without 17

t-SNE visualizations 18

Optimize latent code from measurement(s) SVBRDF auto-encoder SVBRDFs Multiple measurements 19

Bootstrap the optimization State-of-the art single input network [Deschaintre et al.] SVBRDF auto-encoder SVBRDFs Multiple measurements 20

Optimize in latent space SVBRDF auto-encoder SVBRDFs Multiple measurements 21

Detail refinement After Before Reference SVBRDF auto-encoder SVBRDFs Multiple measurements 22

Improved quality with single input Normal Diffuse Roughness Specular Top View Novel view render Reference Deschaintr e et al. Ours N=1 23

Improved quality with single input Deschaintr e et al. Ours Diffuse Novel view render Normal Roughness Specular 24

Normal Diffuse Roughness Specular Top View Novel view render Reference Ours N=1 Plausible Ours N=5 Accurate Ours N=20 25

Comparison with class inverse rendering Classic inverse rendering ours Reference Classic inverse rendering Ours 26

Comparison with class inverse rendering Classic inverse rendering ours Reference Classic inverse rendering Ours 27

High resolution results Support arbitrary resolution! Estimated SVBRDF with 20 input photos Novel view rendering 28

High resolution results Support arbitrary resolution! Estimated SVBRDF with 20 input photos Novel view rendering 29

High resolution results Support arbitrary resolution! Estimated SVBRDF with 20 input photos Novel view rendering 30

Real captured results GT Leather, 1k resolution, 2 inputs Render Diffuse Normal Roughness Specular Novel view 31

Real captured results GT Wood, 1k resolution, 10 inputs Render Diffuse Normal Roughness Specular Novel view 32

Real captured results GT Metal Plate, 1k resolution, 30 inputs Render Diffuse Normal Roughness Specular Novel view 33

Robustness LDR HDR Reference Reference Top view init. Side view init. 34

Robustness Reference 35

Robustness 10% 5% 0% Reference 36

Conclusion • Unified deep inverse rendering framework for estimating SVBRDF from arbitrary number of input photographs. • Learned latent space + optimization in latent space 37

Thanks 38