Estimation from an Arbitrary Number of Images Duan Gao 1,3 , Xiao Li - - PowerPoint PPT Presentation

▶
estimation from an arbitrary number of images
SMART_READER_LITE
LIVE PREVIEW

Estimation from an Arbitrary Number of Images Duan Gao 1,3 , Xiao Li - - PowerPoint PPT Presentation

Jun 03, 2023 132 likes •540 views

Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images Duan Gao 1,3 , Xiao Li 2,3 , Yue Dong 3 , Pieter Peers 4 , Kun Xu 1 , Xin Tong 3 1 Tsinghua University 2 University of Science and Technology of China 3

slide-1
SLIDE 1

Deep Inverse Rendering for High-resolution SVBRDF Estimation from an Arbitrary Number of Images

Duan Gao 1,3, Xiao Li 2,3, Yue Dong 3, Pieter Peers 4, Kun Xu 1, Xin Tong 3

1 Tsinghua University 2 University of Science and Technology of China 3 Microsoft Research Asia 4 College of William & Mary

slide-2
SLIDE 2

Rendering

1

slide-3
SLIDE 3

Materials

Material Geometry

2

slide-4
SLIDE 4

Appearance Estimation

3

slide-5
SLIDE 5

Related Work

  • Classic Inverse Rendering

[Deschaintre et al. 2018] [Li et al. 2018] [Aittala et al. 2015] [Dong et al. 2014]

  • Learning-based

Appearance Modeling

  • Multi-Image Heuristics-based

Appearance Modeling.

  • Single/Few Image Reflectance

Modeling

4

slide-6
SLIDE 6

Our goal Unified framework

5

slide-7
SLIDE 7

Challenges

Non-trivial to combine current solutions

6

Reference Render Three lighting conditions

  • Learning-based methods:

hard to extend to arbitrary number of inputs

slide-8
SLIDE 8

Challenges

Non-trivial to combine current solutions

7

Reference Render Three lighting conditions

  • Classic Inverse Rendering:

failed when input number is insufficient.

slide-9
SLIDE 9

Our goal

Plausible Accurate Single Multiple

…

8

slide-10
SLIDE 10

Assumptions

  • Planar object
  • Point light source collocated with the

camera

  • Fix distance between object plane

and camera

9

slide-11
SLIDE 11

Overview

Key Idea: Deep Inverse Rendering

SVBRDF auto-encoder SVBRDFs Multiple measurements

10

slide-12
SLIDE 12

Overview

Key Idea: Deep Inverse Rendering

SVBRDF auto-encoder SVBRDFs Multiple measurements

11

slide-13
SLIDE 13

Overview

Key Idea: Deep Inverse Rendering

SVBRDFs Multiple measurements

12

slide-14
SLIDE 14

SVBRDF auto-encoder

13

slide-15
SLIDE 15

Training SVBRDF auto-encoder

Training Loss:

14

slide-16
SLIDE 16

Training SVBRDF auto-encoder

Training Loss:

15

Normal Diffuse Roughness Specular Render

Ours Reference

slide-17
SLIDE 17

Training SVBRDF auto-encoder

Training Loss: Latent space smoothness:

16

slide-18
SLIDE 18

Training SVBRDF auto-encoder

Training Loss: Latent space smoothness:

17

Normal Diffuse Roughness Specular Render Closeup Ours Reference Without smoothness

slide-19
SLIDE 19

t-SNE visualizations

18

slide-20
SLIDE 20

Optimize latent code from measurement(s)

SVBRDFs Multiple measurements SVBRDF auto-encoder

19

slide-21
SLIDE 21

Bootstrap the optimization

SVBRDFs Multiple measurements SVBRDF auto-encoder

State-of-the art single input network [Deschaintre et al.]

20

slide-22
SLIDE 22

Optimize in latent space

SVBRDF auto-encoder SVBRDFs Multiple measurements

21

slide-23
SLIDE 23

Detail refinement

SVBRDFs Multiple measurements SVBRDF auto-encoder

After Before Reference

22

slide-24
SLIDE 24

Improved quality with single input

Normal Diffuse Roughness Specular Top View Novel view render Reference Deschaintr e et al. Ours N=1

23

slide-25
SLIDE 25

Improved quality with single input

Normal Diffuse Roughness Specular Novel view render Ours Deschaintr e et al.

24

slide-26
SLIDE 26

Normal Diffuse Roughness Specular Top View Novel view render Reference Ours N=1 Ours N=5 Ours N=20

Plausible Accurate

25

slide-27
SLIDE 27

Comparison with class inverse rendering

26

Ours Classic inverse rendering Reference

Classic inverse rendering ours

slide-28
SLIDE 28

Comparison with class inverse rendering

27

Ours Classic inverse rendering Reference

Classic inverse rendering ours

slide-29
SLIDE 29

Estimated SVBRDF with 20 input photos Novel view rendering

Support arbitrary resolution! High resolution results

28

slide-30
SLIDE 30

Estimated SVBRDF with 20 input photos Novel view rendering

Support arbitrary resolution! High resolution results

29

slide-31
SLIDE 31

High resolution results

Estimated SVBRDF with 20 input photos Novel view rendering

Support arbitrary resolution!

30

slide-32
SLIDE 32

Real captured results

Leather, 1k resolution, 2 inputs

Normal Diffuse Roughness Specular Novel view GT Render

31

slide-33
SLIDE 33

Real captured results

Wood, 1k resolution, 10 inputs

Normal Diffuse Roughness Specular Novel view GT Render

32

slide-34
SLIDE 34

Real captured results

Metal Plate, 1k resolution, 30 inputs

Normal Diffuse Roughness Specular Novel view GT Render

33

slide-35
SLIDE 35

Robustness

LDR HDR Reference Reference Top view init. Side view init.

34

slide-36
SLIDE 36

Robustness

Reference

35

slide-37
SLIDE 37

Robustness

10% 5% 0% Reference

36

slide-38
SLIDE 38

Conclusion

  • Unified deep inverse rendering framework for estimating SVBRDF from arbitrary

number of input photographs.

  • Learned latent space + optimization in latent space

37

slide-39
SLIDE 39

38

Thanks