Bayesian View Synthesis and Image-Based Rendering Principles 1 1 - - PowerPoint PPT Presentation

▶
bayesian view synthesis and image based rendering
SMART_READER_LITE
LIVE PREVIEW

Bayesian View Synthesis and Image-Based Rendering Principles 1 1 - - PowerPoint PPT Presentation

Sep 28, 2022 333 likes •1.15k views

Bayesian View Synthesis and Image-Based Rendering Principles 1 1 2 Sergi Pujades, Frdric Devernay, Bastian Goldluecke CVPR 2014 1 2 University of Konstanz Image Based Rendering Input views Input views v 2 v 1 INRIA Grenoble,

slide-1
SLIDE 1

University of Konstanz

Bayesian View Synthesis and Image-Based Rendering Principles

Sergi Pujades, Frédéric Devernay, Bastian Goldluecke CVPR 2014

1 1 2 1 2

slide-2
SLIDE 2

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

2

Input views Input views

v1 v2

slide-3
SLIDE 3

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

u

?

2

Input views Target view Input views

v1 v2

slide-4
SLIDE 4

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

u

?

2

Input views Target view Scene Geometry Input views

v1 v2

slide-5
SLIDE 5

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

u

2

Input views Target view Scene Geometry Input views

v1 v2

x

slide-6
SLIDE 6

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

u

2

Input views Target view Scene Geometry Input views

v1 v2

x

slide-7
SLIDE 7

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Image Based Rendering

u

2

Input views Target view Scene Geometry Input views

v1 v2

x

slide-8
SLIDE 8

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

IBR Continum

3

Scene Geometry less more

Light field Lumigraph Texture-mapped models

slide-9
SLIDE 9

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

8 Desirable Properties

  • Use of geometric proxies
  • Unstructured input
  • Minimal angular deviation
  • Epipole consistency
  • Equivalent ray consistency
  • Resolution sensitivity
  • Continuity
  • Real-time

4

Unstructured Lumigraph Rendering

  • C. Buehler et al. - SIGGRAPH 2001
slide-10
SLIDE 10

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

8 Desirable Properties

  • Use of geometric proxies
  • Unstructured input
  • Minimal angular deviation
  • Epipole consistency
  • Equivalent ray consistency
  • Resolution sensitivity
  • Continuity
  • Real-time

4

Unstructured Lumigraph Rendering

  • C. Buehler et al. - SIGGRAPH 2001
slide-11
SLIDE 11

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Minimal angular deviation

5

Input views Input views

v1 v2

slide-12
SLIDE 12

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Minimal angular deviation

5

Input views Target view Input views

v1 v2 u

slide-13
SLIDE 13

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Minimal angular deviation

5

Input views Target view Input views

v1 v2 u

slide-14
SLIDE 14

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Minimal angular deviation

5

Input views Target view Input views

v1 v2 u

slide-15
SLIDE 15

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Resolution Sensitivity

6

Input views Input views

v1

v2

slide-16
SLIDE 16

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Resolution Sensitivity

6

Input views Input views

u

v1

v2

Target view

slide-17
SLIDE 17

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Resolution Sensitivity

6

Input views Input views

u

v1

v2

Target view

slide-18
SLIDE 18

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Resolution Sensitivity

6

Input views Input views

u

v1

v2

Target view

slide-19
SLIDE 19

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art limitations

7

For both properties:

  • Minimal angular deviation
  • Resolution sensitivity

No formal deduction of heuristics Manual parameter tuning depending on the scene

slide-20
SLIDE 20

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

New properties proposed

8

  • Use of geometric proxies
  • Unstructured input
  • Minimal angular deviation
  • Epipole consistency
  • Equivalent ray consistency
  • Resolution sensitivity
  • Formal deduction of heuristics
  • Physics-based parameters
  • Continuity
  • Real-time
slide-21
SLIDE 21

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

New properties proposed

8

  • Use of geometric proxies
  • Unstructured input
  • Minimal angular deviation
  • Epipole consistency
  • Equivalent ray consistency
  • Resolution sensitivity
  • Formal deduction of heuristics
  • Physics-based parameters
  • Continuity
  • Real-time
slide-22
SLIDE 22

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

slide-23
SLIDE 23

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

slide-24
SLIDE 24

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

slide-25
SLIDE 25

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

Wanner and Goldluecke

ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields

slide-26
SLIDE 26

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

Wanner and Goldluecke

ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields

slide-27
SLIDE 27

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

Wanner and Goldluecke

ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields

Our method

CVPR 2014

slide-28
SLIDE 28

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

State of the art

9

Method

Formal deduction Physics-Based Parameters Resolution sensitivity Minimal angular deviation Buehler et al.

SIGGRAPH 2001 Unstructured Lumigraph Rendering

Keita Takahashi

ECCV 2010 Theory of Optimal View Interpolation with Depth Inaccuracy

Wanner and Goldluecke

ECCV 2012 Spatial and Angular Variational Super-resolution of 4D Light Fields

Our method

CVPR 2014

slide-29
SLIDE 29

CVPR 2014 - 27 June 2014 INRIA Grenoble, France 10

v1 u

?

Bayesian Approach: Inverse Problem

Input view Target view

slide-30
SLIDE 30

CVPR 2014 - 27 June 2014 INRIA Grenoble, France 10

v1 u

? ?

Bayesian Approach: Inverse Problem

Input view Target view

slide-31
SLIDE 31

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Input view Target view

x

slide-32
SLIDE 32

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Input view Target view Scene Geometry

x

slide-33
SLIDE 33

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Input view Target view Scene Geometry

x ˜ τi

slide-34
SLIDE 34

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Input view Target view Scene Geometry

x ˜ τi ˜ vi(x) = (u

Perfect image

˜ τi

˜ τi)(x)

slide-35
SLIDE 35

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Generative Model Perfect image formation description Input view Target view Scene Geometry

x ˜ τi ˜ vi(x) = (u

Perfect image

˜ τi

˜ τi)(x)

slide-36
SLIDE 36

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

11

Generative Model Perfect image formation description

assuming Lambertian model

Input view Target view Scene Geometry

x ˜ τi ˜ vi(x) = (u

Perfect image

˜ τi

˜ τi)(x)

slide-37
SLIDE 37

˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Perfect image

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-38
SLIDE 38

vi(x) = ˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Perfect image Observed image

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-39
SLIDE 39

vi(x) = ˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Perfect image Observed image

) = ˜ vi(x) +

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-40
SLIDE 40

vi(x) = ˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Perfect image Observed image Sensor noise

) = ˜ vi(x) + ) + es(x)

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-41
SLIDE 41

vi(x) = ˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Gaussian distribution Perfect image Observed image Sensor noise

) = ˜ vi(x) + ) + es(x)

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-42
SLIDE 42

vi(x) = ˜ vi(x) = (u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

12

Least squares minimisation problem

Gaussian distribution Perfect image Observed image Sensor noise

) = ˜ vi(x) + ) + es(x)

Input view Target view Scene Geometry

x ˜ τi

˜ τi

˜ τi)(x)

slide-43
SLIDE 43

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

13

Physics based Resolution sensibility Minimal angular deviation

vi(x) = ˜ vi(x) + es(x)

Spatial and Angular Variational Super-resolution of 4D Light Fields

  • S. Wanner and B. Goldluecke ECCV 2012

Input view Target view Scene Geometry

˜ τi x

slide-44
SLIDE 44

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Inverse Problem

v1 u

13

Physics based Resolution sensibility Minimal angular deviation

WHY?

vi(x) = ˜ vi(x) + es(x)

Spatial and Angular Variational Super-resolution of 4D Light Fields

  • S. Wanner and B. Goldluecke ECCV 2012

Input view Target view Scene Geometry

˜ τi x

slide-45
SLIDE 45

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

14

Input view Target view Scene Geometry

x

slide-46
SLIDE 46

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

14

Input view Target view

x

slide-47
SLIDE 47

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Input view Target view

x

slide-48
SLIDE 48

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Input view Target view

x ˜ τi

slide-49
SLIDE 49

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Input view Target view

x

slide-50
SLIDE 50

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Observed image

vi(x) =

Input view Target view

x

slide-51
SLIDE 51

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Perfect image Observed image

vi(x) = ) = ˜ vi(x) +

Input view Target view

x

slide-52
SLIDE 52

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Perfect image Observed image Sensor noise

vi(x) = ) = ˜ vi(x) + ) + es(x) +

Input view Target view

x

slide-53
SLIDE 53

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Perfect image Observed image Sensor noise Geometric Noise

vi(x) = ) = ˜ vi(x) + ) + es(x) + ) + eg(x)

Input view Target view

x

slide-54
SLIDE 54

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian Approach: Proposed Method

Depth distribution

14

Perfect image Observed image Sensor noise Geometric Noise

vi(x) = ) = ˜ vi(x) + ) + es(x) + ) + eg(x)

details in the paper Input view Target view

x

slide-55
SLIDE 55

v1 u

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Support of the projected Gaussian depends on the angular deviation

Bayesian Approach: Proposed Method

Depth distribution

14

Perfect image Observed image Sensor noise Geometric Noise

vi(x) = ) = ˜ vi(x) + ) + es(x) + ) + eg(x)

details in the paper Input view Target view

x

slide-56
SLIDE 56

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

15

Minimal angular deviation Physics based Resolution sensitivity

slide-57
SLIDE 57

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

15

Minimal angular deviation Physics based Resolution sensitivity

slide-58
SLIDE 58

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

15

Minimal angular deviation Physics based Resolution sensitivity

slide-59
SLIDE 59

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

correspondence confidence

15

Minimal angular deviation Physics based Resolution sensitivity

Weighting factor depends on

slide-60
SLIDE 60

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

correspondence confidence

15

Minimal angular deviation Physics based Resolution sensitivity

Weighting factor depends on

slide-61
SLIDE 61

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Deducted weights

  • σ2

zi

✓∂ (u τi) ∂zi ◆2

  • −1

|det Dτi|−1

correspondence confidence

15

Minimal angular deviation Physics based Resolution sensitivity

image content (color gradient along epipolar line) Weighting factor depends on

slide-62
SLIDE 62

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Experiments

Implementation of a simplified camera configuration 4D Light Field

16

Stanford multi-camera array

slide-63
SLIDE 63

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

The (New) Stanford Light Field Archive

Tarot Truck

17

slide-64
SLIDE 64

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

HCI Lightfield Dataset

18

Maria Still Life

slide-65
SLIDE 65

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

?

19

slide-66
SLIDE 66

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Previous method Ground truth

20 Wanner and Goldluecke ECCV 2012

Results

slide-67
SLIDE 67

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Previous method Ground truth

20 Wanner and Goldluecke ECCV 2012

Results

slide-68
SLIDE 68

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Previous method Proposed method Ground truth

20 Wanner and Goldluecke ECCV 2012

Results

slide-69
SLIDE 69

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement

21

slide-70
SLIDE 70

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement

21

slide-71
SLIDE 71

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement

21

slide-72
SLIDE 72

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

22

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement
slide-73
SLIDE 73

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

23

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement
slide-74
SLIDE 74

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

What is happening?

24

Better selection of the contributing views based on :

  • View distance
  • color gradient aligned with view displacement
slide-75
SLIDE 75

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Previous method Proposed method

25

Ground truth

Results

Wanner and Goldluecke ECCV 2012

slide-76
SLIDE 76

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Status and future work

26

  • Use of geometric proxies
  • Unstructured input
  • Epipole consistency
  • Equivalent ray consistency
  • Minimal angular deviation
  • Resolution sensitivity
  • Formal deduction
  • Physics-based parameters
  • Continuity
  • Real-time
slide-77
SLIDE 77

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Status and future work

26

  • Use of geometric proxies
  • Unstructured input
  • Epipole consistency
  • Equivalent ray consistency
  • Minimal angular deviation
  • Resolution sensitivity
  • Formal deduction
  • Physics-based parameters
  • Continuity
  • Real-time
slide-78
SLIDE 78

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

New generative model for IBR Unify current knowledge Improve results Code available as part of cocolib library http://sourceforge.net/projects/cocolib/

Conclusion

27

slide-79
SLIDE 79

CVPR 2014 - 27 June 2014 INRIA Grenoble, France

Bayesian formulation: Use physically-sound parameters! Uncertainty is helpful: Don’t throw away your covariance matrices!

Take home messages

28

slide-80
SLIDE 80

University of Konstanz

Bayesian View Synthesis and Image-Based Rendering Principles

Sergi Pujades, Frédéric Devernay, Bastian Goldluecke CVPR 2014

1 1 2 1 2