3D Photography: Stereo Matching Kevin Kser, Marc Pollefeys Spring - - PowerPoint PPT Presentation

▶
3d photography
SMART_READER_LITE
LIVE PREVIEW

3D Photography: Stereo Matching Kevin Kser, Marc Pollefeys Spring - - PowerPoint PPT Presentation

Dec 03, 2022 388 likes •1.63k views

3D Photography: Stereo Matching Kevin Kser, Marc Pollefeys Spring 2012 http://cvg.ethz.ch/teaching/2012spring/3dphoto/ Stereo & Multi-View Stereo Tsukuba dataset http://cat.middlebury.edu/stereo/ Stereo Standard stereo geometry

slide-1
SLIDE 1

3D Photography: Stereo Matching

Kevin Köser, Marc Pollefeys Spring 2012

http://cvg.ethz.ch/teaching/2012spring/3dphoto/

slide-2
SLIDE 2

Stereo & Multi-View Stereo

http://cat.middlebury.edu/stereo/ Tsukuba dataset

slide-3
SLIDE 3

Stereo

  • Standard stereo geometry
  • Stereo matching
  • Correlation
  • Optimization (DP, GC)
  • General camera configuration
  • Rectifications
  • Plane-sweep
  • Multi-view stereo
slide-4
SLIDE 4

Stereo

slide-5
SLIDE 5

Occlusions

(Slide from Pascal Fua)

slide-6
SLIDE 6

Exploiting scene constraints

slide-7
SLIDE 7

Ordering constraint

1 2 3 4,5 6 1 2,3 4 5 6 2 1 3 4,5 6 1 2,3 4 5 6

surface slice surface as a path

  • cclusion right
  • cclusion left
slide-8
SLIDE 8

Uniqueness constraint

  • In an image pair each pixel has at most
  • ne corresponding pixel
  • In general one corresponding pixel
  • In case of occlusion there is none
slide-9
SLIDE 9

Disparity constraint

surface slice surface as a path

bounding box

use reconstructed features to determine bounding box

constant disparity surfaces

slide-10
SLIDE 10

Stereo matching

Optimal path (dynamic programming ) Similarity measure (SSD or NCC) Constraints

  • epipolar
  • ordering
  • uniqueness
  • disparity limit

Trade-off

  • Matching cost (data)
  • Discontinuities (prior)

Consider all paths that satisfy the constraints pick best using dynamic programming

slide-11
SLIDE 11

Hierarchical stereo matching

Downsampling

(Gaussian pyramid)

Disparity propagation

Allows faster computation Deals with large disparity ranges

slide-12
SLIDE 12

Disparity map

image I(x,y) image I´(x´,y´) Disparity map D(x,y)

(x´,y´)=(x+D(x,y),y)

slide-13
SLIDE 13

Example: reconstruct image from neighboring images

slide-14
SLIDE 14
slide-15
SLIDE 15

Energy minimization

(Slide from Pascal Fua)

slide-16
SLIDE 16

Graph Cut

(Slide from Pascal Fua)

(general formulation requires multi-way cut!)

slide-17
SLIDE 17

(Boykov et al ICCV‘99) (Roy and Cox ICCV‘98)

Simplified graph cut

slide-18
SLIDE 18
slide-19
SLIDE 19

Stereo matching with general camera configuration

slide-20
SLIDE 20

Image pair rectification

slide-21
SLIDE 21

Planar rectification

Bring two views to standard stereo setup (moves epipole to ) (not possible when in/close to image)

~ image size

(calibrated)

Distortion minimization

(uncalibrated)

slide-22
SLIDE 22
slide-23
SLIDE 23

Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed

  • riginal image

rectified image

Polar rectification

(Pollefeys et al. ICCV’99) Works for all relative motions Guarantees minimal image size

slide-24
SLIDE 24

polar rectification planar rectification

  • riginal

image pair

slide-25
SLIDE 25

Example: Béguinage of Leuven

Does not work with standard Homography-based approaches

slide-26
SLIDE 26

Example: Béguinage of Leuven

slide-27
SLIDE 27

Stereo camera configurations

(Slide from Pascal Fua)

slide-28
SLIDE 28

Multi-camera configurations

Okutami and Kanade

(illustration from Pascal Fua)

slide-29
SLIDE 29
  • Multi-baseline, multi-resolution
  • At each depth, baseline and resolution

selected proportional to that depth

  • Allows to keep depth accuracy constant!

Variable Baseline/Resolution Stereo

(Gallup et al., CVPR08)

slide-30
SLIDE 30

Variable Baseline/Resolution Stereo: comparison

slide-31
SLIDE 31

Multi-view depth fusion

  • Compute depth for every

pixel of reference image

  • Triangulation
  • Use multiple views
  • Up- and down sequence
  • Use Kalman filter

(Koch, Pollefeys and Van Gool. ECCV‘98)

Allows to compute robust texture

slide-32
SLIDE 32

Plane-sweep multi-view matching

  • Simple algorithm for multiple cameras
  • no rectification necessary
  • doesn’t deal with occlusions

Collins’96; Roy and Cox’98 (GC); Yang et al.’02/’03 (GPU)

slide-33
SLIDE 33

Space Carving

slide-34
SLIDE 34

3D Reconstruction from Calibrated Images

Scene Volume V Input Images (Calibrated)

Goal: Determine transparency, radiance of points in V

slide-35
SLIDE 35

Discrete Formulation: Voxel Coloring

Discretized Scene Volume Input Images (Calibrated)

Goal: Assign RGBA values to voxels in V

photo-consistent with images

slide-36
SLIDE 36

Complexity and Computability

Discretized Scene Volume N voxels C colors

3

All Scenes (CN3) Photo-Consistent Scenes True Scene

slide-37
SLIDE 37

Issues

Theoretical Questions

Identify class of all photo-consistent scenes

Practical Questions

How do we compute photo-consistent models?

slide-38
SLIDE 38
  • 1. C=2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-39
SLIDE 39
  • 1. C=2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-40
SLIDE 40
  • 1. Choose voxel
  • 2. Project and correlate
  • 3. Color if consistent

Voxel Coloring Approach

Visibility Problem: in which images is each voxel visible?

slide-41
SLIDE 41

The Global Visibility Problem

Inverse Visibility

known images

Unknown Scene

Which points are visible in which images?

Known Scene

Forward Visibility

known scene

slide-42
SLIDE 42

Layers

Depth Ordering: visit occluders first!

Scene Traversal

Condition: depth order is view-independent

slide-43
SLIDE 43

Compatible Camera Configurations

Depth-Order Constraint

Scene outside convex hull of camera centers

Outward-Looking

cameras inside scene

Inward-Looking

cameras above scene

slide-44
SLIDE 44

Calibrated Image Acquisition

Calibrated Turntable

360° rotation (21 images)

Selected Dinosaur Images Selected Flower Images

slide-45
SLIDE 45

Voxel Coloring Results (Video)

Dinosaur Reconstruction

72 K voxels colored 7.6 M voxels tested 7 min. to compute

  • n a 250MHz SGI

Flower Reconstruction

70 K voxels colored 7.6 M voxels tested 7 min. to compute

  • n a 250MHz SGI
slide-46
SLIDE 46

Limitations of Depth Ordering

A view-independent depth order may not exist

p q

Need more powerful general-case algorithms

Unconstrained camera positions Unconstrained scene geometry/topology

slide-47
SLIDE 47
  • 1. C=2 (silhouettes)

Volume intersection [Martin 81, Szeliski 93]

  • 2. C unconstrained, viewpoint

constraints

Voxel coloring algorithm [Seitz & Dyer 97]

  • 3. General Case

Space carving [Kutulakos & Seitz 98]

Voxel Coloring Solutions

slide-48
SLIDE 48

Space Carving Algorithm

Space Carving Algorithm

Image 1 Image N

…... Initialize to a volume V containing the true scene Repeat until convergence Choose a voxel on the current surface Carve if not photo-consistent Project to visible input images

slide-49
SLIDE 49

Convergence

Consistency Property

The resulting shape is photo-consistent

all inconsistent points are removed

Convergence Property

Carving converges to a non-empty shape

a point on the true scene is never removed

V’ V

p

slide-50
SLIDE 50

What is Computable?

The Photo Hull is the UNION of all photo-consistent scenes in V

  • It is a photo-consistent scene reconstruction
  • Tightest possible bound on the true scene
  • Computable via provable Space Carving Algorithm

True Scene V Photo Hull V

slide-51
SLIDE 51

Space Carving Algorithm

The Basic Algorithm is Unwieldy

  • Complex update procedure

Alternative: Multi-Pass Plane Sweep

  • Efficient, can use texture-mapping

hardware

  • Converges quickly in practice
  • Easy to implement
slide-52
SLIDE 52

Space Carving Algorithm

Step 1: Initialize V to volume containing true scene Step 2: For every voxel on surface of V

test photo-consistency of voxel if voxel is inconsistent, carve it

Step 3: Repeat Step 2 until all voxels consistent Convergence:

Always converges to a photo-consistent model (when all assumptions are met) Good results on difficult real-world scenes

slide-53
SLIDE 53

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence

True Scene Reconstruction

slide-54
SLIDE 54

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-55
SLIDE 55

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-56
SLIDE 56

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-57
SLIDE 57

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-58
SLIDE 58

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-59
SLIDE 59

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-60
SLIDE 60

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-61
SLIDE 61

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-62
SLIDE 62

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-63
SLIDE 63

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-64
SLIDE 64

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-65
SLIDE 65

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-66
SLIDE 66

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-67
SLIDE 67

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-68
SLIDE 68

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-69
SLIDE 69

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-70
SLIDE 70

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-71
SLIDE 71

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-72
SLIDE 72

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-73
SLIDE 73

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-74
SLIDE 74

Multi-Pass Plane Sweep

  • Sweep plane in each of 6 principle directions
  • Consider cameras on only one side of plane
  • Repeat until convergence
slide-75
SLIDE 75

Space Carving Results: African Violet

Input Image (1 of 45) Reconstruction Reconstruction Reconstruction

slide-76
SLIDE 76

Space Carving Results: Hand

Input Image (1 of 100) Views of Reconstruction

slide-77
SLIDE 77

Other Features

Coarse-to-fine Reconstruction

  • Represent scene as octree
  • Reconstruct low-res model first, then refine

Hardware-Acceleration

  • Use texture-mapping to compute voxel

projections

  • Process voxels an entire plane at a time

Limitations

  • Need to acquire calibrated images
  • Restriction to simple radiance models
  • Bias toward maximal (fat) reconstructions
  • Transparency not supported
slide-78
SLIDE 78
slide-79
SLIDE 79

voxel occluded

Probalis listic ic Space Carving ng

Broadhurst et al. ICCV’01

slide-80
SLIDE 80

Bayesian

The Master's Lodge Image Sequence

slide-81
SLIDE 81

I

Light Intensity Object Color

N

Normal vector

L

Lighting vector

V

View Vector

R

Reflection vector

color of the light Diffuse color Saturation point 1 1 1

Reflected Light in RGB color space Dielectric Materials (such as plastic and glass)

C 

Space-carving for specular surfaces

(Yang, Pollefeys & Welch 2003)

Extended photoconsistency:

slide-82
SLIDE 82

Experiment

slide-83
SLIDE 83

Animated Views

Our result

slide-84
SLIDE 84

Other Approaches

Level-Set Methods [Faugeras & Keriven 1998]

Evolve implicit function by solving PDE’s

More recent level-set/PDE approaches by Pons et al., CVPR05, Gargallo et al. ICCV07, Kalin and Kremers ECCV08, …

slide-85
SLIDE 85

Volumetric Graph cuts

(x)

  • 1. Outer surface
  • 2. Inner surface (at

constant offset)

  • 3. Discretize

middle volume

  • 4. Assign

photoconsistency cost to voxels Slides from [Vogiatzis et al. CVPR2005]

slide-86
SLIDE 86

Volumetric Graph cuts

Source Sink Slides from [Vogiatzis et al. CVPR2005]

slide-87
SLIDE 87

Volumetric Graph cuts

Source Sink Cost of a cut   (x) dS S

S

cut  3D Surface S

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

slide-88
SLIDE 88

Volumetric Graph cuts

Source Sink Minimum cut  Minimal 3D Surface under photo-consistency metric

[Boykov and Kolmogorov ICCV 2001]

Slides from [Vogiatzis et al. CVPR2005]

slide-89
SLIDE 89

Photo-consistency

  • Occlusion
  • 1. Get nearest point
  • n outer surface
  • 2. Use outer

surface for

  • cclusions
  • 2. Discard occluded

views Slides from [Vogiatzis et al. CVPR2005]

slide-90
SLIDE 90

Photo-consistency

  • Occlusion

Self occlusion Slides from [Vogiatzis et al. CVPR2005]

slide-91
SLIDE 91

Photo-consistency

  • Occlusion

Self occlusion Slides from [Vogiatzis et al. CVPR2005]

slide-92
SLIDE 92

Photo-consistency

  • Occlusion

N threshold on angle between normal and viewing direction threshold= ~60 Slides from [Vogiatzis et al. CVPR2005]

slide-93
SLIDE 93

Photo-consistency

  • Score

Normalised cross correlation Use all remaining cameras pair wise Average all NCC scores Slides from [Vogiatzis et al. CVPR2005]

slide-94
SLIDE 94

Photo-consistency

  • Score

Average NCC = C Voxel score  = 1 - exp( -tan2[(C-1)/4] / 2 )

0    1  = 0.05 in all experiments

Slides from [Vogiatzis et al. CVPR2005]

slide-95
SLIDE 95

Example

Slides from [Vogiatzis et al. CVPR2005]

slide-96
SLIDE 96

Example - Visual Hull

Slides from [Vogiatzis et al. CVPR2005]

slide-97
SLIDE 97

Example - Slice

Slides from [Vogiatzis et al. CVPR2005]

slide-98
SLIDE 98

Example - Slice with graphcut

Slides from [Vogiatzis et al. CVPR2005]

slide-99
SLIDE 99

Example – 3D

Slides from [Vogiatzis et al. CVPR2005]

slide-100
SLIDE 100

[Vogiatzis et al. PAMI2007]

slide-101
SLIDE 101

Protrusion problem

  • ‘Balooning’ force
  • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d

  • images. PAMI, 15(11):1131–1147, November 1993.

Slides from [Vogiatzis et al. CVPR2005]

slide-102
SLIDE 102

Protrusion problem

  • ‘Balooning’ force
  • favouring bigger volumes that fill the visual hull

L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131– 1147, November 1993.



(x) dS -   dV S V Slides from [Vogiatzis et al. CVPR2005]

slide-103
SLIDE 103

Protrusion problem

Slides from [Vogiatzis et al. CVPR2005]

slide-104
SLIDE 104

Protrusion problem

Slides from [Vogiatzis et al. CVPR2005]

slide-105
SLIDE 105

wij SOURCE

wb wb

Graph

h j i

wb = h3 wij = 4/3h2 * (i+j)/2

[Boykov and Kolmogorov ICCV 2001] Slides from [Vogiatzis et al. CVPR2005]

slide-106
SLIDE 106

116 Address Memory and Computational Overhead (Sinha et. al. 2007)

– Compute Photo-consistency only where it is needed – Detect Interior Pockets using Visibility

Graph-cut on Dual of Adaptive Tetrahedral Mesh

slide-107
SLIDE 107

117

Adaptive Mesh Refinement

Cell (Tetrahedron) Face (triangle) Unknown surface

slide-108
SLIDE 108

118

Adaptive Mesh Refinement

Detect crossing faces by testing photo-consistency

  • f points sampled on the faces
slide-109
SLIDE 109

119

Adaptive Mesh Refinement

If none of the faces of a cell are crossing faces, that cell cannot any surface element.

slide-110
SLIDE 110

120

Adaptive Mesh Refinement

  • Prune the Inactive cells
  • Refine the Active cells
slide-111
SLIDE 111

121

Mesh with Photo-consistency

Final Mesh shown with Photo-consistency

slide-112
SLIDE 112

122

Detect Interior

Use Visibility of the Photo-consistent Patches Also proposed by Hernandez et. al. 2007, Labatut et.

  • al. 2007
slide-113
SLIDE 113

123

First Graph-Cut

  • Compute the Min-cut

Source Sink Solution re-projected into the original silhouette

slide-114
SLIDE 114

124

Silhouette Constraints

Every such ray must meet the real surface at least once

slide-115
SLIDE 115

125

Silhouette Constraints

  • Test photo-consistency

along the ray

  • Robustly find an interior cell

Photo-consistency Surface

slide-116
SLIDE 116

126

Source Sink

Final Graph-Cut

slide-117
SLIDE 117

127

Final Graph-Cut

Before After

slide-118
SLIDE 118

128

Results

After graph-cut

  • ptimization

After local refinement

36 images 2000 x 3000 pixels

slide-119
SLIDE 119

20 images Running Time: Graph Construction : 25 mins Graph-cut : 5 mins Local Refinement : 20 mins 20 images 640 x 480 pixels

slide-120
SLIDE 120

130 36 images 36 images

slide-121
SLIDE 121

131

48 images 47 images 24 images

slide-122
SLIDE 122
  • Optimize photo-consistency while exactly

enforcing silhouette constraint

Photo-consistency + silhouettes

[Sinha and Pollefeys ICCV05]

slide-123
SLIDE 123

Multi-view stereo through convex optimization

(Kolev et al. ECCV2008) includes relaxed silhouette constraint

slide-124
SLIDE 124

Interesting comparison of multi- view stereo approaches

http://vision.middlebury.edu/mview/