CS 4495 Computer Vision Stereo: Disparity and Matching Aaron Bobick - - PowerPoint PPT Presentation

▶
cs 4495 computer vision stereo disparity and matching
SMART_READER_LITE
LIVE PREVIEW

CS 4495 Computer Vision Stereo: Disparity and Matching Aaron Bobick - - PowerPoint PPT Presentation

Mar 04, 2024 348 likes •1.11k views

Stereo 1: Disparity and Matching CS 4495 Computer Vision A. Bobick CS 4495 Computer Vision Stereo: Disparity and Matching Aaron Bobick School of Interactive Computing Stereo 1: Disparity and Matching CS 4495 Computer Vision A. Bobick

slide-1
SLIDE 1

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Aaron Bobick School of Interactive Computing

CS 4495 Computer Vision Stereo: Disparity and Matching

slide-2
SLIDE 2

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Administrivia

  • PS2 will be out tomrrow. Due Sunday Sept 22nd ,

11:55pm

  • There is *no* grace period. We can either:

a) leave submission open and have 50% penalty b) or close it, require email and have 50% penalty You choose…

  • Read; FP chapter 7
slide-3
SLIDE 3

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo: A Special case of Multiple views

Hartley and Zisserman Lowe

Multi-view geometry, matching, invariant features, stereo vision

slide-4
SLIDE 4

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Why multiple views?

  • Structure and depth are inherently ambiguous

from single views.

Images from Lana Lazebnik

slide-5
SLIDE 5

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Why multiple views?

  • Structure and depth are inherently ambiguous

from single views.

Optical center

P1 P2 P1’=P2’

slide-6
SLIDE 6

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

How do we see depth?

  • What cues help us to perceive 3d shape and

depth?

  • What about one eye first?
slide-7
SLIDE 7

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Perspective effects

  • S. Seitz
slide-8
SLIDE 8

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Shading

  • K. Grauman
slide-9
SLIDE 9

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Texture

[From A.M. Loh. The recovery of 3-D structure using visual texture patterns. PhD thesis]

slide-10
SLIDE 10

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Focus/defocus

[figs from H. Jin and P. Favaro, 2002]

Images from same point

  • f view,

different camera parameters 3d shape / depth estimates

slide-11
SLIDE 11

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Motion

Figures from L. Zhang http://www.brainconnection.com/teasers/?main=illusion/motion-shape

slide-12
SLIDE 12

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Estimating scene shape from one eye

  • “Shape from X”: Shading, Texture, Focus, Motion…
  • Very popular circa 1980
slide-13
SLIDE 13

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

But we (and lots of creatures) have two eyes!

  • Stereo:
  • shape from “motion” between two views
  • infer 3d shape of scene from two (multiple) images

from different viewpoints

scene point

  • ptical center

image plane Main idea:

slide-14
SLIDE 14

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo photography and stereo viewers

Take two pictures of the same subject from two slightly different viewpoints and display so that each eye sees only one

  • f the images.

Invented by Sir Charles Wheatstone 1838

slide-15
SLIDE 15

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

People fascinated by 3D

slide-16
SLIDE 16

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

http://www.johnsonshawmuseum.org

slide-17
SLIDE 17

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick Public Library, Stereoscopic Looking Room, Chicago, by Phillips, 1923

slide-18
SLIDE 18

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Teesta suspension bridge-Darjeeling, India

slide-19
SLIDE 19

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick Mark Twain at Pool Table", no date, UCR Museum of Photography

slide-20
SLIDE 20

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo photography and stereo viewers

When I grew up…

slide-21
SLIDE 21

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo photography and stereo viewers

When I grew up…

You guys..

slide-22
SLIDE 22

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

If you like to cross (wall-eye) your eyes…

slide-23
SLIDE 23

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Single Image Stereo: Autostereogram

Single image stereogram, by Niklas Een

  • S. Seitz
slide-24
SLIDE 24

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

The Basic Idea: Two slightly different images

http://www.well.com/~jimg/stereo/stereo_list.html

slide-25
SLIDE 25

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

So how do humans do it?

slide-26
SLIDE 26

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Random dot stereograms

  • Julesz 1960: Do we identify local brightness

patterns before fusion (monocular process) or after (binocular)?

  • To test: pair of synthetic images obtained by

randomly spraying black dots on white objects

slide-27
SLIDE 27

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Random dot stereograms

Forsyth & Ponce

slide-28
SLIDE 28

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Random dot stereograms

slide-29
SLIDE 29

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Random dot stereograms

  • When viewed monocularly, they appear random;

when viewed stereoscopically, see 3d structure.

  • Conclusion: human binocular fusion not based

upon matching large scale structures or any processing of the individual images

  • Imaginary “cyclopean retina” that combines the

left and right image stimuli as a single unit. Later discovered the cells in the brain’s visual cortex that create this “percept”

slide-30
SLIDE 30

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Estimating depth with stereo

  • Stereo: shape from “motion” between two views
  • We’ll need to consider:
  • Info on camera pose (“calibration”)
  • Image point correspondences

scene point

  • ptical

center image plane

slide-31
SLIDE 31

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Estimating depth with stereo

  • Stereo: shape from “motion” between two views
  • We’ll need to consider:
  • scene point
  • ptical

center image plane Info on camera pose (“calibration”) Image point correspondences

slide-32
SLIDE 32

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Geometry for a simple stereo system

  • First, assuming parallel
  • ptical axes, known

camera parameters (i.e., calibrated cameras)

  • Figure is looking down
  • n the cameras and

image planes

  • Baseline B,

focal length f

  • Point P is distance Z in

camera coordinate systems

B f P Z COPL COPR

Optic Axis Optic Axis

slide-33
SLIDE 33

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Geometry for a simple stereo system

  • Point P projects into

left and right images.

  • Distance is positive in

left image, and negative in right

B f

xl xr

Z pl pr P COPL COPR

slide-34
SLIDE 34

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Geometry for a simple stereo system

  • What is the expression

for Z?

  • Similar triangles

(pl, P, pr) and (CL,P, Cr):

COPL B f

xl xr

Z pl pr COPR P

l r

B x x Z Z f B − + − =

l r

B Z f x x = −

Disparity

slide-35
SLIDE 35

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Depth from disparity

image I(x,y) image I´(x´,y´) Disparity map D(x,y) (x´,y´)=(x+D(x,y), y) So if we could find the corresponding points in two images, we could estimate relative depth…

slide-36
SLIDE 36

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

General case, with calibrated cameras

  • The two cameras need not have parallel optical axes and

image planes. Vs.

slide-37
SLIDE 37

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

  • Given p in left image, where can corresponding point

p’ be?

Stereo correspondence constraints

slide-38
SLIDE 38

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo correspondence constraints

  • In perspective projection, lines project into lines. So

the line containing the center of projection and the point p in the left image must project to a line in the right image.

slide-39
SLIDE 39

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Geometry of two views constrains where the corresponding pixel for some image point in the first view must occur in the second view.

  • It must be on the line carved out by a plane connecting

the world point and optical centers.

Epipolar constraint

slide-40
SLIDE 40

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Epipolar constraint

slide-41
SLIDE 41

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Epipolar geometry: terms

  • Baseline: line joining the

camera centers

  • Epipole: point of intersection
  • f baseline with image plane
  • Epipolar plane: plane

containing baseline and world point

  • Epipolar line: intersection of

epipolar plane with the image plane

Why is the epipolar constraint useful?

  • All epipolar lines intersect at the epipole
  • An epipolar plane intersects the left and right image

planes in epipolar lines

slide-42
SLIDE 42

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

This is useful because it reduces the correspondence problem to a 1D search along an epipolar line.

Image from Andrew Zisserman

Epipolar constraint

slide-43
SLIDE 43

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Example

slide-44
SLIDE 44

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

What do the epipolar lines look like?

Ol Or Ol Or

1. 2.

slide-45
SLIDE 45

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Figure from Hartley & Zisserman

Example: converging cameras

slide-46
SLIDE 46

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Figure from Hartley & Zisserman

Where are the epipoles?

Example: parallel cameras

slide-47
SLIDE 47

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

For now assume parallel image planes…

  • Assume parallel image planes…
  • Assume same focal lengths…
  • Assume epipolar lines are horizontal…
  • Assume epipolar lines are at the same y location

in the image…

  • That’s a lot of assuming, but it allows us to move

to the correspondence problem – which you will be solving!

slide-48
SLIDE 48

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correspondence problem

Multiple match hypotheses satisfy epipolar constraint, but which is correct?

Figure from Gee & Cipolla 1999

slide-49
SLIDE 49

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correspondence problem

  • Beyond the hard constraint of epipolar geometry,

there are “soft” constraints to help identify corresponding points

  • Similarity
  • Uniqueness
  • Ordering
  • Disparity gradient
  • To find matches in the image pair, we will assume
  • Most scene points visible from both views
  • Image regions for the matches are similar in

appearance

slide-50
SLIDE 50

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Dense correspondence search

For each epipolar line For each pixel / window in the left image

  • compare with every pixel / window on same epipolar line

in right image

  • pick position with minimum match cost (e.g., SSD,

normalized correlation)

Adapted from Li Zhang

slide-51
SLIDE 51

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Matching cost disparity Left Right scanline

Correspondence search with similarity constraint

  • Slide a window along the right scanline and

compare contents of that window with the reference window in the left image

  • Matching cost: SSD or normalized correlation
slide-52
SLIDE 52

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Left Right scanline

Correspondence search with similarity constraint

SSD

slide-53
SLIDE 53

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Left Right scanline

Correspondence search with similarity constraint

  • Norm. corr
slide-54
SLIDE 54

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correspondence problem

Source: Andrew Zisserman

Intensity profiles

slide-55
SLIDE 55

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correspondence problem

Neighborhoods of corresponding points are similar in intensity patterns.

Source: Andrew Zisserman

slide-56
SLIDE 56

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correlation-based window matching

Source: Andrew Zisserman

slide-57
SLIDE 57

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Textureless regions

Textureless regions are non-distinct; high ambiguity for matches.

Source: Andrew Zisserman

slide-58
SLIDE 58

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Effect of window size

Source: Andrew Zisserman

slide-59
SLIDE 59

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

W = 3 W = 20

Figures from Li Zhang

Want window large enough to have sufficient intensity variation, yet small enough to contain only pixels with about the same disparity.

Effect of window size

slide-60
SLIDE 60

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Correspondence problem

  • Beyond the hard constraint of epipolar geometry,

there are “soft” constraints to help identify corresponding points

  • Similarity
  • Disparity gradient – depth doesn’t change too quickly.
  • Uniqueness
  • Ordering
slide-61
SLIDE 61

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Uniqueness constraint

  • Up to one match in right image for every point in left

image

Figure from Gee & Cipolla 1999

slide-62
SLIDE 62

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Problem: Occlusion

  • Uniqueness says “up to match” per pixel
  • When is there no match?

Occluded pixels

slide-63
SLIDE 63

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Ordering constraint

  • Points on same surface (opaque object) will be in same
  • rder in both views

Figure from Gee & Cipolla 1999

slide-64
SLIDE 64

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Ordering constraint

Figures from Forsyth & Ponce

  • Won’t always hold, e.g. consider transparent object, or

an occluding surface

slide-65
SLIDE 65

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo results

Ground truth Scene

  • Data from University of Tsukuba
  • Similar results on other images without ground truth
slide-66
SLIDE 66

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Results with window search

Window-based matching (best window size) Ground truth

slide-67
SLIDE 67

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Better solutions

  • Beyond individual correspondences to estimate

disparities:

  • Optimize correspondence assignments jointly
  • Scanline at a time (DP)
  • Full 2D grid (graph cuts)
slide-68
SLIDE 68

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Scanline stereo

  • Try to coherently match pixels on the entire

scanline

  • Different scanlines are still optimized

independently

Left image Right image

intensity

slide-69
SLIDE 69

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

“Shortest paths” for scan-line stereo

Left image Right image

Can be implemented with dynamic programming

Ohta & Kanade ’85, Cox et al. ’96, Intille & Bobick, ‘01

left

S

right

S

q p

Left

  • cclusion

t

Right

  • cclusion

s

I I′

Slide credit: Y. Boykov

slide-70
SLIDE 70

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Coherent stereo on 2D grid

  • Scanline stereo generates streaking artifacts
  • Can’t use dynamic programming to find

spatially coherent disparities/ correspondences on a 2D grid

slide-71
SLIDE 71

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo as energy minimization

  • What defines a good stereo correspondence?

1.

Match quality

  • Want each pixel to find a good match in the other image

2.

Smoothness

  • If two pixels are adjacent, they should (usually) move

about the same amount

slide-72
SLIDE 72

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Stereo matching as energy minimization

I1 I2 D

  • Energy functions of this form can be minimized

using graph cuts

  • Y. Boykov, O. Veksler, and R. Zabih, Fast Approximate

Energy Minimization via Graph Cuts, PAMI 2001 W1(i ) W2(i+D(i )) D(i )

) ( ) , , (

smooth 2 1 data

D E D I I E E β α + =

( )

∑

− =

j i

j D i D E

, neighbors smooth

) ( ) ( ρ

( )

2 2 1 data

)) ( ( ) (

∑

+ − =

i

i D i W i W E

Source: Steve Seitz

slide-73
SLIDE 73

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Better results…

State of the art method

Boykov et al., Fast Approximate Energy Minimization via Graph Cuts, International Conference on Computer Vision, September 1999.

Ground truth

For the latest and greatest: http://www.middlebury.edu/stereo/

slide-74
SLIDE 74

Stereo 1: Disparity and Matching CS 4495 Computer Vision – A. Bobick

Challenges

  • Low-contrast ; textureless image regions
  • Occlusions
  • Violations of brightness constancy (e.g., specular

reflections)

  • Really large baselines (foreshortening and

appearance change)

  • Camera calibration errors