Stereo Vision INEL 6088 - Computer Vision 1 Some References - - PowerPoint PPT Presentation

Stereo Vision

INEL 6088 - Computer Vision

1

Some References

• Secs.11.1 and 11.2 on Jain et. al.
• Davies, 4th ed.: secs. 15.1-15.3 & 18.6-18.17
• Davies, 3rd ed.: secs.16.1-16.3 & 21.6-21.17
• Chapter 11 on Szeliski's book

2

3

Stereo vision

4

Epipolar plane and lines

Epipolar plane: plane passing through the

camera centers and the feature point

Epipolar line: intersection between epipolar

plane and image plane

Conjugate pair: two points in different images

that are the projection of a single scene point

Disparity: difference in the conjugate pair’s

pixel position if the images are superimposed

5

Consideration of similar triangles gives

6

Stereo Matching

Correspondence problem: main problem in stereo

vision; to match a point in one image with the corresponding point in the other image.

Features that can be matched must be located.

Sometimes they are created by lightening a pattern.

Depth is computed only for this features. Others are

estimated by interpolation.

Epipolar constrain limits the search space Obstruction means that points might be missing

from one of the images – this makes solving the problem more difficult

Baseline = 10in Taken with iPad (focal length = 3.2mm) Pleft = (1006, 641) Pright = (691,635)

7

8

xl - xr = 1006 - 691 pixels = 315 pixels Pixel size = 1.12 um xl - xr = 352.8 um = 0.3528 mm Baseline = 10in Z = (10in)(3.3mm)/(.3528mm) = 93.5 in Actual distance = 70 in (measured with tape) Difference is to large. Error due to small device rotation?

lc2=im2(411:962,449:1000); crop_im1 = im1(529:579,976:1026); 9

xc12 = normxcorr2(double(crop_im1),double(lc2)); plot(max(xc12)); figure; plot(max(xc12')); figure; imshow(lc2) figure; imshow(crop_im1)

peaks at (257,159)

10

11

12

Common metrics to compare image patches:

• Sum of square differences: (Il − Ir)2
• Absolute difference: |Il − Ir|
• Cross-correlation: IlIr
• Normalized Cross-correlation

IlIr

√ I2

l

I2

r

See this link for other similarity measures.

13

14

Solutions to the Correspondence problem

Edge Matching

Filter each image with a Gaussian at four different

scales

Compute edge positions within row Match non-horizontal edges by comparing

• rientation and magnitude

Refine disparity estimates by using finer scales However, strong edges tend to be silhouette

edges and correspond to different parts of the

• bject in the two images

15

Region Correlation

Match areas of the image with high variance

Calculate variance along different directions using all

pixels in a window centered about a point.

Select minimum value as the interest value of the central

pixel

Chose pixels whose interest measure is a local maxima

and is above a threshold as the feature points

Compute correlation coefficient between a small window

• f pixels centered around the selected features

Consider only features that satisfy the epipolar constrain Features with highest correlation coefficient are

considered a match

16

Correlation coefficient: match if r>th - look along epipolar lines Directional variances: look for points with large min(I1, I2, I3, I4)

Other feature based methods

edge points lines (length, orientation, midpoint, average

contrast)

corners

points where edge lines meet formed by intensity patterns

select method based on

kind of object of interest and illumination image contrast, etc.

17

18

Epipolar lines generalized


pl = fl Zl Pl pr = fr Zr Pr

19

Rectification

• Transformation to make epipolar lines collinear

and parallel to horizontal axis

• Steps:
• Rotate left camera so that epipole goes to infinity along the

horizontal axis

• Apply the same rotation to right camera
• Rotate right camera by R
• Adjust scale in both camera reference frames

20

Rectification

• Rectification: make the epipolar lines collinear and parallel to the hori-

zontal image axis

• This transformation can be described in terms of the matrix R defined in

the equation Pr = R(Pl − T) and a rotation matrix Q such that the epipolar lines become parallel.

• Assume same f for both cameras.
• The Q is a 3 × 3 matrix given by

Q = ⇥ qT

1

qT

2

qT

3

⇤T where q1 = T

• T
• q2 =

⇥ −T2, T1, ⇤T p T 2

1 + T 2 2

and q3 = q1 × q2

21

Rectification

• 1. For each left-camera point pl = [x, y, f]T compute

[x, y, z] = Qpl and the coordinates of the corresponding rectified point p

l,

p

l = f

z [x, y, z]

• 2. Let Rr = RQ.
• 3. Repeat step 1 for the right-camera using Rr and pr.

22

23

Essential and Fundamental Matrices

• A translation vector T = Or − Ol and an orthogonal rotational matrix R

bring the left camera’s coordinate system into the right’s. Pr = R(Pl − T)

• Since Pl and T are in Πe, Pl − T is also in Πe
• Thus

(Pl − T) · (T × Pl) = 0

• From the definition of the cross product, a matrix A can be defined

T × Pl =

• T1

T2 T3 Pl1 Pl2 Pl3 i j k

• =

i(T2Pl3 − T3Pl2) − j(T1Pl3 − T3Pl1) + k(T1Pl2 − T2Pl1) = 2 4 −T3Pl2 + T2Pl3 T3Pl1 − T1Pl3 −T2Pl1 + T1Pl2 3 5 2 4 i j k 3 5 = 2 4 −T3 T2 T3 −T1 −T2 T1 3 5 2 4 Pl1 i Pl2 j Pl3 k 3 5 = APl

24

• The previous result in step 2 can be expressed as

(R−1Pr)T APl = 0

• Since R is orthogonal, RT = R−1.
• In general,

(AB)T = BT AT

• Thus

PT

r RAPl = PT r EPl = 0

• The essential matrix is E = RA
• The image points pl and pr are also related by E

pT

r Epl = 0

25

• The pixel coordinates of a given point pk are given by

c pk = Mint,kpk where Mint,k is a matrix that represent the intrinsic parameters.

• The above expression

pT

r Epl = 0

(M−1

int,rc

pr)T EM−1

int,l b

pl = 0 c pr

T M−T int,rEM−1 int,l b

pl = 0 c pr

T F b

pl = 0

• The fundamental matrix F = M−T

int,rEM−1 int,l describes the epipolar geom-

etry in terms of pixel coordinates.

• The essential matrix is defined in terms of camera coordinates.
• The essential matrix depends on the extrinsic parameters.

26

MATLAB page on how to obtain the epipolar line from the Fundamental matrix

F pl describes the epipolar line on which the corresponding point pr

• n the other image must lie.

27

Reconstruction

• by triangulation - possible if intrinsic and

extrinsic parameters are known

• up to a scale factor - intrinsic parameters are

known

28

29

• Assume that: stereo system’s intrinsic and extrinsic parameters are known.
• Compute: 3-D location for points from their projections pl and pr
• A point P lies in the intersection of the rays from Ol and pl, and Or and

pr

• Problem: since parameters and image locations are known only approxi-

mately, the two rays will not intersect

• Solution: estimate the intersection as the point of minimum distance from

both rays.

• Let

– apl be the ray l going from Ol through pl – T+bRT pr be the ray, r, through Or through pr expressed in the left reference frame. – w be a vector orthogonal to both l and r

• Problem reduces to finding the midpoint , P 0, of the segment parallel to

w that joins l and r

Reconstruction by Triangulation

30

• w can be expressed as

w = pl × RT pr

• apl + cw represent a line through apl parallel to w.
• Determine the endpoints of the segment, s, belonging to the line parallel

to w that joins l (a0pl) and r ( T + b0RT pr) from apl − (T + bRT pr) + cw = 0 by solving apl − bRT pr + c(pl × RT pr) = T for a0, b0 and c0

• The triangulated point P 0 is the midpoint of s.

31

32

Zero disparity means that the point is on the surface; positive disparity indicates points farther than the surface, etc. ACTIVE VISION SYSTEM- camera is controlled dynamically by the analysis algorithm to facilitate depth (or other) computation.