Fundamental Matrix & Structure from Motion Instructor - Simon - - PowerPoint PPT Presentation

Fundamental Matrix & Structure from Motion

Instructor - Simon Lucey

16-423 - Designing Computer Vision Apps

Today

• Transformations between images
• Structure from Motion
• The Essential Matrix
• The Fundamental Matrix

Transformations between images

• So far we have considered

transformations between the image and a plane in the world

• Now consider two cameras

viewing the same plane

• There is a homography between

camera 1 and the plane and a second homography between camera 2 and the plane

• It follows that the relation between the two images is also a

homography

Camera under pure rotation

Special case is camera under pure rotation. Homography can be showed to be

Camera under pure rotation

Special case is camera under pure rotation. Homography can be showed to be

Why is this?

Panorama Example

a) b) c) d)

Review - Homography

• Start with basic projection equation:
• Combining these two matrices we get:

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Review - Homography

Homogeneous: Cartesian:

For short:

Review - Homography

Homogeneous: Cartesian:

For short:

How many unknowns?

Homography Estimation

• Re-arrange cartesian equations,

           −u1 −v1 −1 y1u1 y1v1 y1 u1 v1 1 −x1u1 −x1v1 −x1 −u2 −v2 −1 y2u2 y2v2 y2 u2 v2 1 −x2u2 −x2v2 −x2 . . . . . . . . . . . . . . . . . . . . . . . . . . . −uI −vI −1 yIuI yIvI yI uI vI 1 −xIuI −xIvI −xI                          φ11 φ12 φ13 φ21 φ22 φ23 φ31 φ32 φ33               = 0,

• Form linear system

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Homography Estimation

• In MATLAB this becomes,

>> [U,S,V] = svd(A); >> Phi = reshape(V(:,end),[3,3])’;

• Both sides are 3x1 vectors; should be parallel, so cross

product will be zero

• For you to try MATLAB,

>> x = [randn(2,1);1]; cross(x,4*x)

Caution…

• Approach only minimizes algebraic error NOT the re-

projection error!!!

• Need to employ non-linear optimization.

Estimating Extrinsics

Estimating Extrinsics

• Writing out the camera equations in full
• Estimate the homography from matched points
• Factor out the intrinsic parameters

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Estimating Extrinsics

• Find the last column using the cross product of first two

columns

• Make sure the determinant is 1. If it is -1, then multiply

last column by -1.

• Find translation scaling factor between old and new

values

• Finally, set

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Augmented Reality

Today

• Transformations between images
• Structure from Motion
• The Essential Matrix
• The Fundamental Matrix

Taken from Agarwal et al. “Building Rome in a Day”, ICCV 2009.

Taken from Agarwal et al. “Building Rome in a Day”, ICCV 2009.

Structure from Motion

For simplicity, we’ll start with simpler problem

• Just J=2 images
• Known intrinsic matrix

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Epipolar lines

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Epipole

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Special configurations

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Today

• Transformations between images
• Structure from Motion
• The Essential Matrix
• The Fundamental Matrix

Review - Cross Product

• Cross product between two 3D vectors is defined as,
• Operation is equivalent to the,
• Can be written in short as,

2 4 c1 c2 c3 3 5 = 2 4 a3 a2 a3 a1 a2 a1 3 5 2 4 b1 b2 b3 3 5

• n c = a ⇥ b

c = [a]×b

Review - Cross Product

• In other words,

aT (a ⇥ b) = bT (a ⇥ b) = 0.

Right-hand rule

• Given three orthonormal 3D vectors (x, y, z)

z = x × y

• Then,

First camera: Second camera: Substituting: This is a mathematical relationship between the points in the two images, but it’s not in the most convenient form.

The Essential Matrix

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

The Essential Matrix

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

The cross product term can be expressed as a matrix Defining: We now have the essential matrix relation

The Essential Matrix

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

• Rank 2:
• 5 degrees of freedom
• Non-linear constraints between elements

Properties of the Essential Matrix

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Recovering Epipolar Lines

Equation of a line:

• r
• r

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

T

Recovering Epipolar Lines

Equation of a line: Now consider This has the form where So the epipolar lines are

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

T T

Every epipolar line in image 1 passes through the epipole e1. In other words for ALL This can only be true if e1 is in the nullspace of E. Similarly: We find the null spaces by computing , and taking the last column of and the last row of .

Recovering Epipolar Lines

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Essential matrix: To recover translation and rotation use the matrix: We take the SVD and then we set

Decomposition of E

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Four Interpretations

To get the different solutions, we mutliply τ by -1 and substitute

Today

• Transformations between images
• Structure from Motion
• The Essential Matrix
• The Fundamental Matrix

The Fundamental Matrix Song

Taken from Daniel Wedge’s home page - http://danielwedge.com/fmatrix/

The Fundamental Matrix Song

Taken from Daniel Wedge’s home page - http://danielwedge.com/fmatrix/

The Fundamental Matrix

Now consider two cameras that are not normalised By a similar procedure to before, we get the relation

• r

where Relation between essential and fundamental

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Estimation of the Fundamental Matrix

Estimation of Fundamental Matrix

When the fundamental matrix is correct, the epipolar line induced by a point in the first image should pass through the matching point in the second image and vice-versa. This suggests the criterion If and then Unfortunately, there is no closed form solution for this quantity.

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

The 8 Point Algorithm

Approach:

• solve for fundamental matrix using homogeneous

coordinates

• closed form solution (but to wrong problem!)
• Known as the 8 point algorithm

Start with fundamental matrix relation Writing out in full:

• r

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

The 8 Point Algorithm

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Can be written as: where Stacking together constraints from at least 8 pairs of points, we get the system of equations

Minimum direction problem of the form Find minimum of subject to . To solve, compute the SVD and then set to the last column of .

The 8 Point Algorithm

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Fitting Concerns

• This procedure does not ensure that solution is rank 2.

Solution: set last singular value to zero.

• Can be unreliable because of numerical problems to do with

the data scaling – better to re-scale the data first

• Needs 8 points in general positions (cannot all be planar).
• Fails if there is not sufficient translation between the views
• Use this solution to start non-linear optimisation of true

criterion (must ensure non-linear constraints obeyed).

• There is also a 7 point algorithm.

More to read…

• Prince et al.
• Chapter 16, Sections 1-3.