Fundamental Matrix & Structure from Motion Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps
Today • Transformations between images • Structure from Motion • The Essential Matrix • The Fundamental Matrix
Transformations between images • So far we have considered transformations between the image and a plane in the world • Now consider two cameras viewing the same plane • There is a homography between camera 1 and the plane and a second homography between camera 2 and the plane • It follows that the relation between the two images is also a homography
Camera under pure rotation Special case is camera under pure rotation. Homography can be showed to be
Camera under pure rotation Special case is camera under pure rotation. Homography can be showed to be Why is this?
Panorama Example a) b) c) d)
Review - Homography • Start with basic projection equation: • Combining these two matrices we get: Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Review - Homography Homogeneous: Cartesian: For short:
Review - Homography Homogeneous: Cartesian: For short: How many unknowns?
Homography Estimation • Re-arrange cartesian equations, • Form linear system φ 11 0 0 0 − 1 − u 1 − v 1 y 1 u 1 y 1 v 1 y 1 φ 12 1 0 0 0 u 1 v 1 − x 1 u 1 − x 1 v 1 − x 1 φ 13 0 0 0 − 1 − u 2 − v 2 y 2 u 2 y 2 v 2 y 2 φ 21 1 0 0 0 u 2 v 2 − x 2 u 2 − x 2 v 2 − x 2 = 0 , φ 22 . . . . . . . . . . . . . . . . . . φ 23 . . . . . . . . . φ 31 0 0 0 − 1 − u I − v I y I u I y I v I y I φ 32 1 0 0 0 u I v I − x I u I − x I v I − x I φ 33 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Homography Estimation • In MATLAB this becomes, >> [U,S,V] = svd(A); >> Phi = reshape(V(:,end),[3,3])’; • Both sides are 3x1 vectors; should be parallel, so cross product will be zero • For you to try MATLAB, >> x = [randn(2,1);1]; cross(x,4*x)
Caution… • Approach only minimizes algebraic error NOT the re- projection error!!! • Need to employ non-linear optimization.
Estimating Extrinsics
Estimating Extrinsics • Writing out the camera equations in full • Estimate the homography from matched points • Factor out the intrinsic parameters Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Estimating Extrinsics • Find the last column using the cross product of first two columns • Make sure the determinant is 1. If it is -1, then multiply last column by -1. • Find translation scaling factor between old and new values • Finally, set Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Augmented Reality
Today • Transformations between images • Structure from Motion • The Essential Matrix • The Fundamental Matrix
Taken from Agarwal et al. “Building Rome in a Day”, ICCV 2009.
Taken from Agarwal et al. “Building Rome in a Day”, ICCV 2009.
Structure from Motion For simplicity, we’ll start with simpler problem • Just J=2 images • Known intrinsic matrix Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Epipolar lines Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Epipole Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Special configurations Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Today • Transformations between images • Structure from Motion • The Essential Matrix • The Fundamental Matrix
Review - Cross Product • Cross product between two 3D vectors is defined as, on c = a ⇥ b • Operation is equivalent to the, 2 3 2 3 2 3 0 � a 3 c 1 a 2 b 1 5 = 0 � a 1 c 2 a 3 b 2 4 4 5 4 5 0 � a 2 c 3 a 1 b 3 • Can be written in short as, c = [ a ] × b
Review - Cross Product • In other words, a T ( a ⇥ b ) = b T ( a ⇥ b ) = 0 .
Right-hand rule • Given three orthonormal 3D vectors ( x , y , z ) • Then, z = x × y
The Essential Matrix First camera: Second camera: Substituting: This is a mathematical relationship between the points in the two images, but it’s not in the most convenient form. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
The Essential Matrix Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
The Essential Matrix The cross product term can be expressed as a matrix Defining: We now have the essential matrix relation Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Properties of the Essential Matrix • Rank 2: • 5 degrees of freedom • Non-linear constraints between elements Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Recovering Epipolar Lines Equation of a line: or T or Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Recovering Epipolar Lines T Equation of a line: Now consider T This has the form where So the epipolar lines are Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Recovering Epipolar Lines Every epipolar line in image 1 passes through the epipole e 1 . In other words for ALL This can only be true if e 1 is in the nullspace of E . Similarly: We find the null spaces by computing , and taking the last column of and the last row of . Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Decomposition of E Essential matrix: To recover translation and rotation use the matrix: We take the SVD and then we set Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Four Interpretations To get the different solutions, we mutliply τ by -1 and substitute
Today • Transformations between images • Structure from Motion • The Essential Matrix • The Fundamental Matrix
The Fundamental Matrix Song Taken from Daniel Wedge’s home page - http://danielwedge.com/fmatrix/
The Fundamental Matrix Song Taken from Daniel Wedge’s home page - http://danielwedge.com/fmatrix/
The Fundamental Matrix Now consider two cameras that are not normalised By a similar procedure to before, we get the relation or where Relation between essential and fundamental Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Estimation of the Fundamental Matrix
Estimation of Fundamental Matrix When the fundamental matrix is correct, the epipolar line induced by a point in the first image should pass through the matching point in the second image and vice-versa. This suggests the criterion If and then Unfortunately, there is no closed form solution for this quantity. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
The 8 Point Algorithm Approach: • solve for fundamental matrix using homogeneous coordinates • closed form solution (but to wrong problem!) • Known as the 8 point algorithm Start with fundamental matrix relation Writing out in full: or Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
The 8 Point Algorithm Can be written as: where Stacking together constraints from at least 8 pairs of points, we get the system of equations Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
The 8 Point Algorithm Minimum direction problem of the form Find minimum of subject to . To solve, compute the SVD and then set to the last column of . Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince
Fitting Concerns • This procedure does not ensure that solution is rank 2. Solution: set last singular value to zero. • Can be unreliable because of numerical problems to do with the data scaling – better to re-scale the data first • Needs 8 points in general positions (cannot all be planar). • Fails if there is not sufficient translation between the views • Use this solution to start non-linear optimisation of true criterion (must ensure non-linear constraints obeyed). • There is also a 7 point algorithm.
More to read… • Prince et al. • Chapter 16, Sections 1-3.
Recommend
More recommend
