Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices CS 4495 Computer Vision N-Views (2) – Essential and Fundamental Matrices Aaron Bobick School of Interactive Computing
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Administrivia • Today: Second half of N-Views (n = 2) • PS 3: Will hopefully be out by Thursday • Will be due October 6 th . • Will be based upon last week and today’s material • We may revisit the logistics – suggestions?
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Two views…and two lectures • Projective transforms from image to image • Some more projective geometry • Points and lines and planes • Two arbitrary views of the same scene • Calibrated – “Essential Matrix” • Two uncalibrated cameras “Fundamental Matrix” • Gives epipolar lines
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time • Projective Transforms: Matrices that provide transformations including translations, rotations, similarity, affine and finally general (or perspective) projection. • When 2D matrices are 3x3; for 3D they are 4x4.
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time: Homographies • Provide mapping between images (image planes) taken from same center of projection; also mapping between any images of a planar surface. PP2 PP1
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Last time: Projective geometry • A line is a plane of rays through origin – all rays (x,y,z) satisfying: ax + by + cz = 0 x [ ] = in vector notation : 0 a b c y z l p • A line is also represented as a homogeneous 3-vector l
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Projective Geometry: lines and points p 1 + + = ax by c 0 2D Lines: p 2 x = [ ] [ ] a b c y 0 p 1 = x 1 y 1 1 l = p 1 × p 2 1 [ ] Eq of line p 2 = x 2 y 2 1 = T l x 0 x 12 [ ] = ⇒ l a b c n n d x y (n x , n y ) [ ] 1 = a 1 d l b c 1 1 x 12 = l 1 × l 2 [ ] l 2 = a 2 b 2 c 2
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Motivating the problem: stereo • Given two views of a scene (the two cameras not necessarily having optical axes) what is the relationship between the location of a scene point in one image and its location in the other?
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo correspondence • Determine Pixel Correspondence • Pairs of points that correspond to same scene point P epipolar line epipolar line epipolar plane CP 1 CP 2 Epipolar Constraint • Reduces correspondence problem to 1D search along conjugate epipolar lines
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Example: converging cameras Figure from Hartley & Zisserman
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Epipolar geometry: terms • Baseline: line joining the camera centers • Epipole: point of intersection of baseline with image plane • Epipolar plane: plane containing baseline and world point • Epipolar line: intersection of epipolar plane with the image plane • All epipolar lines intersect at the epipole • An epipolar plane intersects the left and right image planes in corresponding epipolar lines
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From Geometry to Algebra • So far, we have the explanation in terms of geometry. • Now, how to express the epipolar constraints algebraically?
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras Main idea
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to camera reference frame 2. Rotation: 3 x 3 matrix R ; translation: 3 vector T .
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Stereo geometry, with calibrated cameras If the stereo rig is calibrated, we know : how to rotate and translate camera reference frame 1 to get to = + camera reference frame 2. X ' RX T c c
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices An aside: cross product × = a b c Vector cross product takes two vectors and returns a third vector that’s perpendicular to both inputs. So here, c is perpendicular to both a and b, which means the dot product = 0. ⋅ = a c 0 ⋅ = b c 0
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From geometry to algebra ( ) ( ) ′ ′ ′ = + ⋅ × = ⋅ × X' RX T X T X X T RX ′ × = × + × T X T RX T T ( ) ′ = ⋅ × 0 X T RX Normal to the plane = T × RX
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Another aside: Matrix form of cross product − 0 a a b 3 2 1 × = − = a b a 0 a b c 3 1 2 − a a 0 b 2 1 3 Can be expressed as a matrix multiplication!!! − 0 a a Notation: 3 2 [ ] [ ] = − × = a x a 0 a a b a b × 3 1 − a a 0 Has rank 2! 2 1
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From geometry to algebra ( ) ( ) ′ ′ ′ = + ⋅ × = ⋅ × X' RX T X T X X T RX ′ × = × + × T X T RX T T ( ) ′ = ⋅ × 0 X T RX Normal to the plane = T × RX
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Essential matrix ( ) ′ ⋅ × = X T RX 0 ( ) ′ ⋅ = X RX [T ] 0 x = E [T x ] R Let ′ EX = X T 0 E is called the essential matrix , and it relates corresponding image points between both cameras, given the rotation and translation. Note: these points are in each camera coordinate systems . We know if we observe a point in one image, its position in other image is constrained to lie on line defined by above.
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Essential matrix example: parallel cameras Z = p [ Zx Zy f , , ] P = R I Z Τ = − = T [ B ,0,0] p' [ Zx Zy ', ', ] x x r f Z 0 0 0 l = = E [ T ]R x 0 0 B p p 0 – B 0 f l r B COP L COP R 0 0 0 x = [ ] x ' y ' 1 0 0 B y 0 ′ Τ Ep = p 0 Given a known − 0 B 0 1 point (x,y) in the original 0 image, this is a = [ ] line in the x ' y ' 1 B 0 For the parallel cameras, (x’,y’) image. − By image of any point must lie on same horizontal line in = ⇒ y' = y By ' By each image plane.
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices Weak calibration • Want to estimate world geometry without requiring calibrated cameras • Archival videos (already have the pictures) • Photos from multiple unrelated users • Dynamic camera system • Main idea : • Estimate epipolar geometry from a (redundant) set of point correspondences between two uncalibrated cameras
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From before: Projection matrix X w wx • This can be rewritten as a im Y = matrix product using K Φ w wy im int ext homogeneous coordinates: Z w w where: 1 − Τ r r r R T 11 12 13 1 = − Φ R T Τ r r r ext 21 22 23 2 − Τ r r r R T 31 32 33 3 − f / s 0 o x x Note: Invertible, scale x = − K 0 f / s o int y y and y, assumes no skew 0 0 1
Two Views Part 2: Essential CS 4495 Computer Vision – A. Bobick and Fundamental Matrices From before: Projection matrix X • This can be rewritten as a w wx im matrix product using Y = K Φ w homogeneous coordinates: wy im int ext Z w w 1 = K Φ P p im int ext w p c = p K p im int c
Recommend
More recommend
