The Eight-Point Algorithm COMPSCI 527 Computer Vision COMPSCI 527 - PowerPoint PPT Presentation

The Eight-Point Algorithm COMPSCI 527 — Computer Vision COMPSCI 527 — Computer Vision The Eight-Point Algorithm 1 / 17

Outline 1 Summary: The Epipolar Constraint 2 The Eight-Point Algorithm: t , R 3 Triangulation: P m 4 Bundle Adjustment COMPSCI 527 — Computer Vision The Eight-Point Algorithm 2 / 17

Summary: The Epipolar Constraint Summary: The Epipolar Constraint P to compute E 1 How How to break projection ray projection ray 2 up E into R E epipolar plane I e b p i p o p l a I r a l a i n center of e o epipolar line of p f p projection a T b p a baseline epipole e b b center of camera a camera b b projection epipole e a The two projection rays and the baseline are coplanar for corresponding points • a = a p a b = b p b R = a R b t = a t b e = a e b , , , , O • Epipolar constraint: b T E a = 0 O where E = R [ t ] × • Linear in E but not in R and t • The epipole e belongs to every epipolar line λ T = b T E O O λ T x = 0 with COMPSCI 527 — Computer Vision The Eight-Point Algorithm 3 / 17

Summary: The Epipolar Constraint The Structure of E • E has rank 2 and null ( E ) = span ( t ) = span ( e ) • Geometry: TEe • The epipole e is in all epipolar lines • Therefore, λ T e = 0 for all b in λ T = b T E 00 • Therefore E e = 0 , so e 2 null ( E ) if • Algebra: 00 [ t ] × t = t ⇥ t = 0 t ⇥ v 6 = 0 if v is not parallel to t • Therefore, the rank of [ t ] × is 2 for nonzero t • Since R is full rank, the solutions of [ t ] × x = 0 and E x = 0 are the same E REE • Therefore, rank ( E ) = 2 for nonzero t • null ( E ) = span ( t ) = span ( e ) COMPSCI 527 — Computer Vision The Eight-Point Algorithm 4 / 17

Summary: The Epipolar Constraint Ii iii it i The Structure of E • E has two equal singular values and one zero singular value • Proof • Let v be perpendicular to t . Then k [ t ] × v k = k t k k v k (geometric definition of cross product) • Let k v k = 1. Then k [ t ] × v k = k t k • v ? t means that v 2 row space ([ t ] × ) HIE Ir E II • Therefore, unit-norm vectors v 2 row space ([ t ] × ) are HEH mapped to a circle T • Therefore [ t ] × has two equal singular values UI V T ft • Third is zero because t 2 null ([ t ] × ) T • Ditto for E , since E = R [ t ] × and R is orthogonal R E Y • Therefore v 3 ⇠ e ⇠ t g • If we have E , we can find t by SVD COMPSCI 527 — Computer Vision The Eight-Point Algorithm 5 / 17

Summary: The Epipolar Constraint A Fundamental Ambiguity • The equation b T E a = 0 is homogeneous in E k • Therefore, we cannot tell the magnitude of E , or of t in E = R [ t ] × • Absolute scale cannot be determined from images alone • This ambiguity is general, has nothing to do with the specifics of the formulation • Cameras fundamentally measure angles, not distances • This ambiguity is often exploited in movie special effects 0 • W.l.o.g., let k t k = 1 • Measure everything in units of inter-camera distance COMPSCI 527 — Computer Vision The Eight-Point Algorithm 6 / 17

Summary: The Epipolar Constraint Switching Cameras • Everything holds when a and b are switched • b T E a = 0 a T E T b = 0 or • Switch null space and row space of E with left null space and column space • Therefore, u 3 ⇠ b e a ⇠ b t a be • Also, b E a = a E T Eb b O O O a E b ⇠ U Σ V T = ⇤ T ⇥ ⇤ ⇥ u 1 u 2 u 3 diag ( 1 , 1 , 0 ) v 1 v 2 v 3 COMPSCI 527 — Computer Vision The Eight-Point Algorithm 7 / 17

Summary: The Epipolar Constraint Next Problem: How to Find E ? b T E a = 0 • Given pairs ( a 1 , b 1 ) , . . . ( a n , b n ) (tracking) • Write one epipolar constraint equation per pair • Linear in E COMPSCI 527 — Computer Vision The Eight-Point Algorithm 8 / 17

The Eight-Point Algorithm: t, R The Eight-Point Algorithm • H. C. Longuet-Higgins, Nature , 293:133–135, 1981 • Needs at least 8 corresponding point pairs • Preferably many more • Overview: • Solve b T 1 E a 1 = 0, . . . , b T n E a n = 0 for E H • Solve E = R [ t ] × for t , R • Compute the 3D structure (points P m ) from a m , b m , t , R e in • The last step is called triangulation ops BASELINE 0 COMPSCI 527 — Computer Vision The Eight-Point Algorithm 9 / 17

The Eight-Point Algorithm: t, R Rewriting the Epipolar Constraint 0 b T E a = 0 IEEE 2 3 2 3 e 11 e 12 e 13 a 1 ⇥ b 1 5 = 0 ⇤ b 2 b 3 e 21 e 22 e 23 a 2 4 5 4 e 31 e 32 e 33 a 3 outer gb flatten up a 1 b 1 e 11 + a 1 b 2 e 21 + a 1 b 3 e 31 + f a 2 b 1 e 12 + a 2 b 2 e 22 + a 2 b 3 e 32 + y a 3 b 1 e 13 + a 3 b 2 e 23 + a 3 b 3 e 33 = 0 I 8 e 11 2 3 l e 21 6 7 e 31 r 6 7 6 7 e 12 6 7 6 7 ⇥ ⇤ a 1 b 1 a 1 b 2 a 1 b 3 a 2 b 1 a 2 b 2 a 2 b 3 a 3 b 1 a 3 b 2 a 3 b 3 e 22 = 0 6 7 6 7 e 32 6 7 ee 6 7 e 13 6 7 6 7 e 23 4 5 e 33 O c T η = 0 • With n point pairs, c T m η = 0 for m = 1 , . . . , n COMPSCI 527 — Computer Vision The Eight-Point Algorithm 10 / 17

The Eight-Point Algorithm: t, R Solving for E c T m η = 0 for m = 1 , . . . , n C η = 0 where C is n ⇥ 9 • Because of the scale ambiguity, we cannot tell the norm of η Ill • Set η = 1 • Homogeneous, least squares problem on the unit sphere • We know how to solve that! of e O I vases i COMPSCI 527 — Computer Vision The Eight-Point Algorithm 11 / 17

The Eight-Point Algorithm: t, R Solving for t VE I E VI E E = R [ t ] × I Ve fi 3 E • We have E now • We saw that null ( E ) = span ( t ) • So we know how to find t with k t k = 1, up to a sign O ee I • ± t (and also ± [ t ] × ) I COMPSCI 527 — Computer Vision The Eight-Point Algorithm 12 / 17

The Eight-Point Algorithm: t, R Solving for R ET TR xT RT ET RI E Tri E E = R [ t ] × NON INVERTIBLE O • We have both E and T = [ t ] × • Linear system in R , but with the constraints R T R = RR T = I and det ( R ) = 1 00 • The Procrustes problem , arg min R T R = RR T = I k E � RT k F • Appendix in the notes gives a solution based on the SVD • Since T has rank 2, it turns out that the there are two 00 solutions, R 1 and R 2 COMPSCI 527 — Computer Vision The Eight-Point Algorithm 13 / 17

The Eight-Point Algorithm: t, R The Fourfold Ambiguity ( t , R 1 ) , ( − t , R 2 ) , ( t , R 2 ) , ( − t , R 1 ) P 15 ER Z D Z C Z C Z D X D X D X C X C D C C D P P Z C Z C he D’ D’ X C X C X D’ C C X D’ Z D’ Z D’ P • Only one solution places all world points in front of both cameras • Try all four solutions, and reconstruct world points by triangulation • Pick the one solution that makes sense COMPSCI 527 — Computer Vision The Eight-Point Algorithm 14 / 17

Triangulation: P m FEET're Triangulation 2 3 2 3 a 0 a 1 O 1 • For simplicity, divide a 0 = 5 by f so that now a = a 0 a 2 4 4 5 2 1 f  a 1 Q � def • Let α = (coordinates in canonical image reference system) a 2 • Ditto for b , β • Projection equations in each camera reference frame: A is P in frame a  A 1  B 1 O � � 1 1 I α = and β = A 3 A 2 B 3 B 2 G A  A 1  B 1 � � • Rewrite as α A 3 = 4 1 and β B 3 = 4 3 34 A 2 B 2 ALEE.BE EllGA FH Plug B = R ( A − t ) into second set of equations • All equations are linear. Four equations, 3 unknowns • Solve in the LSE sense, get a modicum of noise rejection COMPSCI 527 — Computer Vision The Eight-Point Algorithm 15 / 17

Bundle Adjustment Summary of Eight-Point Algorithm • Given n ≥ 8 image point pairs ( a m , b m ) for m = 1 , . . . , n • Solve n × 9 linear homogeneous system b T m E a m = 0 for E • Compute ± t as the third right singular vector of ± E • Solve ± E = R ± [ t ] × for R by Procrustes (linear problem with orthogonality constraint) to obtain R 1 , R 2 • Triangulate scene points P m from a m , b m , t , R and for all four combinations of t and R ( n separate problems, one per point pair) • Choose the one combination of t , R that places world points in front of both cameras • Keep the corresponding triangulated P m • Everything is found up to a single, global scale factor COMPSCI 527 — Computer Vision The Eight-Point Algorithm 16 / 17

Bundle Adjustment Bundle Adjustment • Let π be the perspective projection function. We are after HI n 1 X ( a m − π ( A m )) 2 + ( b m − π ( R ( A m − t ))) 2 ⇤ ⇥ arg min n t , R , A 1 ,..., A n | {z } m = 1 reprojection error arg t , R , A 1 ,..., A n ρ ( t , R , A 1 , . . . , A n ) min • Eight-point algorithm solves this single optimization problem in multiple steps • This greedy approach leads to a suboptimal solution • Use solution t , R , P 1 , . . . , P n to initialize a gradient-descent search for an optimal solution to the full problem • This fine-tuning step is called bundle adjustment COMPSCI 527 — Computer Vision The Eight-Point Algorithm 17 / 17