Two-View Geometry: Epipolar Geometry and the Fundamental Matrix - PowerPoint PPT Presentation

Two-View Geometry: Epipolar Geometry and the Fundamental Matrix 簡韶逸 Shao-Yi Chien Department of Electrical Engineering National Taiwan University Fall 2018 1

Outline • Epipolar geometry and the fundamental matrix [Slides credit: Marc Pollefeys] 2

Three Questions • Correspondence geometry: Given an image point x in the first view, how does this constrain the position of the corresponding point x’ in the second image? • Camera geometry (motion): Given a set of corresponding image points {x i ↔x’ i }, i =1,…,n, what are the cameras P and P’ for the two views ? • Scene geometry (structure): Given corresponding image points x i ↔x’ i and cameras P, P’, what is the position of (their pre -image) X in space? 3

The Epipolar Geometry C,C’,x,x’ and X are coplanar 4

The Epipolar Geometry What if only C,C’,x are known? 5

The Epipolar Geometry All points on p project on l and l’ 6

The Epipolar Geometry Family of planes p and lines l and l’ Intersection in e and e’ 7

The Epipolar Geometry Epipoles e,e ’ = intersection of baseline with image plane = projection of projection center in other image = vanishing point of camera motion direction an epipolar plane = plane containing baseline (1-D family) an epipolar line = intersection of epipolar plane with image (always come in corresponding pairs) 8

Example: Converging Cameras 9

Example: Motion Parallel with Image Plane 10

Example: Forward Motion e’ e 11

The Fundamental Matrix F Algebraic representation of epipolar geometry x  l' we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F 12

The Fundamental Matrix F geometric derivation If a =(a 1 , a 2 , a 3 ) T  0 −𝑏 3 𝑏 2 x' H x π 𝑏 3 0 −𝑏 1 [𝒃] × =       −𝑏 2 𝑏 1 0 e' H x Fx l' e' x'  π mapping from 2-D to 1-D family (rank 2) 13

The Fundamental Matrix F algebraic derivation          λ λC X P x P P I    P  l P' C P' P x x   λ e' X     F e' P' P  (note: doesn’t work for C=C’  F=0) 14

The Fundamental Matrix F correspondence condition The fundamental matrix satisfies the condition that for any pair of corresponding points x ↔x ’ in the two images    x' T  x' T Fx 0 l' 0 15

The Fundamental Matrix F F is the unique 3x3 rank 2 matrix that satisfies x’ T Fx=0 for all x ↔x’ Transpose: if F is fundamental matrix for (P,P’), then F T is (i) fundamental matrix for (P’,P) (ii) Epipolar lines: l’= Fx & l=F T x ’ (iii) Epipoles: on all epipolar lines, thus e’ T Fx=0,  x  e’ T F=0, similarly Fe=0 (iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2) (v) F is a correlation, projective mapping from a point x to a line l’= Fx (not a proper correlation, i.e. not invertible) 16

The Epipolar Line Geometry l,l ’ epipolar lines, k line not through e  l’=F[k] x l and symmetrically l=F T [k’] x l’ k k  l Fk  l l e e' (pick k=e, since e T e ≠0)   l'   l  T  l F e' l' F e   17

Fundamental Matrix for Pure Translation 18

Fundamental Matrix for Pure Translation 19

Fundamental Matrix for Pure Translation          F e' H e'   1 H K RK    example: P=K[ I | 0], P’=K[ I | t] Translation is parallel to the x-axis   0 0 0   e'  T    1,0,0 F 0 0 - 1   0 1 0      y  x' T Fx 0 y' 20

Fundamental Matrix for Pure Translation T    -1 ( ) K x/ x PX K[I | 0]X X,Y,Z Z   -1 K x     x' P' X K[I | t] x' x Kt/ Z     Z motion starts at x and moves towards e, faster depending on Z pure translation: F only 2 d.o.f., x T [e] x x=0  auto-epipolar 21

General Motion    T x ' e' Hx 0     ˆ T x ' e' x 0  -1   x' K' RK x K' t/ Z 22

Projective Transformation and Invariance Derivation based purely on projective concepts -T FH ˆ     ˆ ˆ -1 x Hx, x ' H' x' F H' F invariant to transformations of projective 3-space     ˆ ˆ    -1 x PX PH H X P X     Same matching point! ˆ ˆ    -1 x' P' X P' H H X P ' X    P, P' F unique   F  P, P' not unique canonical form   M  P [I | 0]  F m   P' [M | m] 26

Projective Ambiguity of Cameras Given F previous slide: at least projective ambiguity this slide: not more! ~ ~ Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that ~ ~ P=PH and P ’=P’H ~ ~ ~ ~     P [I | 0] P' [A | a] P [I | 0] P ' [ A | a ]     A ~ ~   F a A a     ~ ~     1 T lemma: a ka A A av k   rank 2  ~ ~     aF a a A 0 a F a ka            ~ ~ ~ ~      T a A a A a k A - A 0 k A - A av       1 0 k I  H    1 T   v k k      ~ 1 0 k I     1 T P' H [A | a] [ A - av | a ] P ' k k    1 T  v  k k 27

Canonical Cameras Given F F matrix corresponds to P,P’ iff P’ T FP is skew-symmetric     T T X P' FPX 0, X F matrix, S skew-symmetric matrix   P [I | 0] P' [SF | e' ] (fund.matrix=F)       T T T T F S F 0 F S F 0     T [SF | e' ] F[I | 0]     T  0 0   e' F 0    Possible choice:   P [I | 0] P' [[e' ] F | e' ]  Canonical representation:    T λe' P [I | 0] P' [[e' ] F e' v | ]  28

The Essential Matrix ≡ fundamental matrix for calibrated cameras (remove K)     T E t R R[R t]      ˆ T ˆ   ˆ ˆ x ' E x 0 -1 -1 x K x; x ' K x'  T E K' FK 5 d.o.f. (3 for R; 2 for t up to scale) E is essential matrix if and only if two singularvalues are equal (and third=0) E  T Udiag(1,1, 0)V Given E, P=[I|0], there are 4 possible choices for the second camera matrix P’ 29

Four Possible Reconstructions from E 30 (only one solution where points is in front of both cameras)