Computation of the Fundamental Matrix F Shao-Yi Chien Department - - PowerPoint PPT Presentation

Two-View Geometry: Computation of the Fundamental Matrix F

簡韶逸 Shao-Yi Chien Department of Electrical Engineering National Taiwan University Fall 2019

1

Outline

• Computation of the fundamental matrix F

2

[Slides credit: Marc Pollefeys]

Fx x'T 

separate known from unknown

' ' ' ' ' '

33 32 31 23 22 21 13 12 11

         f yf xf f y yf y xf y f x yf x xf x

  

, , , , , , , , 1 , , , ' , ' , ' , ' , ' , '

T 33 32 31 23 22 21 13 12 11

 f f f f f f f f f y x y y y x y x y x x x

(data) (unknowns) (linear)

Af 

f 1 ' ' ' ' ' ' 1 ' ' ' ' ' '

1 1 1 1 1 1 1 1 1 1 1 1

        

n n n n n n n n n n n n

y x y y y x y x y x x x y x y y y x y x y x x x         

Epipolar Geometry: Basic Equation

F e'T  Fe  detF  2 F rank 

T 3 3 3 T 2 2 2 T 1 1 1 T 3 2 1

V σ U V σ U V σ U V σ σ σ U F            

SVD from linearly computed F matrix (rank 3)

T 2 2 2 T 1 1 1 T 2 1

V σ U V σ U V σ σ U F'           

F

F'

• F

min

Compute closest rank-2 approximation

The Singularity Constraint

Enforcing singularity!

Effect of enforcing singularity

f 1 ' ' ' ' ' ' 1 ' ' ' ' ' '

7 7 7 7 7 7 7 7 7 7 7 7 1 1 1 1 1 1 1 1 1 1 1 1

         y x y y y x y x y x x x y x y y y x y x y x x x         

 

T 9x9 7 1 7x7

V , , σ ,..., σ diag U A 

9x2 9 8

] V A[V  

 

T 8 T

] 000000010 [ V e.g.V 

1...7 0, )x λF F ( x

2 1 T

    i

i i

• ne parameter family of solutions

but F1+lF2 not automatically rank 2

The Minimum Case – 7 Point Correspondences

F1 F2 F F7pts

λ λ λ ) λF F det(

1 2 2 3 3 2 1

      a a a a

(obtain 1 or 3 solutions) (cubic equation)

The Minimum Case – Impose Rank 2

1 ´ ´ ´ ´ ´ ´ 1 ´ ´ ´ ´ ´ ´ 1 ´ ´ ´ ´ ´ ´

33 32 31 23 22 21 13 12 11 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1

                                         f f f f f f f f f y x y y y y x x x y x x y x y y y y x x x y x x y x y y y y x x x y x x

n n n n n n n n n n n n

        

~10000 ~10000 ~10000 ~10000 ~100 ~100 1 ~100 ~100

!

Orders of magnitude difference Between column of data matrix  least-squares yields poor results

NOT Normalized 8-point Algorithm

The Normalized 8-point Algorithm

Transform image to ~[-1,1]x[-1,1]

(0,0) (700,500) (700,0) (0,500) (1,-1) (0,0) (1,1) (-1,1) (-1,-1)

                  1 1 500 2 1 700 2

Least squares yields good results (Hartley, PAMI´97)

The Normalized 8-point Algorithm

Or nomalization with the previous method:

(0, 0) 2

The Normalized 8-point Algorithm

11

Geometric Distance

• Gold standard
• Sampson error
• Symmetric epipolar distance

Maximum Likelihood Estimation

   





i i i i i

d d

2 2

' x ˆ , x' x ˆ , x

(= least-squares for Gaussian noise)

x ˆ F ' x ˆ subject to

T



i

X t], | [M P' 0], | [I P  

Parameterize: Initialize: normalized 8-point, (P,P‘) from F, reconstruct Xi

i i i i

X P' x ˆ , PX x ˆ  

Minimize cost using Levenberg-Marquardt (preferably sparse LM, see book) (overparametrized)

Gold Standard

’

15

(Expensive Method)

 



 e

e

1 T T JJ



T T

JJ e e

(one eq./point JJT scalar)

Fx x'T   e    

   

2 2 2 1 2 2 T 2 1 T T

Fx Fx F x' F x' JJ    



T T

JJ e e

   

   



  

2 2 2 1 2 2 T 2 1 T T

Fx Fx F x' F x' Fx x'

(problem if some x is located at epipole) advantage: no subsidiary variables required where 𝐺𝑦𝑗 𝑘

2 represents the square of the j-th entry of the vector Fxi

First-order Geometric Error (Sampson Error)

   

   



           

2 2 2 1 2 2 T 2 1 T T

Fx Fx 1 F x' F x' 1 Fx x'

 

 





i i i i i

d F d

2 T 2

x' F , x x , x'

Symmetric Epipolar Error

Some experiments:

Some experiments:

Some experiments:

Some experiments:

 

 





i i i i i

d F d

2 T 2

x' F , x x , x'

(for all points!) Residual error:

Recommendations

• Do not use unnormalized algorithms
• Quick and easy to implement: 8-point

normalized

• Better: enforce rank-2 constraint during

minimization

• Best: Maximum Likelihood Estimation

(minimal parameterization, sparse implementation)

Automatic Computation of F

• 1. Interest points
• 2. Putative correspondences
• 3. RANSAC
• 4. Non-linear re-estimation of F
• 5. Guided matching

(repeat 4 and 5 until stable)

Automatic Computation of F

28

homogeneous edge corner

(e.g.Harris&Stephens´88; Shi&Tomasi´94)

Find points that differ as much as possible from all neighboring points

Interest Points

Interest Points

Select strongest features (e.g. 1000/image)

Interest Points

Evaluate NCC for all features with similar coordinates

Keep mutual best matches Still many wrong matches!

  

  

10 10 10 10

, , ´ ? e.g.

h h w w

y y x x y x      

?

0.96

• 0.40
• 0.16
• 0.39

0.19

• 0.05

0.75

• 0.47

0.51 0.72

• 0.18
• 0.39

0.73 0.15

• 0.75
• 0.27

0.49 0.16 0.79 0.21 0.08 0.50

• 0.45

0.28 0.99

1 5 2 4 3 1 5 2 4 3

Gives satisfying results for small image motions

Interest Points

RANSAC

Step 1. Extract features Step 2. Compute a set of potential matches Step 3. do

Step 3.1 select minimal sample (i.e. 7 matches) Step 3.2 compute solution(s) for F Step 3.3 determine inliers

until (#inliers,#samples)<95%

Step 4. Compute F based on all inliers Step 5. Look for additional matches by guided matching Step 6. Refine F based on all correct matches

(generate hypothesis) (verify hypothesis)



RANSAC

• Why choose 7-point algorithm instead of 8-point

algorithm?

• A rank 2 matrix is produced without enforcement
• The number of samples that must be tried in order to

ensure a high probability of the no outliers is exponential in the size of the sample set

• Distance measure
• Reprojection error
• Sampson approximation

34

   





i i i i i

d d

2 2

' x ˆ , x' x ˆ , x    

   



  

2 2 2 1 2 2 T 2 1 T T

Fx Fx F x' F x' Fx x'

restrict search range to neighborhood of epipolar line

(1.5 pixels)

relax disparity restriction (along epipolar line)

Guided Matching

geometric relations between two views is fully described by recovered 3x3 matrix F

37

500 corners 500 corners

38

188 matches 89 outliers 99 inliers 157 matches with guided matching

Degenerate Cases

• Degenerate cases
• Planar scene
• Pure rotation
• No unique solution
• Remaining DOF filled by noise
• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV´98, Kanatani, Akaike)
• Compare H and F according to expected residual error

(compensate for model complexity)

simplify stereo matching by warping the images Apply projective transformation so that epipolar lines correspond to horizontal scanlines e e map epipole e to (1,0,0) try to minimize image distortion problem when epipole in (or close to) the image

Image Pair Rectification

Planar Rectification

Bring two views to standard stereo setup (moves epipole to ) (not possible when in/close to image)

~ image size

(calibrated) Distortion minimization (uncalibrated) (standard approach)

Rectification

• Two steps:
• Mapping the epipolar to infinity
• Finding a projective transformation H of an image that maps

the epipole to a point at infinity

• Avoid distortion: better to have rigid transformation, to first-
• rder the neighborhood of x0 may undergo rotation and

translation only

• Matching transformation
• Match the epiplolar lines
• Find a match pair that

H-Tl=H’-Tl’

42

Mapping the Epipolar to Infinity

• For example, given epipole e=(f, 0, 1)T
• A good transform is
• For an arbitrary x0 and epipole e
• H=GRT: R: rotate to x-axis, T: translate to (f, 0, 1)T

43

=I if x=y=0

Matching Transformation

• Target: to minimize
• To solve a, b, c, minimize

44

Algorithm Outline

45

Polar re-parameterization around epipoles Requires only (oriented) epipolar geometry Preserve length of epipolar lines Choose  so that no pixels are compressed

• riginal image

rectified image

Polar Rectification

(Pollefeys et al. ICCV’99)

Works for all relative motions Guarantees minimal image size

Polar Rectification: Example

Polar Rectification: Example