C18 Computer Vision Lecture 7 Recovering 3D from two images I: - - PowerPoint PPT Presentation

Victor Adrian Prisacariu

http://www.robots.ox.ac.uk/~victor

C18 Computer Vision

Lecture 7

Recovering 3D from two images I: epipolar geometry

Computer Vision: This time…

• 5. Imaging geometry, camera calibration.
• 6. Salient feature detection and description.

7.

• 7. Recoveri

ring 3D from two im images I: I: epip ipola lar geometry ry.

1. Introduction 2. Epipolar Geometry 3. Algebraic Representation and the Fundamental Matrix 4. Computing the Fundamental Matrix 5. The Essential Matrix and Ego-Motion

• 8. Recovering 3D from two images II: stereo

correspondences, triangulation, neural nets.

7.1 Introduction: Forward and Inverse Mappings

• Geometric image entities back-proje

ject to entities of higher dimensionality in the scene.

• So points in the image back-project to lin

lines in the scene, lines to planes, etc.

Introduction: What do single-view ambiguities tell us?

• Sing

ngle le vie views ar are e NOT T suf suffic icie ient to solve geometric problem in data- driven vision.

• We need methods of understanding mult

ultip iple vie views.

– Shape-from-stereo: different cameras, different viewpoints, same time. – Structure-from-motion: same camera, different viewpoints, different times.

• Note that single views may be sufficient to solve geometric problems

in model (easy) or data driven vision (just recently) vision.

Reconstruction from monocular views

Introduction: Reconstruction from two views

In principle, recovering 3D structure is straightforward: Fin Find a a bit bit of

• f the

the tw two

• sce

scenes tha that is s ob

• bservable by tw

two

• or
• r mor
• re

e cameras, and and bac backproject the the tw two

• rays to
• fi

find the their ir intersectio ion in n the the worl

• rld.

There are three things to cover: 1. Understanding the geometry – epip ipolar geometry ry. 2. Determining which points in the images are from the same scene location – th the e cor

• rresponden

ence problem. 3. Determining the 3D structure by back-projecting rays – tri triangu gulation.

Simple case: two identical parallel cameras

• Assume:

– Some 3D world. – Two views. – Two identical cameras – same camera params (f). – Separation only on x: 𝑢𝑦

• 𝑍

𝐷1 = 𝑍 𝐷2, 𝑎𝐷1 = 𝑎𝐷2, 𝑌𝐷1 = 𝑌𝐷2 + 𝑢𝑦

• 𝑦𝐷1 = 𝑔𝑌𝐷1/𝑎 and 𝑦𝐷2 = 𝑔𝑌𝐷2/𝑎 so

1 𝑎 = 1 𝑔𝑢𝑦 (𝑦𝐷1 − 𝑦𝐷2)

• Recip

eciprocal dep epth th 1/Z is proportional to the hori

• rizontal

l dis isparity 𝑦𝐷1 − 𝑦𝐷2.

View 1 View 2 𝑢𝑦 𝐷1 𝐷2 3D World

Simple case: two identical parallel cameras

൞ 𝑦𝐷2 = 𝑔 𝑎 (𝑌 + 𝑢𝑦) 𝑧𝐷2 = 𝑔 𝑎 𝑍 ⇒ 𝑦𝐷2 = 𝑦 + 𝑔𝑢𝑦 𝑎 𝑧𝐷2 = 𝑔 𝑎 𝑍

View 1 View 2 𝐷1 𝐷2 3D World 𝑢𝑦 𝑎 ranges from 0 to ∞ All possible matches lie on a straight lin ine. Finding a correspondence is a 1D 1D sear search.

7.2 Arbitrary case: Epipolar Geometry

• The locus of the matches is the projection onto one view of the

backprojected ray in the other one.

• This is al

always a a str traig ight line, and is called the ep epip ipola lar line, which we label with 𝐦’.

• As 𝐘 moves along the ray, the other ray sweeps across the ep

epip ipolar pla plane.

• The intersection of the epipolar plane with the image plane is the

epipolar line.

Arbitrary case: Epipolar Geometry

• If 𝐘 is moved, the entire plane moves too, generating a new epipolar

line.

• Epipolar planes hinge about the camera bas

baseli line, forming a pencil of planes.

• All epipolar lines in C’ meet at its ep

epip ipole le 𝐟’, where the baseline pierces the image plane.

• The epipole point is the pr

proje

• jectio

ion of

• f the

the op

• ptic

ical l cen center of camera C into C’.

Epipolar geometry examples

1. Converging cameras Notice that the epipoles must often lie off the physical image planes. What would be a quick test you could carry out to see whether the epipole was on the image plane?

Epipolar geometry examples

2. (Close to) parallel cameras Epipolar geometry depends only on the relative post of the cameras (i.e. the rotation and translation between them) and

• n the camera’s intrinsic parameters.

It It does not t depend on th the sc scene str truct ctrure. Can you reason qualitatively why not?

7.3 Algebraic representation and the F matrix

Before formulating the algebraic representation we must cover three things:

• 1. How homogeneous notation handles points at infinity.
• 2. How homogeneous notation can be generalised to

lines.

You must notice a duality between lines and points …

• 3. How vector product can be represented using

matrices.

1> Points at ∞ in homogeneous coordinates

• A line of points in 3D through the point 𝐁 with direction 𝐄 is:

𝐘 = 𝐁 + μ𝐄

• In homogeneous notation this becomes:

𝑌1(𝜈) 𝑌2(𝜈) 𝑌3(𝜈) 𝑌4(𝜈) = 𝐁 + 𝜈𝐄 1 = 𝐁 1 + 𝜈 𝐄 0 = 1 𝜈 𝐁 1 + 𝐄 0 .

• As 𝜈 → ∞, the point on the line becomes 𝐄

0 .

• So homogeneous vectors with 𝒀𝟓=0 represent points at “infinity”.

– Points at infinity are equivalent to directions. – Parallel lines in the scene meet at the same point.

1> Points at ∞ in homogeneous coordinates

• The projection of a point at ∞ into the image is its vanishing

poin int.

• To find it, simply project the point-at-∞ into the image:

𝐰 = 𝑳 𝑺 𝐮 𝐄

1> Points at ∞ in homogeneous coordinates

2> Homogeneous notation for lines

• In 2D, a point 𝑦, 𝑧 T is represented in

homogeneous coordinates as 𝐲 = 𝑦1, 𝑦2, 𝑦3 T.

• where 𝑦 = 𝑦1/𝑦3 and 𝑧 = 𝑧1/𝑧3.
• The lin

line l1𝑦 + l2y + l3 = 0 in 2D is represented by the homogeneous 3-vector: 𝐦 = 𝑚1, 𝑚2, 𝑚3 T

• Any point 𝐲 on the line 𝐦 has:

𝐉T𝐲 = 𝐲T𝐉 = 0

2> Homogeneous notation for lines

Reminders:

• A point 𝐲 on the line 𝐦 has 𝐦T𝐲 = 0, 𝐲T𝐦 = 0 and

𝐦 ∙ 𝐲 = 0.

• The line through two points 𝐪 and 𝐫 is given by:

𝐦 = 𝐪 × 𝐫

• The intersection of two lines is the point:

𝐲 = 𝐦1 × 𝐦2

3> Matrix representation of vector products

The vector product 𝐛 × 𝐜 is: Ƹ 𝐣 Ƹ 𝐤 መ 𝐥 𝑏1 𝑏2 𝑏3 𝑐1 𝑐2 𝑐3 = 𝑏2𝑐3 − 𝑏3𝑐2 𝑏3𝑐1 − 𝑏1𝑐3 𝑏1𝑐2 − 𝑏2𝑐1 = −𝑏3 𝑏2 𝑏3 −𝑏1 −𝑏2 𝑏1 𝑐1 𝑐2 𝑐3 = 𝒃 𝑦𝐜 𝒃 𝒚 is a 3 × 3 skew-symmetric matrix and has rank=2 𝒃 is the kernel of 𝒃 𝒚𝐜 Example: compute the vector product of I=(1,2,3) and m=(2,3,4). Pseudo-determinant method gives (-1, 2, 1)T Skew-sym method gives: −3 2 3 −1 −2 1 2 3 4 =

Algebraic representation of Epipolar Geometry

• We now know that epipolar geometry defines the mapping

fr from poin int t 𝐲 to lin line 𝐦’.

• The mapping depends only on the cameras, not on the

structure, so the mapping depends on th the overall proje jection matr trices P and P’.

• We will next show that this mapping is linear, and can be

written as: 𝑱’ = 𝑮𝒚, , where 𝑮 is is th the fu fundamental matr trix

Algebraic representation of Epipolar Geometry

• We use the first camera C to define the world

coordinate system, so the overall projection matrix is: 𝑸 = 𝑳[𝑱|𝟏]

• We then define the rotation and translation

between camera frames as: 𝐘′ = 𝑺 𝐮 𝟏T 1 𝐘

• So that:

𝑸′ = 𝑳′[𝑺|𝐮]

• Note that the camera intrinsics 𝑳 and 𝑳’ can be

different.

Algebraic representation of Epipolar Geometry

• Step

ep 1: back project a ray from C.

• The back-projected 𝐘(𝜂) satisfies

𝐲 = 𝑳 𝑱 𝟏 𝐘 𝜂

• where we use 𝜂 as a parameter.
• Now:

𝑦 = 𝑦 𝑧 1 = 𝑳 𝑱 𝟏 X 𝑍 𝑎 1 = 𝑳 𝑌 𝑍 𝑎 ⇒ ⇒ 𝐘 𝜂 = 𝜂[𝑳]−1𝐲 1 and 𝐘 ∞ = 𝑳−1𝐲

In effect, 𝑳 −1 corrects the direction of the ray. Direction 𝐲

• we be incorrect, because

𝐲 was measured in a non-ideal camera.

Algebraic representation of Epipolar Geometry

• Step

ep 2: Chose two points on the ray and project into second camera C’.

– Point 1 with 𝜂 = 0 is the optical center 𝟏 1 . – Point 2 with 𝜂 = ∞ is the point at infinity 𝑳−1 .

• Projecting into C’ we get:
• Point 1:

𝐟′ = 𝑳′ 𝑺 𝐮 𝟏 1 = 𝑳′𝐮.

• Point 2:

𝐫 = 𝑳′ 𝑺 𝐮 𝑳−𝟐 = 𝑳′𝑺𝑳−1𝐲.

• Next we tidy up:

– Use 𝑵𝐛 × 𝑵𝐜 = 𝑵−T(𝐛 × 𝐜), where 𝑵−T = 𝑵−1 T – Use the matrix representation of a vector product i.e. the matrix 𝐮 𝑦

• So 𝐦′ = 𝑳′ −T 𝐮 × 𝑺𝑳−𝟐𝐲 = 𝑳′ −𝐔 𝐮 𝐲𝑺𝑳−1𝐲
• 𝑮 = 𝑳′ −𝐔 𝐮 𝐲𝑺𝑳−1 is known as the fu

fundamental l matrix, with the epipole line 𝐦′ = 𝑮𝐲 and the property that 𝐲′𝑮𝐲 = 0

Algebraic representation of Epipolar Geometry

• Step 3: Use vector product

to find the epipolar line 𝐦′ = 𝑳′𝐮 × 𝑳′𝑺𝑳−1𝐲

Summary of Properties

• 𝑮 is a rank 2 matrix with 7 DoF
• det 𝑮 = 0
• If 𝐲 ↔ 𝐲′ then 𝐲′T𝑮𝐲 = 0
• The epipolar lines in 𝐷 and 𝐷′ are obtained from 𝑮𝐟 = 𝟏

and 𝑮T𝐟’ = 𝟏.

• For 𝑸 = 𝑳[𝑱|𝟏] and 𝑸′ = 𝑳′[𝑺|𝐮] the fundamental matrix

is derived as F = 𝑳′ −𝐔 𝐮 𝐲𝑺𝑳−1𝐲 where – T denotes the transpose of the inverse.

7.4 Computing F: Algebraic minimizations

The basis for several methods of computing 𝑮 lies in re-writing the constraint 𝐲′T𝑮𝐲 = 0 for each match 𝐲 ↔ 𝐲′ as:

𝑦′ 𝑧′ 1 𝐺

1

𝐺

2

𝐺

3

𝐺

4

𝐺

5

𝐺

6

𝐺

7

𝐺

8

𝐺

9

𝑦 𝑧 1 = 0

𝑦′𝑦𝐺

1 + 𝑦′𝑧𝐺2 + 𝑦′𝐺3 + 𝑧′𝑦𝐺 4 + 𝑧′𝑧𝐺5 + 𝑧′𝐺6 + 𝑦𝐺7 + 𝑧𝐺8 + 𝐺9 = 0

𝑦′𝑦 𝑦′𝑧 𝑦′ 𝑧′𝑦 𝑧′𝑧 𝑧′ 𝑦 𝑧 1 𝐺

11

⋮ 𝐺

33

= 0

Computing F: Algebraic minimizations

• For n correspondences we build up the 𝑜 × 9 system.

𝑩𝑜×9𝐠 = 𝑦1

′𝑦1

𝑦1

′𝑧1

𝑦1

′

𝑧1

′𝑦1

𝑧1

′𝑧1

𝑧1

′

𝑦1 𝑧1 1 ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ 𝑦𝑜

′ 𝑦𝑜

𝑦𝑜

′ 𝑧𝑜

𝑦𝑜

′

𝑧𝑜

′𝑦𝑜

𝑧𝑜

′𝑧𝑜

𝑧𝑜

′

𝑦𝑜 𝑧𝑜 1 𝐺

1

⋮ 𝐺

9

• 𝑮 has 7 degrees of freedom, so minimal solution needs 7-

poin ints ts: 7 point algorithm – nonlinear equations.

• Robust solutions will need 8+ poin

ints: 8 point algorithm – linear solution.

7-Point Algorithm

• With 7 entries, matrix 𝑩7×9 has rank 7 and a two

dimensional null-space.

• Let vectors 𝐰 and 𝐱 be two vectors spanning the null-

space of A.

• Every 𝐠 = 𝛽𝐰 + 𝛾𝐱 satisfies 𝑩𝐠 = 𝟏, but we need to

find an 𝐠 that gives det 𝑮 = 0.

• We don’t need to consider every 𝛽 and 𝛾, since for any

𝐠 which is a solution, any scaling (+ve of –ve) of 𝐠 is also a solution.

• We only need to write 𝐠 = 𝛽𝐰 + 1 − 𝛽 𝐱, where 𝛽

runs from −∞ to +∞.

7-Point Algorithm

• We want to find an 𝛽 along that path that makes det 𝑮 = 0.
• det 𝑮 = 0 ⇒ det 𝑮 = 𝑏0 + 𝑏1𝛽 + 𝑏2𝛽2 + 𝑏3𝛽3 = 0
• There are 3 solutions, so the algorithm goes like:

– Generate the matches. – Statistically center all sets of 𝑦 and 𝑧 values. – Build 𝑩 from seven of the matches. – Use SVD to find the two vectors 𝐰 and 𝐱 spanning the null- space. – Use 𝐰 and 𝐱 to find coeffs of the cubic. – Solve the cubic, and test which 𝛽 is best.

Skeleton Matlab Code

Cubic coefficients …

• Writing 𝐞 = 𝐰 − 𝐱, then 𝐠 = 𝛽𝐞 + 𝐱, and

det 𝑮 = 𝛽𝑒1 + 𝑥1 𝛽𝑒2 + 𝑥2 𝛽𝑒3 + 𝑥3 𝛽𝑒4 + 𝑥4 𝛽𝑒5 + 𝑥5 𝛽𝑒6 + 𝑥6 𝛽𝑒7 + 𝑥7 𝛽𝑒8 + 𝑥8 𝛽𝑒9 + 𝑥9

• Lots of fun later you can write 𝑏1…4 as a

function of 𝐞 and 𝐱.

• In Matlab it looks like:

Computing F: Algebraic minimizations

• For n correspondences we build up the 𝑜 × 9 system.

𝑩𝑜×9𝐠 = 𝑦1

′𝑦1

𝑦1

′𝑧1

𝑦1

′

𝑧1

′𝑦1

𝑧1

′𝑧1

𝑧1

′

𝑦1 𝑧1 1 ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ 𝑦𝑜

′ 𝑦𝑜

𝑦𝑜

′ 𝑧𝑜

𝑦𝑜

′

𝑧𝑜

′𝑦𝑜

𝑧𝑜

′𝑧𝑜

𝑧𝑜

′

𝑦𝑜 𝑧𝑜 1 𝐺

1

⋮ 𝐺

9

• 𝑮 has 7 degrees of freedom, so minimal solution needs 7-

poin ints ts.

• Robust solutions will need 8+ poin

ints.

8-Point Algorithm

• For 𝑜 = 8 points, the 𝐆 can be found as the

null-space of 𝐵, and so 𝑮 is determined up to a scale (as expected).

• Since points are noisy, one would prefer to use

𝑜 > 8. This is done using le least squares.

A least squares version of the 8-point algorithm

Due to noise, there will not be an exact solution to 𝑩𝐆 = 𝟏 (𝑩 has full rank) Least square formulation:

Find the unit vector 𝐆 that minimizes the norm of the residual 𝐬 = 𝑩𝐠: 𝐆∗ = argmin𝐆: 𝐆 =1 𝑩𝐆 𝟑

Solution with eigenvalues:

Compute the eig eigen-decomposit ition of the matrix 𝐍 = 𝑩𝑈𝑩 and set 𝐆 to the (unit) eigenvector 𝐟1 corresponding to the smallest eigenvalue 𝛽1.

Solution with SVD:

Compute the SVD of the matrix 𝑩 and set 𝐆 to the (unit) right singular vector 𝐟1 corresponding to the smallest singular value 𝜏1.

Proof of eigen-decomposition solution

• The squared sum of the residuals 𝐬 = 𝑩𝐠 is 𝐬 2 = 𝐬T𝐬 = 𝐆T𝑩T𝑩𝐆 =

𝐆T𝐍𝐠.

• 𝑵 = 𝑩T𝑩 is a 𝑜 × 𝑜 symmetric real matrix, hence it can be decomposed

as: 𝑵 = 𝑾𝜧𝑾T = 𝑾 𝜇1 ⋱ 𝜇𝑜 𝑾T = ෍

𝑗=1 𝑜

𝜇𝑗[𝐟i𝐟i

T]

• 𝑾 = 𝐟1

… 𝐟𝑜 is the orthonormal matrix of eigenvectors and 0 ≤ 𝜇1 ≤ … ≤ 𝜇𝑜 are non-decreasing eigenvalues.

• The eigenvalues are non-negative because

𝑵𝐟i = 𝐟i𝝁i ⇒ 𝐟i

T𝑵𝐟i = 𝐟i T𝐟i𝜇𝑗 ⇒ 𝑩𝐟i T 𝑩𝐟i = 𝜇𝑗 > 0

• Therefore:

: 𝐆T𝐍𝐠 = 𝜇𝑗 𝑮T𝐟1

2 + ⋯ + 𝜇𝑜 𝑮T𝐟n 2

This is minimised when 𝐆 = 𝐟1.

Proof of SVD solution

• Any 𝑛 × 𝑜 matrix 𝑩 where 𝑛 ≥ 𝑜 can be decomposed as:

𝑩𝑛×𝑜 = 𝑽𝑛×𝑜 𝜏1 ⋱ 𝜏𝑜 𝑜×𝑜 𝑾𝑜×𝑜

T

• 𝑽 is column-orthogonal, 𝑾 is fully orthogonal and 𝜯 contains

the singular values ordered so 0 ≤ 𝜏1 ≤ … ≤ 𝜏𝑜

• The singular vectors 𝑾 of 𝑩 are the same as the eigenvectors
• f 𝑵 = 𝑩T𝑩 = 𝑾𝜯T𝑽T𝜯𝑾T = 𝑾𝜯𝑾T.
• In particular 𝐠 = 𝐟1 is the first column of 𝑾.
• The SVD is

is usu sually preferred to the eigenvalue decomposition because it is numerically more stable.

7.5 A note on the Essential Matrix

• The fundamental matrix 𝑮 = 𝑳′ −T 𝐮 𝐲𝑺𝑳−1

requires 𝐲′T𝑮𝐲 = 0.

• If however we know the in

intrinsic cali libration, we can transform the matching points into their respective ideal images.

• Points in the ideal images are related by 𝐲′T𝑭𝐲 =

0, where 𝑭 = 𝐮 𝑦𝑺 is the Essential Matrix.

7.5 Determining the Ego-Motion in 𝑮

We know that 𝑮 = 𝑳′ −T 𝐮 𝐲𝑺𝑳−1. We now show how to recover 𝑺 and 𝐮 from 𝑮 (given 𝑳 and 𝑳′). 1. Compute the essential matrix 𝑭 = 𝐮 𝑦𝑺 = 𝑳′𝑮𝑳. 2. Compute 𝐮 as the null-vector of 𝑭T (i.e. 𝑭T𝐮 = 0).

– We can only determine 𝐮 up to a scaling factor 𝜈. – there are two solutions ±𝜈𝐮.

3. Compute 𝑺 from 𝑭.

– the algorithm for this step is given later. – it returns two solutions 𝑺1 and 𝑺2.

4. Overall, there are four solutions for the projection matrix: 𝑸′ = 𝑳′[𝑺1|𝜈𝒖] 𝑸′ = 𝑳′[𝑺2|𝜈𝐮] 𝑸′ = 𝑳′ 𝑺1 − 𝜈𝐮] 𝑸′ = 𝑳′[𝑺2| − 𝜈𝐮] 5. Exclude 3 of these using a visibility test.

The four solutions

• The 3D poi

point t is s in n fr fron

• nt of
• f bot

both cam ameras in n only

• nly one
• ne case.
• Note these are “computer vision” cameras, so to be visible a ray

must pass through the image on its way to the optic centre!

Computing 𝑺1,2 from the essential matrix 𝑭

Recall that 𝑭 = 𝐮 𝑦𝑺; we now recover 𝑺 from 𝑭. Algorithm:

• 1. Compute the Singular Value Decomposition

(SVD) of 𝑭. 𝑽 1 1 𝑾T ← 𝑵

• 2. Set 𝑿 =

−1 1 1 .

• 3. The two solutions are 𝑺1 = 𝑽𝑿𝑾⊤, 𝑺2 =

𝑽𝑿⊤𝑾⊤.

Summary of Lecture 7

Introduction Epipolar Geometry

• Clues from Parallel Cameras; the Epipolar Plane and lines; Cat’s

whiskers for converging cameras. Algebraic Representation and the Fundamental Matrix

• Representing the epipolar plane; linear relationship between point

in one image and epipolar line in the other Computing the F matrix

• The 7-point algorithm: a minimal nonlinear method.
• The 8-point algorithm: least squares, with eigen-decomposition or

SVD. The Essential Matrix and the ego-motion from 𝑮.