Lecture 7 Measurement Using a Single Camera Lin ZHANG, PhD School - - PowerPoint PPT Presentation

Lin ZHANG, SSE, 2016

Lecture 7 Measurement Using a Single Camera

Lin ZHANG, PhD School of Software Engineering Tongji University Fall 2016

Lin ZHANG, SSE, 2016

If I have an image containing a coin, can you tell me the diameter

• f that coin?

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Vector representation

{ , , } a xi y j zk x y z        

Length (or norm) of a vector

2 2 2

a x y z    

Normalized vector (unit vector)

{ , , } a x y z a a a a      

We say if and only if

, a  0  0, 0, x y z   

Vector operations

Lin ZHANG, SSE, 2016

if

1 1 1

( , , ), a x y z  

2 2 2

( , , ), b x y z  

1 2 1 2 1 2

( , , ), a b x x y y z z       

then

1 2 1 2 1 2

cos a b a b x x y y z z          

Dot product (inner product) Laws of dot product:

, ( )= + a b b a a b c a b a c                  

Theorem

a b a b        

(why?)

a  b  

Vector operations

Lin ZHANG, SSE, 2016

Cross product

1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2

i j k y z z x x y a b x y z i j k y z z x x y x y z       

Vector operations

Lin ZHANG, SSE, 2016

c a b     

Cross product is also a vector, whose direction is determined by the right‐hand law and

a  b  c  , c a c b       sin c a b      

represents the oriented area of the parallelogram taking and as two sides

c  a  b 

(easy to prove)

1 2 2 1

r r r r            

(why?)

Vector operations

Lin ZHANG, SSE, 2016

they are not equal to zero at the same time, and Cross product Theorem

|| a b a b       

(why?) Theorem

|| , , a b       a b     0  

1 2 3 1 2 1 3

( ) r r r r r r r                    

Property (easy to understand)

Vector operations

Lin ZHANG, SSE, 2016

Mixed product (scalar triple product or box product) Geometric Interpretation: it is the (signed) volume of the

parallelepiped defined by the three vectors given

1 2 3 1 2 3 1 2 3

( , , ) ( ) a a a b b b c c c     a b c a b c

Vector operations

Lin ZHANG, SSE, 2016

Mixed product (scalar triple product or box product)

1 2 3 1 2 3 1 2 3

( , , ) ( ) a a a b b b c c c     a b c a b c ( ) cos sin cos          a b c a b c a b c

a c

Base h

Vector operations

Lin ZHANG, SSE, 2016

Mixed product (scalar triple product or box product)

1 2 3 1 2 3 1 2 3

( , , ) ( ) a a a b b b c c c     a b c a b c

Property:

( , , ) ( , , ) ( , , )   a b c b c a c a b ( , , ) ( , , ) ( , , )     a b c b a c a c b

Vector operations

why?

Lin ZHANG, SSE, 2016

Mixed product (scalar triple product or box product) Theorem

, , a b c are coplanar ( , , )   a b c , , a b c are coplanar , , , v    

they are not equal to zero at the same time, and

v      a b c

Vector operations

why?

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

For a normal point on a plane

 

, ,

T

x y 

Its homogenous coordinate is where k can be any non‐zero real number

 

, ,1 ,

T

k x y

For a homogenous coordinate Homogenous coordinate for a point is not only one

 

' ' '

, ,

T

x y z

we usually rewrite it as 



' ' ' '

/ , / ,1

T

x z y z

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Converting from homogenous coordinate to inhomogeneous coordinate

' ' ' ' ' ' '

x x z y y z z                       

For a normal point on a plane



Its homogenous coordinate is where k can be any non‐zero real number

 

, ,1 ,

T

k x y

 

, ,

T

x y

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Geometric interpretation

1

e

2

e

• 1
• 1



( , ) M x y

3

e

1

e

2

e

In plane , in the 2D frame



1 1 2

( : , )

• e e

:( , ) M x y

Coordinate of any point on line OM in the frame is the homogeneous coordinate of M

1 2 3

( : , , )

• e e e

These points can be represented as

 

, ,1

T

k x y

, one point

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Geometric interpretation

1

e

2

e

• 1
• 1



( , ) M x y

3

e

1

e

2

e

How about a line passing through O and parallel to ?



In plane , in the 2D frame



1 1 2

( : , )

• e e

:( , ) M x y

Coordinate of any point on line OM in the frame is the homogeneous coordinate of M , one point These points can be represented as

 

, ,1

T

k x y

1 2 3

( : , , )

• e e e

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Geometric interpretation

1

e

2

e

• 1
• 1



3

e

1

e

2

e

How about a line passing through O and parallel to ?



Consider a line passing through O and M(x0, y0, 0)T

( , ) M x y

We define: it meets at an infinity point, and also the homogeneous coordinate of such a point can be represented as points on OM



So, infinity point has the form (kx0, ky0, 0)T

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Normal case:

line k (x0, y0, 1) a normal point (x0, y0) on the plane



The homogeneous coordinate of this normal point is k(x0, y0 ,1)

abnormal case:

line (kx0, ky0, 0) Define: it meets at an infinity point



The homogeneous coordinate of this infinity point is k(x0, y0 ,0) make an analogy

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• What is homogeneous coordinate?

Geometric interpretation

1

e

2

e

• 1
• 1



3

e

1

e

2

e

How about a line passing through O and parallel to ?



( , ) M x y

One infinity point determines an

• rientation

We define: all infinity points on comprise an infinity line



In fact, plane meets at the infinity line



1 2

• e e

Homogeneous equation of the infinity line is 0x + 0y + 1z = 0

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

Properties of a projective plane

• Two points determine a line; two lines determine a point (the

second claim is not correct in the normal Euclidean plane)

• Two parallel lines intersect at an infinity point; that means one

infinity point corresponds to a specific orientation

• Two parallel planes intersect at the infinity line

 + infinity line = Projective plane

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

JingHu High‐speed railway: rails will “meet” at the vanishing point

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

On a projective plane, please determine the line passing two points

1

e

2

e

• 1



M

'

,

• x

x determine two lines

xx’ actually is the intersection between oxx’ and



' 1 1 1 2 2 2

( , , ) , ( , , )

T T

x y z x y z   x x

x

'

x

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

1

e

2

e

• 1



x

'

x M

Thus, locates on xx’

( , , ) M x y z  oM resides on the plane oxx’ 

'

, ,

• M o

x x are coplanar 

1 1 1 2 2 2

x y z x y z x y z 

On a projective plane, please determine the line passing two points

' 1 1 1 2 2 2

( , , ) , ( , , )

T T

x y z x y z   x x

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

1 1 1 2 2 2

x y z x y z x y z 



1 1 1 1 1 1 2 2 2 2 2 2

y z z x x y x y z y z z x x y   

1 1 1 1 1 1 2 2 2 2 2 2

, ,

T

y z z x x y y z z x x y      

Homogeneous coordinate of the line Homogeneous coordinate of the infinity line is (0,0,1)T On a projective plane, please determine the line passing two points

' 1 1 1 2 2 2

( , , ) , ( , , )

T T

x y z x y z   x x

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

1 1 1 2 2 2

x y z x y z x y z 



1 1 1 1 1 1 2 2 2 2 2 2

y z z x x y x y z y z z x x y   

Theorem On the projective plane, the line passing two points is

'

, x x

'

  l x x

1 1 1 1 1 1 2 2 2 2 2 2

, ,

T

y z z x x y y z z x x y      

On a projective plane, please determine the line passing two points

' 1 1 1 2 2 2

( , , ) , ( , , )

T T

x y z x y z   x x

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

A point is on the line

( , , )T x y z  x 

T 

x l

( , , )T a b c  l

  x l

(It is )

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

Theorem: On the projective plane, the intersection of two lines is the point

'

  x l l

'

, l l

Proof: Two lines

1 1 1

0, a x b y c z   

2 2 2

a x b y c z   

' 1 1 1 2 2 2

( , , ) , ( , , )

T T

a b c a b c   l l

Inhomogeneous form

, x y X Y z z        

1 1 1 2 2 2

a X bY c a X b Y c         

1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 2

, c b a c c b a c X Y a b a b a b a b      

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

1 1 1 1 2 2 2 2 1 1 1 1 2 2 2 2

, ,1 c b a c c b a c k a b a b a b a b                    x

Homogenous form of the cross point is

1 1 1 1 1 1 2 2 2 2 2 2

, , c b a c a b c b a c a b            x

let

1 1 2 2

a b k a b 



1 1 1 1 1 1 2 2 2 2 2 2

, , b c c a a b b c c a a b        x

'

  x l l

Theorem: On the projective plane, the intersection of two lines is the point

'

  x l l

'

, l l

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

Example: find the cross point of the lines

1, 1 x y  

• x

y

1 x  1 y 

1 2 3 1 2 3

1 ( 1) 1 ( 1) x x x x x x           

Homogeneous coordinates of the two lines are (1,0, 1) ,(0,1, 1)

T T

 

Cross point is

 

(1,0, 1) (0,1, 1) 1,1,1

T T

   

Homogeneous form

Theorem: On the projective plane, the intersection of two lines is the point

'

  x l l

'

, l l

1 2 3 3

, x x x y x x        

Lin ZHANG, SSE, 2016

 

(1,0, 1) (1,0, 2) 1 0 1 0,1,0 1 0 2

T T

i j k       

Foundations of Projective Geometry

• Lines in the homogeneous coordinate

Example: find the cross point of the lines

1, 2 x x  

• x

y

1 x 

Homogeneous coordinates of the two lines are (1,0, 1) ,(1,0, 2)

T T

 

Cross point is

2 x 

1 2 3 1 2 3

1 ( 1) 1 ( 2) x x x x x x           

Homogeneous form

Theorem: On the projective plane, the intersection of two lines is the point

'

  x l l

'

, l l

1 2 3 3

, x x x y x x        

Lin ZHANG, SSE, 2016

Foundations of Projective Geometry

• Duality

In projective geometry, lines and points can swap their positions

T 

x l

If x is a variable, it represents the points lying on the line l; If l is a variable, it represents the lines passing a fixed point x

How to interpret?

The line passing two points is

'

, x x

'

  l x x

The cross point of two lines is

'

, l l

'

  x l l

Duality Principle: To any theorem of projective geometry, there corresponds a dual theorem, which may be derived by interchanging the roles of points and lines in the original theorem

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Matrix differentiation

• Function is a vector and the variable is a scalar

 

1 2

( ) ( ), ( ),..., ( )

T n

f t f t f t f t 

Definition

1 2

( ) ( ) ( ) , ,...,

T n

df t df t df t df dt dt dt dt       

Lin ZHANG, SSE, 2016

• Function is a matrix and the variable is a scalar

11 12 1 21 22 2 1 2

( ) ( ),..., ( ) ( ) ( ),..., ( ) ( ) ( ) ( ) ( ),..., ( )

m m ij n m n n nm

f t f t f t f t f t f t f t f t f t f t f t



                  

Definition

1 11 12 2 21 22 1 2

( ) ( ) ( ) ,..., ( ) ( ) ( ) ( ) ,..., ( ) ( ) ( ) ,...,

m m ij n m n n nm

df t df t df t dt dt dt df t df t df t df t df dt dt dt dt dt df t df t df t dt dt dt



                            

Matrix differentiation

Lin ZHANG, SSE, 2016

• Function is a scalar and the variable is a vector

1 2

( ), ( , ,..., )T

n

f x x x  x x

Definition

1 2

, ,...,

T n

df f f f d x x x              x

In a similar way,

1 2

( ), ( , ,..., )

n

f x x x  x x

1 2

, ,...,

n

df f f f d x x x              x

Matrix differentiation

Lin ZHANG, SSE, 2016

• Function is a vector and the variable is a vector

   

1 2 1 2

, ,..., , ( ), ( ),..., ( )

T T n m

x x x y y y   x y x x x

Definition

1 1 1 1 2 2 2 2 1 2 1 2

( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,...,

n n T m m m n m n

y y y x x x y y y d x x x d y y y x x x



                                         x x x x x x y x x x x 

Matrix differentiation

Lin ZHANG, SSE, 2016

• Function is a vector and the variable is a vector

   

1 2 1 2

, ,..., , ( ), ( ),..., ( )

T T n m

x x x y y y   x y x x x

In a similar way,

1 2 1 1 1 1 2 2 2 2 1 2

( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,..., ( ) ( ) ( ) , ,...,

m m T m n n n n m

y y y x x x y y y d x x x d y y y x x x



                                         x x x x x x y x x x x 

Matrix differentiation

Lin ZHANG, SSE, 2016

• Function is a vector and the variable is a vector

Example:

1 1 2 2 2 1 1 2 2 3 2 2 3

( ) , , ( ) , ( ) 3 ( ) x y x y x x y x x y x                       x y x x x x

1 2 1 1 1 1 2 2 2 3 1 2 3 3

( ) ( ) 2 ( ) ( ) 1 3 2 ( ) ( )

T

y y x x x d y y d x x x y y x x                                            x x y x x x x x

Matrix differentiation

Lin ZHANG, SSE, 2016

• Function is a scalar and the variable is a matrix

11 12 1 1 2

( )

n m m mn

f f f x x x df d f f f x x x                              X X   

( ),

m n

f



 X X  Definition

Matrix differentiation

Lin ZHANG, SSE, 2016

• Useful results

1

,

n

 x a  ,

T T

d d d d   a x x a a a x x Then, How to prove? (1)

Matrix differentiation

Lin ZHANG, SSE, 2016

• Useful results

1

,

m n n

A

 

  x   (2) Then,

T

dA A d  x x

1

,

m n n

A

 

  x   (3) Then,

T T T

d A A d  x x

1

,

n n n

A

 

  x   (4) Then, ( )

T T

d A A A d   x x x x

1 1

, ,

m n m n   

   X a b    (5) Then,

T T

d d  a Xb ab X

1 1

, ,

n m m n   

   X a b    (6) Then,

T T T

d d  a X b ba X

1 n

 x  (7) Then, 2

T

d d  x x x x

Matrix differentiation

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• What is Camera Calibration
• Single Camera Calibration
• Bird‐view Generation

Lin ZHANG, SSE, 2016

Lagrange multiplier

• Single‐variable function

( ) f x is differentiable in (a, b). At , f(x) achieves an

extremum

( , ) x a b 

|

x

df dx 

• Two‐variables function

( , ) f x y is differentiable in its domain. At , f(x, y) achieves

an extremum

( , ) x y

( , ) ( , )

| 0, |

x y x y

f f x y      

Lin ZHANG, SSE, 2016

Lagrange multiplier

• In general case

1

( ),

n

f



 x x 

If is a stationary point of

x

1 2

| 0, | 0,..., |

n

f f f x x x         

x x x

Lin ZHANG, SSE, 2016

Lagrange multiplier

• Lagrange multiplier is a strategy for finding the local

extremum of a function subject to equality constraints

Problem: find stationary points for

1

( ), x x

n

y f



 

under m constraints

( ) 0, 1,2,...,

k

g k m   x

is a stationary point of with constraints Solution:

1 1

( ; ,..., ) ( ) ( )

m m k k k

F f g   



  x x x

If is a stationary point

• f F, then,

10 20

( , , ..., )

m

   x x

( ) f x

Joseph‐Louis Lagrange

• Jan. 25, 1736~Apr.10, 1813

Lin ZHANG, SSE, 2016

Lagrange multiplier

• Lagrange multiplier is a strategy for finding the local

extremum of a function subject to equality constraints

Solution:

1 1

( ; ,..., ) ( ) ( )

m m k k k

F f g   



  x x x

is a stationary point of F

10

( , ,..., )

m

  x

1 2 1 2

0, 0,..., 0, 0, 0,...,

n m

F F F F F F x x x                     

n + m equations! at that point Problem: find stationary points for

1

( ), x x

n

y f



 

under m constraints

( ) 0, 1,2,...,

k

g k m   x

Lin ZHANG, SSE, 2016

Lagrange multiplier

• Example

Problem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0. y=x p0 ?

The distance is

2 2

( , ) ( 1) ( 0) f x y x y    

Now we want to find the stationary point

• f f(x, y) under the constraint

( , ) g x y y x   

According to Lagrange multiplier method, construct another function

2 2

( , , ) ( ) ( ) ( 1) ( ) F x y f x g x x y y x          

Find the stationary point for

( , , ) F x y 

Lin ZHANG, SSE, 2016

Lagrange multiplier

• Example

Problem: for a given point p0 = (1, 0), among all the points lying on the line y=x, identify the one having the least distance to p0. y=x p0 ?

F x F y F                 2( 1) 2 x y x y               0.5 0.5 1 x y          (0.5,0.5,1) is a stationary point of

( , , ) F x y 

(0.5,0.5)is a stationary point of f(x,y) under constraints

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• Homograpy Estimation
• What is Camera Calibration
• Single Camera Calibration
• Bird‐view Generation

Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System

Consider the following linear equations system

1 2 1 1 2 2

3 1 1 3 2 4 2 1 4 x x x x x x                           

Matrix form: A 

x b A x b

It can be easily solved

1 2

1 2 x x     

Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System

How about the following one?

1 2 1 1 2 2 1 2

3 1 1 3 2 4 2 1 4 1 2 6 2 6 x x x x x x x x                                       

It does not have a solution! What is the condition for a linear equation system can be solved?

A  x b

Can we solve it in an approximate way? A: we can use least squares technique! Carl Friedrich Gauss

Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System

Let’s consider a system of p linear equations with q unknowns

11 1 12 2 1 1 21 1 22 2 2 2 1 1 2 2

... ... ... ...

q q q q p p pq q p

a x a x a x a x a x a x A a x a x a x                      b b x b b

We consider the case: p>q, and rank(A)=q In general case, there is no solution! Instead, we want to find a vector x that minimizes the error:

2 2 1 1 1

( ) ( ... )

p i iq q i i

E a x a x A



     



x b x b

unknowns

Lin ZHANG, SSE, 2016

LS for Inhomogeneous Linear System

2 * 2

arg min ( ) arg min E A   

x x

x x x b

 

1 * T T

A A A



 x b

Pseudoinverse of A

How about the pseudoinverse of A when A is square and non-singular?

Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System

Let’s consider a system of p linear equations with q unknowns

11 1 12 2 1 21 1 22 2 2 1 1 2 2

... ... ... ...

q q q q p p pq q

a x a x a x a x a x a x A a x a x a x                      x

We consider the case: p>q, and rank(A)=q unknowns Theoretically, there is only a trivial solution: x = 0 So, we add a constraint to avoid the trivial solution

2

1 x 

Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System

We want to minimize , subject to

2

1 x 

2 2

( ) x x E A 

* 2

arg min ( ), . ., 1

x

x x x E s t  

Use the Lagrange multiplier to solve it,

 

2 2 * 2 2

arg min 1

x

x x x A        

(1) Solving the stationary point of the Lagrange function,

   

2 2 2 2 2 2 2 2

1 1 x x x x x A A                             

(3) (2)

Lin ZHANG, SSE, 2016

LS for Homogeneous Linear System

Then, we have

 

2 2 2 2

1 x x x A          

(3)

x x

T

A A  

x is the eigen‐vector of ATA associated with the eigenvalue 

 

2 2

x x x x x x

T T T

E A A A      

The unit vector x is the eigenvector associated with the minimum eigenvalue of

T

A A

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• Homograpy Estimation
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Homography Estimation

Problem definition: Given a set of points and a corresponding set of points in a projective plane, compute the projective transformation that takes to

 

i

x

 

' i

x

i

x

' i

x

'

, 1,2,...,

i i

H i n   x x We know there existing an H satisfying Coordinates of and are known,

 

i

x

 

' i

x

we need to find H where H is a homography matrix

11 12 13 21 22 23 31 32 33

a a a H a a a a a a           

It has 8 degrees of freedom

Lin ZHANG, SSE, 2016

Homography Estimation

11 12 13 21 22 23 31 32 33

1 a a a cu x cv a a a y c a a a                               

11 12 13 21 22 23 31 32 33

a x a y a cu a x a y a cv a x a y a c              

11 12 13 31 32 33 21 22 23 31 32 33

a x a y a u a x a y a a x a y a v a x a y a                 

4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom

Lin ZHANG, SSE, 2016

Homography Estimation

4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom

11 12 13 21 22 23 31 32 33

1 1 a a a a x y ux uy u a x y vx vy v a a a a                                           

Thus, four correspondence pairs generate 8 equations

Lin ZHANG, SSE, 2016

Homography Estimation

4 point‐correspondence pairs can uniquely determine a homography matrix since each correspondence pair solves two degrees of freedom

(1) A  x 8 9  9 1 

Normally, ; thus (1) has 1 (9‐8) solution vector in its solution space

( ) 8 Rank A 

Lin ZHANG, SSE, 2016

Homography Estimation

• How about the case when there are more than 4

correspondence pairs?

• Use the LS method (for homogeneous case) to solve the

model

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• Homograpy Estimation
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Camera calibration is a necessary step in 3D computer

vision in order to extract metric information from 2D images

• It estimates the parameters of a lens and image

sensor of the camera; you can use these parameters to correct for lens distortion, measure the size of an

• bject in world units, or determine the location of the

camera in the scene

• These tasks are used in applications such as machine

vision to detect and measure objects. They are also used in robotics, for navigation systems, and 3‐D scene reconstruction

What is camera calibration?

Lin ZHANG, SSE, 2016

What is camera calibration?

Lin ZHANG, SSE, 2016

• Camera parameters include
• Intrinsics
• Extrinsics
• Distortion coefficients

What is camera calibration?

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• Homograpy Estimation
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• For simplicity, usually we use a pinhole camera model

Single Camera Calibration

Lin ZHANG, SSE, 2016

• To model the image formation process, 4 coordinate

systems are required

• World coordinate system (3D space)
• Camera coordinate system (3D space)
• Retinal coordinate system (2D space)
• Pixel coordinate system (2D space)

Single Camera Calibration

Lin ZHANG, SSE, 2016

• To model the image formation process, 4 coordinate

systems are required

Single Camera Calibration

Lin ZHANG, SSE, 2016

• From the world CS to the camera CS

Single Camera Calibration

 

, ,

T w w w

X Y Z

is a 3D point represented in the WCS In the camera CS, it is represented as,

R t

c w c w c w

X X Y Y Z Z                      

a rotation matrix (orthogonal)

3 3 

a translation vector

3 1 

Lin ZHANG, SSE, 2016

• From the world CS to the camera CS

Single Camera Calibration

 

, ,

T w w w

X Y Z

is a 3D point represented in the WCS In the camera CS, it is represented as,

R t

c w c w c w

X X Y Y Z Z                       1 1 1 R t

c w c w T c w

X X Y Y Z Z                               

Homogeneous form

(1)

Lin ZHANG, SSE, 2016

• From the camera CS to the retinal CS

Single Camera Calibration

We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS

Lin ZHANG, SSE, 2016

• From the camera CS to the retinal CS

Single Camera Calibration

We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS

x y z

f O is the optical center, f is the focal length, P = [Xc, Yc, Zc]T is a scene point while P’=(x, y) is its image on the retinal plane

Lin ZHANG, SSE, 2016

• From the camera CS to the retinal CS

Single Camera Calibration

We can use a pin‐hole model to represent the mapping from the camera CS to the retinal CS

c c c c c c c

X f X Z x Y y Y f Z Z                               0 0 0 0 0 1 0 0 10 1

c c c c

X x f Y Z y f Z                                 

Homogeneous form

(2)

Lin ZHANG, SSE, 2016

• From the retinal CS to the pixel CS

Single Camera Calibration

The unit for retinal CS (x-y) is physical unit (e.g., mm, cm) while the unit for pixel CS (u-v) is pixel One pixel represents dx physical units along the x‐axis and represents dy physical units along the y‐axis; the image of the

• ptical center is (u0, v0)

1 1 1 1 0 1 u dx u x v v y dy                                       

Lin ZHANG, SSE, 2016

• From the retinal CS to the pixel CS

Single Camera Calibration

If the two axis of the pixel plane are not perpendicular to each

• ther, another parameter s is introduced to represent the

skewness of the two axis

1 1 1 1 0 1 u dx u x v v y dy                                        1 1 1 1 0 1 s u dx u x v v y dy                                       

(3)

Lin ZHANG, SSE, 2016

Single Camera Calibration

From Eqs.1~3, we can have

1 0 0 0 1 . 0 0 0 0 . 1 0 0 10 0 1 1 0 1 1 0 . 0 0 1 0 1

c c c c c c c c c c

f s u sf u X X dx dx u f Y Y f Z v v f v Z Z dy dy X u u Y v Z                                                                                                               

 

0 . . 1 0 0 1 0 0 0 1 1 1 R t R t

w w w w T w w

X X u Y Y v v Z Z                                                       

Intrinsic Extrinsic

Lin ZHANG, SSE, 2016

Single Camera Calibration

[ ] [ ] 1 0 0 1 1 1 K R t R t

w w w w c w w

X X u u Y Y Z v v Z Z                                                   

Intrinsic Extrinsic

, the coordinates of the principal point in the image plane

 

, u v

, the scale factors in image u and v axes

 and

, describing the skewness of the two image axes



R and t determines the rigid transformation from the world coordinate system to the camera coordinate system Altogether, there are 11 parameters to be determined

Lin ZHANG, SSE, 2016

Single Camera Calibration

• To accurately represent an ideal camera, the camera

model can include the radial and tangential lens distortion

• Radial distortion occurs when light rays bend more near the

edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion

Lin ZHANG, SSE, 2016

Single Camera Calibration

• To accurately represent an ideal camera, the camera

model can include the radial and tangential lens distortion

• Radial distortion occurs when light rays bend more near the

edges of a lens than they do at its optical center; the smaller the lens, the greater the distortion

   

2 4 6 1 2 3 2 4 6 1 2 3

1 1

distorted distorted

u u k r k r k r v v k r k r k r        

where

2 2 2

r u v  

are the radial distortion coefficients of the lens

1 2 3

, , k k k

Lin ZHANG, SSE, 2016

Single Camera Calibration

• To accurately represent an ideal camera, the camera

model can include the radial and tangential lens distortion

• Tangential distortion occurs when the lens and the image plane

are not parallel

Lin ZHANG, SSE, 2016

Single Camera Calibration

• To accurately represent an ideal camera, the camera

model can include the radial and tangential lens distortion

• Tangential distortion occurs when the lens and the image plane

are not parallel

 

 

 

 

2 2 1 2 2 2 2 1

+ 2 2 2 2

distorted distorted

u u uv r u v v uv r v           

are the tangential distortion coefficients of the lens

1 2

,  

Lin ZHANG, SSE, 2016

Single Camera Calibration

• The purpose of the camera calibration is to determine

the values for the extrinsics, intrinsics, and distortion coefficients

• How to do?
• Zhengyou Zhang’s method[1] is a commonly used modern

approach

• A calibration board is needed; several images of the board need

to be captured; based on the correspondence pairs (pixel coordinate and world coordinate of a feature point), equation systems can be obtained; by solving the equation systems, parameters can be determined

[1] Z. Zhang, A flexible new technique for camera calibration, IEEE Trans. Pattern Analysis and Machine Intelligence, 2000

Lin ZHANG, SSE, 2016

Single Camera Calibration

Calibration board

Lin ZHANG, SSE, 2016

Single Camera Calibration

A set of Calibration board images (50~60)

Lin ZHANG, SSE, 2016

Single Camera Calibration

• Matlab provides a “Camera Calibrator”
• Straightforward to use
• However, based on my experience, it is not as accurate as the

routine provided in openCV3.0, especially for large FOV cameras (such as fisheye camera); thus, for some accuracy critical applications, I recommend to use the openCV function, though a little more complicated

• It exports “cameraParams” as the calibration result

Lin ZHANG, SSE, 2016

Single Camera Calibration

Lin ZHANG, SSE, 2016

Single Camera Calibration

• For our purpose (measuring geometric metrics of a planar
• bject), we use the camera parameters to undistort the

image

• The essence of this step is to make sure the

transformation from a physical plane to the image plane can be represented by a linear projective matrix; or in

• ther words, a straight line should be mapped to a

straight line

Lin ZHANG, SSE, 2016

Single Camera Calibration

Original image Undistorted image

Lin ZHANG, SSE, 2016

Contents

• Foundations of Projective Geometry
• Matrix Differentiation
• Lagrange Multiplier
• Least‐squares for Linear Systems
• Homograpy Estimation
• What is Camera Calibration
• Single Camera Calibration
• Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Our task is to measure the geometric properties of
• bjects on a plane (e.g., conveyor belt)
• Such a problem can be solved if we have its bird‐view

image; bird’s‐eye‐view is easy for object detection and measurement

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Three coordinate systems are required
• Bird’s‐eye‐view image coordinate system
• World coordinate system
• Undistorted image coordinate system

X Y X Y X Y

Bird’s‐eye‐view image WCS Undistorted image Similarity Projective

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is

B W

P 

W I

P 

Then, given a position on bird’s‐eye‐view, we can get its corresponding position in the undistorted image as

 

, ,1

T B B

x y 1 x

B I W I B W B

x P P y

 

          

• Basic idea for bird’s‐eye‐view generation

Then, the intensity of the pixel can be determined using some interpolation technique based on the neighborhood around on the undistorted image

 

, ,1

T B B

x y xI

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Suppose that the transformation matrix from bird’s‐eye‐view to WCS is and the transformation matrix from WCS to the undistorted image is

B W

P 

W I

P 

• Basic idea for bird’s‐eye‐view generation

The key problem is how to obtain and ?

B W

P 

W I

P 

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Determine

B W

P 

X Y X Y H (mm) M (pixels) N (pixels)

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Determine

B W

P  2 2 1 1 1 1

W B B W B B W B

H HN M M x x x H H y y P y M



                                                  For a point on bird’s‐eye‐view, the corresponding point on the world coordinate system is,

 

, ,1

T B B

x y

Please verify!!

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Determine

W I

P 

The physical plane (in WCS) and the undistorted image plane can be linked via a homography matrix

W I

P  x x

I W I W

P  

If we know a set of correspondence pairs ,

 

1

, x x

N Ii Wi i W I

P  can be estimated using the least‐square method

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

• Determine

W I

P 

A set of point correspondence pairs; for each pair, we know its coordinate on the undistorted image plane and its coordinate in the WCS

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

When and are known, the bird’s‐eye‐view can be generated via,

W I

P 

B W

P  1 1 x

B B I W I B W B B I B

x x P P y P y

  

                     

Bird’s‐eye‐view Generation

Lin ZHANG, SSE, 2016

Bird’s‐eye‐view Generation

Original image Bird’s‐eye‐view

Lin ZHANG, SSE, 2016

Bird‐view Generation

Another example Original fish‐eye image Undistorted image

Lin ZHANG, SSE, 2016

Bird‐view Generation

Another example Bird’s‐eye‐view Original fish‐eye image

Lin ZHANG, SSE, 2016

Thanks for your attention