Camera calibration Digital Visual Effects, Spring 2006 Yung-Yu - - PowerPoint PPT Presentation

Camera calibration

Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/4/19

with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes

Announcements

• Artifacts for assignment #1 voting

http:/ / www.csie.ntu.edu.tw/ ~cyy/ vfx/ assignm ents/ proj 1/ artifacts/ index.html

Outline

• Camera proj ection models
• Camera calibration
• Nonlinear least square methods
• Bundle adj ustment

Camera projection models

Pinhole camera

Pinhole camera model

(optical center)

• rigin

principal point

Pinhole camera model

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ 1 1 ~ 1 Z Y X f f Z fY fX y x Z fY y Z fX x = =

Pinhole camera model

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ 1 1 1 1 1 ~ 1 Z Y X f f Z fY fX y x

Principal point offset

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ 1 1 1 1 1 ~ 1 Z Y X y f x f Z fY fX y x

[ ]X

I K x ~

intrinsic matrix

Intrinsic matrix

• non-square pixels (digital video)
• skew
• radial distortion

⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 y f x f K ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ = 1 y f x s fa K

Is this form of K good enough?

Radial distortion

Camera rotation and translation

t R + ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ = ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛

×

Z Y X Z Y X

3 3

' ' '

[ ]

⎟ ⎟ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎜ ⎜ ⎝ ⎛ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ ⎟ ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎜ ⎝ ⎛ 1 1 ~ 1 Z Y X y f x f y x t R

[ ]X

t R K x ~

extrinsic matrix

Two kinds of parameters

• internal or intrinsic parameters such as focal

length, optical center, aspect ratio:

what kind of camera?

• external or extrinsic (pose) parameters

including rotation and translation:

where is the camera?

Other projection models

Orthographic projection

• S

pecial case of perspective proj ection

– Distance from the COP to the PP is infinite – Also called “parallel proj ection”: (x, y, z) → (x, y)

Image World

Other types of projections

• S

caled orthographic

– Also called “weak perspective”

• Affine proj ection

– Also called “paraperspective”

Fun with perspective

Perspective cues

Perspective cues

Fun with perspective

Ames room

Forced perspective in LOTR

Camera calibration

Camera calibration

• Estimate both intrinsic and extrinsic parameters
• Mainly, two categories:
• 1. Photometric calibration: uses reference obj ects

with known geometry

• 2. S

elf calibration: only assumes static scene, e.g. structure from motion

Camera calibration approaches

• 1. linear regression (least squares)
• 2. nonlinear optinization
• 3. multiple planar patterns

Chromaglyphs (HP research)

Linear regression

[ ]

MX X t R K x = ~

Linear regression

• Directly estimate 11 unknowns in the M matrix

using known 3D points (X

i,Yi,Zi) and measured

feature positions (ui,vi)

Linear regression

Linear regression

Linear regression

S

• lve for Proj ection Matrix M using least-square

techniques

Normal equation

Given an overdetermined system

b Ax = b A Ax A

T T

=

the normal equation is that which minimizes the sum of the square differences between left and right sides Why?

Normal equation

( )

2

) ( b Ax x − = E ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎣ ⎡ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡

n m nm n m

b b x x a a a a : : : : ... : : : : : : ...

1 1 1 1 11

nxm, n equations, m variables

Normal equation

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ = ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ − ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ ⎡ = −

∑ ∑ ∑ ∑ ∑ ∑

= = = = = = n m j j nj i m j j ij m j j j n i m j j nj m j j ij m j j j

b x a b x a b x a b b b x a x a x a

1 1 1 1 1 1 1 1 1 1

: : : : : : b Ax

( )

∑ ∑

= =

⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ = − =

n i i m j j ij

b x a E

1 2 1 2

) ( b Ax x

Normal equation

( )

∑ ∑

= =

⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ = − =

n i i m j j ij

b x a E

1 2 1 2

) ( b Ax x = ∂ ∂ =

1

x E

∑ ∑ ∑

= = =

− =

n i i i n i j m j ij i

b a x a a

1 1 1 1 1

2 2

1 1 1

2

i n i i m j j ij

a b x a

∑ ∑

= =

⎥ ⎥ ⎦ ⎤ ⎢ ⎢ ⎣ ⎡ − ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ ) ( 2 b A Ax A x

T T

− = ∂ ∂ = E

b A Ax A

T T

=

→

Linear regression

• Advantages:

– All specifics of the camera summarized in one matrix – Can predict where any world point will map to in the image

• Disadvantages:

– Doesn’ t tell us about particular parameters – Mixes up internal and external parameters

• pose specific: move the camera and everything breaks

Nonlinear optimization

• Feature measurement equations
• Likelihood of M given {(ui,vi)}

Optimal estimation

• Log likelihood of M given {(ui,vi)}
• How do we minimize C?
• Non-linear regression (least squares), because

ûi and vi are non-linear functions of M

• We can use Levenberg-Marquardt method to

minimize it

Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

Advantage

• Only requires a plane
• Don’ t have to know positions/ orientations
• Good code available online!

– Intel’ s OpenCV library:

http:/ / www.intel.com/ research/ mrl/ research/ opencv/

– Matlab version by Jean-Yves Bouget:

http:/ / www.vision.caltech.edu/ bouguetj / calib_doc/ index.html

– Zhengyou Zhang’ s web site: http:/ / research.microsoft.com/ ~zhang/ Calib/

Step 1: data acquisition

Step 2: specify corner order

S tep 3: corner extraction

S tep 3: corner extraction

Step 4: minimize projection error

Step 4: camera calibration

Step 4: camera calibration

Step 5: refinement

Nonlinear least square methods

Least square fitting

number of data points number of parameters

Linear least square fitting

y t

t x x t x M t y

1

) , ( ) ( + = = ) , ( ) (

i i i

t x M y x f − =

3 2 1

) , ( t x t x x t x M + + =

is linear, too. residual prediction

Nonlinear least square fitting

Function minimization

It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum. Least square is related to function minimization.

Function minimization

Quadratic functions

Approximate the function with a quadratic function within a small neighborhood

Quadratic functions

A is positive definite. All eigenvalues are positive. Fall all x, xTAx>0. negative definite A is indefinite A is singular

Function minimization

Descent methods

Descent direction

Steepest descent method

It has good performance in the initial stage of the iterative

• process. Converge very slow with a linear rate.

the decrease of F(x) per unit along h direction

→

hsd is a descent direction because hT

sd F’ (x)=- F’ (x)2<0

Steepest descent method

isocontour gradient

Line search

Line search

Hh h h h

T T

= α

Steepest descent method

Newton’s method

It has good performance in the final stage of the iterative process, where x is close to x*.

→ → → →

Hybrid method

This needs to calculate second-order derivative which might not be available.

Levenberg-Marquardt method

• LM can be thought of as a combination of

steepest descent and the Newton method. When the current solution is far from the correct one, the algorithm behaves like a steepest descent method: slow, but guaranteed to converge. When the current solution is close to the correct solution, it becomes a Newton’ s method.

Nonlinear least square ). ( ˆ with , ˆ Here, minimal. is distance squared that the so vector parameter best the find try to , ts measuremen

• f

set a Given p x x x p x f

T

= − = ε ε ε

Levenberg-Marquardt method

Levenberg-Marquardt method

• μ=0 → Newton’ s method
• μ→∞ → steepest descent method
• S

trategy for choosing μ

– S tart with some small μ – If F is not reduced, keep trying larger μ until it does – If F is reduced, accept it and reduce μ for the next iteration

Bundle adjustment

Bundle adjustment

• Bundle adj ustment (BA) is a technique for

simultaneously refining the 3D structure and camera parameters

• It is capable of obtaining an optimal

reconstruction under certain assumptions on image error models. For zero-mean Gaussian image errors, BA is the maximum likelihood estimator.

Bundle adjustment

• n 3D points are seen in m views
• xij is the proj ection of the i-th point on image j
• aj is the parameters for the j -th camera
• bi is the parameters for the i-th point
• BA attempts to minimize the proj ection error

Euclidean distance predicted proj ection

Bundle adjustment

Bundle adjustment

3 views and 4 points

Typical Jacobian

Block structure of normal equation

Bundle adjustment

Bundle adjustment

Multiplied by

Reference

• Manolis Lourakis and Antonis Argyros, The Design and

Implementation of a Generic S parse Bundle Adj ustment S

• ftware

Package Based on the Levenberg-Marquardt Algorithm, FORTH- ICS / TR-320 2004.

• K. Madsen, H.B. Nielsen, O. Timgleff, Methods for Non-Linear Least

S quares Problems, 2004.

• Zhengyou Zhang, A Flexible New Techniques for Camera

Calibration, MS R-TR-98-71, 1998.

• Bill Triggs, Philip McLauchlan, Richard Hartley and Andrew

Fitzgibbon, Bundle Adj ustment - A Modern S ymthesis, Proceedings

• f the International Workshop on Vision Algorithms: Theory and

Practice, pp298-372, 1999.