Camera calibration Digital Visual Effects, Spring 2006 Yung-Yu - PowerPoint PPT Presentation

Camera calibration Digital Visual Effects, Spring 2006 Yung-Yu Chuang 2005/4/19 with slides by Richard Szeliski, Steve Seitz, and Marc Pollefyes

Announcements • Artifacts for assignment #1 voting http:/ / www.csie.ntu.edu.tw/ ~cyy/ vfx/ assignm ents/ proj 1/ artifacts/ index.html

Outline • Camera proj ection models • Camera calibration • Nonlinear least square methods • Bundle adj ustment

Camera projection models

Pinhole camera

principal point Pinhole camera model (optical center) origin

Pinhole camera model fX = x Z fY = y Z ⎛ ⎞ X ⎜ ⎟ ⎛ ⎞ ⎛ ⎞ ⎡ ⎤ x fX f 0 0 0 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ Y = ⎜ ⎟ ⎜ ⎟ y ~ fY 0 f 0 0 ⎜ ⎟ ⎢ ⎥ Z ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ 1 Z 0 0 1 0 ⎜ ⎟ ⎝ ⎠ 1

Pinhole camera model ⎛ ⎞ X ⎜ ⎟ ⎞ ⎡ ⎤ ⎡ ⎤ ⎛ ⎞ ⎛ x fX f 0 0 1 0 0 0 ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ Y = ⎜ ⎟ ⎜ ⎟ y ~ fY 0 f 0 0 1 0 0 ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ Z ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ ⎣ ⎦ 1 Z 0 0 1 0 0 1 0 ⎜ ⎟ ⎝ ⎠ 1

Principal point offset intrinsic matrix [ ] X x ~ K I 0 ⎛ ⎞ X ⎜ ⎟ ⎞ ⎡ ⎤ ⎡ ⎤ ⎛ ⎞ ⎛ x fX f 0 x 1 0 0 0 ⎜ ⎟ ⎜ ⎟ 0 ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ Y = ⎜ ⎟ ⎜ ⎟ y ~ fY 0 f y 0 1 0 0 ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ 0 Z ⎜ ⎟ ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎢ ⎥ ⎝ ⎠ ⎝ ⎠ ⎣ ⎦ ⎣ ⎦ 1 Z 0 0 1 0 0 1 0 ⎜ ⎟ ⎝ ⎠ 1

Intrinsic matrix ⎡ ⎤ f 0 x Is this form of K good enough? 0 ⎢ ⎥ = K 0 f y ⎢ ⎥ 0 ⎢ ⎥ ⎣ ⎦ 0 0 1 • non-square pixels (digital video) • skew ⎡ ⎤ fa s x 0 ⎢ ⎥ • radial distortion = K 0 f y ⎢ ⎥ 0 ⎢ ⎥ ⎣ ⎦ 0 0 1

Radial distortion

Camera rotation and translation ⎛ ⎞ ⎛ ⎞ X ' X ⎜ ⎟ ⎜ ⎟ = + ⎜ ⎟ ⎜ ⎟ R t Y ' Y × 3 3 ⎜ ⎟ ⎜ ⎟ ⎝ ⎠ ⎝ ⎠ Z ' Z [ ] X ⎛ ⎞ X ⎜ ⎟ x ~ K R t ⎛ ⎞ ⎡ ⎤ x f 0 x ⎜ ⎟ 0 [ ] ⎜ ⎟ ⎢ ⎥ Y ⎜ ⎟ y ~ 0 f y R t ⎜ ⎟ ⎢ ⎥ 0 Z ⎜ ⎟ ⎜ ⎟ ⎢ ⎥ ⎝ ⎠ ⎣ ⎦ extrinsic matrix 1 0 0 1 ⎜ ⎟ ⎝ ⎠ 1

Two kinds of parameters internal or intrinsic parameters such as focal • length, optical center, aspect ratio: what kind of camera? external or extrinsic (pose) parameters • including rotation and translation: where is the camera?

Other projection models

Orthographic projection • S pecial case of perspective proj ection – Distance from the COP to the PP is infinite Image World – Also called “ parallel proj ection ” : (x, y, z) → (x, y)

Other types of projections • S caled orthographic – Also called “ weak perspective ” • Affine proj ection – Also called “ paraperspective ”

Fun with perspective

Perspective cues

Perspective cues

Fun with perspective Ames room

Forced perspective in LOTR

Camera calibration

Camera calibration • Estimate both intrinsic and extrinsic parameters • Mainly, two categories: 1. Photometric calibration: uses reference obj ects with known geometry 2. S elf calibration: only assumes static scene, e.g. structure from motion

Camera calibration approaches 1. linear regression (least squares) 2. nonlinear optinization 3. multiple planar patterns

Chromaglyphs (HP research)

Linear regression [ ] = x ~ K R t X MX

Linear regression • Directly estimate 11 unknowns in the M matrix using known 3D points ( X i , Y i , Z i ) and measured feature positions ( u i , v i )

Linear regression

Linear regression

olve for Proj ection Matrix M using least-square Linear regression techniques S

Normal equation Given an overdetermined system Ax = b the normal equation is that which minimizes the sum of the square differences between left and right sides = T T A Ax A b Why?

Normal equation ( ) = − 2 E ( x ) Ax b ⎡ ⎤ ⎡ ⎤ a ... a b 11 1 m 1 ⎢ ⎥ ⎢ ⎥ ⎡ ⎤ : : x : ⎢ ⎥ ⎢ ⎥ 1 ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ = : : : : ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ : : x : ⎢ ⎥ ⎢ ⎥ m ⎢ ⎥ ⎢ ⎥ ⎣ ⎦ ⎣ ⎦ a ... a b n 1 nm n n x m, n equations , m variables

Normal equation ⎡ ⎤ ⎛ ⎞ m ⎡ ⎤ ∑ m ⎜ ⎟ ∑ − ⎢ ⎥ a x b ⎜ ⎟ a x ⎢ ⎥ 1 j j 1 ⎡ ⎤ 1 j j ⎢ ⎝ ⎠ ⎥ b = ⎢ ⎥ j 1 = j 1 1 ⎢ ⎥ ⎢ ⎥ : ⎢ ⎥ : : ⎢ ⎥ ⎢ ⎥ ⎛ ⎞ ⎢ m ⎥ m ∑ ∑ ⎜ ⎟ ⎢ ⎥ − = − = − ⎢ ⎥ a x Ax b a x b b ⎜ ⎟ ⎢ ⎥ ij j i ij j i ⎢ ⎥ ⎢ ⎥ ⎝ ⎠ = = ⎢ ⎥ j 1 j 1 : ⎢ ⎥ ⎢ ⎥ : ⎢ ⎥ : ⎢ ⎥ ⎢ ⎥ m ⎛ ⎞ ⎣ ⎦ ∑ b ⎢ ⎥ m ∑ ⎜ ⎟ n ⎢ ⎥ a x − a x b ⎢ ⎥ ⎜ ⎟ nj j ⎣ ⎦ nj j n ⎢ ⎥ = ⎝ ⎠ ⎣ ⎦ j 1 = j 1 2 ⎡ ⎤ ⎛ ⎞ n m ( ) ∑ ∑ ⎜ ⎟ = − = − 2 ⎢ ⎥ E ( x ) Ax b a x b ⎜ ⎟ ij j i ⎢ ⎥ ⎝ ⎠ ⎣ ⎦ = = i 1 j 1

Normal equation 2 ⎡ ⎤ ⎛ ⎞ n m ( ) ∑ ∑ ⎜ ⎟ = − = − 2 ⎢ ⎥ x Ax b E ( ) a x b ⎜ ⎟ ij j i ⎢ ⎥ ⎝ ⎠ ⎣ ⎦ = = i 1 j 1 ⎡ ⎤ ⎛ ⎞ ∂ n m ∑ ∑ E ⎜ ⎟ − = = ⎢ ⎥ 2 a x b a 0 ⎜ ⎟ ∂ ij j i i 1 ⎢ ⎥ ⎝ ⎠ x ⎣ ⎦ = = 1 i 1 j 1 n m n ∑ ∑ ∑ = − 2 a a x 2 a b i 1 ij j i 1 i = = = i 1 j 1 i 1 ∂ = E = = − T T T T → A Ax A b 0 2 ( A Ax A b ) ∂ x

Linear regression • Advantages: – All specifics of the camera summarized in one matrix – Can predict where any world point will map to in the image • Disadvantages: – Doesn’ t tell us about particular parameters – Mixes up internal and external parameters • pose specific: move the camera and everything breaks

Nonlinear optimization • Feature measurement equations • Likelihood of M given {( u i , v i )}

Optimal estimation • Log likelihood of M given {( u i , v i )} • How do we minimize C ? • Non-linear regression (least squares), because û i and v i are non-linear functions of M • We can use Levenberg-Marquardt method to minimize it

Multi-plane calibration Images courtesy Jean-Yves Bouguet, Intel Corp. Advantage • Only requires a plane • Don’ t have to know positions/ orientations • Good code available online! – Intel’ s OpenCV library: http:/ / www.intel.com/ research/ mrl/ research/ opencv/ – Matlab version by Jean-Yves Bouget: http:/ / www.vision.caltech.edu/ bouguetj / calib_doc/ index.html – Zhengyou Zhang’ s web site: http:/ / research.microsoft.com/ ~zhang/ Calib/

Step 1: data acquisition

Step 2: specify corner order

tep 3: corner extraction S

tep 3: corner extraction S

Step 4: minimize projection error

Step 4: camera calibration

Step 4: camera calibration

Step 5: refinement

Nonlinear least square methods

number of parameters number of data points Least square fitting

Linear least square fitting y = = + y ( t ) M ( x , t ) x x t 0 1 = − f ( x ) y M ( x , t ) i i i t prediction residual = + + 3 M ( x , t ) x x t x t is linear, too. 0 1 2

Nonlinear least square fitting

Function minimization Least square is related to function minimization. It is very hard to solve in general. Here, we only consider a simpler problem of finding local minimum.

Function minimization

Quadratic functions Approximate the function with a quadratic function within a small neighborhood

Quadratic functions A is positive definite. negative definite All eigenvalues are positive. Fall all x, x T Ax>0. A is singular A is indefinite

Function minimization

Descent methods

Descent direction

Steepest descent method the decrease of F(x) per unit along h direction → h sd is a descent direction because h T sd F’ (x)=- F’ (x) 2 <0 It has good performance in the initial stage of the iterative process. Converge very slow with a linear rate.

gradient Steepest descent method isocontour

Line search

Hh h T T h h = α Line search

Steepest descent method

Newton’s method → → → → It has good performance in the final stage of the iterative process, where x is close to x*.

Hybrid method This needs to calculate second-order derivative which might not be available.