Pin Hole Cameras & Warp Functions Instructor - Simon Lucey - - PowerPoint PPT Presentation

Pin Hole Cameras & Warp Functions

Instructor - Simon Lucey

16-423 - Designing Computer Vision Apps

Today

• Pinhole Camera.
• Homogenous Coordinates.
• Planar Warp Functions.

Example of SLAM for AR

Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Example of SLAM for AR

Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Example of SLAM for AR

Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Motivation

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole Camera

Taken from: http://img.gawkerassets.com/img/18w7i1umpzoa9jpg/original.jpg

Pinhole Camera

Real camera image is inverted Instead model impossible but more convenient virtual image

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole Camera Terminology

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Normalized Camera

By similar triangles:

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Focal length parameters

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Focal length parameters

Can model both

• the effect of the distance to the focal plane
• the density of the receptors

with a single focal length parameter φ In practice, the receptors may not be square: So use different focal length parameter for x and y dims

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Offset parameters

• Current model assumes that pixel (0,0) is where the

principal ray strikes the image plane (i.e. the center)

• Model offset to center

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Skew parameter

• Finally, add skew parameter
• Accounts for image plane being not exactly perpendicular

to the principal ray

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Radial distortion

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Camera & World Coordinates

u

• w

“world coordinate frame”

Camera & World Coordinates

u

• w

“world coordinate frame”

u0 w0

“camera coordinate frame”

u0 w0

• u

w

• +

τx τz

• =

ω1 ω2 ω3 ω4

Camera & World Coordinates

u0 w0

• u

w

• +

τx τz

• =

ω1 ω2 ω3 ω4

• Rotation Matrix Translation Vector

Position and orientation of camera

• Position w=(u,v,w)T of point in the world is generally not

expressed in the frame of reference of the camera.

• Transform using 3D transformation
• r

Point in frame of reference of camera

Point in frame of reference of world

Constraints on

• As is a rotation matrix it is constrained by the following,
• We refer to these matrices as belonging to the Special

Orthogonal Group - SO(3).

• How many degrees of freedom do you think has ?

Ω

Ω ΩT Ω = I det(Ω) = 1 Ω

Something to try

• In MATLAB type,

>> R1 = orth(randn(3,3)); >> R1(:,end) = det(R1)*R1(:,end); >> R2 = orth(randn(3,3)); >> R2(:,end) = det(R2)*R2(:,end);

• If you form a new matrix as a linear combination of R1 & R2,

>> R3 = 0.5*R1 + 0.5*R2;

• Does R3 lie in SO(3) ?

Reminder: Convex Set

18

Reminder: Convex Set

18

Reminder: Convex Set

18

Reminder: Non-Convex Set

19

Reminder: Non-Convex Set

19

Reminder: Non-Convex Set

19

Complete pinhole camera model

• Intrinsic parameters (stored as intrinsic matrix)
• Extrinsic parameters

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Complete pinhole camera model

• For short:
• Question: is a linear function?

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Perspective Transform

Learning extrinsic parameters

ˆ Ω, ˆ τ = min

Ω,τ N

X

n=1

η{xn − pinhole[wn, Λ, Ω, τ]}

e.g. η{x} = ||x||2

2

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Learning intrinsic parameters

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

e.g. η{x} = ||x||2

2

ˆ Λ = min

Λ [min Ω,τ N

X

n=1

η{xn − pinhole[wn, Λ, Ω, τ]}]

Camera Calibration

• Use 3D target with known 3D points.

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

For you to try…..

• There exists camera calibration tools in MATLAB,
• see Bouget’s Calibration Toolbox in MATLAB.
• Or if you prefer,
• you can use OpenCV’s tutorial.
• What are the intrinsics of your device?
• How sensitive are vision algorithms to the correct intrinsics?

Today

• Pinhole Camera.
• Homogenous Coordinates.
• Planar Warp Functions.

Homogeneous Coordinates

• Convert 2D coordinate to 3D
• To convert back

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Geometric interpretation

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole camera

• Camera model:
• In homogeneous coordinates:

(linear!)

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole camera

• Writing out these three equations
• Eliminate λ to retrieve original equations

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Adding in extrinsics

Or for short: Or even shorter:

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Today

• Pinhole Camera.
• Homogenous Coordinates.
• Planar Warp Functions.

Planar Warp Functions

• Consider viewing a planar scene
• There is now a 1 to 1 mapping between points on the plane

and points in the image

• We will investigate models for this 1 to 1 mapping

–Euclidean –Similarity –Affine –Homography

Piecewise planarity

Many scenes are not planar, but are nonetheless piecewise planar Can we match all of the planes to one another?

Euclidean warp

• Consider viewing a fronto-parallel plane at a fixed

distance D.

• In homogeneous coordinates, the imaging equations are:

3D rotation matrix becomes 2D (in plane) Plane at known distance D Point is on plane (w=0)

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp

• Simplifying
• Rearranging the last equation

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp

Homogeneous: Cartesian: For short:

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp

Homogeneous: Cartesian: For short:

Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

How many unknowns?

More to read…

• Prince et al.
• Chapter 14, Section 1 & 3.
• Chapter 15, Section 1.