Pin Hole Cameras & Warp Functions Instructor - Simon Lucey - PowerPoint PPT Presentation

Pin Hole Cameras & Warp Functions Instructor - Simon Lucey 16-423 - Designing Computer Vision Apps

Today • Pinhole Camera. • Homogenous Coordinates. • Planar Warp Functions.

Example of SLAM for AR Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Example of SLAM for AR Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Example of SLAM for AR Taken from: H. Liu et al. “Robust Keyframe-based Monocular SLAM for Augmented Reality”, ISMAR 2016.

Motivation Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole Camera Taken from: http://img.gawkerassets.com/img/18w7i1umpzoa9jpg/original.jpg

Pinhole Camera Instead model impossible but more convenient Real camera image virtual image is inverted Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole Camera Terminology Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Normalized Camera By similar triangles: Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Focal length parameters Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Focal length parameters Can model both • the effect of the distance to the focal plane • the density of the receptors with a single focal length paramete r φ In practice, the receptors may not be square: S o use different focal length parameter for x and y dims Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Offset parameters • Current model assumes that pixel (0,0) is where the principal ray strikes the image plane (i.e. the center) • Model offset to center Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Skew parameter • Finally, add skew parameter • Accounts for image plane being not exactly perpendicular to the principal ray Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Radial distortion Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Camera & World Coordinates w “world coordinate frame” o u

Camera & World Coordinates “camera coordinate frame” w 0 w u 0 o 0 “world coordinate frame” o u  u 0 �  ω 1 �  u �  τ x � ω 2 + = w 0 ω 3 ω 4 w τ z

Camera & World Coordinates Rotation Matrix Translation Vector  u 0 �  ω 1 �  u �  τ x � ω 2 + = w 0 ω 3 ω 4 w τ z

Position and orientation of camera • Position w =(u,v,w) T of point in the world is generally not expressed in the frame of reference of the camera. • Transform using 3D transformation or Point in frame of Point in frame of reference of camera reference of world

Constraints on Ω • As is a rotation matrix it is constrained by the following, Ω Ω T Ω = I det( Ω ) = 1 • We refer to these matrices as belonging to the Special Orthogonal Group - SO(3). • How many degrees of freedom do you think has ? Ω

Something to try • In MATLAB type, >> R1 = orth(randn(3,3)); >> R1(:,end) = det(R1)*R1(:,end); >> R2 = orth(randn(3,3)); >> R2(:,end) = det(R2)*R2(:,end); • If you form a new matrix as a linear combination of R1 & R2, >> R3 = 0.5*R1 + 0.5*R2; • Does R3 lie in SO(3) ?

Reminder: Convex Set 18

Reminder: Convex Set 18

Reminder: Convex Set 18

Reminder: Non-Convex Set 19

Reminder: Non-Convex Set 19

Reminder: Non-Convex Set 19

Complete pinhole camera model • Intrinsic parameters (stored as intrinsic matrix) • Extrinsic parameters Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Complete pinhole camera model • For short: • Question: is a linear function? Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Perspective Transform

Learning extrinsic parameters N ˆ X Ω , ˆ τ = min η { x n − pinhole[ w n , Λ , Ω , τ ] } Ω , τ n =1 e.g. η { x } = || x || 2 2 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Learning intrinsic parameters N ˆ X Λ = min Λ [min η { x n − pinhole[ w n , Λ , Ω , τ ] } ] Ω , τ n =1 e.g. η { x } = || x || 2 2 Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Camera Calibration • Use 3D target with known 3D points. Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

For you to try….. • There exists camera calibration tools in MATLAB, • see Bouget’s Calibration Toolbox in MATLAB. • Or if you prefer, • you can use OpenCV’s tutorial. • What are the intrinsics of your device? • How sensitive are vision algorithms to the correct intrinsics?

Today • Pinhole Camera. • Homogenous Coordinates. • Planar Warp Functions.

Homogeneous Coordinates • Convert 2D coordinate to 3D • To convert back Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Geometric interpretation Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole camera • Camera model: • In homogeneous coordinates: (linear!) Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Pinhole camera • Writing out these three equations • Eliminate λ to retrieve original equations Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Adding in extrinsics Or for short: Or even shorter: Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Today • Pinhole Camera. • Homogenous Coordinates. • Planar Warp Functions.

Planar Warp Functions • Consider viewing a planar scene • There is now a 1 to 1 mapping between points on the plane and points in the image • We will investigate models for this 1 to 1 mapping – Euclidean – Similarity – Affine – Homography

Piecewise planarity Many scenes are not planar, but are nonetheless piecewise planar Can we match all of the planes to one another?

Euclidean warp • Consider viewing a fronto-parallel plane at a fixed distance D. • In homogeneous coordinates, the imaging equations are: Plane at Point is on 3D rotation matrix known plane becomes 2D (in plane) ( w =0) distance D Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp • Simplifying • Rearranging the last equation Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp Homogeneous: Cartesian: For short: Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Euclidean warp Homogeneous: Cartesian: For short: How many unknowns? Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

More to read… • Prince et al. • Chapter 14, Section 1 & 3. • Chapter 15, Section 1.