Robust Gyroscope-Aided Camera Self-Calibration Santiago Cort es - PowerPoint PPT Presentation

Robust Gyroscope-Aided Camera Self-Calibration Santiago Cort´ es Arno Solin Juho Kannala Aalto University July 11, 2018

Motivation ◮ Camera sensors are common in smart devices ◮ Use cases: AR/VR � , games � , odometry � , photography � , etc. ◮ But the observed images are distorted ◮ The distortion can be estimated off-line or be factory-calibrated ◮ We want to estimate the distortion online What the camera sees Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 2/12

Idea Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 3/12

Camera model ◮ World coordinates ( x , y , z ) to image coordinates ( u , v ) : ◮ Pinhole camera model:   x   u y  = K E   v    z   1 1 ◮ where the intrinsic and extrinsic matrices are:   f x 0 c x R T − R T p � � K = 0 f y c y and E =   0 0 1 ◮ Camera pose ( R , p ) : the camera orientation (quaternion) and position Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 4/12

Camera model (non-linear) ◮ Lens distortions are typically non-linear ◮ Radial distortion coefficients k 1 and k 2 : � u ′ � � u � ( 1 + k 1 r 2 + k 2 r 4 ) = v ′ v with the radial component given by �� u − c x � v − c y � 2 � 2 r = + f x f y Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 5/12

Feature tracking ◮ The dense image is not convenient to work with ◮ Choose sparse points by a feature detector ◮ Track the points over frames using a feature tracker ◮ Measurement data consists of tracks of points over frames Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 6/12

Motion model ◮ The gyroscope for drives the orientation dynamics: d q ( t ) = 1 2 Ω ( ω ) q ( t ) d t q ( t ) is the quaternion at t and ω the angular velocity. ◮ The position p ( t ) = ( p 1 ( t ) , p 2 ( t ) , p 3 ( t )) is modeled as a Wiener velocity model: d 2 p j ( t ) = w ( t ) d t 2 w ( t ) is white noise. Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 7/12

State estimation ◮ The state variables are: z T � T c T p T v T q T � x = c = ( f x , f y , c x , c y , k 1 , k 2 ) are the parameters, p position, v velocity, q orientation, and z feature world coordinates. ◮ State space model: x k = A k x k − 1 + ε k , y k = h k ( x k ) + γ k , where A k depends on ω k and y k = ( u 1 , v 2 , . . . ) are the feature image coordinates. ◮ We use an Extended Kalman filter for inference. Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 8/12

Experiments The videos are available at: https://youtu.be/ro7TeQKgfT0 Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 9/12

Real-world experiment Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 10/12

Recap ◮ Online estimation of camera parameters ◮ Information fusion: ◮ Gyroscope-driven ◮ Feature-track observations ◮ Movement constrained by a Wiener velocity motion model ◮ Inference done by an EKF Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 11/12

◮ Link to codes can be found on my homepage! ◮ Homepage: http://arno.solin.fi ◮ Twitter: @arnosolin Robust gyroscope-aided camera self-calibration Cort´ es, Solin, Kannala 12/12