Realtime Gaze Estimation with Online Calibration Li Sun, Mingli - PowerPoint PPT Presentation

Realtime Gaze Estimation with Online Calibration Li Sun, Mingli Song, Zicheng Liu, Ming-Ting Sun

Outline  Introduction  Limitations & goals  Proposed method  Results  Demo

Gaze Estimation Dia iagnostic applications: • Eye disease diagnosis • Human behavior studies, e.g. visual attention analysis In Interactive applications: • Human-computer interaction • Gaze-contingent display

Gaze Estimation Methods 2D regression based Eye images, Screen Pupil-glint vector coordinates Regression Appearance/ Point of features gaze Support Vector Regression, Gaussian Processes, Neural Network Difficult to handle head movements, require many samples for calibration

Gaze Estimation Methods 3D model based Glints, pupil/iris Visual axis, center optical axis 3D model Image Line of gaze features Model the structure of the eye and the geometrical relationship of the eye and the scene Complex/expensive setup, user calibration for person-specific parameters

Limitations & Goals Limitations: Goals: • Intrusion • Remote gaze • Complex, estimation expensive setup • Simple, low-cost • Cannot tolerate setup head movements Broad • Free movements • Cumbersome application • Easy calibration/ calibration calibration-free • Cannot work in • Realtime realtime processing speed • Low accuracy • Accurate (<3 ° ) (>5 ° )

Proposed Method System Calibration Gaze Feature Extraction 3D Face Tracking Screen-Camera Calibration Face & Eye Detection Online (User) Calibration Iris Center Eye Corner Screen-Camera Location Location Parameters & subject-specific Parameters Gaze Features 3D Gaze Estimation

3D Model

3D Gaze Estimation In Input: • Gaze features: 𝐒 ℎ , 𝐪 𝑗 , 𝐪 𝑏 , 𝑨 𝑏 • Screen-camera parameters: 𝐏 s , 𝐎 𝑡 , 𝐎 𝑣 , 𝐎 𝑤 𝑥 𝑡 , ℎ 𝑡 , 𝑥 𝑠 , ℎ 𝑠 , 𝑔, 𝐩 • Personal parameters: 𝐼 𝑠 𝑓 , 𝐖 𝑏𝑓 Output: Gaze point: 𝐐 𝑕 , 𝐪 𝑕

3D Gaze Estimation 1. Compute the 3D position of 𝐐 a : 𝐪 𝑏 − 𝐩 = 𝑔 𝑦 𝑏 −𝑧 𝑏 𝑨 𝑏 2. Calculate the offset vector 𝐖 𝑏𝑓 and obtain the eyeball center 𝐏 𝑓 : 𝐼 𝐖 𝑏𝑓 = 𝐒 ℎ 𝐖 𝑏𝑓 𝐏 𝑓 = 𝐒 ℎ 𝐖 𝑏𝑓 + 𝐐 𝑏 3. Find the iris center 𝐐 𝑗 : 𝐪 𝑗 − 𝐩 = 𝑔 𝑦 𝑗 −𝑧 𝑗 𝑨 𝑗 𝐐 𝑗 − 𝐏 𝑓 2 = 𝑠 𝑓 4. Obtain the gaze direction 𝐎 𝑕 : 𝐎 𝑕 = 𝐐 𝑗 − 𝐏 𝑓 𝑠 𝑓

3D Gaze Estimation 5 obtain the 3D gaze point 𝐐 𝑕 : 𝐐 𝑕 = 𝐏 𝑓 + 𝑚𝐎 𝑕 𝐏 𝒇 − 𝐐 𝒉 ∙ 𝐎 𝑡 λ = 𝑃 𝑓 𝑄 𝑕 2 = − 𝐎 𝑕 ∙ 𝐎 𝑡 = − 𝐏 𝒇 ∙ 𝐎 𝑡 +𝑒 𝐎 𝑕 ∙ 𝐎 𝑡 6 obtain the 2D gaze point 𝐪 𝑕 (𝑣 𝑕 , 𝑤 𝑕 ) : 𝑕 − 𝐏 𝑡 ∙ 𝐎 𝑣 − 𝑥 𝑡 𝐐 𝑣 𝑕 = 0 𝑥 𝑠 𝑕 − 𝐏 𝑡 ∙ 𝐎 𝑤 − ℎ 𝑡 𝐐 𝑤 𝑕 = 0 ℎ 𝑠

Gaze Feature Extraction Gaze features: Gaze Feature Extraction • Rotation matrix 𝐒 ℎ • Depth value 𝑨 𝑏 3D Face Tracking • 2D iris location 𝐪 𝑗 • 2D anchor point location 𝐪 𝑏 The anchor point: Face & Eye Detection Inner corner of the left eye Iris Center Eye Corner Location Location

Gaze Feature Extraction 3D face tracking: Microsoft face tracking SDK[1] Face & eye detection: OpenCV[2] Visual Context Boosting[3] Eye corner detection: Template filtering [4]

Online Calibration Offline calibration: 𝑓 𝐎 𝑕 ∙ 𝐖 𝑦 + 𝑥 𝐼 + 𝑠 𝐼 + 𝑠 𝑔 ∙ 𝑦 𝑏 + 𝐒 𝑦 ∙ 𝐖 𝑏𝑓 2 − 𝑣 𝑝 𝑨 𝑏 + 𝐒 𝑨 ∙ 𝐖 𝑏𝑓 𝑓 𝐎 𝑕 ∙ 𝐖 𝑨 = 0 𝑓 𝐎 𝑕 ∙ 𝐖 𝑧 + 𝑤 𝑝 − ℎ 𝐼 + 𝑠 𝐼 + 𝑠 𝑔 ∙ 𝑧 𝑏 + 𝐒 𝑧 ∙ 𝐖 𝑏𝑓 𝑨 𝑏 + 𝐒 𝑨 ∙ 𝐖 𝑏𝑓 𝑓 𝐎 𝑕 ∙ 𝐖 𝑨 = 0 2 where 𝐖 𝑦 = 1,0,0 T , 𝐖 𝑧 = 0,1,0 T , 𝐖 𝑨 = 0,0,1 T 𝐒 𝑦 𝐒 𝑧 𝐒 ℎ = 𝐒 𝑨 For T calibration points, there are 2T equations with 4 unknowns. • There exist noise calibration points (changes in luminance, head pose, target position, etc), the calibration result varies from time to time. • How many calibration points are required to ensure satisfactory calibration result? • Too many calibration points have a large impact on the usability of the system.

Online Calibration Rewrite the linear system into the form 𝐁𝐲 = 𝐜 , the least square solution can be obtained from 𝐁 𝑈 𝐁𝐲 = 𝐁 𝑈 𝐜 . Online calibration: Given a new calibration point with coefficient matrix 𝐃 𝑢+1 and column vector 𝐞 𝑢+1 , the coefficient matrix and column vector can be updated as follows: 𝑈 𝑈 𝐁 𝑢 + 𝐃 𝑢+1 𝑈 𝐁 𝑢+1 𝐁 𝑢+1 = 𝐁 𝑢 𝐃 𝑢+1 𝑈 𝑈 𝑐 𝑢 + 𝐃 𝑢+1 𝑈 𝐁 𝑢+1 𝐜 𝑢+1 = 𝐁 𝑢 𝐞 𝑢+1 • During online calibration, the eye parameters are constantly improved with increasing gaze estimation accuracy. • The online calibration completes as soon as the updates of eye parameters reach convergence. • Increasing calibration points average out the impact of noise calibration points.

Results System setup: Kinect sensor 19” LCD screen Desktop PC 4GB memory Intel Core2 Quad CPU Line of Q9550 2.83GHz Point of gaze nVidia GT310 gaze Kinect Screen User

Results Online calibration: The updates of eye parameters (unit: mm) Number of calibration points Number of calibration points

Results Online calibration vs. offline calibration Average gaze estimation error (unit: degree) over first n calibration points from the online calibration and offline calibration with fixed number (5, 9, 16) of calibration points.

Results Gaze estimation accuracy: 𝑰 (mm) Offset vector V 𝒃𝒇 Accuracy Eyeball User radius 𝒔 𝒇 (degree) 𝑦 𝑧 𝑨 1 1.7703 11.0 -16.1 4.7 6.1 2 1.7276 9.4 -14.3 2.0 1.0 3 1.9243 10.2 -13.0 3.3 3.8 4 1.8547 10.8 -14.7 5.2 6.4

Results Gaze estimation accuracy:

Results Processing speed: Computational cost in ms of the major steps of the proposed method. Eye Face Eye Blink Iris Gaze Total Corner 5.78 4.98 0.38 18.14 0.23 0.64 30.15

Demo: 3D Chess Game

References [1] Microsoft Kinect SDK, http://www.microsoft.com/en-us/kinectforwindows/develop/, 2013. [2] OpenCV, http://opencv.org/, 2013 [ 3] Z. Sun M. Song, D. Tao and X. Li, “Visual -context boosting for eye detection ,” IEEE Trans. Sys. Man Cyber. Part B , 40(6):1460 – 1467, 2010. [ 4] J. Zhu and J. Yang, “ Subpixel eye gaze tracking,” in AFGR , 2002.

Thank you!