Model-based Visual Tracking: the OpenTL framework Giorgio Panin - PowerPoint PPT Presentation

Technische Universität München Model-based Visual Tracking: the OpenTL framework Giorgio Panin Technische Universität München · Institut für Informatik Lehrstuhl für Echtzeitsysteme und Robotik (Prof. Alois Knoll) Dr.–Ing. Giorgio Panin

Technische Universität München Contents • Object tracking: theory • Building applications with OpenTL Dr.–Ing. Giorgio Panin

Object tracking G. Panin 27.04.2010

Goal: Multi-Target / -Sensor / -Modal localization O 1 O 2 C i C 1 W G. Panin 27.04.2010

Model-based tracking Visual processing Localization (tracking) Model Video surveillance Control/navigation Face tracking ... G. Panin 27.04.2010

Ground shape O 2 O 1 W G. Panin 27.04.2010

Pose parameters – single-body transforms Base Euclidean Similarity Affine Homography 2D 3D Distances Angles Parallel lines Straight lines Invariant Angles Parallel lines Straight lines properties Parallel lines Straight lines Straight lines G. Panin 27.04.2010

Pose parameters – articulated body T W , l 0 T l 0 , l 1 W ρ T l 1 , l 2 T l 2 , l = 3 x T x x W W , l l 3 3 G. Panin 27.04.2010

Pose parameters – deformable shapes x x Link 0 T , 0 ( ) W 0 Link 0 y y T ( p ) W , 0 W z z y y x T , 0 ( ) W 0 x T ( p ) W , 0 W G. Panin 27.04.2010

Active Shape Model (2D) – face tracking Learning deformation modes from examples (PCA) G. Panin 27.04.2010

Object appearance Key-frames Texture map Active Appearance Model G. Panin 27.04.2010

Object dynamics 2nd order, Auto-Regressive model Process noise Damped-spring motion − = − + − + x x F ( x x ) F ( x x ) W w − − t 1 t 1 2 t 2 0 t x Motion type: specified by three main parameters x x • Average state: • Oscillation frequency : f Damping rate : β • G. Panin 27.04.2010

Object dynamics - examples White Noise Accelerati on f β Brownian Motion ( , undefined) = β = f 0 0 = = = 1 2 0 F 1 F 0 W 0 . 1 = = − = 1 2 0 F 2 F 1 W 0 . 1 Periodic, undamped : Aperiodic (criticall y damped) : Periodic, damped : = β = = β = f 0 . 1 0 = β = f 0 0 . 5 f 0 . 1 0 . 05 1 = 2 = − 0 = F 1 . 99 F 1 W 0 . 1 1 = 2 = − 0 = 1 = 2 = − 0 = F 1.9 F 0.9 W 2 . 12 F 1.98 F 0.99 W 0 . 88 G. Panin 27.04.2010

( ) = + − + s F s I F s W w Object dynamics – multi-dim. − 1 t t t t=10000 t=50 t=1000 Constrained   1.9 I - 0.9 I =   F   I 0    2 . 12 I  =   W   0   Unconstrained   I 0   = F   I 0    2 . 12 I  =   W   0   G. Panin 27.04.2010

Object model Shape Appearance Pose t 2 t 1 t 0 Dynamics t 1 t i t 3 t 2 t 0 t 0 t k G. Panin 27.04.2010

Camera model Intrinsic parameters Radial distortion Pin-hole model x C x c optical axis z c c y c y f retinal plane G. Panin 27.04.2010

Camera model Extrinsic parameters c c 2 1 x w w T T , c , w c w 2 1 G. Panin 27.04.2010

Camera model Object-to-image projection (and back-projection) y 1 x y 2 y C Depth map O T , W T , c w w o G. Panin 27.04.2010

Visual modalities Shape moments Intensity gradients Contour lines Color statistics Texture template Optical flow Local keypoints (and others: Background subtraction / CCD / Harris keypoints / Histogram of oriented gradients / SIFT) G. Panin 27.04.2010

The „tracking pipeline“ Data Pre- Targets Matching Output acquisition processing update Targets Off-line features Data On-line features prediction sampling fusion update Back-projection − s − s t t s t Rendered view New features Prediction Sampling model features Update model features − s t Re-projection Image features Data acquisition Pre-processing Matching G. Panin 27.04.2010

Abstraction: visual modality processing Rule: any modality class must implement • Model free pre-processing • Off-line and on-line features sampling and back-projection • Multi-level data association (Pixel-, Feature-, State-space) • Residuals, covariances and Jacobians computation Additional classes: • Multi-modal, multi-sensor data fusion (cascade, parallel) • Likelihood computation G. Panin 27.04.2010

Features sampling – GPU assisted G. Panin 27.04.2010

Multi-level visual processing e = z-h h(s) z Pixel-level measurement h(s) z Feature-level measurement z = s* (LSE estimate) Object-level measurement h = s - (predicted pose) G. Panin 27.04.2010

Measurement: pixel- vs. feature-level Analogy with fluid mechanics Eulerian = pixel-level Lagrangian = feature-level v v Dense optical flow (HS) Sparse optical flow (LK) G. Panin 27.04.2010

Feature-level: re-projection vs. tracking Model feature re-projection Feature tracking (= „flow“) G. Panin 27.04.2010

Feature-level: validation gates (local search) Prior density (state-space) − P x − ( s , ) 1 x 3 Innovation densities x 2 (measurement space) − ( y i S , ) i − y i S , i − y 1 , S 1 − − y 3 , S y 2 , S 3 2 G. Panin 27.04.2010

Example: color histograms Model Color segmentation Pixel-level matching Feature-level matching = histogram distance Object-level matching = mean-shift optimization G. Panin 27.04.2010

Example: intensity edges Pixel-level Draw the silhouette Match silhouette to the Distance Transform Feature-level sample contour points Re-project and search in the image G. Panin 27.04.2010

Building in OpenTL Multi-camera, multi-level data fusion Pix Color segmentation View 1 Pix Feat Weighted Blobs Average s - Motion Pix Joint likelihood − = P ( Z | s ) − − = ⋅ P ( Z | s ) P ( Z | s ) − blobs obj MLE Feat Edges View 2 Joint MLE Obj Keypoints Feat G. Panin 27.04.2010

Building processing trees in OpenTL Generalization of the tree Pix Mod 1 I 1 Pix Feat Static Mod 5 fusion s - Mod 2 Pix − P ( Z | s ) Dynamic fusion Feat Mod 3 I 2 Static fusion Obj Mod 4 Feat G. Panin 27.04.2010

Data fusion – multi-modal Background Color model Pixel-wise (AND) Benefits: • Combine independent information sources • Increase robustness (tracking fails if ALL modalities fail) Drawbacks: • Need to define a proper fusion scheme, and parameters • Higher computational effort  slower frame rate G. Panin 27.04.2010

Data fusion – multi-camera Complimentary setup Redundant setup G. Panin 27.04.2010

Data fusion – multi-camera Complimentary setup: Indoor people tracking Redundant setup: 3D hand tracking G. Panin 27.04.2010

Multi-target – occlusion handling Pixel-level Data (multi-class segmentation) Feature-level G. Panin 27.04.2010

Bayesian Tracking: prediction - correction Gaussian filters • (Extended) Kalman filter • Information filter • Unscented Kalman/Information filter Monte-Carlo filters • S-I-R particle filter • MCMC particle filter G. Panin 27.04.2010

Representing the target distribution P ( s | z ,..., z ) P ( s | z , z ,..., z ) Prior density Posterior density − − t t 1 0 t t t 1 0 True density ML / MAP Kalman Mix. Of Gaussians Unscented Kalman Kernel Particle Histogram ( Eulerian ) G. Panin 27.04.2010

Flow diagram of OpenTL-based applications Track Maintainance t I Local − Meas Obj t t processing t Bayesian Obj Post-processing t tracking Track Initiation I t ∆ t + Detection/ Obj − 1 Recognition t Obj − t 1 Models Degrees of Dynamics Shape Appearance Sensors Environment freedom G. Panin 27.04.2010

Object detection (examples) • General-purpose Monte-Carlo sampling in state-space • People detection based on foreground blobs clustering • Invariant keypoints matching (for textured objects) • Marker detection based on intensity edges • Hand detection based on color and edge lines • Face detection based on a trained classifier (with Haar features) G. Panin 27.04.2010

Resume: model-based object tracking Sensors Objects Environment Models Target Pre-processing Prediction Object Tracking Features Measurement Sampling Visual Tracking processing Data Occlusion fusion Handling Detection/ Target Data Recognition Update association G. Panin 27.04.2010

Building applications with OpenTL G. Panin 27.04.2010

G. Panin 27.04.2010

Features of OpenTL • Modularized, object-oriented software architecture • Common abstractions for layers • Real-time performance • Different Bayesian filters • A large variety of visual modalities, with multiple processing levels • Robust improvements (multi-hypotheses, data fusion, …) • Generic sensor abstraction • Support multi-camera, multi-target and multi-modal applications • Support for GPU acceleration G. Panin 27.04.2010