Model-based Visual Tracking: the OpenTL framework Giorgio Panin - - PowerPoint PPT Presentation

Technische Universität München

Dr.–Ing. Giorgio Panin

Model-based Visual Tracking: the OpenTL framework

Technische Universität München · Institut für Informatik Lehrstuhl für Echtzeitsysteme und Robotik (Prof. Alois Knoll)

Giorgio Panin

Technische Universität München

Dr.–Ing. Giorgio Panin

Contents

• Object tracking: theory
• Building applications with OpenTL

• G. Panin

27.04.2010

Object tracking

• G. Panin

27.04.2010

O1 O2 W Ci C1

Goal: Multi-Target / -Sensor / -Modal localization

• G. Panin

27.04.2010

Model-based tracking

Video surveillance Control/navigation Face tracking

...

Model Visual processing Localization (tracking)

• G. Panin

27.04.2010

Ground shape

O1 O2 W

• G. Panin

27.04.2010

Pose parameters – single-body transforms

Base Euclidean Similarity Affine Homography 2D 3D Invariant properties Distances Angles Parallel lines Straight lines Angles Parallel lines Straight lines Parallel lines Straight lines Straight lines

• G. Panin

27.04.2010

Pose parameters – articulated body

,l W

T

1 0 ,l

l

T

ρ

2 1,l

l

T

3 2 ,l

l

T

3 3

, l l W W

x T x =

x

W

• G. Panin

27.04.2010

Pose parameters – deformable shapes

) (

,

p T

W

W

) (

, 0 W

T

x y y x z z x y y x

) (

,

p T

W

W

Link 0 Link 0

) (

, 0 W

T

• G. Panin

27.04.2010

Active Shape Model (2D) – face tracking Learning deformation modes from examples (PCA)

• G. Panin

27.04.2010

Object appearance

Key-frames Texture map Active Appearance Model

• G. Panin

27.04.2010

Object dynamics 2nd order, Auto-Regressive model

Damped-spring motion Motion type: specified by three main parameters

• Average state:
• Oscillation frequency: f
• Damping rate: β

t t t t

w W x x F x x F x x

2 2 1 1

) ( ) ( + − + − = −

− −

x

x x

Process noise

• G. Panin

27.04.2010

Object dynamics - examples

1 . 1 2

• n

Accelerati Noise White

2 1

= − = = = = W F F f β 1 . 1 99 . 1 1 . : undamped Periodic,

2 1

= − = = = = W F F f β 88 . 0.99 1.98 05 . 1 . : damped Periodic,

2 1

= − = = = = W F F f β

1 . 1 undefined) , ( Motion Brownian

2 1

= = = W F F f β

12 . 2 0.9 1.9 5 . : damped) y (criticall Aperiodic

2 1

= − = = = = W F F f β

• G. Panin

27.04.2010

Object dynamics – multi-dim.

        =         = 12 . 2 0.9

• 1.9

I W I I I F         =         = 12 . 2 I W I I F

t=50 t=1000 Unconstrained Constrained

( )

t t t

W F I F w s s s + − + =

−1

t=10000

• G. Panin

27.04.2010

Object model

t0 t1 t2 t3 t0 ti tk t0 t2 t1

Shape Appearance Pose Dynamics

• G. Panin

27.04.2010

Camera model

C c

• ptical axis

retinal plane f x y

Intrinsic parameters

Radial distortion Pin-hole model

zc xc yc

• G. Panin

27.04.2010

Camera model

w c

T ,

1

w c

T

,

2

w

x

1

c

2

c w

Extrinsic parameters

• G. Panin

27.04.2010

Camera model

Depth map

y1 y2

W

C

O

x y

w c

T ,

• w

T ,

Object-to-image projection (and back-projection)

• G. Panin

27.04.2010

Visual modalities

Local keypoints Texture template Optical flow Color statistics Shape moments Intensity gradients Contour lines (and others: Background subtraction / CCD / Harris keypoints / Histogram of oriented gradients / SIFT)

• G. Panin

27.04.2010

The „tracking pipeline“

− t

s

Image features Rendered view

− t

s

− t

s

Re-projection

t

s

New features Prediction Data acquisition Pre-processing Sampling model features Matching Update model features

Targets prediction Data acquisition Off-line features sampling Matching Data fusion Targets update On-line features update Pre- processing Output

Back-projection

• G. Panin

27.04.2010

Abstraction: visual modality processing

Rule: any modality class must implement

• Model free pre-processing
• Off-line and on-line features sampling and back-projection
• Multi-level data association (Pixel-, Feature-, State-space)
• Residuals, covariances and Jacobians computation

Additional classes:

• Multi-modal, multi-sensor data fusion (cascade, parallel)
• Likelihood computation

• G. Panin

27.04.2010

Features sampling – GPU assisted

• G. Panin

27.04.2010

Multi-level visual processing

h = s- (predicted pose) z = s* (LSE estimate) Object-level measurement Feature-level measurement Pixel-level measurement h(s) z e = z-h h(s) z

• G. Panin

27.04.2010

Measurement: pixel- vs. feature-level

v v Lagrangian = feature-level Eulerian = pixel-level

Analogy with fluid mechanics

Dense optical flow (HS) Sparse optical flow (LK)

• G. Panin

27.04.2010

Feature-level: re-projection vs. tracking

Model feature re-projection Feature tracking (= „flow“)

• G. Panin

27.04.2010

Feature-level: validation gates (local search)

) , (

− − P

s

Prior density (state-space)

1

x

2

x

3

x

1 1 ,S −

y

2 2,S −

y

3 3 ,S −

y

i i S

,

−

y

) , (

i i S −

y

Innovation densities (measurement space)

• G. Panin

27.04.2010

Example: color histograms

Pixel-level matching Feature-level matching = histogram distance Object-level matching = mean-shift optimization Model Color segmentation

• G. Panin

27.04.2010

Example: intensity edges

Pixel-level Feature-level

sample contour points Re-project and search in the image Draw the silhouette Match silhouette to the Distance Transform

• G. Panin

27.04.2010

Weighted Average Blobs Joint likelihood

Feat Pix

Joint MLE

Feat

) | ( ) | ( ) | (

− − − −

⋅ = = s s s

MLE

• bj

blobs

Z P Z P Z P

Feat Pix Pix Obj

Edges Motion Keypoints Color segmentation

s-

Building in OpenTL

Multi-camera, multi-level data fusion

View 1 View 2

• G. Panin

27.04.2010

Static fusion Mod5 Dynamic fusion

Feat Pix

Static fusion

Feat

) | (

−

s Z P

I1 I2 Feat Pix Pix Obj

Mod3 Mod2 Mod4 Mod1

s-

Building processing trees in OpenTL

Generalization of the tree

• G. Panin

27.04.2010

Data fusion – multi-modal

Background Color model Pixel-wise (AND) Benefits:

• Combine independent information sources
• Increase robustness (tracking fails if ALL modalities fail)

Drawbacks:

• Need to define a proper fusion scheme, and parameters
• Higher computational effort  slower frame rate

• G. Panin

27.04.2010

Data fusion – multi-camera

Complimentary setup Redundant setup

• G. Panin

27.04.2010

Data fusion – multi-camera

Redundant setup: 3D hand tracking Complimentary setup: Indoor people tracking

• G. Panin

27.04.2010

Multi-target – occlusion handling

Pixel-level Feature-level Data (multi-class segmentation)

• G. Panin

27.04.2010

Bayesian Tracking: prediction - correction

Gaussian filters

• (Extended) Kalman filter
• Information filter
• Unscented Kalman/Information filter

Monte-Carlo filters

• S-I-R particle filter
• MCMC particle filter

• G. Panin

27.04.2010

Representing the target distribution

) ,..., , | (

1

z z z s P

t t t −

Prior density

) ,..., | (

1

z z s P

t t −

Posterior density

ML / MAP Kalman Unscented Kalman

• Mix. Of Gaussians

Kernel Particle Histogram (Eulerian) True density

• G. Panin

27.04.2010

Flow diagram of OpenTL-based applications

Local processing Detection/ Recognition Bayesian tracking

t

Meas

t

Obj

t

I

t

I t

Shape Appearance Degrees of freedom Dynamics Sensors Environment Models

− t

Obj

1 − t

Obj

∆t

+ −1 t

Obj

Post-processing

Track Maintainance Track Initiation

• G. Panin

27.04.2010

Object detection (examples)

• General-purpose Monte-Carlo sampling in state-space
• People detection based on foreground blobs clustering
• Invariant keypoints matching (for textured objects)
• Marker detection based on intensity edges
• Hand detection based on color and edge lines
• Face detection based on a trained classifier (with Haar features)

• G. Panin

27.04.2010

Object Tracking Pre-processing Visual processing Data fusion Tracking Target Update Measurement Detection/ Recognition Target Prediction Models Objects Sensors Environment Features Sampling Occlusion Handling Data association

Resume: model-based object tracking

• G. Panin

27.04.2010

Building applications with OpenTL

• G. Panin

27.04.2010

• G. Panin

27.04.2010

Features of OpenTL

• Modularized, object-oriented software architecture
• Common abstractions for layers
• Real-time performance
• Different Bayesian filters
• A large variety of visual modalities, with multiple processing levels
• Robust improvements (multi-hypotheses, data fusion, …)
• Generic sensor abstraction
• Support multi-camera, multi-target and multi-modal applications
• Support for GPU acceleration

• G. Panin

27.04.2010

Application: planar object tracking

Color histograms Likelihood Particle Filter ) | (

−

s Z P

feat

• Meas. processing

−

s

s

feat

Z

Modeling Tracking

Single-target, single-camera, color-based

• G. Panin

27.04.2010

• G. Panin

27.04.2010

• G. Panin

27.04.2010

Background subtraction Color statistics Image fusion Blobs Likelihood Zfeat Zpix

s-

Zpix Zpix Particle Filter

s

) | (

−

s Z P

feat

Application: planar object tracking

Fusion color+background (pixel-level), and blob detection

• G. Panin

27.04.2010

• Distributed, multi-camera

setup

• People detection and

tracking in real-time

• 3D localization
• Goal:

Human-Robot Interaction (coffee-break scenario) CoTeSys – ITrackU

People tracking with distributed cameras

• G. Panin

27.04.2010

Hierarchical grid-based localization

• G. Panin

27.04.2010

Stereo tracking of a quadcopter

• G. Panin

27.04.2010

• G. Panin

27.04.2010

Object-level upgrade Kalman Filter CCD

D

s2

) ( 2 i D

s

− D

s2

feat

Z

• bj

Z

Object-level upgrade Kalman Filter Template matching

D

s3

) ( 3 i D

s

− D

s3

feat

Z

• bj

Z

2D3D pose upgrade

D

s2

) ( 3D

s

Application: face tracking

Cascade of two pipelines, with 3D pose upgrade

2D tracking 3D tracking 3D model

• G. Panin

27.04.2010

• G. Panin

27.04.2010

Application: 3D hand tracking

• G. Panin

27.04.2010

Technische Universität München

Dr.–Ing. Giorgio Panin

Our Tracking Projects

• Pedestrian Tracking
• Vivid cell tracking
• Human Robot Interaction
• VR TV Studio Automation
• Quadcopter tracking
• ITrackU (CCRL)

www.opentl.org

Technische Universität München

Dr.–Ing. Giorgio Panin

Our Team

• graduate and post-doc coworkers
• student coworkers
• manifold research projects
• funded by DFG, EU, industry partners
• focus on robust, real-world applications

ITrackU (Panin)