Tracking moving objects form image sequences Janno Jgeva Mihkel - - PowerPoint PPT Presentation

Tracking moving objects form image sequences

Janno Jõgeva Mihkel Pajusalu

University of Tartu Faculty of Mathematics and Computer Science MTAT.03.260 Pattern Recognition and Image Analysis Tartu, 2011

GENERAL EFFECTS OF TIME

Static to dynamic

• Adding the fourth dimension
• Each picture is not a separate isolated

event

• Sequence of 2D pictures -> reconstruct 4D
• Picture is now in 3D: f(x,y,t)
• Human brain handles still image processing

as a motion problem

What happens if time passes?

• Objects move (or transform) and/or

camera moves

• Lighting changes
• Noise background changes

Movements

• Effectively produces different views of
• bjects and the general scene
• Helps to remove ambiguities
• Dynamic occlusions
• Objects might be missing in some pictures and (re)appear in
• thers producing problems

Lighting changes

• Positions of light
• Possibility of examining surface features
• The color of light
• Spectral properties
• Diffusion of light (sharp shadows to smooth

diffused ambient light)

• Different lighting conditions optimal for different goals

Noise changes

• Each picture has variety of noises
• Sensor noise
• Rounding errors (also under/over exposure)
• Shutter effects (rolling vs. global shutter)
• Analyzing time series allows time-domain

smoothing

• Reduces noise
• Makes analysis more precise

Camera's time resolution

• Camera stores a single picture
• Each pixel is average over time
• Shutter exposes pixels
• Global shutter: all pixels are exposed during the

same time window

• Rolling shutter: rows/columns of pixels or single

pixels are exposed in sequence

Different shutter speeds

http://upload.wikimedia.org/wikipedia/commons/b/b2/Windflower-05237-nevit.JPG

Rolling shutter: Skew

http://en.wikipedia.org/wiki/File:CMOS_rolling_shutter_distortion.jpg

Rolling shutter: Smear & Skew

http://upload.wikimedia.org/wikipedia/commons/4/46/Focalplane_shutter_distortions.jpg

Rolling shutter: Partial exposure

http://en.wikipedia.org/wiki/File:Lightning_rolling_shutter.jpg

MODELING TIME

Time sequence

• Each picture holds a projection of 3D

space at a time value

• A different view of 4D environment
• Goal: to construct the 4D environment from a

sequence of 2D pictures

• Requires modeling of time

Time: the most general case

• Time is a continuous variable
• Just like a spatial coordinate

Simplifications

• Causality:
• Each moment can only depend on previous moments, not

future moments

• Discretization of time:
• Time is divided into discrete moments
• Markov chain:
• Each future moment depends only on the current moment
• Knowing the current state is used to predict future
• Simplifies analysis greatly

Applications of Markov processes

• Markov filter
• Particle filter
• Kalman filter

Markov filter

• Generate a map and

visualize the probabilities

• Propagate model using

control and error estimation

• Multiply by a sensor data
• Changing of the scene

in time decreases ambiguities

• 1. R. Siegwart, I. Nourbakhsh, "Introduction to Autonomous Mobile Robots", The MIT Press,2004

Kalman filter

• Single estimate
• Updating the estimate probabilistically using

sensor data and belief

• 1. R. Siegwart, I. Nourbakhsh, "Introduction to Autonomous Mobile Robots", The MIT Press,2004

http://www.lce.hut.fi/~ssarkka/course_k2011/pdf/ handout3.pdf

Frequency domain

• Many problems are simplified in frequency domain
• Fourier transform properties
• Translation:
• Only phase changes upon translation!
• Moving object changes only phase, not amplitude of

Fourier transform

Frequency domain 2

• Convolution
• Much faster
• Both properties apply in 2D case
• 2D translation = shift in phase

MODELS OF MOVEMENT

Models

• Simple model
• Only translation between frames
• More complex approach
• Affine transformations

Brightness constancy constraint

• The brightness constancy constraint

(BCC) is satisfied if the colors of the spatio–temporal image points representing the same 3D points, remain unchanged throughout the spatio– temporal image

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Simple model

• Patches of image are translated by a vector
• Sufficient if time differences are small
• Points transform to lines
• Lines to planes
• A good book: Bigun J., "Vision with Direction A

Systematic Introduction to Image Processing and Computer Vision"

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Affine transformations

• Modeling only translation might not be enough
• In reality picture regions are translated, rotated and

scaled

• Affine transformations for point s=(x y)T :
• Speed vector field
• Matrix:

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

METHODS

Methods for analyzing image sequences

• Simple
• Using object/feature detection and

constructing time dependences

• Model fitting
• Optical flow
• Sparse optical flow
• Dense optical flow
• Frequency domain optical flow

Simple and straightforward

• Detect object to track in sequences
• Increase precision using Markov/Kalman

filtering

• Simple case: unambiguous
• Easily distinguishable objects, no occlusion
• Occlusion:
• A model of movement must be applied
• Hard to distinguish objects, bad noise

tolerance, bad precision

Example

"Handbook of Computer Vision and Applications, Volume 3: Systems and Applications", Bernd Jähne, Horst Haussecker, Peter Geissler Academic Press (January 1999)

Model fitting based

• Create a 3D model and fit
• Partially helps in case of occlusion
• Ambiguities still remain

Feature detection + prediction + fitting example

• Process
• Detect edges
• Predict transformation using edges
• Project transformed model
• Generate control points on model
• Fit control points and edges
• http://sites.google.com/site/jbarandiaran/

3dtracking

Optical flow

• Motion field analysis
• Try to deconstruct the image sequence into moving object

surface patches

• Each patch moves by a 2D vector

Enables estimation of Movement of objects Movement of camera Depth (3D vision using a single moving camera)

Motion Estimation by Differentials in Two Frames

• Definition of speed
• Difference of intensity f(x,y,t)

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Motion Estimation by Differentials in Two Frames

• Let's find a path s(t), where intensity is

constant (BCC): df/dt=0

• Equation system in a neighborhood: d=-Dv

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Motion Estimation by Differentials in Two Frames: Lucas-Kanade

• Solving the system: d=-Dv

DTd=DTDv v=(DTD)-1DTd dk=f(xk,yk,t1) - f(xk,yk,t0)

• S=DTD is called the structure tensor
• Solution exists if S is not singular

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

What are good features to track?

• The key is the structure tensor
• The larger the eigenvalues the better
• Corners
• Textures

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Sparse optical flow

• Only small portion of pixels are tracked

(mostly detected features)

• Demo (OpenCV)
• cvGoodFeaturesToTrack
• Uses corner detection
• cvOpticalFlowPyrLK
• Uses Pyramidal Lucas-Kanade

algorithm

Dense optical flow

• Every pixel is tracked
• Requires more complex algorithms in

regions where good features are absent

Motion Estimation by Spatial Correlation

• Basically find a similar patch near the
• riginal patch in next frame

„Vision with Direction: A Systematic Introduction to Image Processing and Computer Vision“ by Josef Bigün, Springer-Verlag Berlin Heidelberg 2006

Phase correlation optical flow

• Amplitudes of 2D Fourier transform do not

change when an object moves

• Only phases change
• An area where phase changes correlate likely belongs to the

same object

• Can be used to identify moving objects

Phase correlation optical flow

• The math
• Only phase changes when image is shifted (Ga and Gb are 2D

Fourier transforms of two frames)

• Calculating correlation in Fourier space
• Inverse Fourier transform

http://en.wikipedia.org/wiki/Phase_correlation

Cool applications

• http://www.2d3.com/
• Multiple temporal view advantages
• 3D/2D mapping from single camera
• Super resolution (synthetic aperture)
• Image stabilization
• NASA VISAR
• Virtualdub deshaker
• SLAM
• Optical glyph tracking