CS 4495 Computer Vision Tracking 2: Particle Filters Aaron Bobick - - PowerPoint PPT Presentation

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Aaron Bobick School of Interactive Computing

CS 4495 Computer Vision Tracking 2: Particle Filters

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Administrivia

• PS5 due on Sunday Nov 10, 11:55pm
• PS6 out Thurs Nov 14, due Nov 24th
• EXAM: Tues before Thanksgiving, Nov 26th Covers

concepts and basics

• Problem set resubmission policy:
• Full questions only
• You get 50% credit to replace whatever you got last time on that

question.

• Must be submitted by: DEC 1. NO EXCEPTIONS.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

First some Matlab flow…

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking

• Slides still “adapted” from Kristen Grauman, Deva

Ramanan, but mostly from Svetlana Lazebnik

• And now more adaptations from Sebastian Thrun, Dieter

Fox and someone who did great particle filter illustrations but whose name is lost to web thievery….

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Recall: Some examples

• http://www.youtube.com/watch?v=InqV34BcheM

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Detection vs. tracking

…

t=1 t=2 t=20 t=21

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Detection vs. tracking

… Detection: We detect the object independently in each frame and can record its position over time, e.g., based on blob’s centroid or detection window coordinates

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Detection vs. tracking

… Tracking with dynamics: We use image measurements to estimate position of object, but also incorporate position predicted by dynamics, i.e., our expectation of object’s motion pattern.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking with dynamics

• Use model of expected motion to predict where objects

will occur in next frame, even before seeing the image.

• Intent:
• Do less work looking for the object, restrict the search.
• Get improved estimates since measurement noise is tempered by

smoothness, dynamics priors.

• Assumption: continuous motion patterns:
• Camera is not moving instantly to new viewpoint
• Objects do not disappear and reappear in different places in the

scene

• Gradual change in pose between camera and scene

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking as induction

• Base case:
• Assume we have initial prior that predicts state in absence of any

evidence: P(X0)

• At the first frame, correct this given the value of Y0=y0
• Given corrected estimate for frame t:
• Predict for frame t+1
• Correct for frame t+1

predict correct

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Steps of tracking

• Prediction: What is the next state of the object given past

measurements?

• Correction: Compute an updated estimate of the state

from prediction and measurements

• Tracking can be seen as the process of propagating the

posterior distribution of state given measurements across time

( )

1 1

, ,

− − =

=

t t t

y Y y Y X P 

( )

t t t t t

y Y y Y y Y X P = = =

− −

, , ,

1 1



Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Simplifying assumptions

• Only the immediate past matters
• Measurements depend only on the current state

( ) ( )

t t t t t t

X Y P X Y X Y X Y P =

− −

, , , ,

1 1



( ) ( )

1 1

, ,

− −

=

t t t t

X X P X X X P 

• bservation model

dynamics model

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Linear Dynamic Models

• Dynamics model: state undergoes linear tranformation

plus Gaussian noise

• Observation model: measurement is linearly transformed

state plus Gaussian noise

( )

t

d t t t

x D N X Σ

− ,

~

1

( )

t

m t t t

x M N Y Σ , ~

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Example: Constant velocity (1D)

• State vector is position and velocity
• Measurement is position only

ξ ε + = + ∆ + =

− − − 1 1 1

) (

t t t t t

v v v t p p

      =

t t t

v p x

noise v p t noise x D x

t t t t t

+             ∆ = + =

− − − 1 1 1

1 1

(greek letters denote noise terms)

[ ]

noise v p noise Mx y

t t t t

+       = + = 1

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Example: Constant acceleration (1D)

• State vector is position, velocity, and acceleration
• Measurement is position only

[ ]

noise a v p noise Mx y

t t t t t

+           = + = 1

ζ ξ ε + = + ∆ + = + ∆ + =

− − − − − 1 1 1 1 1

) ( ) (

t t t t t t t t

a a a t v v v t p p

          =

t t t t

a v p x

noise a v p t t noise x D x

t t t t t t

+                     ∆ ∆ = + =

− − − − 1 1 1 1

1 1 1

(greek letters denote noise terms)

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

The Kalman filter

• Method for tracking linear dynamical models in Gaussian

noise

• The predicted/corrected state distributions are Gaussian
• You only need to maintain the mean and covariance
• The calculations are easy (all the integrals can be done in closed

form)

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Propagation of Gaussian densities

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

The Kalman Filter: 1D state

( )

1

, ,

− t t

y y X P 

( )

t t

y y X P , ,

0 

Predict Correct

Given corrected state from previous time step and all the measurements up to the current one, predict the distribution over the current step Given prediction of state and current measurement, update prediction of state

Time advances (from t–1 to t)

− − t t σ

µ ,

Mean and std. dev.

• f predicted state:

+ + t t σ

µ ,

Mean and std. dev.

• f corrected state:

Make measurement

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking with KFs: Gaussians!

update initial estimte x y x y prediction x y measurement x y

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Kalman Filters

prior Measurement evidence posterior

) ( ) | ( ) | ( x p x z p z x p ∝ dx x p x x p x p ) ( ) | ' ( ) ' (

∫

=

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

1D Kalman filter: Prediction

• Linear dynamic model defines predicted state evolution,

with noise

• Want to estimate distribution for next predicted state
• Update the mean:
• Update the variance:

+ − − = 1 t t

dµ µ

( )

( )

2 1

) ( , , ,

− − −

=

t t t t

N y y X P σ µ 

2 1 2 2

) ( ) (

+ − −

+ =

t d t

dσ σ σ

( )

2 1,

~

d t t

dx N X σ

−

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

1D Kalman filter: Correction

• Mapping of state to measurements:
• Predicted state:
• Want to estimate corrected distribution
• Update the mean:
• Update the variance:

( )

2

, ~

m t t

mx N Y σ

( )

( )

2 1

) ( , , ,

− − −

=

t t t t

N y y X P σ µ 

( )

( )

2

) ( , , ,

+ +

=

t t t t

N y y X P σ µ 

2 2 2 2 2

) ( ) (

− − − +

+ + =

t m t t m t t

m my σ σ σ σ µ µ

2 2 2 2 2 2

) ( ) ( ) (

− − +

+ =

t m t m t

m σ σ σ σ σ

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

A Quiz

prior Measurement evidence

( | ) ( | ) ( ) p x y p y x p x ∝

posterior?

Does this agree with your intuition?

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Kalman filter pros and cons

• Pros
• Simple updates, compact and efficient
• Cons
• Unimodal distribution, only single hypothesis
• Restricted class of motions defined by linear model
• So what might we do if not Gaussian?
• E.g. Object seems to be detected (noisy) in two locations in the next image

that are both plausible in terms of dynamics and belief where the object was in the last image.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Propagation of general densities

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Before we go any further…

• In particle filtering the measurements are written as 𝑨𝑢 and

not as 𝑧𝑢

• So we’ll start seeing z’s

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Particle Filters: Basic Idea

) (x p

t

x

Goal: 𝑞 𝑦𝑢 ∈ 𝑌𝑢 ≈ 𝑞 𝑦𝑢 𝑨 1…𝑢 with equality when 𝑜 → ∞ set of n (weighted) particles Xt Density is represented by both where the particles are and their

• weight. 𝑞(𝑦 = 𝑦0)is now probability of drawing an x with value 𝑦0.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

A slight shift…

• For just tracking we had the notion of a dynamics model –

and for the Kalman filter a linear dynamics model.

• At each time step the prediction was based upon linear transform of

state so could easily include velocity, acceleration, etc.

• The linear form was necessary to make the math work out. To do

non-linear dynamics need to linearize about the current state (remember Jacobians?).

• In general, the state transition matrix represented what

was known or expected to change from t to t+1.

• We can generalize that notion of action or input ut at time

t.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

• Given:
• Stream of observations z and action data u:
• Sensor model 𝑞(𝑨|𝑦) .
• Action model 𝑞(𝑦𝑢|𝑣 𝑢−1 , 𝑦 𝑢−1 )
• Prior probability of the system state p(x).
• Wanted:
• Estimate of the state X of a dynamical system.
• The posterior of the state is also called Belief:

Bayes Filters: Framework

) , , , | ( ) (

1 2 1 t t t t

z u z u x P x Bel

−

= 

1 2 1

{ , , , }

t t t

data u z u z

−

= 

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Underlying Assumptions

• Static world
• Independent noise

Graphical Model Representation

1: 1 1: 1 1: 1

( | , , ) ( | , )

t t t t t t t

p x x z u p x x u

− − −

=

0: 1: 1 1:

( | , , ) ( | )

t t t t t t

p z x z u p z x

−

=

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Bayes Rule reminder

( | ) ( ) ( ( | ) ) | ) ( ) ( p z x p x p x z p z p z x p x η = =

Prior before measurement

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Bayes Filters

) , , , | ( ) , , , , | (

1 2 1 1 2 1 − −

=

t t t t t

u z u x P u z u x z P   η

Bayes z = observation u = action x = state

) , , , | ( ) (

1 2 1 t t t t

z u z u x P x Bel

−

= 

Markov

) , , , | ( ) | (

1 2 1 −

=

t t t t

u z u x P x z P  η

1 1 1 1

) ( ) , | ( ) | (

− − − −

∫

=

t t t t t t t

dx x Bel x u x P x z P η

Markov 1 1 2 1 1 1 1

) , , , | ( ) , | ( ) | (

− − − − −

∫

=

t t t t t t t t

dx u z u x P x u x P x z P  η

1 2 1 1 1 1 2 1 1

( | ) ( | , , , , ) ( | , , , · )

t t t t t t t t

P z x P x u z u x P x u z u dx η

− − − − −

=

∫

 

Total Probability of “Prior”

Prediction before taking measurement

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

To illustrate…

• Assume a simple robot with a noisy range sensor looking

to its side…

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

When density is not easily represented analytically (ie a couple of Gaussians), represent by a set of (possibly weighted) samples

Particle Filters

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

( " "| ) p z hole x =

Sensor Information (aka Importance Sampling) Robot sees a hole

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

∫

←

−

' d ) ' ( ) ' | ( ) (

,

x x Bel x u x p x Bel

Robot Motion

Move and Resample

u

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Next Sensor Reading Robot sees a hole again

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Robot Moves Again

Move and Resample again

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

• 1. Algorithm particle_filter

2.

• 3. For Resample (generate i new samples)

4. Sample index j(i) from the discrete distribution given by wt-1 5. Sample from using and Control 6. Compute importance weight (or reweight) 7. Update normalization factor 8. Insert

• 9. For

10. Normalize weights

Particle Filter Algorithm (Sequential Importance Resampling)

, = ∅ = η

t

S n i  1 = } , { > < ∪ =

i t i t t t

w x S S

i t

w + =η η

i t

x

1

( | , )

t t t

p x x u

− ) ( 1 i j t

x −

t

u ) | (

i t t i t

x z p w = n i  1 = η /

i t i t

w w =

1 1 1

{ } , , ,

j j t t t t t

S x w u z

− − −

< > =

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Graphical steps particle filtering

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

A real robot sensing problem

• Assume a robot knows a 3D map of its world
• It has a noisy depth sensor or sensors whose sensing

uncertainty is known.

• It moves from frame to frame.
• How well can it know where it is (x,y,θ) ?

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Proximity Sensor Model

Laser sensor Sonar sensor No return!

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Sonar Example: Initial Distribution

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

After Incorporating Ten Ultrasound Scans

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

After Incorporating 65 Ultrasound Scans

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Estimated Path

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Sample-based Localization (sonar)

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Smithsonian Museum of American History…

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

63

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

64

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

65

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

66

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

How about simple vision….

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Using Ceiling Maps for Localization

Dellaert, et al. 1997

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Vision-based Localization

P(z|x) h(x) z

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Under a Light

Measurement z: P(z|x):

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Next to a Light

Measurement z: P(z|x):

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Elsewhere

Measurement z: P(z|x):

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Global Localization Using Vision

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Odometry Only

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Using Vision

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

• Given: Set S of weighted samples.
• Wanted : Random sample, where the probability
• f drawing xi is given by wi.
• Typically done n times with replacement to

generate new sample set S’.

• Or even not done except when needed…too many low

weight particles.

A detail: Resampling method can matter

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick w2 w3 w1 wn Wn-1

Resampling

w2 w3 w1 wn Wn-1

• Roulette wheel
• Binary search, n log n
• Stochastic universal sampling
• Systematic resampling
• Linear time complexity
• Easy to implement, low variance

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

• 1. Algorithm systematic_resampling(S,n):

2.

• 3. For

Generate cdf 4. 5. Initialize threshold

• 6. For

Draw samples … 7. While ( ) Skip until next threshold reached 8. 9. Insert

• 10. Increment threshold
• 11. Return S’

Resampling Algorithm

1 1

, ' w c S = ∅ = n i  2 =

i i i

w c c + =

−1 1 1 ~

[0, ], 1 u U n i

−

= n j  1 =

1 1 − +

+ = n u u

j j i j

c u >

{ }

> < ∪ =

−1

, ' ' n x S S

i

1 + = i i

(Also called stochastic universal sampling)

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

• If dealing with highly peaked observations
• Add noise to observation and prediction models
• Better proposal distributions: e.g., perform Kalman filter step to

determine proposal

• Overestimating noise often reduces number of required

samples

• Recover from failure by selectively adding samples from
• bservations
• Recover from failure by uniformly adding some samples
• Can Resample only when necessary (efficiency of

representation measured by variance of weights)

PF: Practical Considerations

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Next time?

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Initialization
• Manual
• Background subtraction
• Detection

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Initialization
• Obtaining observation and dynamics model
• Dynamics model: learn (very difficult) or specify using domain

knowledge

• Generative observation model: “render” the state on top of the

image and compare

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Initialization
• Obtaining observation and dynamics model
• Prediction vs. correction
• If the dynamics model is too strong, will end up ignoring the

data

• If the observation model is too strong, tracking is reduced to

repeated detection

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Initialization
• Obtaining observation and dynamics model
• Prediction vs. correction
• Data association
• What if we don’t know which measurements to associate with

which tracks?

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Data association

• So far, we’ve assumed the entire

measurement to be relevant to determining the state

• In reality, there may be

uninformative measurements (clutter) or measurements may belong to different tracked objects

• Data association: task of

determining which measurements go with which tracks

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Source: Lana Lazebnik

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Doesn’t always work… Alternative: keep track of multiple hypotheses at once…

Source: Lana Lazebnik

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

• More sophisticated strategy: keep track of multiple

state/observation hypotheses

• Can be done with particle filtering
• This is a general problem in computer vision, there is no

easy solution

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking issues

• Initialization
• Obtaining observation and dynamics model
• Prediction vs. correction
• Data association
• Drift
• Errors caused by dynamical model, observation model, and

data association tend to accumulate over time

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Drift

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking people by learning their appearance

• Person model = appearance + structure

(+ dynamics)

• Structure and dynamics are generic, appearance is

person-specific

• Trying to acquire an appearance model “on the fly” can

lead to drift

• Instead, can use the whole sequence to initialize the

appearance model and then keep it fixed while tracking

• Given strong structure and appearance models, tracking

can essentially be done by repeated detection (with some smoothing)

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking people by learning their appearance

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracker

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Representing people

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Bottom-up initialization: Clustering

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Top-down initialization: Exploit “easy” poses

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Tracking by model detection

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Tracking 2: Particle Filters CS 4495 Computer Vision – A. Bobick

Example results

http://www.ics.uci.edu/~dramanan/papers/pose/index.html