: 5.1.2 - - PowerPoint PPT Presentation

הביקע

רועישב רמוחל תורוקמ

• דומילה רפס:

– קרפ5.1.2

• Forsyth & Ponce

– קרפ18

• םינוש םירמאמ
• טנרטניאב רמוח!

Today

• …Tracking with Dynamics

– Detection vs. Tracking – Tracking as probabilistic inference – Prediction and Correction

• Linear Dynamic Models
• The Kalman Filter

– Kalman filter for 1D state – General Kalman filter – Limitations

Detection vs. Tracking

…

t=1 t=2 t=20 t=21

Detection vs. Tracking

• Detection

– We detect the object independently in each frame and can record its position over time, e.g., based on blob’s centroid

• r detection window coordinates.

…

t=1 t=2 t=20 t=21

Detection vs. Tracking

• Tracking with dynamics:

– We use image measurements to estimate the object position, but also incorporate the position predicted by dynamics, i.e.,

• ur expectation of the object’s motion pattern.

…

t=1 t=2 t=20 t=21

Tracking with Dynamics

• Key idea

– Given a model of expected motion, predict where objects will occur in next frame, even before seeing the image. – In next frame, update prediction using actual measurements

Illustration

update initial position x y x y prediction x y measurement x y

Tracking with Dynamics

• Key idea

– Given a model of expected motion, predict where objects will occur in next frame, even before seeing the image. – In next frame, update prediction using actual measurements

• Goals

– Restrict search for the object – Improved estimates since measurement noise is reduced by trajectory smoothness.

• Assumption: continuous motion patterns

– Camera is not moving instantly to new viewpoint. – Objects do not disappear and reappear in different places. – Gradual change in pose between camera and scene.

General Model for Tracking

state state x4 state x3 state x2 state x1

y2 y3 y4 measurement y1

state x4 state x3 state x2 state x1

General Model for Tracking

• The moving object of interest is characterized by an

underlying state X

• State X gives rise to measurements or observations Y
• At each time t, the state changes to Xt and we get a

new observation Yt

X1 X2 Y1 Y2 Xt Yt

…

State vs. Observation

• Hidden state : parameters of interest (e.g., location of

point, contour of shape, etc.)

• Measurement : what we get to directly observe (e.g., pixel

values, image features)

Tracking as Inference

• Our goal: recover most likely state Xt given

– All observations seen so far. – Knowledge about dynamics of state transitions.

• In other words maximize
• Different approaches include:

– HMMs – Kalman filters – Condensation

1...

( | )

t t

p x y

Steps of tracking

• Prediction: What is the next state of the object

given past measurements?

 

1 1

, ,

  



t t t

y Y y Y X P 

Steps of tracking

• Prediction: What is the next state of the object

given past measurements?

• Correction: Compute an updated estimate of

the state from prediction and measurements

 

1 1

, ,

  



t t t

y Y y Y X P 

 

t t t t t

y Y y Y y Y X P   

 

, , ,

1 1



Steps of tracking

• Prediction: What is the next state of the object

given past measurements?

• Correction: Compute an updated estimate of

the state from prediction and measurements

 

1 1

, ,

  



t t t

y Y y Y X P 

 

t t t t t

y Y y Y y Y X P   

 

, , ,

1 1



Simplifying assumptions

• Only the immediate past matters

   

1 1

, ,

   t t t t

X X P X X X P 

dynamics model

Simplifying assumptions

• Only the immediate past matters
• Measurements depend only on the current state

   

1 1

, ,

   t t t t

X X P X X X P 

   

t t t t t t

X Y P X Y X Y X Y P 

 

, , , ,

1 1



• bservation model

dynamics model

Simplifying assumptions

• Only the immediate past matters
• Measurements depend only on the current state

   

1 1

, ,

   t t t t

X X P X X X P 

   

t t t t t t

X Y P X Y X Y X Y P 

 

, , , ,

1 1



• bservation model

dynamics model X1 X2 Y1 Y2 Xt Yt

…

Xt-1 Yt-1

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Tracking as Induction

• Base case:
• Assume we have initial prior that predicts state in absence of

any evidence: P(X0)

• At the first frame, correct this given the value of Y0=y0

) ( ) | ( ) ( ) ( ) | ( ) | ( X P X y P y P X P X y P y Y X P   

Posterior prob.

• f state given

measurement Likelihood of measurement Prior of the state

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Tracking as Induction

• Base case:
• Assume we have initial prior that predicts state in absence of

any evidence: P(X0)

• At the first frame, correct this given the value of Y0=y0
• Given corrected estimate for frame t:
• Predict for frame t+1
• Correct for frame t+1

predict correct

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Prediction

• Prediction involves representing

given

 

1

, ,

 t t

y y X P 

 

1 1

, ,

  t t

y y X P 

 

       

1 1 1 1 1 1 1 1 1 1 1 1

, , | | , , | , , , | , , ,

           

  

  

t t t t t t t t t t t t t t t

dX y y X P X X P dX y y X P y y X X P dX y y X X P    

 

1

, ,

 t t

y y X P 

Law of total probability

   

, P A P A B dB 

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Prediction

• Prediction involves representing

given

 

1

, ,

 t t

y y X P 

 

1 1

, ,

  t t

y y X P 

 

       

1 1 1 1 1 1 1 1 1 1 1 1

, , | | , , | , , , | , , ,

           

  

  

t t t t t t t t t t t t t t t

dX y y X P X X P dX y y X P y y X X P dX y y X X P    

 

1

, ,

 t t

y y X P 

Conditioning on Xt–1

     

, | P A B P A B P B 

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Prediction

• Prediction involves representing

given

 

1

, ,

 t t

y y X P 

 

1 1

, ,

  t t

y y X P 

 

       

1 1 1 1 1 1 1 1 1 1 1 1

, , | | , , | , , , | , , ,

           

  

  

t t t t t t t t t t t t t t t

dX y y X P X X P dX y y X P y y X X P dX y y X X P    

 

1

, ,

 t t

y y X P 

Independence assumption

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Correction

• Correction involves computing

given predicted value

 

t t

y y X P , ,

0 

 

1

, ,

 t t

y y X P 

                   



      

  

t t t t t t t t t t t t t t t t t t t t t t

dX y y X P X y P y y X P X y P y y y P y y X P X y P y y y P y y X P y y X y P

1 1 1 1 1 1 1

, , | | , , | | , , | , , | | , , | , , | , , , |       

 

t t

y y X P , ,

0 

Bayes rule

       

| | P B A P A P A B P B 

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Correction

• Correction involves computing

given predicted value

 

t t

y y X P , ,

0 

 

1

, ,

 t t

y y X P 

                   



      

  

t t t t t t t t t t t t t t t t t t t t t t

dX y y X P X y P y y X P X y P y y y P y y X P X y P y y y P y y X P y y X y P

1 1 1 1 1 1 1

, , | | , , | | , , | , , | | , , | , , | , , , |       

 

t t

y y X P , ,

0 

Independence assumption (observation yt depends only on state Xt)

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Correction

• Correction involves computing

given predicted value

 

t t

y y X P , ,

0 

 

1

, ,

 t t

y y X P 

                   



      

  

t t t t t t t t t t t t t t t t t t t t t t

dX y y X P X y P y y X P X y P y y y P y y X P X y P y y y P y y X P y y X y P

1 1 1 1 1 1 1

, , | | , , | | , , | , , | | , , | , , | , , , |       

 

t t

y y X P , ,

0 

Conditioning on Xt

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Induction Step: Correction

• Correction involves computing

given predicted value

 

t t

y y X P , ,

0 

 

1

, ,

 t t

y y X P 

                   



      

  

t t t t t t t t t t t t t t t t t t t t t t

dX y y X P X y P y y X P X y P y y y P y y X P X y P y y y P y y X P y y X y P

1 1 1 1 1 1 1

, , | | , , | | , , | , , | | , , | , , | , , , |       

 

t t

y y X P , ,

0 

• bservation

model predicted estimate normalization factor

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Summary: Prediction and Correction

• Prediction:

     

1 1 1 1 1

, , | | , , |

    





t t t t t t t

dX y y X P X X P y y X P  

Dynamics model Corrected estimate from previous step

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Summary: Prediction and Correction

• Prediction:
• Correction:

31

• B. Leibe

     

1 1 1 1 1

, , | | , , |

    





t t t t t t t

dX y y X P X X P y y X P  

Dynamics model Corrected estimate from previous step

         



 



t t t t t t t t t t t

dX y y X P X y P y y X P X y P y y X P

1 1

, , | | , , | | , , |   

Observation model Predicted estimate

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Topics of This Lecture

• Tracking with Dynamics
• Detection vs. Tracking
• Tracking as probabilistic inference
• Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter
• Kalman filter for 1D state
• General Kalman filter
• Limitations

32

• B. Leibe

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Notation Reminder

• Random variable with Gaussian probability distribution

that has the mean vector μ and covariance matrix Σ.

• x and μ are d-dimensional, Σ is d x d.

33

• B. Leibe

) , ( ~ Σ μ x N

d=2 d=1 If x is 1D, we just have one Σ parameter: the variance σ2

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Linear Dynamic Models

• Dynamics model
• State undergoes linear tranformation Dt plus Gaussian noise
• Observation model
• Measurement is linearly transformed state plus Gaussian noise

34

• B. Leibe

 

1

~ ,

t

t t t d

N

 

x D x

 

~ ,

t

t t t m

N  y M x

nn n1 n1 m1 mn n1

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Example 1: Randomly Drifting Points

• Consider a stationary object, with state as position.
• Position is constant, only motion due to random noise term.

 State evolution is described by identity matrix D=I

35

• B. Leibe

1 t t

p p 



 

t t

x p 

1 1 t t t t

x D x noise Ip noise

 

   

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Example 2: Constant Velocity (1D Points)

36

• B. Leibe

time Measurements States

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Example 2: Constant Velocity (1D Points)

• State vector: position p and velocity v
• Measurement is position only

       

   1 1 1

) (

t t t t t

v v v t p p

      

t t t

v p x noise v p t noise x D x

t t t t t

                

   1 1 1

1 1

(greek letters denote noise terms)

 

noise v p noise Mx y

t t t t

          1

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Example 3: Constant Accel. (1D Points)

38

• B. Leibe

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Example 3: Constant Accel. (1D Points)

• State vector: position p, velocity v, and acceleration a.
• Measurement is position only

39

            

     1 1 1 1 1

) ( ) (

t t t t t t t t

a a a t v v v t p p

          

t t t t

a v p x

noise a v p t t noise x D x

t t t t t t

                         

    1 1 1 1

1 1 1

(greek letters denote noise terms)

 

noise a v p noise Mx y

t t t t t

              1

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Topics of This Lecture

• Tracking with Dynamics
• Detection vs. Tracking
• Tracking as probabilistic inference
• Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter
• Kalman filter for 1D state
• General Kalman filter
• Limitations

40

• B. Leibe

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

The Kalman Filter

• Method for tracking linear dynamical models in

Gaussian noise

• The predicted/corrected state distributions are

Gaussian

• You only need to maintain the mean and covariance.
• The calculations are easy (all the integrals can be done in

closed form).

41

• B. Leibe

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Propagation of Gaussian densities

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

The Kalman Filter

43

• B. Leibe

Know prediction of state, and next measurement Update distribution over current state. Know corrected state from previous time step, and all measurements up to the current one  Predict distribution over next state. Time advances: t++ Time update (“Predict”) Measurement update (“Correct”) Receive measurement

 

1

, ,

 t t

y y X P 

  t t 

 ,

Mean and std. dev.

• f predicted state:

 

t t

y y X P , ,

0 

  t t 

 ,

Mean and std. dev.

• f corrected state:

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Kalman Filter for 1D State

44

• B. Leibe

Want to represent and update

 

 

2 1

) ( , , ,

    t t t t

N y y x P   

 

 

2

) ( , , ,

 



t t t t

N y y x P   

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

1D Kalman Filter: Prediction

• Have linear dynamic model defining predicted state

evolution, with noise

• Want to estimate predicted distribution for next state
• Update the mean:
• Update the variance:

45

• B. Leibe

    1 t t

d 

 

 

2 1

) ( , , ,

    t t t t

N y y X P   

 

2 1,

~

d t t

dx N X 



2 1 2 2

) ( ) (

  

 

t d t

d  

for derivations, see F&P Chapter 18

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

1D Kalman Filter: Correction

• Have linear model defining the mapping of state to

measurements:

• Want to estimate corrected distribution given latest

measurement:

• Update the mean:
• Update the variance:

46

• B. Leibe

 

2

, ~

m t t

mx N Y 

 

 

2

) ( , , ,

 



t t t t

N y y X P   

2 2 2 2 2

) ( ) (

   

  

t m t t m t t

m my      

2 2 2 2 2 2

) ( ) ( ) (

  

 

t m t m t

m     

Derivations: F&P Chapter 18

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Prediction vs. Correction

• What if there is no prediction uncertainty
• What if there is no measurement uncertainty

47

• B. Leibe

   t t

  ) (

2   t



2 2 2 2 2

) ( ) (

   

  

t m t t m t t

m my      

2 2 2 2 2 2

) ( ) ( ) (

  

 

t m t m t

m     

? ) ( 

m



? ) ( 

 t



m yt

t  

 ) (

2   t



The measurement is ignored! The prediction is ignored!

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Recall: Constant Velocity Example

48

• B. Leibe

State is 2D: position + velocity Measurement is 1D: position measurements state time position

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Constant Velocity Model

49

• B. Leibe
• state

x measurement * predicted mean estimate + corrected mean estimate bars: variance estimates before and after measurements

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Constant Velocity Model

50

• B. Leibe
• state

x measurement * predicted mean estimate + corrected mean estimate bars: variance estimates before and after measurements

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Constant Velocity Model

51

• B. Leibe
• state

x measurement * predicted mean estimate + corrected mean estimate bars: variance estimates before and after measurements

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Constant Velocity Model

52

• B. Leibe
• state

x measurement * predicted mean estimate + corrected mean estimate bars: variance estimates before and after measurements

Figure from Forsyth & Ponce

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Kalman Filter: General Case (>1dim)

• What if state vectors have more than one dimension?

53

• B. Leibe

PREDICT CORRECT

    1 t t t

x D x

t

d T t t t t

D D     

   1

 

  

  

t t t t t t

x M y K x x

 

 

   

t t t t

M K I

 

1   

    

t

m T t t t T t t t

M M M K

More weight on residual when measurement error covariance approaches 0. Less weight on residual as a priori estimate error covariance approaches 0. “residual” for derivations, see F&P Chapter 18

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Summary: Kalman Filter

• Pros:
• Gaussian densities everywhere
• Simple updates, compact and efficient
• Very established method, very well understood
• Cons:
• Unimodal distribution, only single hypothesis
• Restricted class of motions defined by linear model

54

• B. Leibe

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Why Is This A Restriction?

• Many interesting cases don’t have linear dynamics
• E.g. pedestrians walking
• E.g. a ball bouncing

55

• B. Leibe

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Ball Example: What Goes Wrong Here?

• Assuming constant acceleration model
• Prediction is too far

from true position to compensate…

• Possible solution:

Extended Kalman Filter

• Keeps multiple different

motion models in parallel

• I.e. would check for

bouncing at each time step

56

• B. Leibe

Prediction t1 Prediction t3 Prediction t2 Prediction t4 Prediction t5 Correct prediction

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Tracking issues

• Deciding on the structure of the model
• Initialization
• Specifying the dynamics model
• Specifying the observation model
• Data association problem: which measurements tell

us about the object(s) being tracked?

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Doesn’t always work…

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Data association

• Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

• More sophisticated strategy: keep track of multiple

state/observation hypotheses

• This is a general problem in computer vision, there is no

easy solution

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Tracking issues

• Deciding on the structure of the model
• Initialization
• Specifying the dynamics model
• Specifying the observation model
• Data association problem
• Prediction vs. correction
• If the dynamics model is too strong, will end up

ignoring the data

• If the observation model is too strong, tracking is

reduced to repeated detection

• Drift

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

Drift

• D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
• Appearance. PAMI 2007.

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

םויה וניאר המ זא?

Perceptual and Sensory Augmented Computing Computer Vision WS 08/09

םוכיס

• תיתורבתסה הקסה תייעבכ הביקע
• תכרעמ בצמ יונישל םייראניל םילדומ
• Kalman filtering
• Particle Filtering

Good luck!