CS 4495 Computer Vision Tracking 3: Follow the pixels Aaron Bobick - - PowerPoint PPT Presentation

▶
cs 4495 computer vision tracking 3 follow the pixels
SMART_READER_LITE
LIVE PREVIEW

CS 4495 Computer Vision Tracking 3: Follow the pixels Aaron Bobick - - PowerPoint PPT Presentation

Feb 25, 2023 332 likes •958 views

Tracking 3: Real tracking CS 4495 Computer Vision A. Bobick CS 4495 Computer Vision Tracking 3: Follow the pixels Aaron Bobick School of Interactive Computing Tracking 3: Real tracking CS 4495 Computer Vision A. Bobick

slide-1
SLIDE 1

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Aaron Bobick School of Interactive Computing

CS 4495 Computer Vision Tracking 3: Follow the pixels…

slide-2
SLIDE 2

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Administrivia

  • PS6 out Thurs Nov 14, due Nov 24th
  • EXAM: Tues before Thanksgiving, Nov 26th (not 21st !!)
  • Covers concepts and basics
  • Example questions will be posted on calendar by November 19th
  • PS7 out Tue Nov 26th, due technically Friday, December 5.
  • Extension available until Sunday, Dec 7, a day that will live in infamy,

11:55pm.

  • Problem set resubmission policy:
  • Full questions only
  • You get 50% credit to replace whatever you got last time on that

question.

  • Must be submitted by: SUNDAY DECEMBER 7, 11:55pm
  • No prior notice needed.
  • Grades posted as available.
slide-3
SLIDE 3

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Tracking as induction

  • Base case:
  • Assume we have initial prior that predicts state in absence of any

evidence: P(X0)

  • At the first frame, correct this given the value of Y0=y0
  • Given corrected estimate for frame t:
  • Predict for frame t+1
  • Correct for frame t+1

predict correct

slide-4
SLIDE 4

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Kalman: Propagation of Gaussian densities

slide-5
SLIDE 5

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Propagation of general densities

slide-6
SLIDE 6

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Particle Filters: Basic Idea

) (x p

t

x

set of n (weighted) particles Xt Density is represented by both where the particles are and their weight.

slide-7
SLIDE 7

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Graphical steps particle filtering

slide-8
SLIDE 8

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • Given:
  • Stream of observations z and action data u:
  • Sensor model P(zt|xt).
  • Action model P(xt+1|ut,xt).
  • Prior probability of the system state P(x).
  • Wanted:
  • Estimate of the state X of a dynamical system.
  • The posterior of the state is also called Belief:

Bayes Filters: Framework

) , , , | ( ) (

1 2 1 t t t t

z u z u x P x Bel

−

= 

1 2 1

{ , , , }

t t t

data u z u z

−

= 

slide-9
SLIDE 9

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Bayes Rule reminder

( | ) ( ) ( ( | ) ) | ) ( ) ( p z x p x p x z p z p z x p x η = =

Prior before measurement

slide-10
SLIDE 10

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • 1. Algorithm particle_filter

2.

  • 3. For Resample (generate i new samples)

4. Sample index j(i) from the discrete distribution given by wt-1 5. Sample from using and Control 6. Compute importance weight (or reweight) 7. Update normalization factor 8. Insert

  • 9. For

10. Normalize weights

Particle Filter Algorithm (Sequential Importance Resampling)

, = ∅ = η

t

S n i  1 = } , { > < ∪ =

i t i t t t

w x S S

i t

w + =η η

i t

x

1

( | , )

t t t

p x x u

− ) ( 1 i j t

x −

t

u ) | (

i t t i t

x z p w = n i  1 = η /

i t i t

w w =

1 1 1

{ } , , ,

j j t t t t t

S x w u z

− − −

< > =

slide-11
SLIDE 11

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Smithsonian Museum of American History…

slide-12
SLIDE 12

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-13
SLIDE 13

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-14
SLIDE 14

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-15
SLIDE 15

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-16
SLIDE 16

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-17
SLIDE 17

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-18
SLIDE 18

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-19
SLIDE 19

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-20
SLIDE 20

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-21
SLIDE 21

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-22
SLIDE 22

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-23
SLIDE 23

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-24
SLIDE 24

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

slide-25
SLIDE 25

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

25

slide-26
SLIDE 26

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

26

slide-27
SLIDE 27

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

27

slide-28
SLIDE 28

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

28

slide-29
SLIDE 29

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

How about simple vision….

slide-30
SLIDE 30

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Using Ceiling Maps for Localization

Dellaert, et al. 1997

slide-31
SLIDE 31

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Vision-based Localization

P(z|x) h(x) z

slide-32
SLIDE 32

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Under a Light

Measurement z: P(z|x):

slide-33
SLIDE 33

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Next to a Light

Measurement z: P(z|x):

slide-34
SLIDE 34

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Elsewhere

Measurement z: P(z|x):

slide-35
SLIDE 35

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Global Localization Using Vision

slide-36
SLIDE 36

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Odometry Only

slide-37
SLIDE 37

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Using Vision

slide-38
SLIDE 38

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • Given: Set S of weighted samples.
  • Wanted : Random sample, where the probability
  • f drawing xi is given by wi.
  • Typically done n times with replacement to

generate new sample set S’.

  • Or even not done except when needed…too many low

weight particles.

A detail: Resampling method can matter

slide-39
SLIDE 39

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick w2 w3 w1 wn Wn-1

Resampling

w2 w3 w1 wn Wn-1

  • Roulette wheel
  • Binary search, n log n
  • Stochastic universal sampling
  • Systematic resampling
  • Linear time complexity
  • Easy to implement, low variance
slide-40
SLIDE 40

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • 1. Algorithm systematic_resampling(S,n):

2.

  • 3. For

Generate cdf 4. 5. Initialize threshold

  • 6. For

Draw samples … 7. While ( ) Skip until next threshold reached 8. 9. Insert

  • 10. Increment threshold
  • 11. Return S’

Resampling Algorithm

1 1

, ' w c S = ∅ = n i  2 =

i i i

w c c + =

−1 1 1 ~

[0, ], 1 u U n i

−

= n j  1 =

1 1 − +

+ = n u u

j j i j

c u >

{ }

> < ∪ =

−1

, ' ' n x S S

i

1 + = i i

(Also called stochastic universal sampling)

slide-41
SLIDE 41

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • If dealing with highly peaked observations
  • Add noise to observation and prediction models
  • Better proposal distributions: e.g., perform Kalman filter step to

determine proposal

  • Overestimating noise often reduces number of required

samples

  • Recover from failure by selectively adding samples from
  • bservations
  • Recover from failure by uniformly adding some samples
  • Can Resample only when necessary (efficiency of

representation measured by variance of weights)

PF: Practical Considerations

slide-42
SLIDE 42

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

To do real tracking…

  • x is the “state”. But of what? The object? Some

representation of the object?

  • z is the “measurement”. But what measurement? And

how does it relate to the state?

  • Where do you get your dynamics from?

( | ) ( ) ( ( | ) ) | ) ( ) ( p z x p x p x z p z p z x p x η = =

1

( | , )

t t t

p x x u

−

State State dynamics Sensor model

slide-43
SLIDE 43

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

The source…

slide-44
SLIDE 44

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

39

  • The “object” to be tracked here is a hand initialized
  • contour. Could have been the image pixels. Which is

better?

  • Its state is its affine deformation. How many parameters?
  • Each particle represents those six parameters.

Particle filter tracking - state

[Isard 1998]

Picture of the states represented by the top weighted particles The mean state

slide-45
SLIDE 45

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

More complex state

  • Tracking of a hand movement using an edge detector
  • State is translation and rotation of hand plus angle of

each finger; 12 DOF

slide-46
SLIDE 46

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

  • Suppose x is a hand

initialized contour.

  • What is z?
  • Gaussian in Distance to nearest high-

contrast feature summed over the contour.

Particle filter tracking - measurement

[Isard 1998]

slide-47
SLIDE 47

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

More tracking contours

  • Head tracking with contour

models (Zhihong et al. 2002)

  • How did it do occlusion?
  • With velocity?
  • Without velocity?
  • How did you get

“dynamics”?

slide-48
SLIDE 48

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Getting the dynamics

  • Why are we doing all this work? ‘Cuz tracking is hard.
  • If it weren’t hard we could just detect the contour.
  • If we could just detect the contour we could get out correct

trajectories,

  • If we can get correct trajectories we can….?
slide-49
SLIDE 49

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

A different model

  • Hands and head movement tracking using color models

and optical flow (Tung et al. 2008)

  • State: location of colored blob (x,y)
  • Prediction based upon flow.
  • Sensor model: color match
slide-50
SLIDE 50

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

How about a really, really simple model?

  • State is just location of

an image patch: x,y

  • Dynamics: just random

noise

  • Sensor model: avg

squared difference of pixel intensities.

  • Really a similarity model:

more similar is more likely.

  • Oh, you need a patch…
slide-51
SLIDE 51

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

An even better model

  • Suppose you want to

track a region of colors.

  • What would be a good

model/State:

  • Location
  • Region size?
  • Distribution of colors
  • What would be a good

sensor model?

  • Similarity of distributions
slide-52
SLIDE 52

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Recapping: Tracking issues

  • Initialization
  • Manual
  • Background subtraction
  • Detection
slide-53
SLIDE 53

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Recapping: Tracking issues

  • Initialization
  • Obtaining observation and dynamics model
  • Dynamics model: learn (difficult) or specify using domain

knowledge

  • Can cheat if you have “easy” tracking case
  • Generative observation model: “render” the state on top of the

image and compare. E.g. put down the contour and evaluate.

slide-54
SLIDE 54

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Recapping: Tracking issues

  • Initialization
  • Obtaining observation and dynamics model
  • Prediction vs. correction
  • If the dynamics model noise is too low, will end up ignoring the

data

  • If the observation noise model is too low, tracking is reduced to

repeated detection (Kalman). If too peaked, only a few particles survive (PF).

slide-55
SLIDE 55

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Recapping: Tracking issues

  • Initialization
  • Obtaining observation and dynamics model
  • Prediction vs. correction
  • Data association
  • What if we don’t know which measurements to associate with

which tracks?

slide-56
SLIDE 56

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Data association

  • So far, we’ve assumed the entire

measurement to be relevant to determining the state

  • In reality, there may be

uninformative measurements (clutter) or measurements may belong to different tracked objects

  • Data association: task of

determining which measurements go with which tracks

slide-57
SLIDE 57

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Data association

  • Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Source: Lana Lazebnik

slide-58
SLIDE 58

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Data association

  • Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

Doesn’t always work… Alternative: keep track of multiple hypotheses at once…

Source: Lana Lazebnik

slide-59
SLIDE 59

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Data association

  • Simple strategy: only pay attention to the measurement

that is “closest” to the prediction

  • More sophisticated strategy: keep track of multiple

state/observation hypotheses

  • Can be done with particle filtering
  • This is a general problem in computer vision, there is no

easy solution

slide-60
SLIDE 60

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Tracking issues

  • Initialization
  • Obtaining observation and dynamics model
  • Prediction vs. correction
  • Data association
  • Drift
  • Errors caused by dynamical model, observation model, and

data association tend to accumulate over time

slide-61
SLIDE 61

Tracking 3: Real tracking CS 4495 Computer Vision – A. Bobick

Drift

  • D. Ramanan, D. Forsyth, and A. Zisserman. Tracking People by Learning their
  • Appearance. PAMI 2007.