[PPT] - Designing Applications that See Designing Applications that See PowerPoint Presentation

SLIDE 1

stanford hci group / cs377s

Designing Applications that See Designing Applications that See Lecture 5: Motion and Tracking

Dan Maynes-Aminzade 22 January 2008 22 January 2008

Designing Applications that See http://cs377s.stanford.edu

SLIDE 2

R i d Reminders

A i t # d Assignment #1 due now Assignment #2 available next Tuesday g y Bring your webcams on Thursday for the P i T t i l Processing Tutorial Sunday is the add deadline y

22 January 2008 2 Lecture 5: Motion and Tracking

SLIDE 3

T d ’ G l Today’s Goals

L h t d t t d di t Learn how to detect, measure, and predict motion in a video sequence Get a high-level overview of some different tactics for tracking moving objects tactics for tracking moving objects

22 January 2008 3 Lecture 5: Motion and Tracking

SLIDE 4

O tli Outline

L k t f id f Look at some of your videos from Assignment #1 Learn about some motion and tracking techniques and try them out on your videos techniques and try them out on your videos

Frame differencing Background subtraction Motion templates p Optical flow Color tracking Color tracking

22 January 2008 4 Lecture 5: Motion and Tracking

SLIDE 5

T i B ll Tennis Balls

Carl Marcello Yangfan Eric Michael

22 January 2008 5 Lecture 5: Motion and Tracking

SLIDE 6

I t ti Intersection

22 January 2008 6 Lecture 5: Motion and Tracking

SLIDE 7

F ’ M k t Farmers’ Market

22 January 2008 7 Lecture 5: Motion and Tracking

SLIDE 8

F b ll Foosball

22 January 2008 8 Lecture 5: Motion and Tracking

SLIDE 9

Fi h Fish

22 January 2008 9 Lecture 5: Motion and Tracking

SLIDE 10

A d th H Around the House

22 January 2008 10 Lecture 5: Motion and Tracking

SLIDE 11

Bik Bikes

22 January 2008 11 Lecture 5: Motion and Tracking

SLIDE 12

Cl th Clothes

22 January 2008 12 Lecture 5: Motion and Tracking

SLIDE 13

D i i Driving

22 January 2008 13 Lecture 5: Motion and Tracking

SLIDE 14

Fi h Fish

22 January 2008 14 Lecture 5: Motion and Tracking

SLIDE 15

Kit h Kitchen

22 January 2008 15 Lecture 5: Motion and Tracking

SLIDE 16

L d Laundry

22 January 2008 16 Lecture 5: Motion and Tracking

SLIDE 17

Pi P Ping-Pong

22 January 2008 17 Lecture 5: Motion and Tracking

SLIDE 18

Pl t Plate

22 January 2008 18 Lecture 5: Motion and Tracking

SLIDE 19

S d i h Sandwich

22 January 2008 19 Lecture 5: Motion and Tracking

SLIDE 20

T ffi Traffic

22 January 2008 20 Lecture 5: Motion and Tracking

SLIDE 21

T f M ti D t i ti Types of Motion Determination

M ti D t ti id tif i h th Motion Detection: identifying whether or not image points are moving Motion Estimation: identifying how image points are moving points are moving Motion Segmentation: identifying moving

bjects from moving points

22 January 2008 21 Lecture 5: Motion and Tracking

SLIDE 22

Extracting Moving Objects Extracting Moving Objects

Si l t ti b k d ith l Simple case: static background, with only the object of interest in motion

22 January 2008 22 Lecture 5: Motion and Tracking

SLIDE 23

S l ti F Diff i Solution: Frame Differencing

S bt t t f f i f Subtract current frame from previous frame, and threshold the result

22 January 2008 23 Lecture 5: Motion and Tracking

SLIDE 24

Accumulative Frame Differencing Accumulative Frame Differencing

E ti t ti di ti b l ti Estimate motion direction by accumulating motion history over a range of frames

22 January 2008 24 Lecture 5: Motion and Tracking

SLIDE 25

M ti Hi t I Motion History Image

22 January 2008 25 Lecture 5: Motion and Tracking

SLIDE 26

M lti l M i Obj t ? Multiple Moving Objects?

22 January 2008 26 Lecture 5: Motion and Tracking

(courtesy of Sebastian Thrun)

SLIDE 27

M ti S t ti Motion Segmentation

Add ti t t t ti hi t i Add timestamp to current motion history image, and overlay it on top of the older ones

22 January 2008 27 Lecture 5: Motion and Tracking

SLIDE 28

M ti S t ti Motion Segmentation

M th di t f th t k f Measure the gradients of the stack of motion history images

22 January 2008 28 Lecture 5: Motion and Tracking

SLIDE 29

M ti S t ti Motion Segmentation

I ti t l t d lti Ignore motion template edges resulting from too large of a time delay

22 January 2008 29 Lecture 5: Motion and Tracking

SLIDE 30

M ti S t ti Motion Segmentation

Fi d b d i f t t ti d Find boundaries of most recent motions and fill them in to segment motion regions

Segmented Motion Segmented Motion

22 January 2008 30 Lecture 5: Motion and Tracking

SLIDE 31

L t’ T It O t! Let’s Try It Out!

22 January 2008 31 Lecture 5: Motion and Tracking

SLIDE 32

B k d S bt ti Background Subtraction

If k h t th b k d l k lik If we know what the background looks like, we can ignore it to focus on things that are moving or changing

=

=

22 January 2008 32 Lecture 5: Motion and Tracking

SLIDE 33

Bl S Blue Screen

22 January 2008 33 Lecture 5: Motion and Tracking

SLIDE 34

Vid E l Video Example

22 January 2008 34 Lecture 5: Motion and Tracking

(courtesy of Frank Dellaert)

SLIDE 35

S bt ti d Th h ldi Subtraction and Thresholding

22 January 2008 35 Lecture 5: Motion and Tracking

SLIDE 36

B i B k d S bt ti Basic Background Subtraction

A b k d i tl t ti Assume background is mostly static Build a background model by averaging g y g g pixel values across a range of frames Gi i t Given a new image, generate a silhouette by marking the pixels that are significantly different from the “background” value

22 January 2008 36 Lecture 5: Motion and Tracking

SLIDE 37

Fi di S b t Finding Subparts

L k t t h d k i t Look at contour shape and mark points farthest from the center as hands Can be combined with a skin color model for better results for better results

22 January 2008 37 Lecture 5: Motion and Tracking

SLIDE 38

Pfi d E l Pfinder Example

22 January 2008 38 Lecture 5: Motion and Tracking

SLIDE 39

D i B k d ? Dynamic Backgrounds?

22 January 2008 39 Lecture 5: Motion and Tracking

(courtesy of Kentaro Toyama)

SLIDE 40

L t’ T It O t! Let’s Try It Out!

22 January 2008 40 Lecture 5: Motion and Tracking

SLIDE 41

K i T k f Obj t Keeping Track of Objects

22 January 2008 41 Lecture 5: Motion and Tracking

SLIDE 42

Bl b T ki Blob Tracking

22 January 2008 42 Lecture 5: Motion and Tracking

SLIDE 43

L t’ T it O t! Let’s Try it Out!

22 January 2008 43 Lecture 5: Motion and Tracking

SLIDE 44

M C l M ti More Complex Motion

22 January 2008 44 Lecture 5: Motion and Tracking

(courtesy of J.M. Rehg)

SLIDE 45

M C l M ti More Complex Motion

22 January 2008 45 Lecture 5: Motion and Tracking

(courtesy of J.M. Rehg)

SLIDE 46

M C l M ti More Complex Motion

22 January 2008 46 Lecture 5: Motion and Tracking

(courtesy of J.M. Rehg)

SLIDE 47

O ti l Fl Optical Flow

A 2 D l it fi ld d ibi th ti i A 2-D velocity field describing the motion in an image sequence A t t h i l i di t it ti A vector at each pixel indicates its motion direction between neighboring frames

22 January 2008 47 Lecture 5: Motion and Tracking

SLIDE 48

Ch t i i M ti Characterizing Motion

Image Sequence Flow Vectors

22 January 2008 48 Lecture 5: Motion and Tracking

(courtesy of Sebastian Thrun)

SLIDE 49

C ti O ti l Fl Computing Optical Flow

22 January 2008 49 Lecture 5: Motion and Tracking

(courtesy of Michael Black)

SLIDE 50

T ki L l F t Tracking Local Features

1

p

2

p

3

p

1

v r

2

v r

3

v r

) 1 ( + t I

Optical Optical Flow Flow

1

p

4

p

1

v

4

v r

) 1 ( + t I } { ), (

i

p t I } { i v r

Velocity vectors Velocity vectors

22 January 2008 50 Lecture 5: Motion and Tracking

SLIDE 51

O ti l Fl A ti Optical Flow Assumptions

B i ht t th h i Brightness constancy: though regions may move around, the brightness within a small i ill t h region will not change

22 January 2008 51 Lecture 5: Motion and Tracking

SLIDE 52

O ti l Fl A ti Optical Flow Assumptions

T l i t d l ti ti Temporal persistence: gradual motion over time

22 January 2008 52 Lecture 5: Motion and Tracking

SLIDE 53

A t P bl Aperture Problem

22 January 2008 53 Lecture 5: Motion and Tracking

SLIDE 54

A t P bl Aperture Problem

M ti l j t d i bi Motion along just an edge is ambiguous

22 January 2008 54 Lecture 5: Motion and Tracking

(courtesy of Sebastian Thrun)

SLIDE 55

A th E l Another Example

22 January 2008 55 Lecture 5: Motion and Tracking

SLIDE 56

H i C Harris Corners

22 January 2008 56 Lecture 5: Motion and Tracking

SLIDE 57

L t’ T It O t! Let’s Try It Out!

22 January 2008 57 Lecture 5: Motion and Tracking

SLIDE 58

S t ti b Cl t i Segmentation by Clustering

Image Clusters on intensity Clusters on color

22 January 2008 58 Lecture 5: Motion and Tracking

SLIDE 59

Si l Cl t i Al ith Simple Clustering Algorithms

22 January 2008 59 Lecture 5: Motion and Tracking

(courtesy of Marc Pollefeys)

SLIDE 60

Cl t i E l Clustering Example

22 January 2008 60 Lecture 5: Motion and Tracking

(courtesy of Marc Pollefeys)

SLIDE 61

M Shift S t ti Mean Shift Segmentation

Original Image Segmented Image Original Image Segmented Image

22 January 2008 61 Lecture 5: Motion and Tracking

(courtesy of D. Comaniciu)

SLIDE 62

M Shift Al ith Mean Shift Algorithm

G l fi d th i t f hi h t d it Goal: find the points of highest density (“modes”) in the data distribution

22 January 2008 62 Lecture 5: Motion and Tracking

(courtesy of D. Comaniciu)

SLIDE 63

M Shift Al ith Mean Shift Algorithm

1 Choose a search window 1. Choose a search window size. 2. Choose the initial location f h h d

f the search window.

3. Compute the mean location (centroid of the data) in the (centroid of the data) in the search window. 4. Center the search window t th l ti at the mean location computed in Step 3. 5. Repeat Steps 3 and 4 until 5. Repeat Steps 3 and 4 until convergence.

22 January 2008 63 Lecture 5: Motion and Tracking

SLIDE 64

M Shift R lt Mean Shift Results

22 January 2008 64 Lecture 5: Motion and Tracking

(courtesy of D. Comaniciu)

SLIDE 65

Continuously Adaptive Mean Shift Continuously Adaptive Mean Shift

A i f th hift l ith b A version of the mean shift algorithm can be applied to object tracking based on color Start with a object location and an object color profile (hue distribution histogram) color profile (hue distribution histogram)

Video Frame Object Rectangle

22 January 2008 65 Lecture 5: Motion and Tracking

Object ecta g e

SLIDE 66

Continuously Adaptive Mean Shift Continuously Adaptive Mean Shift

C l l t “b k j ti ” i b bilit th t Calculate “backprojection” image: probability that each pixel came from the h di t ib ti th t k d bj t same hue distribution as the tracked object Use Mean Shift Algorithm to find the new object center given its back projection and the initial center given its back projection and the initial position of search window

22 January 2008 66 Lecture 5: Motion and Tracking

(courtesy of G. Bradski)

SLIDE 67

CAMSHIFT E l CAMSHIFT Example

22 January 2008 67 Lecture 5: Motion and Tracking

(courtesy of Robin Hewitt)

SLIDE 68

L t’ T It O t! Let’s Try It Out!

22 January 2008 68 Lecture 5: Motion and Tracking

SLIDE 69

Summary

M ti i ft h f l th Motion is often much more useful than static image features for understanding h t i h i what is happening There are many tactics for detecting, measuring, and segmenting motion

You’ll try some out in Assignment #2 y g We’ll have more hands-on practice during the next two workshop sessions p

Think about how you might use motion sensing in your project sensing in your project

22 January 2008 69 Lecture 5: Motion and Tracking