Designing Applications that See Designing Applications that See - PowerPoint PPT Presentation

stanford hci group / cs377s Designing Applications that See Designing Applications that See Lecture 5: Motion and Tracking Dan Maynes-Aminzade 22 January 2008 22 January 2008 Designing Applications that See http://cs377s.stanford.edu

R Reminders i d � Assignment #1 due now A i t # d � Assignment #2 available next Tuesday g y � Bring your webcams on Thursday for the P Processing Tutorial i T t i l � Sunday is the add deadline y Lecture 5: Motion and Tracking 22 January 2008 2

T d Today’s Goals ’ G l � Learn how to detect, measure, and predict L h t d t t d di t motion in a video sequence � Get a high-level overview of some different tactics for tracking moving objects tactics for tracking moving objects Lecture 5: Motion and Tracking 22 January 2008 3

O tli Outline � Look at some of your videos from L k t f id f Assignment #1 � Learn about some motion and tracking techniques and try them out on your videos techniques and try them out on your videos � Frame differencing � Background subtraction � Motion templates p � Optical flow � Color tracking � Color tracking Lecture 5: Motion and Tracking 22 January 2008 4

T Tennis Balls i B ll Carl Marcello Eric Yangfan Michael Lecture 5: Motion and Tracking 22 January 2008 5

I t Intersection ti Lecture 5: Motion and Tracking 22 January 2008 6

F Farmers’ Market ’ M k t Lecture 5: Motion and Tracking 22 January 2008 7

F Foosball b ll Lecture 5: Motion and Tracking 22 January 2008 8

Fi h Fish Lecture 5: Motion and Tracking 22 January 2008 9

A Around the House d th H Lecture 5: Motion and Tracking 22 January 2008 10

Bik Bikes Lecture 5: Motion and Tracking 22 January 2008 11

Cl th Clothes Lecture 5: Motion and Tracking 22 January 2008 12

D i i Driving Lecture 5: Motion and Tracking 22 January 2008 13

Fi h Fish Lecture 5: Motion and Tracking 22 January 2008 14

Kit h Kitchen Lecture 5: Motion and Tracking 22 January 2008 15

L Laundry d Lecture 5: Motion and Tracking 22 January 2008 16

Pi Ping-Pong P Lecture 5: Motion and Tracking 22 January 2008 17

Pl t Plate Lecture 5: Motion and Tracking 22 January 2008 18

S Sandwich d i h Lecture 5: Motion and Tracking 22 January 2008 19

T Traffic ffi Lecture 5: Motion and Tracking 22 January 2008 20

T Types of Motion Determination f M ti D t i ti � Motion Detection: identifying whether or M ti D t ti id tif i h th not image points are moving � Motion Estimation: identifying how image points are moving points are moving � Motion Segmentation: identifying moving objects from moving points Lecture 5: Motion and Tracking 22 January 2008 21

Extracting Moving Objects Extracting Moving Objects � Simple case: static background, with only Si l t ti b k d ith l the object of interest in motion Lecture 5: Motion and Tracking 22 January 2008 22

S l ti Solution: Frame Differencing F Diff i � Subtract current frame from previous frame, S bt t t f f i f and threshold the result Lecture 5: Motion and Tracking 22 January 2008 23

Accumulative Frame Differencing Accumulative Frame Differencing � Estimate motion direction by accumulating E ti t ti di ti b l ti motion history over a range of frames Lecture 5: Motion and Tracking 22 January 2008 24

M ti Motion History Image Hi t I Lecture 5: Motion and Tracking 22 January 2008 25

M lti l M Multiple Moving Objects? i Obj t ? (courtesy of Sebastian Thrun) Lecture 5: Motion and Tracking 22 January 2008 26

M ti Motion Segmentation S t ti � Add timestamp to current motion history image, Add ti t t t ti hi t i and overlay it on top of the older ones Lecture 5: Motion and Tracking 22 January 2008 27

M ti Motion Segmentation S t ti � Measure the gradients of the stack of M th di t f th t k f motion history images Lecture 5: Motion and Tracking 22 January 2008 28

M ti Motion Segmentation S t ti � Ignore motion template edges resulting I ti t l t d lti from too large of a time delay Lecture 5: Motion and Tracking 22 January 2008 29

M ti Motion Segmentation S t ti � Find boundaries of most recent motions and Fi d b d i f t t ti d fill them in to segment motion regions Segmented Segmented Motion Motion Lecture 5: Motion and Tracking 22 January 2008 30

L t’ T Let’s Try It Out! It O t! Lecture 5: Motion and Tracking 22 January 2008 31

B Background Subtraction k d S bt ti � If we know what the background looks like, If k h t th b k d l k lik we can ignore it to focus on things that are moving or changing - = = Lecture 5: Motion and Tracking 22 January 2008 32

Bl Blue Screen S Lecture 5: Motion and Tracking 22 January 2008 33

Vid Video Example E l (courtesy of Frank Dellaert) Lecture 5: Motion and Tracking 22 January 2008 34

S bt Subtraction and Thresholding ti d Th h ldi Lecture 5: Motion and Tracking 22 January 2008 35

B Basic Background Subtraction i B k d S bt ti � Assume background is mostly static A b k d i tl t ti � Build a background model by averaging g y g g pixel values across a range of frames � Gi � Given a new image, generate i t a silhouette by marking the pixels that are significantly different from the “background” value Lecture 5: Motion and Tracking 22 January 2008 36

Fi di Finding Subparts S b t � Look at contour shape and mark points L k t t h d k i t farthest from the center as hands � Can be combined with a skin color model for better results for better results Lecture 5: Motion and Tracking 22 January 2008 37

Pfi d Pfinder Example E l Lecture 5: Motion and Tracking 22 January 2008 38

D Dynamic Backgrounds? i B k d ? (courtesy of Kentaro Toyama) Lecture 5: Motion and Tracking 22 January 2008 39

L t’ T Let’s Try It Out! It O t! Lecture 5: Motion and Tracking 22 January 2008 40

K Keeping Track of Objects i T k f Obj t Lecture 5: Motion and Tracking 22 January 2008 41

Bl b T Blob Tracking ki Lecture 5: Motion and Tracking 22 January 2008 42

L t’ T Let’s Try it Out! it O t! Lecture 5: Motion and Tracking 22 January 2008 43

M More Complex Motion C l M ti (courtesy of J.M. Rehg) Lecture 5: Motion and Tracking 22 January 2008 44

M More Complex Motion C l M ti (courtesy of J.M. Rehg) Lecture 5: Motion and Tracking 22 January 2008 45

M More Complex Motion C l M ti (courtesy of J.M. Rehg) Lecture 5: Motion and Tracking 22 January 2008 46

O ti Optical Flow l Fl � A 2 D � A 2-D velocity field describing the motion in an l it fi ld d ibi th ti i image sequence � A vector at each pixel indicates its motion � A t t h i l i di t it ti direction between neighboring frames Lecture 5: Motion and Tracking 22 January 2008 47

Ch Characterizing Motion t i i M ti Image Sequence Flow Vectors (courtesy of Sebastian Thrun) Lecture 5: Motion and Tracking 22 January 2008 48

C Computing Optical Flow ti O ti l Fl (courtesy of Michael Black) Lecture 5: Motion and Tracking 22 January 2008 49

T Tracking Local Features ki L l F t r p v 2 2 r p v 3 r 3 p p v v 1 1 1 1 Optical Flow Optical Flow r p v 4 4 ( + ( + 1 1 ) ) I I t t r I ( t ), { p } { i v } Velocity vectors Velocity vectors i Lecture 5: Motion and Tracking 22 January 2008 50

O ti Optical Flow Assumptions l Fl A ti � Brightness constancy: though regions may � B i ht t th h i move around, the brightness within a small region will not change i ill t h Lecture 5: Motion and Tracking 22 January 2008 51

O ti Optical Flow Assumptions l Fl A ti � T � Temporal persistence: gradual motion over time l i t d l ti ti Lecture 5: Motion and Tracking 22 January 2008 52

A Aperture Problem t P bl Lecture 5: Motion and Tracking 22 January 2008 53

A Aperture Problem t P bl � Motion along just an edge is ambiguous M ti l j t d i bi (courtesy of Sebastian Thrun) Lecture 5: Motion and Tracking 22 January 2008 54

A Another Example th E l Lecture 5: Motion and Tracking 22 January 2008 55

H Harris Corners i C Lecture 5: Motion and Tracking 22 January 2008 56

L t’ T Let’s Try It Out! It O t! Lecture 5: Motion and Tracking 22 January 2008 57

S Segmentation by Clustering t ti b Cl t i Image Clusters on intensity Clusters on color Lecture 5: Motion and Tracking 22 January 2008 58

Simple Clustering Algorithms Si l Cl t i Al ith (courtesy of Marc Pollefeys) Lecture 5: Motion and Tracking 22 January 2008 59

Cl Clustering Example t i E l (courtesy of Marc Pollefeys) Lecture 5: Motion and Tracking 22 January 2008 60

M Mean Shift Segmentation Shift S t ti Original Image Original Image Segmented Image Segmented Image (courtesy of D. Comaniciu) Lecture 5: Motion and Tracking 22 January 2008 61