Tracking Tracking Many thanks to: H. Bischof, B. Leibe, V. Ferrari, - - PowerPoint PPT Presentation

▶
tracking tracking
SMART_READER_LITE
LIVE PREVIEW

Tracking Tracking Many thanks to: H. Bischof, B. Leibe, V. Ferrari, - - PowerPoint PPT Presentation

Dec 18, 2022 109 likes •1.5k views

Computer Vision Tracking Tracking Many thanks to: H. Bischof, B. Leibe, V. Ferrari, K. Graumann, Y. Ukrainitz, D. Wagner, V Lepetit, M. Breitenstein, P. Sabzmeydani, Z. Kalal from whom I borrowed many slides and videos. We all know what

slide-1
SLIDE 1

Computer Vision

Tracking Tracking

Many thanks to: H. Bischof, B. Leibe, V. Ferrari, K. Graumann, Y. Ukrainitz, D. Wagner, V Lepetit, M. Breitenstein, P. Sabzmeydani, Z. Kalal from whom I borrowed many slides and videos.

slide-2
SLIDE 2

Computer Vision

’06]

We all know what tracking is, right?

[Grabner et al., VideoProc CVPR’

slide-3
SLIDE 3

Computer Vision

Tracking

actual object position

Time t+1 Time t

„FIND IT AGAIN“

slide-4
SLIDE 4

Computer Vision

What to track?

slide-5
SLIDE 5

Computer Vision

What to track?

center point

slide-6
SLIDE 6

Computer Vision

What to track?

multiple points

slide-7
SLIDE 7

Computer Vision

What to track?

(body) parts

slide-8
SLIDE 8

Computer Vision

What to track?

region

slide-9
SLIDE 9

Computer Vision

What to track?

  • utline
slide-10
SLIDE 10

Computer Vision

What to track?

structure

slide-11
SLIDE 11

Computer Vision

Approaches

(i) Model-based tracking application-specific

human body, faces, space shuttle,…

slide-12
SLIDE 12

Computer Vision

Approaches

(i) Model-based tracking application-specific

human body, faces, space shuttle,…

(ii) Feature tracking more generic

corner tracking blob/contour tracking intensity profile tracking region tracking

slide-13
SLIDE 13

Computer Vision Saliency Object

Tracking Cues

Model/ Tracking History Scene

slide-14
SLIDE 14

Computer Vision

Applications!

  • Structure-from-Motion
  • Gesture/Action Recognition
  • Video editing
  • Augmented Reality
  • Augmented Reality
  • Navigation
  • ….
slide-15
SLIDE 15

Computer Vision

Applications: Game Interface

slide-16
SLIDE 16

Computer Vision

Applications: SfM

  • Tracked Points gives correspondences
slide-17
SLIDE 17

Computer Vision

Applications: SfM

004] [Pollefeyes et al. IJCV 200

slide-18
SLIDE 18

Computer Vision

Applications: Analysis of Motion Pattern

Single-Agent Level Multi-Agent Level Scene Level Detail Level

slide-19
SLIDE 19

Computer Vision

[Ess et al. CVPR’08]

slide-20
SLIDE 20

Computer Vision

Outline

  • Point Tracking
  • Template Tracking
  • Region Tracking
  • Foreground vs. Background
  • Tracking-by-Detection
  • Combining Tracking and Detection
  • Context in Tracking
slide-21
SLIDE 21

Computer Vision

Motion

slide-22
SLIDE 22

Computer Vision

Motion

slide-23
SLIDE 23

Computer Vision

Motion

slide-24
SLIDE 24

Computer Vision

x y

Vision = Inverse Graphics

x y z

slide-25
SLIDE 25

Computer Vision

x y

Vision = Inverse Graphics

x y z

Tracking = Inverse Animation

slide-26
SLIDE 26

Computer Vision

Steps of Tracking

predict predict correct correct

  • Recap: Particle filtering

– Tracking can be seen as the process of propagating the posterior distribution of state given measurements across time.

slide-27
SLIDE 27

Computer Vision

) | , (

1 1 1 − − − t t t

z p p p & ) | , (

1 − t t t

z p p p &

prediction C O N D E N Particle Filter

) | (

t t

p z p

weighing with

) | , (

t t t

z p p p &

update N S A T I O N

slide-28
SLIDE 28

Computer Vision

General Tracking Loop

predict to t+1 time t measure at t+1 update location update model

slide-29
SLIDE 29

Computer Vision

Point Tracking Point Tracking

slide-30
SLIDE 30

Computer Vision

Estimate Optimal Transformation

slide-31
SLIDE 31

Computer Vision

Simple 1D Problem

I0(x) I (x+h) I1(x+h)

slide-32
SLIDE 32

Computer Vision

Sum of Squared Differences

I0(x) I (x+h) h I1(x+h)

E(h) = [I0(x) – I1(x+h) ]2

slide-33
SLIDE 33

Computer Vision

Calculation of Displacement

E(h) [ I0 (x) – I1(x) – hI1’(x) ]2

≈

E(h) = [I0(x) – I1(x+h) ]2

  • 2[I1’(x)(I0(x) – I1(x) ) – hI1’(x)2]

h E ∂ ∂

≈

I0(x) – I1(x) h I1’(x)

≈

slide-34
SLIDE 34

Computer Vision

Interpretation

h I0(x) I (x+h) I0(x) – I1(x) I1’(x) I1(x+h)

I0(x) – I1(x) h I1’(x)

≈

I1’(x)

slide-35
SLIDE 35

Computer Vision

Problem A: Local Minima

(a) (b)

slide-36
SLIDE 36

Computer Vision

Problem A: Local Minima

slide-37
SLIDE 37

Computer Vision

Problem B: Zero Gradient

I(x) - I0(x) h ≈ I0’(x)

?

slide-38
SLIDE 38

Computer Vision

Problem B: Aperture problem

No gradient along

  • ne direction:
slide-39
SLIDE 39

Computer Vision

Problem B: Aperture problem

No gradient along

  • ne direction:
slide-40
SLIDE 40

Computer Vision

Recall: Optical Flow

slide-41
SLIDE 41

Computer Vision

, I I x ∂ ∂ =

, ∂ ∂ = y I I y

t I It ∂ ∂ =

Recall: Optical Flow

= + +

t y x

I v I u I

1 equation in 2 unknowns

, = dt dx u

dt dy v =

, x I x ∂ =

, ∂ = y I y

t It ∂ =

slide-42
SLIDE 42

Computer Vision

“Solving” the Aperture Problem

  • How to get more equations for a

pixel?

  • Spatial coherence constraint: Pixel’s

neighbors have the same movement neighbors have the same movement I0(x) I1(x+h)

slide-43
SLIDE 43

Computer Vision

Least Squares Problem

Pseudo Inverse Over determined System of Equations

slide-44
SLIDE 44

Computer Vision

Eigenvectors of ATA

  • Haven’t we seen an equation like this
  • Haven’t we seen an equation like this

before?

  • Recall the Harris corner detector!
  • “Good Features to Track”
slide-45
SLIDE 45

Computer Vision

Interpreting the Eigenvalues

λ2

“Corner” λ λ λ λ1 ~ λ λ λ λ2 and large “Edge” λ λ λ λ2 >> λ λ λ λ1

λ1

“Edge” λ λ λ λ1 >> λ λ λ λ2 “Flat” region

slide-46
SLIDE 46

Computer Vision

Samples: Edge / Low Texture / High Texture

slide-47
SLIDE 47

Computer Vision

Example

slide-48
SLIDE 48

Computer Vision

Template Tracking Template Tracking

slide-49
SLIDE 49

Computer Vision

Lucas-Kanade Template Tracker

  • From Points to templates
  • Estimate „optimal“ warp W
slide-50
SLIDE 50

Computer Vision

4, Lucas-Kanade ramework] [Baker & Matthews, IJCV’04 20 Years On: A Unifying Fr

slide-51
SLIDE 51

Computer Vision

Example

slide-52
SLIDE 52

Computer Vision

Region Tracking Region Tracking

slide-53
SLIDE 53

Computer Vision

Background Modeling

Input Background Model Moving Foreground Blobs (Objects)

slide-54
SLIDE 54

Computer Vision

Mean Shift Tracking

  • The mean shift tracker tracks a region,

with a prescribed (color) distribution

  • The similarity between the tracked

region and the target region is

9]

region and the target region is maximized, through evolution towards higher density in a parameter space

  • Typically this search only takes a few

iterations

[Comaniciu and Meer, ICCV’99

slide-55
SLIDE 55

Computer Vision

Meanshift Tracking

Region of interest (Kernel) Center of mass Mean Shift vector Measurements

slide-56
SLIDE 56

Computer Vision

Intuitive Description

slide-57
SLIDE 57

Computer Vision

Intuitive Description

slide-58
SLIDE 58

Computer Vision

Intuitive Description

slide-59
SLIDE 59

Computer Vision

Intuitive Description

slide-60
SLIDE 60

Computer Vision

Intuitive Description

slide-61
SLIDE 61

Computer Vision

Intuitive Description

slide-62
SLIDE 62

Computer Vision

Intuitive Desciption

slide-63
SLIDE 63

Computer Vision

Example

slide-64
SLIDE 64

Computer Vision

Elderly People Monitoring

slide-65
SLIDE 65

Computer Vision

Model based Model based Tracking

slide-66
SLIDE 66

Computer Vision

Articulated Tracking with Part-Based Model

  • part appearance + relative geometry.
slide-67
SLIDE 67

Computer Vision

Using Models

  • Goal

– Recover a person’s body articulation – Detailed parameterization in terms of joint locations or joint angles

  • Two basic classes of approaches

– Articulated tracking as high- dimensional inference – Part-based models

slide-68
SLIDE 68

Computer Vision

[Ramanan et al. CVPR’05]

slide-69
SLIDE 69

Computer Vision

Tracking as On-line Foreground vs. Background Background Classification

slide-70
SLIDE 70

Computer Vision

Tracking as Classification

  • Learning current object appearance vs. local

background.

current background background

[Grabner et al. CVPR’06]

current

  • bject

appearance

slide-71
SLIDE 71

Computer Vision

Tracking as Classification

  • bject

background vs.

slide-72
SLIDE 72

Computer Vision

Tracking as Classification

  • bject

background vs.

slide-73
SLIDE 73

Computer Vision

Tracking Loop

search Region actual object position from time t to t+1 evaluate classifier on sub-patches

  • +
  • create confidence map

analyze map and set new

  • bject position

update classifier (tracker)

slide-74
SLIDE 74

Computer Vision

slide-75
SLIDE 75

Computer Vision

“Simple tracking”

slide-76
SLIDE 76

Computer Vision

“Tracking the Invisible”

slide-77
SLIDE 77

Computer Vision

When does it fail…

slide-78
SLIDE 78

Computer Vision

search Region actual object position from time t to t+1 evaluate classifier on sub-patches

  • +
  • create confidence map

analyze map and set new object position update classifier (tracker)

Self-learning!

slide-79
SLIDE 79

Computer Vision

Drifting

Tracked Patches Confidence

slide-80
SLIDE 80

Computer Vision

Drifting

CLICK HERE TO START

slide-81
SLIDE 81

Computer Vision

Drifting

slide-82
SLIDE 82

Computer Vision

Tracking by Tracking by Detection

(specific target)

slide-83
SLIDE 83

Computer Vision

3D Object Detection

Reference image(s) of the object to detect Test image

slide-84
SLIDE 84

Computer Vision

Standard Approach

  • Step 1: Keypoint detection

– invariant to scale, rotation, or perspective

slide-85
SLIDE 85

Computer Vision

Standard Approach

  • Step 2: Patch rectification
slide-86
SLIDE 86

Computer Vision

Standard Approach

  • Step 3: Build descriptor vector
slide-87
SLIDE 87

Computer Vision

Standard Approach

  • Step 4: Match descriptor vectors

Query Database

slide-88
SLIDE 88

Computer Vision

Summary

Search in the Database Search in the Database

Keypoint Detection Keypoint Recognition

Database Database Pre-processing Make the actual classification easier Robust 3D Pose Calculation (RANSAC) Robust 3D Pose Calculation (RANSAC)

Geometric verification

slide-89
SLIDE 89

Computer Vision

[Wagner et al. ISMAR’08]

slide-90
SLIDE 90

Computer Vision

[Wagner et al. ‘09]

slide-91
SLIDE 91

Computer Vision

Tracking by Detection Detection

(object class)

slide-92
SLIDE 92

Computer Vision

Traditional Tracking

t=1 initialization t=2 position in prev. frame candidate new positions (e.g., dynamics) best new position (e.g., max color similarity)

slide-93
SLIDE 93

Computer Vision

Tracking-by-Detection

…

detect object(s) independently in each frame associate detections over time into tracks

slide-94
SLIDE 94

Computer Vision

Multiple Objects

Frame 5 Frame 1 Frame 9

slide-95
SLIDE 95

Computer Vision

Example: Multiple Object Tracking

slide-96
SLIDE 96

Computer Vision

How to get the detections?

Persons Background

Supervised Learning

slide-97
SLIDE 97

Computer Vision

Using the classifier

slide-98
SLIDE 98

Computer Vision

How to link them?

  • Space-Time Analysis:

(a) collect detections

Detections Space Time Volume [Leibe et al. CVPR’07]

slide-99
SLIDE 99

Computer Vision

Trajectory Estimation

(a) collect detections (b) trajectory growing and selection

t x t z

Space Time Volume

slide-100
SLIDE 100

Computer Vision

Trajectory Estimation

(a) collect detections (b) trajectory growing and selection

t x t z

H1 H2

Space Time Volume

slide-101
SLIDE 101

Computer Vision

Result

Input (Object Detections) “Tracking” Result

slide-102
SLIDE 102

Computer Vision

slide-103
SLIDE 103

Computer Vision

More information helps…

  • Articulated tracking

– “walking” people

  • 3D Information

Ground Plane Depth verification

slide-104
SLIDE 104

Computer Vision

[Gammeter et al. ECCV’08]

slide-105
SLIDE 105

Computer Vision

Towards Scene Interpretation

slide-106
SLIDE 106

Computer Vision

Combining Tracking and Tracking and Detection

slide-107
SLIDE 107

Computer Vision

Combination: KLT & TbD

  • Use a KLT Tracker

to explore

  • Learn an object

detector on the fly.

[Kalal et al. CVPR’10]

slide-108
SLIDE 108

Computer Vision

slide-109
SLIDE 109

Computer Vision

slide-110
SLIDE 110

Computer Vision

slide-111
SLIDE 111

Computer Vision

Context in Tracking in Tracking

slide-112
SLIDE 112

Computer Vision

I’m Carl – Track me…

[Grabner et al. CVPR’10]

slide-113
SLIDE 113

Computer Vision

Tracking Carl

slide-114
SLIDE 114

Computer Vision

SUPPORTERS…

  • … came with different strength.
  • … change over time.
slide-115
SLIDE 115

Computer Vision

SUPPORTERS…

  • … came with different strength.
  • … change over time.
slide-116
SLIDE 116

Computer Vision

SUPPORTERS help Tracking of…

  • … objects which

change there appearance very quickly.

  • … occluded
  • bjects or object
  • utside the image.
  • … small and/or

low textured

  • bjects or even

“virtual points”.

slide-117
SLIDE 117

Computer Vision

ETH-Cup Sequenze

slide-118
SLIDE 118

Computer Vision

ETH-Cup: Humans

slide-119
SLIDE 119

Computer Vision

Of the Web Tracker

slide-120
SLIDE 120

Computer Vision

ETH-Cup: Supporters

slide-121
SLIDE 121

Computer Vision

Beyond the Image

Supporters

slide-122
SLIDE 122

Computer Vision

Coupled Motion

Supporters

slide-123
SLIDE 123

Computer Vision

Changing Supporters

Supporters

slide-124
SLIDE 124

Computer Vision

Obviously, there are failure cases…

…. and magician knows that.

Supporters

slide-125
SLIDE 125

Computer Vision

Tracking Issues Tracking Issues

slide-126
SLIDE 126

Computer Vision

Tracking Requirements

  • Strongly depends on the application!

Robust, Accurate, Fast,…

  • Constrain the tracking task!

Information about the object, dynamics, environment,…

slide-127
SLIDE 127

Computer Vision

Tracking Issues

  • Initialization
  • bject position

Time t = 0

  • bject position
slide-128
SLIDE 128

Computer Vision

Tracking Issues

  • Prediction vs. Correction

– If the dynamics model is too strong, will end up ignoring the data – If the observation model is too strong, tracking is reduced to repeated detection is reduced to repeated detection

http://www.ethlife.ethz.ch/archive_articles/091008_kalman_per/index

slide-129
SLIDE 129

Computer Vision

Tracking Issues

  • Obtaining observation…

– Generative: “render” the state on top of the image and compare – Discriminative: classifier or detector score score

  • …and dynamics model

– specify using domain knowledge – learn (very difficult)

slide-130
SLIDE 130

Computer Vision

Tracking Issues

  • Nonlinear

dynamics

– Sometimes needed to keep multiple trackers in parallel trackers in parallel – E.g., for abrupt direction changes („Persons“)

Wrong prediction Correct prediction

slide-131
SLIDE 131

Computer Vision

Tracking Issues

  • Data Association - Multiple Object

Tracking

– What if we don’t know which measurements to associate with which tracks? tracks?

slide-132
SLIDE 132

Computer Vision

Tracking Issues

  • Data Association – Fast Motion
slide-133
SLIDE 133

Computer Vision

Tracking Issues

  • Data Association – Background /

Appearance Change

– Cluttered Background – Changes in shape, orientation, color,… – Changes in shape, orientation, color,…

slide-134
SLIDE 134

Computer Vision

Tracking Issues

  • Data Association – Occlusions / Self

Occlusions

slide-135
SLIDE 135

Computer Vision

Tracking Issues

  • Model- vs. Model-free-Tracking
slide-136
SLIDE 136

Computer Vision

Tracking Issues

  • Drift

– Errors caused by dynamical model,

  • bservation model, and data association

tend to accumulate over time

slide-137
SLIDE 137

Computer Vision

End.