VALSE VA ON ONLINE Tr Tracking Mu Multiple Ob Objects in in - - PowerPoint PPT Presentation

Tr Tracking Mu Multiple Ob Objects in in Im Image Se Sequences

Xinchao Wang

June, 2018

VA VALSE

ON ONLINE

2

3

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

4

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

5

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:

6

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:
• A concrete body, like a person or a car

7

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:
• A concrete body, like a person or a car
• An event, like a person gets into a car

8

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:
• A concrete body, like a person or a car
• An event, like a person gets into a car
• State of object:

9

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:
• A concrete body, like a person or a car
• An event, like a person gets into a car
• State of object:
• State of a concrete body, like the location of a person

10

TPAMI’16

MOT

Definition Tracking by Detection Challenges

Definition of Multiple Objects Tracking (MOT)

• The operation that determines the states of objects over time while

preserving their identities

• Object:
• A concrete body, like a person or a car
• An event, like a person gets into a car
• State of object:
• State of a concrete body, like the location of a person
• State of an event, like the number of people inside a car

TPAMI’16

MOT

Definition Tracking by Detection Challenges

Examples of MOT

• People tracking
• Event tracking

12

MVA’16

t=9 t=10 t=11

TMI’17

MOT

Definition Tracking by Detection Challenges

13

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Challenges of MOT

• Object Interactions
• Track the targets
• Track the interaction events

14

People and Car Player and Basketball Player and Soccer People and Backpack TPAMI’16 CVIU’14 CVPR’16 ECCV’14

MOT

Definition Tracking by Detection Challenges

Challenges of MOT

• Occlusions
• Moving targets may occlude one another

15

Cells occlude one another People occlude one another Players occlude ball TPAMI’16 TIP’17 TIP’18 TMI’17

MOT

Definition Tracking by Detection Challenges

Challenges of MOT

• Motions
• Targets may follow specific motion pattern

16

Ball follows parabolic motion CVPR’16 Pedestrian follow high-order walking pattern ICCV’17

MOT

Definition Tracking by Detection Challenges

TIP’18

Challenges of MOT

• Online
• Tracking multiple targets in an online or semi-online manner

17

TIP’17 Batchn Batchn+1

18

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Solving MOT: the Tracking by Detection Paradigm

• Workflow

Input Images

(A batchor the whole video)

Detection

(on 2D images or the 3D world)

Tracking

(on 2D images or the 3D world)

MOT

Definition Tracking by Detection Challenges

Detection

• Probability Occupancy Map (POM)

20

It

MOT

Definition Tracking by Detection Challenges

Detection

• Probability Occupancy Map (POM)

21

It

MOT

Definition Tracking by Detection Challenges

Detection

• Probability Occupancy Map (POM)

22

Xt

k0 ∈ {0, 1}

ρt

k0 = P(Xt k0 = 1 | It)

MOT

Definition Tracking by Detection Challenges

Tracking

• Network flow programming

23

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1

… … … … …

MOT

Definition Tracking by Detection Challenges

Tracking

• Network flow programming

24

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1

… … … … …

xt

k

xt+1

j

f t

kj ∈ {0, 1}

MOT

Definition Tracking by Detection Challenges

Tracking

• Network flow programming

25

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1

… … … … …

f ∗ = argmax

f∈F

X

t,k,j

log ✓ ρt

k

1−ρt

k

◆ f t

j

MOT

Definition Tracking by Detection Challenges

26

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Handling Object Interactionsin MOT

• Interactions:
• Two or more objects involved
• Our focus:
• One class of object (i.e., container) “contain” the other (i.e., containee)

27

Cars-People People-Bags Players-Basketball Players-Soccer

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Objective

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1 xt

k

yt

k

xt+1

j

yt+1

j

f t

kj ∈ {0, 1}

gt

kj ∈ {0, 1}

(x,y)∗ = argmax

(x,y)∈F

X

t,k

log ✓ ρt

k

1−ρt

k

◆ xt

k+log

✓ βt

k

1−βt

k

◆ yt

k

and are intertwined on each edge f t

kj

gt

kj

Container Containee

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Objective

29

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1 argmax

(f,g)∈F

X

t

X

j∈N (k)

X

k

log ✓ ρt

k

1−ρt

k

◆ f t

kj +log

✓ βt

k

1−βt

k

◆ gt

kj

gt

kj

f t

kj

(x,y)∗ = argmax

(x,y)∈F

X

t,k

log ✓ ρt

k

1−ρt

k

◆ xt

k+log

✓ βt

k

1−βt

k

◆ yt

k

Container Containee

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Objective

30

…

t t + 1 t − 1

…

t t + 1

… … … … … … … …

t − 1 argmax

(f,g)∈F

X

t

X

j∈N (k)

X

k

log ✓ ρt

k

1−ρt

k

◆ f t

kj +log

✓ βt

k

1−βt

k

◆ gt

kj

gt

kj

f t

kj

(x,y)∗ = argmax

(x,y)∈F

X

t,k

log ✓ ρt

k

1−ρt

k

◆ xt

k+log

✓ βt

k

1−βt

k

◆ yt

k

Container Containee

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Constraints

• Flow Conservation on Container Objects
• Spatial Exclusion on the ground
• Consistency of Interacting Flows
• Tracking the Invisible
• Additional Bound Constraint

31

✔

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Constraints

• Tracking the Invisible – Counter Flow Variable

32

h

The person is in the car after he enters it. ht

kj = ht−1 ik

+ 1 ht−1

ik

ht

kj

Container Containee

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Constraints

• Tracking the Invisible – Counter Flow Variable

33

h

The person was in the car before he exits. ht

kj = ht−1 ik

− 1 ht−1

ik

ht

kj

Container Containee

MOT Challenges

Interactions Motions Occlusion

• nline

Formulation: Constraints

• Tracking the Invisible – Counter Flow Variable

34

h

The person was in the car before he exits. The person is in the car after he enters it.

X

k:l=l(k) j∈N (k)

ht

kj =

X

k:l=l(k), i:k∈N (i)

ht−1

ik

+ X

k:l=l(k), i:k∈N (i)

f t−1

ik

− X

k:l=l(k) j∈N (k)

f t

kj, ∀t, l

ht

kj = ht−1 ik

+ 1 ht

kj = ht−1 ik

− 1 ht−1

ik

ht−1

ik

ht

kj

ht

kj

MOT Challenges

Interactions Motions Occlusion

• nline

Simultaneous Approach: Results

35

PETS2006

SSP KSP−fixed KSP−free KSP−sequential TIF−LP TIF−MIP

0.2 0.3 0.4 0.5 0.6 0.7 −0.5 0.5 1 Overlap Threshold MOTA

First track big then small Ours Linear Programming

MOT Challenges

Interactions Motions Occlusion

• nline

38

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Handling Occlusions in MOT

• Occlusions:
• Incomplete image evidence
• How to model occlusions in MOT?

39

MOT Challenges

Interactions Motions Occlusion

• nline

Modeling Occlusions

• Example: tracking cells

40

Source Segmentation Ground Truth t=76 t=77 t=78 KTH [1] CT [2] Ours

[1] K. Magnusson et al., A Batch Algorithm Using Iterative Application of the Viterbi Algorithm to Track Cells and Construct Cell Lineages. ISBI’12. [2] M. Schiegg et al., Conservation Tracking. ICCV’13.

MOT Challenges

Interactions Motions Occlusion

• nline

Workflow

41

Source Images Segmentation Generating conflicting hypotheses Network flow programming to detect and track cells Pixel-based classifier

✔ ✔

MOT Challenges

Interactions Motions Occlusion

• nline

Generating Conflicting Hypotheses

42

Image Segmentation Hierarchy Ellipse fitting conflic t Level 1 Level 2 Level 3 Level 4

MOT Challenges

Interactions Motions Occlusion

• nline

Network Flow Programming: Graph and Objective

43

Migration Appearance Disappearance Division Source Sink Division

fij, fsj, fit, fdj ∈ {0, 1}

f ∗ = argmax

f∈F

X

eij∈E0

log ✓ ρij 1−ρij ◆ fij + X

j∈V

log ✓ ρj 1−ρj ◆ fdj + X

j∈V

log ✓ ρa 1−ρa ◆ fsj + X

i∈V

log ✓ ρd 1−ρd ◆ fit

MOT Challenges

Interactions Motions Occlusion

• nline

Quantitative Results

Division Detection Migration Recall Precision F-Measure Recall Precision F-Measure Recall Precision F-Measure GMM [1] 0.40 0.18 0.24 N/A N/A N/A 0.95 0.98 0.96 KTH [2] 0.65 0.72 0.68 N/A N/A N/A 0.94 0.99 0.97 CT [3] 0.76 0.81 0.78 0.73 0.63 0.67 0.94 0.99 0.96 OURS 0.86 0.83 0.84 0.86 0.78 0.82 0.96 0.99 0.97

44

[1] F. Amat et al., Fast, accurate reconstruction of cell lineages from large-scale fluorescencemicroscopy data. Nature methods, 2014. [2] K. Magnusson et al., A Batch Algorithm Using Iterative Application of the Viterbi Algorithm to Track Cells and Construct Cell Lineages. ISBI’12. [3] M. Schiegg et al., Conservation Tracking. ICCV’13.

F-Measure: harmonic mean of Recall and Precision

MOT Challenges

Interactions Motions Occlusion

• nline

46

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Handling Motions in MOT

• Motions:
• Targets may follow some specific motion patterns

47

MOT Challenges

Interactions Motions Occlusion

• nline

Motion of a Ball

• Model physically constraints and interactions jointly

48

MOT Challenges

Interactions Motions Occlusion

• nline

Results

200 400 600 800 1000

Distance threshold, cm

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Tracking accuracy, % Volley-1

OUR OUR-No-Physics OUR-Two-States InterTrack[28] KSP[2] MaxDetection RANSAC[23] Growth[10] 49

200 400 600 800 1000

Distance threshold, cm

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Tracking accuracy, % Basket-2

OUR OUR-No-Physics OUR-Two-States InterTrack[28] KSP[2] MaxDetection RANSAC[23] FoS[27]

MOT Challenges

Interactions Motions Occlusion

• nline

51

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Handling Online Processing in MOT

• Online:
• Tracking multiple targets in an online or semi-online manner

52

TIP’17 Batchn Batchn+1

MOT Challenges

Interactions Motions Occlusion

• nline

Online Processing

• Tracking on frame batches
• Consecutive batches are not independent

53 Batch m (KSP) Batch m+1 (KSP) Batch m (E-KSP) Batch m+1 (E-KSP) Tracking Results on Both Batches Tracking Results on Both Batches

Hungarian E-KSP

Conventional Approach Our Approach MOT Challenges

Interactions Motions Occlusion

• nline

54

Handling Challenges

Interactions Occlusions Motions

Overview

Definition Challenges Tracking by Detection

MOT

Online

Product and Press: NBA

56

Product and Press: Volleyball Championship

57

Code, Dataset and Application Publicly Available!

• [code and dataset] TPAMI’16
• http://cvlab.epfl.ch/op/preview/TrackInteractObj
• [code] CVPR’16
• https://github.com/cvlab-epfl/balltracking
• [code] ICCV’17
• https://github.com/maksay/ptrack_cpp
• [code] TMI’17
• http://cvlab.epfl.ch/biomed/cell-tracking
• [code and dataset] ECCV’14
• http://cvlab.epfl.ch/op/preview/TrackInteractObj
• [code] TIP’17
• https://www.dropbox.com/s/4w2yweriycyqevh/code_public_TIP_batch_

v1.0.zip?dl=0

• [dataset] MVA’16
• http://campar.in.tum.de/Chair/MultiHumanOR
• [application] CVIU’14
• http://bball-roulette.epfl.ch/
• http://volleyballtracking.com/

58

Reference

TIP’18: Interacting Tracklets for Multi-object Tracking [Motion] ICCV’17: Globally Consistent Multi-People Tracking using Motion Patterns [Motion] TIP’17: Greedy Batch-based Minimum-cost Flows for Tracking Multiple Objects [Online] [Occlusions] TMI’17: Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences [Occlusions] TPAMI’16: Tracking Interacting Objects Using Intertwined Flows [Interaction] [Online] CVPR’16: What Players do with the Ball: A Physically Constrained Interaction Modeling [Interaction] [Motion] THESIS’15 Tracking Interacting Objects in ImageSequences [Interaction] [Motion] ECCV’14: Tracking Interacting Objects Optimally Using Integer Programming [Interaction] CVIU’14: What Players do with the Ball: A Physically Constrained Interaction Modeling [Interaction] [Motion]