Online Multi-Target Tracking Using Recurrent Neural Networks Anton - - PowerPoint PPT Presentation

▶
online multi target tracking using
SMART_READER_LITE
LIVE PREVIEW

Online Multi-Target Tracking Using Recurrent Neural Networks Anton - - PowerPoint PPT Presentation

Feb 25, 2023 332 likes •828 views

Online Multi-Target Tracking Using Recurrent Neural Networks Anton Milan, S. Hamid Rezatofighi , Anthony Dick, Ian Reid, Konrad Schindler Outline Some applications of multi-target tracking The challenges Existing approaches Our

slide-1
SLIDE 1

Online Multi-Target Tracking Using Recurrent Neural Networks

Anton Milan, S. Hamid Rezatofighi, Anthony Dick, Ian Reid, Konrad Schindler

slide-2
SLIDE 2
  • S. Hamid Rezatofighi

2

Outline

  • Some applications of multi-target tracking
  • The challenges
  • Existing approaches
  • Our motivation for using RNN
  • Our idea and contribution
  • Preliminary results
  • Conclusion
slide-3
SLIDE 3
  • S. Hamid Rezatofighi

3

Multi-target tracking: Applications

  • Multiple similar targets in very noisy sequences of sonar
  • r radar
slide-4
SLIDE 4
  • S. Hamid Rezatofighi

4

Multi-target tracking: Applications

  • Tracking several pedestrians in a very crowded scene in

surveillance camera

slide-5
SLIDE 5
  • S. Hamid Rezatofighi

5

Multi-target tracking: Applications

  • Tracking

populated and dense cells in biological sequences

slide-6
SLIDE 6
  • S. Hamid Rezatofighi

6

Multi-target tracking: Applications

  • 3D particle tracking velocimetry for flow measurement
  • etc
slide-7
SLIDE 7
  • S. Hamid Rezatofighi

7

Challenges in Multi-Target Tracking

  • Unknown and time-varying numbers of targets
slide-8
SLIDE 8
  • S. Hamid Rezatofighi

8

Challenges in Multi-Target Tracking

  • Unknown and time-varying numbers of targets
  • The complex behaviors of targets
  • Maneuvering dynamics of the targets
  • Targets occlusion and entering or exiting from scene
  • Interactions with other targets such as targets splitting
slide-9
SLIDE 9
  • S. Hamid Rezatofighi

9

Challenges in Multi-Target Tracking

  • Noisy observations

Applying a detection method, But imperfect detection due to noisy sequences Misdetections, noisy and spurious measurements (clutter) Track-before-detect approach, Detection is not applicable Noisy features

slide-10
SLIDE 10
  • S. Hamid Rezatofighi

10

Multi-Target Tracking with Detection

  • bservation

produced by

  • bjects

state dynamic state space

  • bservation space

5 objects 3 objects

An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of

  • Noisy observations

⎻Clutter noise, Detection uncertainty

  • Data association uncertainty,
slide-11
SLIDE 11
  • S. Hamid Rezatofighi

11

Multi-Target Tracking with Detection

  • bservation

produced by

  • bjects

state dynamic state space

  • bservation space

5 objects 3 objects

Survive or Die?

An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of

  • Noisy observations

⎻Clutter noise, Detection uncertainty

  • Data association uncertainty,
slide-12
SLIDE 12
  • S. Hamid Rezatofighi

12

Multi-Target Tracking with Detection

  • bservation

produced by

  • bjects

state dynamic state space

  • bservation space

5 objects 3 objects

New born or existing target?

An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of

  • Noisy observations

⎻Clutter noise, Detection uncertainty

  • Data association uncertainty,
slide-13
SLIDE 13
  • S. Hamid Rezatofighi

13

Multi-Target Tracking with Detection

  • bservation

produced by

  • bjects

state dynamic state space

  • bservation space

5 objects 3 objects

? ? ?

An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of

  • Noisy observations

⎻Clutter noise, Detection uncertainty

  • Data association uncertainty,
slide-14
SLIDE 14
  • S. Hamid Rezatofighi

14

Multi-Target Tracking with Detection

  • bservation

produced by

  • bjects

state dynamic state space

  • bservation space

5 objects 3 objects

Missed detected?

An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of

  • Noisy observations

⎻Clutter noise, Detection uncertainty

  • Data association uncertainty,
slide-15
SLIDE 15
  • S. Hamid Rezatofighi

15

Existing Approaches

Bayesian Filtering + Data Associations (Online)

slide-16
SLIDE 16
  • S. Hamid Rezatofighi

16

Existing Approaches

Bayesian Filtering + Data Associations (Online) Bayesian Filtering

  • Kalman filter [1] and its variation
  • Particle filter [2] and its variations

[1] Kalman. Journal of Basic Engineering, 1960 [2] Liu and Chen, Journal of the American Statistical Association, 1998

slide-17
SLIDE 17
  • S. Hamid Rezatofighi

17

Existing Approaches

Bayesian Filtering + Data Associations (Online) Bayesian Filtering

  • Kalman filter [1] and its variation
  • Particle filter [2] and its variations

Data association:

  • Multi Assignment Problem (MAP)
  • Multiple Hypothesis Tracking (MHT) [3]
  • Joint Probabilistic Data Association (JPDA,IJPDA) [4]

[1] Kalman, Journal of Basic Engineering, 1960 [2] Liu and Chen, Journal of the American Statistical Association, 1998 [3] Reid, TAC 1979 [4] Fortman et al, CDC 1980

slide-18
SLIDE 18
  • S. Hamid Rezatofighi

18

Existing Approaches

Bayesian Filtering + Data Associations (Online) Bayesian Filtering

  • Kalman filter [1] and its variation
  • Particle filter [2] and its variations

Data association:

  • Multi Assignment Problem (MAP)
  • Multiple Hypothesis Tracking (MHT) [3]
  • Joint Probabilistic Data Association (JPDA,IJPDA) [4]

Parametric approaches: Prior knowledge is required

[1] Kalman, Journal of Basic Engineering, 1960 [2] Liu and Chen, Journal of the American Statistical Association, 1998 [3] Reid, TAC 1979 [4] Fortman et al, CDC 1980

slide-19
SLIDE 19
  • S. Hamid Rezatofighi

19

Existing Approaches

Optimization based approaches (Often Offline) with

slide-20
SLIDE 20
  • S. Hamid Rezatofighi

20

Existing Approaches

Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches

  • Shortest-Path algorithms [1]
  • Min-cost Max Flow algorithms [2]

[1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008

slide-21
SLIDE 21
  • S. Hamid Rezatofighi

21

Existing Approaches

Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches

  • Shortest-Path algorithms [1]
  • Min-cost Max Flow algorithms [2]

Non-Linear and Complex Objectives

  • Alpha-Expansion approaches [3]
  • Discrete-Continuous optimization [4]

[1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008 [3] Leibe et al, ICCV 2007 [4] Milan et al, TPAMI 2014

slide-22
SLIDE 22
  • S. Hamid Rezatofighi

22

Existing Approaches

Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches

  • Shortest-Path algorithms [1]
  • Min-cost Max Flow algorithms [2]

Non-Linear and Complex Objectives

  • Alpha-Expansion approaches [3]
  • Discrete-Continuous optimization [4]
  • 1. Hand-crafted objectives are required to be defined.
  • 2. The approaches are often offline (batch processing).

[1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008 [3] Leibe et al, ICCV 2007 [4] Milan et al, TPAMI 2014

slide-23
SLIDE 23
  • S. Hamid Rezatofighi

23

RNN for Multi-Target Tracking

Our Motivations:

slide-24
SLIDE 24
  • S. Hamid Rezatofighi

24

RNN for Multi-Target Tracking

Our Motivations:

  • A generic and model free approach
slide-25
SLIDE 25
  • S. Hamid Rezatofighi

25

RNN for Multi-Target Tracking

Our Motivations:

  • A generic and model free approach
  • A reliable data driven approach using deep structured

networks

slide-26
SLIDE 26
  • S. Hamid Rezatofighi

26

RNN for Multi-Target Tracking

Our Motivations:

  • A generic and model free approach
  • A reliable data driven approach using deep structured

networks

  • An online multi-target tracking approach
slide-27
SLIDE 27
  • S. Hamid Rezatofighi

27

RNN for Multi-Target Tracking

Our Motivations:

  • A generic and model free approach
  • A reliable data driven approach using deep structured

networks

  • An online multi-target tracking approach

RNN/LSTM

slide-28
SLIDE 28
  • S. Hamid Rezatofighi

28

No Trivial Solution For The Problem

slide-29
SLIDE 29
  • S. Hamid Rezatofighi

29

No Trivial Solution For The Problem

Why?

  • Unknown + Time-varying number of targets
  • Input dimension unknown (for tracking-by-detection)
  • Output dimension unknown
  • “Class” has no semantic meaning
  • Arbitrary assignments
  • Multiple equally correct solutions

1 2 2 1

slide-30
SLIDE 30
  • S. Hamid Rezatofighi

30

RNN for Multi-Target Tracking

Our idea/contribution toward end-to-end model learning:

slide-31
SLIDE 31
  • S. Hamid Rezatofighi

31

RNN for Multi-Target Tracking

Our idea/contribution toward end-to-end model learning:

  • Prediction: An RNN to learn a complex dynamic model for predicting

target motion in the absence of measurements.

slide-32
SLIDE 32
  • S. Hamid Rezatofighi

32

RNN for Multi-Target Tracking

Our idea/contribution toward end-to-end model learning:

  • Prediction: An RNN to learn a complex dynamic model for predicting

target motion in the absence of measurements.

  • Update: An RNN to learn noisy measurement model in presence of

false detections (clutter).

slide-33
SLIDE 33
  • S. Hamid Rezatofighi

33

RNN for Multi-Target Tracking

Our idea/contribution toward end-to-end model learning:

  • Prediction: An RNN to learn a complex dynamic model for predicting

target motion in the absence of measurements.

  • Update: An RNN to learn noisy measurement model in presence of

false detections (clutter).

  • Birth/death: to identify track initiation and termination.
slide-34
SLIDE 34
  • S. Hamid Rezatofighi

34

RNN for Multi-Target Tracking

Our idea/contribution toward end-to-end model learning:

  • Prediction: An RNN to learn a complex dynamic model for predicting

target motion in the absence of measurements.

  • Update: An RNN to learn noisy measurement model in presence of

false detections (clutter).

  • Birth/death: to identify track initiation and termination.
  • Data Association: An LSTM to learn the challenging combinatorial

problem of data association.

slide-35
SLIDE 35
  • S. Hamid Rezatofighi

RNN + LSTM

35

slide-36
SLIDE 36
  • S. Hamid Rezatofighi

RNN: Prediction, Update & Birth/Death

36

slide-37
SLIDE 37
  • S. Hamid Rezatofighi

RNN: Prediction, Update & Birth/Death

37

slide-38
SLIDE 38
  • S. Hamid Rezatofighi

RNN: Prediction, Update & Birth/Death

38

slide-39
SLIDE 39
  • S. Hamid Rezatofighi

RNN: Prediction, Update & Birth/Death

39

slide-40
SLIDE 40
  • S. Hamid Rezatofighi

time time time G T Resul t Existenc e x x p(E)

RNN: Prediction, Update & Birth/Death

40

slide-41
SLIDE 41
  • S. Hamid Rezatofighi

LSTM for Data Association

41

slide-42
SLIDE 42
  • S. Hamid Rezatofighi

Data association loss: Data association Inputs:

LSTM for Data Association

42

slide-43
SLIDE 43
  • S. Hamid Rezatofighi

 Learn Hungarian Algorithm  Learn JPDA

⋮

Pairwise distance

LSTM for Data Association

43

slide-44
SLIDE 44
  • S. Hamid Rezatofighi

Method MOTA Recall Precision ID Sw. Kalman+HA (O) 19.2 28.5 79.0 685 Kalman+HA+Post 22.4 28.3 83.4 105 JPDA𝑛 23.5 30.6 81.7 109 RNN+HA (O) 24.0 37.8 75.2 518 RNN+LSTM (O) 22.3 37.1 73.5 572

MOTChallenge 2015 Training Set

(O) = Online

Results

44

slide-45
SLIDE 45
  • S. Hamid Rezatofighi

Method MOTA FN FP ID Sw. FPS MDP [1] 30.3 32,422 9,717 680 1.1 JPDA𝑛 [2] 23.8 40,084 6,373 365 32.6 TC_ODAL [3] 15.1 38,538 12,970 637 1.7 RNN+LSTM 19.0 38,706 11,578 1,490 165.2

MOTChallenge 2015 Test Set

[1] Xiang et al., ICCV 2015 [2] Rezatofighi et al., ICCV 2015 [3] Bae et al., CVPR 2014

Results

45

slide-46
SLIDE 46
  • S. Hamid Rezatofighi

Results

46

slide-47
SLIDE 47
  • S. Hamid Rezatofighi

Summary

  • The first approach that employs end-to-end learning for

Online multi-target tracking

  • A generic, model free and data-driven approach

Future work

  • Training with a real data
  • Learning more complex measurement models and data association

techniques

  • Testing on other multi-target tracking applications

47

Conclusion

slide-48
SLIDE 48
  • S. Hamid Rezatofighi

Thank you

https://bitbucket.org/amilan/rnntracking 48