Online Multi-Target Tracking Using Recurrent Neural Networks Anton Milan, S. Hamid Rezatofighi , Anthony Dick, Ian Reid, Konrad Schindler
Outline • Some applications of multi-target tracking • The challenges • Existing approaches • Our motivation for using RNN • Our idea and contribution • Preliminary results • Conclusion 2 S. Hamid Rezatofighi
Multi-target tracking: Applications • Multiple similar targets in very noisy sequences of sonar or radar 3 S. Hamid Rezatofighi
Multi-target tracking: Applications • Tracking several pedestrians in a very crowded scene in surveillance camera 4 S. Hamid Rezatofighi
Multi-target tracking: Applications • Tracking populated and dense cells in biological sequences 5 S. Hamid Rezatofighi
Multi-target tracking: Applications • 3D particle tracking velocimetry for flow measurement • etc 6 S. Hamid Rezatofighi
Challenges in Multi-Target Tracking • Unknown and time-varying numbers of targets 7 S. Hamid Rezatofighi
Challenges in Multi-Target Tracking • Unknown and time-varying numbers of targets • The complex behaviors of targets • Maneuvering dynamics of the targets • Targets occlusion and entering or exiting from scene • Interactions with other targets such as targets splitting 8 S. Hamid Rezatofighi
Challenges in Multi-Target Tracking • Noisy observations Track-before-detect approach, Applying a detection method, Detection is not applicable But imperfect detection due to noisy sequences Noisy features Misdetections, noisy and spurious measurements (clutter) 9 S. Hamid Rezatofighi
Multi-Target Tracking with Detection An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of • Noisy observations ⎻ Clutter noise, Detection uncertainty • Data association uncertainty, observation space observation produced by objects state space state dynamic 10 5 objects 3 objects S. Hamid Rezatofighi
Multi-Target Tracking with Detection An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of • Noisy observations ⎻ Clutter noise, Detection uncertainty • Data association uncertainty, observation space observation produced by objects state space state dynamic Survive or Die? 11 5 objects 3 objects S. Hamid Rezatofighi
Multi-Target Tracking with Detection An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of • Noisy observations ⎻ Clutter noise, Detection uncertainty • Data association uncertainty, observation space observation produced by objects state space state dynamic New born or 12 existing target? 5 objects 3 objects S. Hamid Rezatofighi
Multi-Target Tracking with Detection An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of • Noisy observations ⎻ Clutter noise, Detection uncertainty • Data association uncertainty, observation space observation produced by objects ? ? ? state space state dynamic 13 5 objects 3 objects S. Hamid Rezatofighi
Multi-Target Tracking with Detection An automated tracking system should be able to track an unknown and time-varying number of targets in the presence of • Noisy observations ⎻ Clutter noise, Detection uncertainty • Data association uncertainty, observation space observation produced by objects Missed detected? state space state dynamic 14 5 objects 3 objects S. Hamid Rezatofighi
Existing Approaches Bayesian Filtering + Data Associations (Online) 15 S. Hamid Rezatofighi
Existing Approaches Bayesian Filtering + Data Associations (Online) Bayesian Filtering • Kalman filter [1] and its variation • Particle filter [2] and its variations [1] Kalman. Journal of Basic Engineering, 1960 [2] Liu and Chen , Journal of the American Statistical Association, 1998 16 S. Hamid Rezatofighi
Existing Approaches Bayesian Filtering + Data Associations (Online) Bayesian Filtering • Kalman filter [1] and its variation • Particle filter [2] and its variations Data association: • Multi Assignment Problem (MAP) • Multiple Hypothesis Tracking (MHT) [3] • Joint Probabilistic Data Association (JPDA,IJPDA) [4] [1] Kalman, Journal of Basic Engineering, 1960 [2] Liu and Chen , Journal of the American Statistical Association, 1998 17 [3] Reid, TAC 1979 [4] Fortman et al , CDC 1980 S. Hamid Rezatofighi
Existing Approaches Bayesian Filtering + Data Associations (Online) Bayesian Filtering • Kalman filter [1] and its variation • Particle filter [2] and its variations Data association: • Multi Assignment Problem (MAP) • Multiple Hypothesis Tracking (MHT) [3] • Joint Probabilistic Data Association (JPDA,IJPDA) [4] Parametric approaches: Prior knowledge is required [1] Kalman, Journal of Basic Engineering, 1960 [2] Liu and Chen , Journal of the American Statistical Association, 1998 18 [3] Reid, TAC 1979 [4] Fortman et al , CDC 1980 S. Hamid Rezatofighi
Existing Approaches Optimization based approaches (Often Offline) with 19 S. Hamid Rezatofighi
Existing Approaches Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches • Shortest-Path algorithms [1] • Min-cost Max Flow algorithms [2] [1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008 20 S. Hamid Rezatofighi
Existing Approaches Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches • Shortest-Path algorithms [1] • Min-cost Max Flow algorithms [2] Non-Linear and Complex Objectives • Alpha-Expansion approaches [3] • Discrete-Continuous optimization [4] [1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008 21 [3] Leibe et al, ICCV 2007 [4] Milan et al, TPAMI 2014 S. Hamid Rezatofighi
Existing Approaches Optimization based approaches (Often Offline) with Linear Objectives – (Near) Global Optimal Approaches • Shortest-Path algorithms [1] • Min-cost Max Flow algorithms [2] Non-Linear and Complex Objectives • Alpha-Expansion approaches [3] • Discrete-Continuous optimization [4] 1. Hand-crafted objectives are required to be defined. 2. The approaches are often offline (batch processing). [1] Berclaz et al, TPAMI 2011 [2] Zhang and Nevatia, CVPR 2008 22 [3] Leibe et al, ICCV 2007 [4] Milan et al, TPAMI 2014 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our Motivations: 23 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our Motivations: • A generic and model free approach 24 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our Motivations: • A generic and model free approach • A reliable data driven approach using deep structured networks 25 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our Motivations: • A generic and model free approach • A reliable data driven approach using deep structured networks • An online multi-target tracking approach 26 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our Motivations: • A generic and model free approach • A reliable data driven approach using deep structured networks • An online multi-target tracking approach RNN/LSTM 27 S. Hamid Rezatofighi
No Trivial Solution For The Problem 28 S. Hamid Rezatofighi
No Trivial Solution For The Problem Why? Unknown + Time-varying number of targets • Input dimension unknown (for tracking-by-detection) • Output dimension unknown “Class” has no semantic meaning • Arbitrary assignments • Multiple equally correct solutions 1 2 2 1 29 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our idea/contribution toward end-to-end model learning: 30 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our idea/contribution toward end-to-end model learning: Prediction: An RNN to learn a complex dynamic model for predicting • target motion in the absence of measurements. 31 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our idea/contribution toward end-to-end model learning: Prediction: An RNN to learn a complex dynamic model for predicting • target motion in the absence of measurements. Update: An RNN to learn noisy measurement model in presence of • false detections (clutter). 32 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our idea/contribution toward end-to-end model learning: Prediction: An RNN to learn a complex dynamic model for predicting • target motion in the absence of measurements. Update: An RNN to learn noisy measurement model in presence of • false detections (clutter). Birth/death: to identify track initiation and termination. • 33 S. Hamid Rezatofighi
RNN for Multi-Target Tracking Our idea/contribution toward end-to-end model learning: Prediction: An RNN to learn a complex dynamic model for predicting • target motion in the absence of measurements. Update: An RNN to learn noisy measurement model in presence of • false detections (clutter). Birth/death: to identify track initiation and termination. • Data Association: An LSTM to learn the challenging combinatorial • problem of data association. 34 S. Hamid Rezatofighi
RNN + LSTM 35 S. Hamid Rezatofighi
RNN: Prediction, Update & Birth/Death 36 S. Hamid Rezatofighi
Recommend
More recommend
