Performance Measures and the DukeMTMC Benchmark for Multi-Target - - PowerPoint PPT Presentation

Performance Measures and the DukeMTMC Benchmark for Multi-Target Multi-Camera Tracking

Ergys Ristani

In collaboration with:

Francesco Solera Roger Zou Rita Cucchiara Carlo Tomasi

Outline

• Problem Definition
• Performance Measures
• DukeMTMCT Benchmark
• Summary

Problem Definition

• Given n camera streams, determine who/what is where at all times

Multi-Target Multi-Camera Tracking

DukeMTMCT Benchmark

Problem Definition

• Given n camera streams, determine who/what is where at all times

Multi-Target Multi-Camera Tracking

DukeMTMCT Benchmark

Problem Definition

• Given n camera streams, determine who/what is where at all times

Multi-Target Multi-Camera Tracking

• How well does a tracker determine who/what is where at all times?

MTMCT Evaluation

DukeMTMCT Benchmark

Evaluation Paradigms

Suspect

Evaluation Paradigms

Suspect ID1 Tracker (a) Computed IDs

Evaluation Paradigms

Suspect ID1 ID2 Tracker (a) Computed IDs

Evaluation Paradigms

Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch

Evaluation Paradigms

Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification

Evaluation Paradigms

Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification Tracker (b)

Evaluation Paradigms

Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification Tracker (b)

Evaluation Paradigms

Suspect Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification 7 switches 83% identification Tracker (b) Tracker (c)

Evaluation Paradigms

Suspect Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification 7 switches 83% identification Tracker (b) Tracker (c)

• Researcher: How often does the tracker switch identities?

Evaluation Paradigms

Suspect Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification 7 switches 83% identification Tracker (b) Tracker (c)

Evaluation Paradigms

Suspect Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification 7 switches 83% identification Tracker (b) Tracker (c)

• End-user: How well does the tracker explain identity?

Evaluation Paradigms

Suspect Suspect Suspect ID1 ID2 Tracker (a) Computed IDs 1 switch 67% identification 7 switches 67% identification 7 switches 83% identification Tracker (b) Tracker (c)

Outline

• Problem Definition
• Performance Measures
• Existing Measures
• Issues
• Identity Measures
• DukeMTMCT Benchmark
• Summary

Performance Measures

MTMC Tracker Performance

Performance Measures

Object detection Appearance model Motion model Interaction model

Optimization

Parameter tuning Training data

Performance Measures

MTMC Tracker Performance

Performance Measures

ML MT MOTA IDS FRG

[1] Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. Image Video Proc. 2008 [2] Learning to Associate: HybridBoosted Multi-Target Tracker for Crowded Scene. CVPR 2009

MTMC Tracker Performance

Performance Measures

ML MT MOTA IDS MCTA

Handover (X, FRG)

FRG

[3] An Equalized Global Graph Model-Based Approach for Multi-Camera Object Tracking. IEEE TCAS 2016 [4] Inter-camera Association of Multi-target Tracks by On-Line Learned Appearance Affinity Models. ECCV 2010

MTMC Tracker Performance

Performance Measures

MCTA

Handover (X, FRG)

MTMC Tracker Performance

ML MT MOTA IDS FRG

Performance Measures

MCTA

Handover (X, FRG)

[5] Framework for Performance Evaluation of Face, Text, and Vehicle Detection and Tracking in Video: Data, Metrics, and Protocol. TPAMI 2008 [6] A New Benchmark and Protocol for Multi-Object Detection and Tracking. arXiv CoRR 2015 [7] Evaluating Multi-Object Tracking. CVPRWS 2005

MTMC Tracker Performance

ATA PR-MOTA STDA-D

TP

FIT FIO OP ML MT MOTA IDS FRG

Errors in Tracking

• Mismatches
• Fragmentations
• Merges
• False Positives/Negatives

T1 C1 C2 C1 T1 T2 T1 C2 True Computed

Key Evaluation Steps

T1 C1 C2 T2 C3 True Computed

Key Evaluation Steps

• Truth-to-Result Matching

True Computed T1 C1 C2 T2 C3

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

CLEAR MOT Mapping

True Computed T1 C1 C2 T2 C3 time

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

True Computed

1

T1 C1 C2 T2 C3 time

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

3

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

5

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

7

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

2 9

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

2 1 11

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

T1 C1 C2 T2 C3

2 2 13

time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

2 1 2 14

T1 C1 C2 T2 C3 time True Computed

CLEAR MOT Mapping

• At each frame t
• Preserve mapping of detections from frame t-1 if still valid
• Solve bipartite matching at frame t for unmapped detections

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Key Evaluation Steps

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Common Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors
• Trajectory Scores

MT, PT, ML

Outline

• Problem Definition
• Performance Measures
• Existing Measures
• Issues
• Identity Measures
• DukeMTMCT Benchmark
• Summary

Issues

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Issues

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

CLEAR MOT Mapping

• Identity mapping bijective at each frame
• Not bijective overall

T1 T2 C1 C2 T1 C1 C2 T2 C3 time True Computed

Issues

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors
• Trajectory Scores

MT, PT, ML

Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors

Brittle due to mapping

• Trajectory Scores

MT, PT, ML

A A A A A A A A A A A A A A A A Camera 1 Camera 2 Truth

Handover Errors

A A A A A A A A A A A A A A A A Camera 1 Camera 2 Truth b b b b b b b b b b b b Tracker 1 c b b b time

Handover Errors

A A A A A A A A A A A A A A A A Camera 1 Camera 2 Truth b b b b b b b b b b b b Tracker 1 c b b b time

Handover Errors

A A A A A A A A A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b Truth Tracker 1 c b b b b b b b b b b b b b b b Tracker 2 b b b time

Handover Errors

A A A A A A A A A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b Truth Tracker 1 c b b b b b b b b b b b b b b b Tracker 2 b b b time

Handover Errors

Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors
• Trajectory Scores

MT, PT, ML Ignore identity

Trajectory Scores

A A A A A A A A time Truth A A

A A A A A A A A time Truth A A

Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores

A A A A A A A A time Truth Tracker 1 b c c e e g g b A A f f

Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores

A A A A A A A A time Truth Tracker 1 b c c e e g g b MT = 1 A A f f

Mostly Tracked: GT trajectories which are covered by tracker output for more than 80% in length

Trajectory Scores

A A A A A A A A time b c c b g g Truth Tracker g b MT = 1 A A b b FRG = 4

Mostly Tracked: Percentage of GT trajectories which are covered by tracker output for more than 80% in length Fragments: The number of times that a GT trajectory is interrupted in tracking result

Trajectory Scores

A A A A A A A A time Truth Tracker b c c e e g g b A A f f FRG = 4 MT = 1

Trajectory Scores

A A A A A A A A time b c b b g b Truth Tracker g b b c c e e g g b Tracker++ MT = 1 A A b b f f FRG = 4

Trajectory Scores

A A A A A A A A time b c b b g b Truth Tracker g b Tracker++ MT = 1 A A b b FRG = 4 b c c e e g g b f f MT = 1 FRG = 4

Trajectory Scores

Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors

Bizarre combination

• Trajectory Scores

MT, PT, ML

MCTA

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b Truth Tracker 1 c b b b b b b b b b b b b b b b c b b b Tracker 2

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b Truth Tracker 1 c b b b b b b b b b b b b b b b c b b b Tracker 2 Tracker 1 Tracker 2

Fragmentations

Handover | Within camera

MCTA

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b Truth Tracker 1 c b b b b b b b b b b b b b b b c b b b Tracker 2 Tracker 1 Tracker 2

Fragmentations

Handover | Within camera

MCTA

Scoring Functions

• Multiple Object Tracking Accuracy (MOTA)
• Multiple Camera Tracking Accuracy (MCTA)
• Handover Errors

Mapping not one-to-one

• Trajectory Scores

MT, PT, ML

MOTA

A A A A A A A A time b b b True Computed c b c c c A A A A A A A A time b b b True Computed b c b b b

A A A A A A A A time b b b True Computed c b c c c A A A A A A A A time b b b True Computed b c b b b

Half explanation Nearly full explanation

MOTA

Outline

• Problem Definition
• Performance Measures
• Existing Measures
• Issues
• Identity Measures
• DukeMTMCT Benchmark
• Summary

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Identity Measures

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A

Cost on each edge is number of mis-assigned frames between two trajectories

4 14 16 12 2

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b b b b b b b b A

Cost on each edge is number of mis-assigned frames between two trajectories

4

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A

Cost on each edge is number of mis-assigned frames between two trajectories

4 14 16 12 2

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 c c c A

Cost on each edge is number of mis-assigned frames between two trajectories

14

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A

Cost on each edge is number of mis-assigned frames between two trajectories

4 14 16 12 2

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 A

Cost on each edge is number of mis-assigned frames between two trajectories

16

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A

Cost on each edge is number of mis-assigned frames between two trajectories

4 14 16 12 2

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

time Camera 1 Camera 2 b b b b b b b b b b b b b

Cost on each edge is number of mis-assigned frames between two trajectories

12

True Computed

• Proposed Truth-to-Result Matching

Identity Measures

A A A A A A A A time A A A A A A A A Camera 1 Camera 2 b b b b b b c b b b b b b c b c A

Cost on each edge is number of mis-assigned frames between two trajectories

4 14 16 12 2

True Computed

• Proposed Truth-to-Result Matching

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Identity Measures

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Identity Measures

• Truth-to-Result Matching
• Scoring Function

True Computed T1 C1 C2 T2 C3

MCTA Handover (X, FRG)

MTMC Tracker Performance

ATA PR-MOTA STDA-D TP FIT FIO OP

ML MT MOTA IDS FRG

Identity Measures

• ID Precision
• ID Recall
• F1-score

Identity Measures

True Detections Computed Detections

True Positives False Positives False Negatives

• Proposed Scores

• ID Precision
• ID Recall
• F1-score

Identity Measures

True Detections Computed Detections

True Positives False Positives False Negatives

• Proposed Scores
• IDP: Fraction of computed detections that are correctly identified.

• ID Precision
• ID Recall
• F1-score

Identity Measures

True Detections Computed Detections

True Positives False Positives False Negatives

• Proposed Scores
• IDP: Fraction of computed detections that are correctly identified.
• IDR: Fraction of ground-truth detections that are correctly identified.

• ID Precision
• ID Recall
• F1-score

Identity Measures

True Detections Computed Detections

True Positives False Positives False Negatives

• Proposed Scores
• IDP: Fraction of computed detections that are correctly identified.
• IDR: Fraction of ground-truth detections that are correctly identified.
• IDF1: Ratio of correctly identified detections over the average

number of ground-truth and computed detections

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

b c A

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

b c A b A c

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

b c A

4 14 16 12 2

Min Cost Matching

b A c

4 2

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

T1 C1 C2 T2 C3

Identity Measures

• Properties
• True and computed identities are mapped 1-1
• Notion of identity switch is not present in ID measures
• The truth-to-result matching is the most favorable to the algorithm
• Applicable to single- and multi-camera tracking

T1 C1 C2 T2 C3

Scope and Limitations

• ID measures are useful in:
• Surveillance
• Sport
• MTT for autonomous driving
• Not necessary when:
• Tracking indistinguishable targets (ants, sheep)
• Not applicable to:
• Tracking targets that merge/split (cells)

Practical Implications

MOT 16 Challenge

Tracker MOTA ↑

HCC 0.492 LMP 0.487 FWT 0.477 NLLMPa 0.475 MDPNN16 0.471 NOMT_16 0.464 JMC 0.462 QuadMOT16 0.441

• ICF_16

0.432 MHT_DAM_16 0.429 LINF1_16 0.410 EAMTT_pub 0.388 OVBT 0.384 LTTSC-CRF 0.375 LP2D_16 0.357 TBD_16 0.337 CEM_16 0.331 DP_NMS_16 0.321 GMPHD_HDA 0.305 SMOT_16 0.297 JPDA_m_16 0.261

Practical Implications

MOT 16 Challenge

Tracker MOTA ↑

HCC 0.492 LMP 0.487 FWT 0.477 NLLMPa 0.475 MDPNN16 0.471 NOMT_16 0.464 JMC 0.462 QuadMOT16 0.441

• ICF_16

0.432 MHT_DAM_16 0.429 LINF1_16 0.410 EAMTT_pub 0.388 OVBT 0.384 LTTSC-CRF 0.375 LP2D_16 0.357 TBD_16 0.337 CEM_16 0.331 DP_NMS_16 0.321 GMPHD_HDA 0.305 SMOT_16 0.297 JPDA_m_16 0.261

Tracker IDF1 ↑

NOMT_16 0.533 LMP 0.512 HCC 0.506

• ICF_16

0.493 NLLMPa 0.473 JMC 0.463 MDPNN16 0.462 MHT_DAM_16 0.457 LINF1_16 0.456 FWT 0.442 EAMTT_pub 0.424 LTTSC-CRF 0.420 QuadMOT16 0.382 OVBT 0.378 CEM_16 0.357 LP2D_16 0.341 GMPHD_HDA 0.333 JPDA_m_16 0.311 DP_NMS_16 0.287 TBD_16 0.251 SMOT_16 0.185

Practical Implications

Tracker MOTA ↑

HCC 0.492 LMP 0.487 FWT 0.477 NLLMPa 0.475 MDPNN16 0.471 NOMT_16 0.464 JMC 0.462 QuadMOT16 0.441

• ICF_16

0.432 MHT_DAM_16 0.429 LINF1_16 0.410 EAMTT_pub 0.388 OVBT 0.384 LTTSC-CRF 0.375 LP2D_16 0.357 TBD_16 0.337 CEM_16 0.331 DP_NMS_16 0.321 GMPHD_HDA 0.305 SMOT_16 0.297 JPDA_m_16 0.261

MOT 16 Challenge

Tracker IDF1 ↑

NOMT_16 0.533 LMP 0.512 HCC 0.506

• ICF_16

0.493 NLLMPa 0.473 JMC 0.463 MDPNN16 0.462 MHT_DAM_16 0.457 LINF1_16 0.456 FWT 0.442 EAMTT_pub 0.424 LTTSC-CRF 0.420 QuadMOT16 0.382 OVBT 0.378 CEM_16 0.357 LP2D_16 0.341 GMPHD_HDA 0.333 JPDA_m_16 0.311 DP_NMS_16 0.287 TBD_16 0.251 SMOT_16 0.185

Rank Improvement

+5

• 2

+5

• 1

+1

• 2

+2 +2

• 7

+1 +2

• 5
• 1

+2

• 1

+2 +3

• 1
• 4
• 1

Private detections Public detections

IDP/IDR Curves

NOMT LMP HCC

• ICF

FWT CEM JPDA NLLMPa JMC POI NOMTwSDP DeepSORT KDNT

Performance Measures

[8] Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking. ECCV 2016 BMTT Workshop

ML MT MOTA IDS IDF1 IDP IDR MCTA

Handover (X, FRG)

FRG

MTMC Tracker Performance

ATA PR-MOTA STDA-D

TP

FIT FIO OP

Outline

• Problem Definition
• Performance Measures
• Existing Measures
• Issues
• Identity Measures
• DukeMTMCT Benchmark
• Summary

Previous MTMC Data Sets

EPFL APIDIS

NLPR 1, 2 NLPR 3

ISSIA

PETS2009

DukeMTMC

• 8 static cameras x 85 minutes of 1080p 60 fps video
• More than 2,000,000 manually annotated frames
• More than 2,000 identities
• Annotation by 5 people over 1 year
• More identities than all existing datasets combined
• Unconstrained paths, diverse appearance

DukeMTMC@MOTChallenge

DukeMTMC@MOTChallenge

• Evaluation
• Single Camera
• Multi-Camera
• Ranking
• IDF1 score

DukeMTMC@MOTChallenge

Trainval 0-50 min Test-hard 50-60 min Test-easy 60-85 min

Baseline

Baseline: BIPCC

Camera i Person detection Multi-Camera Identities

• 0.8
• ∞

+1 +1 +1 +0.4 +1 +1 +1 +1 +1

• 0.7

MTMC Tracking as Correlation Clustering

Tracking Multiple People Online and in Real Time, E. Ristani and C. Tomasi, ACCV 2014

Baseline

Baseline

Incorrect Matches

Baseline

Correct Matches

IDP IDR

Fragmentations

Merges

DukeMTMCT

IDP IDR

Fragmentations

Merges

DukeMTMCT

• ID switches in CLEAR MOT don’t correlate with ID measures

Outline

• Problem Definition
• Performance Measures
• DukeMTMCT Benchmark
• Summary

Summary

ML MT MOTA IDS IDF1 IDP IDR MCTA

Handover (X, FRG)

FRG

MTMC Tracker Performance

• MTMC trackers are complex - multiple uncorrelated measures are required to

understand tracker performance

ATA PR-MOTA STDA-D

TP

FIT FIO OP

Object detection Appearance model Motion model Interaction model

Optimization

Parameter tuning Training data

Summary

• Large-scale realistic benchmarks make comparisons meaningful

Suspect Suspect Suspect Tracker (a) 1 switch 67% identification 7 switches 67% identification 83% identification Tracker (b) Tracker (c)

Summary

• Different applications require different measures
• ID measures show bottom line performance: They are useful for end users of MTMC

tracking systems

[1] Performance Measures and Dataset for Multi-Target, Multi-Camera Tracking Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, Carlo Tomasi ECCV 2016 Workshop on Benchmarking Multi-Target Tracking [3] Tracking Multiple People Online and in Real Time Ergys Ristani and Carlo Tomasi ACCV 2014 [2] Tracking Social Groups Within and Across Cameras Francesco Solera, Simone Calderara, Ergys Ristani, Carlo Tomasi, Rita Cucchiara IEEE Transactions on Circuits and Systems 2016

Code/Data: vision.cs.duke.edu/DukeMTMC/

Thank you!