VALSE VA ON ONLINE Tr Tracking Mu Multiple Ob Objects in in Im Image Se Sequences Xinchao Wang June, 2018
2
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 3
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 4
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities 5
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: 6
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: • A concrete body, like a person or a car 7
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: • A concrete body, like a person or a car • An event , like a person gets into a car 8
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: • A concrete body, like a person or a car • An event , like a person gets into a car • State of object: 9
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: • A concrete body, like a person or a car • An event , like a person gets into a car • State of object: TPAMI’16 • State of a concrete body, like the location of a person 10
MOT Tracking by Definition Challenges Detection Definition of Multiple Objects Tracking (MOT) • The operation that determines the states of objects over time while preserving their identities • Object: • A concrete body, like a person or a car • An event , like a person gets into a car • State of object: • State of a concrete body, like the location of a person TPAMI’16 • State of an event , like the number of people inside a car
MOT Tracking by Definition Challenges Detection Examples of MOT MVA’16 • People tracking • Event tracking TMI’17 t=9 t=10 t=11 12
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 13
MOT Tracking by Definition Challenges Detection Challenges of MOT TPAMI’16 CVIU’14 • Object Interactions • Track the targets • Track the interaction events People and Car Player and Basketball CVPR’16 ECCV’14 Player and Soccer People and Backpack 14
MOT Tracking by Definition Challenges Detection Challenges of MOT • Occlusions • Moving targets may occlude one another TPAMI’16 TIP’17 TIP’18 TMI’17 Players occlude ball Cells occlude one another People occlude one another 15
MOT Tracking by Definition Challenges Detection Challenges of MOT • Motions • Targets may follow specific motion pattern CVPR’16 ICCV’17 TIP’18 Ball follows parabolic motion Pedestrian follow high-order walking pattern 16
Challenges of MOT • Online • Tracking multiple targets in an online or semi-online manner TIP’17 Batchn Batchn+1 17
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 18
MOT Tracking by Definition Challenges Detection Solving MOT: the Tracking by Detection Paradigm • Workflow Input Images Detection Tracking (A batchor the whole video) (on 2D images or the 3D world) (on 2D images or the 3D world)
MOT Tracking by Definition Challenges Detection Detection • Probability Occupancy Map (POM) I t 20
MOT Tracking by Definition Challenges Detection Detection • Probability Occupancy Map (POM) I t 21
MOT Tracking by Definition Challenges Detection Detection • Probability Occupancy Map (POM) X t k 0 ∈ { 0 , 1 } ρ t k 0 = P ( X t k 0 = 1 | I t ) 22
MOT Tracking by Definition Challenges Detection Tracking • Network flow programming t − 1 t t + 1 … … … t − 1 t t + 1 … … … … … … … … … … … … 23
MOT Tracking by Definition Challenges Detection Tracking • Network flow programming t − 1 t t + 1 … … … t − 1 t t + 1 … … … … … … … … … … … … x t x t +1 f t kj ∈ { 0 , 1 } k j 24
MOT Tracking by Definition Challenges Detection Tracking • Network flow programming t − 1 t t + 1 … … … t − 1 t t + 1 … … … … … … … … … … … … ρ t ✓ ◆ f ∗ = argmax X f t k log j 1 − ρ t f ∈ F k t,k,j 25
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 26
MOT Challenges Interactions Occlusion Motions online Handling Object Interactionsin MOT • Interactions: • Two or more objects involved • Our focus: • One class of object (i.e., container) “contain” the other (i.e., containee) People-Bags Cars-People Players-Basketball Players-Soccer 27
MOT Challenges Interactions Occlusion Motions online Formulation: Objective ρ t β t ✓ ◆ ✓ ◆ ( x , y ) ∗ = argmax X x t y t k k log k +log k 1 − ρ t 1 − β t ( x , y ) ∈ F k k t,k Container Containee t − 1 t t + 1 … … … t − 1 t t + 1 … … … … … … … x t x t +1 f t kj ∈ { 0 , 1 } k j g t y t +1 kj ∈ { 0 , 1 } y t j k and are intertwined on each edge f t g t kj kj
MOT Challenges Interactions Occlusion Motions online Formulation: Objective ρ t β t ✓ ◆ ✓ ◆ ( x , y ) ∗ = argmax X x t y t k k log k +log k 1 − ρ t 1 − β t ( x , y ) ∈ F k k t,k Container Containee t − 1 t t + 1 … … … t − 1 t t + 1 … … … … f t kj g t kj … … … ρ t β t ✓ ◆ ✓ ◆ X X X f t g t k k argmax log kj +log kj 1 − ρ t 1 − β t ( f , g ) ∈ F k k t j ∈ N ( k ) k 29
MOT Challenges Interactions Occlusion Motions online Formulation: Objective ρ t β t ✓ ◆ ✓ ◆ ( x , y ) ∗ = argmax X x t y t k k log k +log k 1 − ρ t 1 − β t ( x , y ) ∈ F k k t,k Container Containee t − 1 t t + 1 … … … t − 1 t t + 1 … … … … f t kj g t kj … … … ρ t β t ✓ ◆ ✓ ◆ X X X f t g t k k argmax log kj +log kj 1 − ρ t 1 − β t ( f , g ) ∈ F k k t j ∈ N ( k ) k 30
MOT Challenges Interactions Occlusion Motions online Formulation: Constraints • Flow Conservation on Container Objects • Spatial Exclusion on the ground • Consistency of Interacting Flows • Tracking the Invisible ✔ • Additional Bound Constraint 31
MOT Challenges Interactions Occlusion Motions online Formulation: Constraints • Tracking the Invisible – Counter Flow Variable h The person is in the car after he enters it. h t − 1 h t kj ik h t kj = h t − 1 + 1 ik Container Containee 32
MOT Challenges Interactions Occlusion Motions online Formulation: Constraints • Tracking the Invisible – Counter Flow Variable h The person was in the car before he exits. h t − 1 h t ik kj h t kj = h t − 1 − 1 ik Container Containee 33
MOT Challenges Interactions Occlusion Motions online Formulation: Constraints • Tracking the Invisible – Counter Flow Variable h The person is in the car after he enters it. The person was in the car before he exits. h t − 1 h t − 1 h t h t ik kj ik kj h t kj = h t − 1 h t kj = h t − 1 + 1 − 1 ik ik X X h t − 1 X f t − 1 X h t f t − kj , ∀ t, l kj = + ik ik k : l = l ( k ) k : l = l ( k ) , k : l = l ( k ) , k : l = l ( k ) j ∈ N ( k ) i : k ∈ N ( i ) i : k ∈ N ( i ) j ∈ N ( k ) 34
MOT Challenges Interactions Occlusion Motions online Simultaneous Approach: Results Linear First track big then small Ours Programming SSP KSP − fixed KSP − free KSP − sequential TIF − LP TIF − MIP 1 0.5 MOTA 0 − 0.5 0.2 0.3 0.4 0.5 0.6 0.7 Overlap Threshold PETS2006 35
MOT Overview Tracking by Definition Challenges Detection Handling Challenges Interactions Occlusions Motions Online 38
MOT Challenges Interactions Occlusion Motions online Handling Occlusions in MOT • Occlusions: • Incomplete image evidence • How to model occlusions in MOT? 39
MOT Challenges Interactions Occlusion Motions online Modeling Occlusions [1] K. Magnusson et al., A Batch Algorithm Using Iterative Application of the • Example: tracking cells Viterbi Algorithm to Track Cells and Construct Cell Lineages. ISBI’12. [2] M. Schiegg et al., Conservation Tracking. ICCV’13. Source Segmentation CT [2] KTH [1] Ours Ground Truth t=76 t=77 t=78 40
Recommend
More recommend
