Haishan Wu School of Computer Science, Fudan University Shanghai, - - PowerPoint PPT Presentation

Haishan Wu School of Computer Science, Fudan University Shanghai, China

1

When simulation meets empirical data

2

Computer vision techniques: pattern recognition, feature extraction, image processing, stereo reconstruction, machine learning Different sensors

Measurement data

Related works on motion measurement methods

ž On small groups

— Human, fish, ants, bees, cells, bacteria

ž On large groups

— Bacterial swarms, Drosophila groups, bird

flock, human crowd, bat flock

3

There is no robust implementation publicly available for tracking large groups of targets yet.

General Procedure

4

Detecting each individual Tracking each individual in 2D or 3D throughout the whole data sequences Target Detection is a task-specific and challenging problem. Maintaining the identity individually is very difficult. (false positive detections, occlusions etc.). Computation speed should also be considered. Computer vision techniques that may be used: image segmentation,

• bject detection, image denoising

feature extraction Computer vision techniques that may be used: multi-view geometry, multi-target tracking, combinatorial

• ptimization, feature extraction and

matching

Very useful references

ž Four excellent books

• 1. Multiple view geometry in computer vision
• 2. Computer Vision: Algorithms and

Applications （the latest draft can be downloaded in http://szeliski.org/Book/)

• 3. Pattern Recognition and Machine

Learning

• 4. Multiple-target tracking with radar

applications

5

Multi-view geometry and stereo reconstruction

6

Epipolar geometry Image rectification

Framework for 3D tracking :1

7

Image sequences of multi-camera setup,

• ften acquired by two or more CCD-cameras in

a convergent arrangement Tracking of particles in 2D image space with different approaches (filter + data association) Stereoscopic correspondences problem, matching of the 2D tracks to generate particle trajectories in object space, requires system calibration and orientation

Framework for 3D tracking: 2

8

Image sequences of multi-camera (more than two) setup Establishment of particle correspondences with consequent use of epipolar constraints and determination of 3D particle positions, treating each single time step separately, requires system calibration. Tracking is performed in 3D object space, one 3D point cloud as input for each time step

Framework for 3D tracking :3

9

Establishment of particle correspondences with epipolar constraints and determination of 3D particle positions, in addition storage of all relevant data from particle detection process

My work

ž Drosophila group tracking in 3D ž Fish school tracking in 2D (I bring my poster here )

10

Tracking fruit fly group in 3D

Why it is a challenging problem:

11

Frequent

• cclusions make

many tracking methods failed. Appearance feature will be also ineffectual for tracking or matching across various views Tracking or matching failures will lead to the broken or incomplete trajectories in 3D space

Tracking fruit fly group in 3D (cont.)

ž By using two high-speed cameras, we

solved the abovementioned problems by using one compact framework: linear assignment problem (LAP)

12

Tracking fruit fly group in 3D (cont.) The first LAP

ž 2 steps: single particle state updating and

data association

ž State updating: alpha-beta or Kalman filter

13

Project 1: Tracking fruit fly group in 3D (cont.) The first LAP

ž The first LAP is:

14

To handle these difficulties:

Tracking fruit fly group in 3D (cont.) The second LAP

ž Now our problem is how to assign (match) a 2D trajectory in

• ne view to that of another view

ž The appearance feature of targets, however, is useless in our

experiments, so how to define the matching cost?

ž Observations:

l Two true matching trajectories will submit to epipolar constrains l The longer the length of matched trajectories, the larger the

possibility of correct matching is.

15

Project 1: Tracking fruit fly group in 3D (cont.) The second LAP

ž Our matching cost: MECL(maximum

epipolar co-motion length)

16

Tracking fruit fly group in 3D (cont.) The third LAP

ž 3D trajectories can be

recovered by using multi- view geometry technique.

ž However, tracking errors

make these trajectories broken into segments

ž We solve this problem by

formulating it as pairwise tracklet matching problem

17

Tracking fruit fly group in 3D (cont.) The third LAP

ž We incorporate temporal and kinematic

information into linking cost.

l temporal information l kinematic information

18

Tracking fruit fly group in 3D (cont.)

19

Large scale fish school tracking (I bring the poster here )

20

Acknowledgements

ž Thank Prof. Tamas Vicsek for offering me the

• pportunity to give a talk here. Many thanks also

go to Zsuzsa Ákos.

ž Thank Qiyuan Tian, Qi Xu, Yihao Zhou, Miaohui

Wang, Ye Liu, Jinlong Shi, Iain Couzin for their valuable comments in Drosophila group tracking.

ž Thank Linguo Li and Wei Li at School of Life

Sciences of Fudan University for providing fruit flies for the experiments.

ž Thank Simon Leblanc in Princeton University for

providing the fish school videos

21