Instance Learning Boris Babenko, Ming-Hsuan Yang, Serge Belongie - - PowerPoint PPT Presentation

Visual Tracking with Online Multiple Instance Learning

Kelsie Zhao

Boris Babenko, Ming-Hsuan Yang, Serge Belongie

Content

• Goal
• Background: Tracking by Detection
• Previous Work
• New Tracking Solution
• MILTrack
• Online MILBoost
• Experiments & Results

Goal

Track one arbitrary object in video, given its location in first frame

Background: Tracking by detection

• Frame 1 is labeled, tracker location known

Background: Tracking by detection

• Crop one positive and some negative patches

near tracker

Negative Positive

x2 x1 x3

Background: Tracking by detection

• Use patches to train the classifier

Negative Positive

Classifier {(x1, 1), (x2, 0), (x3, 0)}

x2 x1 x3

Background: Tracking by detection

• Frame 2 comes

Classifier

Background: Tracking by detection

• Calculate classifier response within a range of

the old tracker location

Classifier

X Old location

Background: Tracking by detection

• Find the maximum response location

Classifier

X Old location X New location

Background: Tracking by detection

• Move tracker

Frame 1 Frame 2

Background: Tracking by detection

• Repeat

Negative Positive Frame 2

Background: Tracking by detection

• Problem: If tracker location is not precise, might

select bad training examples

Background: Tracking by detection

• Problem: If tracker location is not precise, might

select bad training examples Model start to degrade!

Background: Tracking by detection

• Problem: If tracker location is not precise, might

select bad training examples

• How to select good training

examples? Model start to degrade!

Previous Work

• Solution 1: multiple positive examples around

tracker location x4 x2 x5 x1 x3

{(x1, 1), (x2, 1), (x3, 1), (x4, 0), (x5, 0)}

Classifier

Previous Work

• Solution 1

Might confuse classifier! x4 x2 x5 x1 x3

{(x1, 1), (x2, 1), (x3, 1), (x4, 0), (x5, 0)}

Classifier

Previous Work

• Solution 2: Multiple Instance Learning (MIL)

[Keeler ‘90, Dietterich et al. ‘97]

Previous Work: Multiple Instance Learning

• Multiple examples in one bag

[Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11 x21 x31

Previous Work: Multiple Instance Learning

X3 X2 X1

• Multiple examples in one bag

[Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11 x21 x31

Previous Work: Multiple Instance Learning

(X3 , 0) (X2 , 0) (X1 , 1)

• Multiple examples in one bag
• One bag one label

[Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11 x21 x31

Previous Work: Multiple Instance Learning

(X3 , 0) (X2 , 0) (X1 , 1)

• Multiple examples in one bag
• One bag one label
• Bag Positive if at least one example is Positive

[Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11 x21 x31

Previous Work: Multiple Instance Learning

{(X1, 1), (X2, 0), (X3, 0)} Classifier

(X3 , 0) (X2 , 0) (X1 , 1)

[Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11 x21 x31

Previous Work: Multiple Instance Learning

MIL training input: 𝒀𝟐, 𝑧1 … 𝒀𝒐, 𝑧𝑜 , where, 𝒀𝒋 = 𝑦𝑗1 … 𝑦𝑗𝑛 , 𝑧𝑗 = 𝑛𝑏𝑦𝑘 𝑧𝑗𝑘

(X3 , 0) (X2 , 0) (X1 , 1) x13 x12 x11 x21 x31 [Keeler ‘90, Dietterich et al. ‘97]

x13 x12 x11

Previous Work: Multiple Instance Learning

MIL training input: 𝒀𝟐, 𝑧1 … 𝒀𝒐, 𝑧𝑜 , where, 𝒀𝒋 = 𝑦𝑗1 … 𝑦𝑗𝑛 , 𝑧𝑗 = 𝑛𝑏𝑦𝑘 𝑧𝑗𝑘

(X3 , 0) (X2 , 0) (X1 , 1)

Bag babel is 1 if at least one instance is 1

[Keeler ‘90, Dietterich et al. ‘97] x21 x31

Now we have training examples! How to train the classifier?

Previous Work: MILBoost

• MIL + boosting

Train a boosting classifier that maximizes log likelihood of bags 𝑚𝑝𝑕𝑀 = log (𝑞 𝑧𝑗 𝒀𝒋))

𝑗

where, 𝑞 𝑧𝑗 𝒀𝒋) = 1 − (1 − 𝑞 𝑧𝑗 𝑦𝑗𝑘))

𝑘

[Viola et al. ‘05]

Previous Work: MILBoost

• MIL + boosting

Train a boosting classifier that maximizes log likelihood of bags 𝑚𝑝𝑕𝑀 = log (𝑞 𝑧𝑗 𝒀𝒋))

𝑗

where, 𝑞 𝑧𝑗 𝒀𝒋) = 1 − (1 − 𝑞 𝑧𝑗 𝑦𝑗𝑘))

𝑘

[Viola et al. ‘05]

1 ~1

Previous Work: MILBoost

• Problem: need all training examples

[Viola et al. ‘05]

Previous Work: MILBoost

But in tracking, only current frame available

[Viola et al. ‘05]

Previous Work: MILBoost

But in tracking, only current frame available Need an online training algorithm for MIL

[Viola et al. ‘05]

Main Contribution of this paper

• Online-MILBoost:

Online training for MIL-based classifier

• MILTrack

New tracking solution using Online-MILBoost

MILTrack workflow

New frame comes in:

• 1. Crop out a set of image patches

𝑌𝑡 = {𝑦| < 𝑡1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

X

Babenko et al., 09 1: s = 35 in authors’ experiment

MILTrack workflow

New frame comes in:

• 1. Crop out a set of image patches

𝑌𝑡 = {𝑦| < 𝑡1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

X

Babenko et al., 09 1: s = 35 in authors’ experiment

MILTrack workflow

New frame comes in:

• 1. Crop out a set of image patches

𝑌𝑡 = {𝑦| < 𝑡1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

X

1: s = 35 in authors’ experiment Babenko et al., 09

MILTrack workflow

New frame comes in:

• 1. Crop out a set of image patches

𝑌𝑡 = {𝑦| < 𝑡1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

X

Babenko et al., 09 1: s = 35 in authors’ experiment

MILTrack workflow

New frame comes in:

• 1. Crop out a set of image patches

𝑌𝑡 = {𝑦| < 𝑡1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

Babenko et al., 09 1: s = 35 in authors’ experiment

MILTrack workflow

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

• 2. Use MIL classifier to find new tracker location

𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

Babenko et al., 09

MILTrack workflow

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜} 2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

• 3. 1) Crop positive examples

𝑌𝑠 = {𝑦| < 𝑠1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

Babenko et al., 09 1: r = 5 in authors’ experiment

MILTrack workflow

(X1 , 1) Babenko et al., 09

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜} 2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

• 3. 1) Crop positive examples

𝑌𝑠 = {𝑦| < 𝑠1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

1: r = 5 in authors’ experiment x13 x12 x11

MILTrack workflow

Babenko et al., 09

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜} 2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

3. 1) Crop positive examples 𝑌𝑠 = {𝑦| < 𝑠 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

• 3. 2) Crop Negative examples

𝑌𝑠, 𝛾 = 𝑦 𝑠 𝑢𝑝 𝛾1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑏𝑥𝑏𝑧 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

1: 𝛾 = 50 in authors’ experiment

MILTrack workflow

(X3 , 0) x31 (X2 , 0) x21 Babenko et al., 09

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜} 2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

3. 1) Crop positive examples 𝑌𝑠 = {𝑦| < 𝑠 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

• 3. 2) Crop Negative examples

𝑌𝑠, 𝛾 = 𝑦 𝑠 𝑢𝑝 𝛾1 𝑞𝑗𝑦𝑓𝑚𝑡 𝑏𝑥𝑏𝑧 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

1: 𝛾 = 50 in authors’ experiment

MILTrack workflow

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜} 2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

3. Crop positive and negative examples near new

• bject location
• 4. Online MILBoost:

Update MIL classifier with positive and negative example bags

(X3 , 0) (X2 , 0) (X1 , 1) Babenko et al., 09 x31 x21 x13 x12 x11

Classifier

Online-MILBoost:

f1 f2 f3 …

Image patch x

Babenko et al., 09

Online-MILBoost:

• ℎ𝑙: a weak classifier using one feature

Image patch x

Babenko et al., 09

f1 f2 f3 … f9 …

ℎ1(𝑦) ℎ2(𝑦)

Online-MILBoost:

• ℎ𝑙: a weak classifier using one feature

ℎ𝑙(𝑦) = log 𝑞(𝑧 = 1|𝑔𝑙(𝑦)) 𝑞(𝑧 = 0|𝑔𝑙(𝑦))

Image patch x

Babenko et al., 09

f1 f2 f3 … ft …

ℎ𝑙(𝑦)

Online-MILBoost:

• ℎ𝑙: a weak classifier using one feature

ℎ𝑙(𝑦) = log 𝑞(𝑧 = 1|𝑔𝑙(𝑦)) 𝑞(𝑧 = 0|𝑔𝑙(𝑦)) with,

𝑞 𝑔𝑙 𝑦 𝑧 = 1 ~ 𝒪(𝜈1, 𝜏1) 𝑞 𝑔𝑙 𝑦 𝑧 = 0 ~ 𝒪(𝜈0, 𝜏0) 𝑞 𝑧 = 1 = 𝑞 𝑧 = 0 Image patch x

Babenko et al., 09

f1 f2 f3 … ft …

ℎ𝑙(𝑦)

Online-MILBoost:

• 𝑰(𝒚) : the MIL classifier made from weak

classifiers 𝑰 𝒚 = ℎ𝑙(𝑦)

𝐿 𝑙=1

Babenko et al., 09 K = 50 in authors’ experiment

Image patch x ℎ1(𝑦) ℎ𝑙(𝑦) … …

f1 f2 f3 … ft …

Online-MILBoost:

• Always keep a pool of M >> K weak classifier

candidates h3 h1 h2 … hM

Babenko et al., 09 M = 250 & K = 50 in authors’ experiment

Online-MILBoost:

• Update all M weak classifiers with positive and

negative bags

X3

{(X1, 1), (X2, 0), (X3, 0)}

X2 X1 x13 x12 x11 x21 x31

Classifier h1 Classifier h2 Classifier hM

…

Babenko et al., 09

Online-MILBoost:

• Pick best K weak classifiers to form 𝑰(𝒚),

where ℎ𝑙 = argmax

ℎ ∈ {ℎ1 … ℎ𝑁}

log 𝑀(𝐼𝑙 − 1 + ℎ) 𝑰 𝒚 = ℎ𝑙(𝑦)

𝐿 𝑙=1

where 𝐼𝑙 − 1 is the classifier made up of the first 𝑙 − 1 weak classifiers

Babenko et al., 09

Online-MILBoost:

• Prediction : 𝑞 𝑧 = 1 𝑦 = 𝜏(𝑰(𝒚))

𝜏 𝑦 = 1 1 + 𝑓 − 𝑦

Babenko et al., 09

Online-MILBoost:

• Prediction : 𝑞 𝑧 = 1 𝑦 = 𝜏(𝑰(𝒚))

𝜏 𝑦 = 1 1 + 𝑓 − 𝑦

Babenko et al., 09

ℎ1 𝑦 = 2, ℎ2 𝑦 = 1.8, ℎ3 𝑦 = 0.6

f1 f2 f3

ℎ1(𝑦) ℎ2(𝑦) ℎ3(𝑦)

Online-MILBoost:

• Prediction : 𝑞 𝑧 = 1 𝑦 = 𝜏(𝑰(𝒚))

𝜏 𝑢 = 1 1 + 𝑓 − 𝑢

Babenko et al., 09

𝑰 𝒚 = ℎ𝑙(𝑦)

𝐿 𝑙=1

= 4.4

f1 f2 f3

ℎ1(𝑦) ℎ2(𝑦) ℎ3(𝑦)

𝑞 𝑧 = 1 𝑦 = 𝜏 𝑰 𝒚 = 0.99 ℎ1 𝑦 = 2, ℎ2 𝑦 = 1.8, ℎ3 𝑦 = 0.6

MILTrack workflow

New frame comes in:

1. Crop out a set of image patches 𝑌𝑡 = {𝑦| < 𝑡 𝑞𝑗𝑦𝑓𝑚𝑡 𝑔𝑠𝑝𝑛 𝑢𝑠𝑏𝑑𝑙𝑓𝑠 𝑚𝑝𝑑𝑏𝑢𝑗𝑝𝑜}

2. Use MIL classifier to find new tracker location 𝑚𝑜𝑓𝑥 = 𝑚(argmax

𝑦 ∈ 𝑌𝑡 𝑞(𝑧 = 1|𝑦))

3. Crop positive and negative examples near new object location

4. Online MILBoost: Update MIL classifier with positive and negative example bags

Babenko et al., 09

Experiments

Datesets: 8 publicly available videos,

• Grayscale, 320 x 240 pixels
• Ground truth labeled every 5 frames by hand

− Babenko et al., 2009

Coke can Girl Occluded face David Occluded face 2 Tiger 1 Sylvester Tiger 2

Experiments

Compared with:

• OAB1

Online AdaBoost w/ 1 positive example per frame

• OAB5

Online AdaBoost w/ 45 positive examples per frame

• SemiBoost

Label in 1st frame only.

• FragTrack

Static appearance model

Babenko et al., 2009

Experiments

Evaluation criterion: Tracker position error (pixels) w.r.t. Ground truth

Babenko et al., 2009 X X

Results

Video David

Babenko et al., 09

Results

Position error versus Frame #, Video David

Babenko et al., 2009

Results

Video Occluded Face

Babenko et al., 09

Results

Position error versus Frame #, Video Occluded Face

Babenko et al., 2009

Results

Average Center location errors (pixels)

Best Second Best Babenko et al., 2009

Conclusion

• Online MILBoost:

Online algorithm to update MIL-based classifier

• Performance of “MILTrack” is stable

Babenko et al., 2009

Discussion

• Why it can handle occlusion?
• Possible improvements
• Motion Model
• Features
• Part based representation

Babenko et al., 2009

Note…

Wu et al., 2013

Note…

Wu et al., 2013

Note…

Wu et al., 2013 Project 2 use this!

Thank You! Q&A