Boris Babenko, Steve Branson, Serge Belongie University of - - PowerPoint PPT Presentation

Boris Babenko, Steve Branson, Serge Belongie

University of California, San Diego ICCV 2009, Kyoto, Japan

• Recognizing multiple categories

Recognizing multiple categories

– Need meaningful similarity metric / feature space

• Recognizing multiple categories

Recognizing multiple categories

– Need meaningful similarity metric / feature space

• Idea use training data to learn metric
• Idea: use training data to learn metric

– Goes by many names:

i l i

• metric learning
• cue combination/weighting
• kernel combination/learning
• kernel combination/learning
• feature selection

• Learn a single global similarity metric

Learn a single global similarity metric

Labeled Dataset Query Image Similarity Metric

Monolithic

Category 1 Category 2 C 4 Category 3

[ Jones et al. ‘03, Ch t l ‘05

Category

Chopra et al. ‘05, Goldberger et al. ‘05, Shakhnarovich et al. ‘05 Torralba et al. ‘08]

• Learn similarity metric for each category (1‐vs‐all)

Learn similarity metric for each category (1 vs all)

Labeled Dataset Query Image Similarity Metric

Monolithic

Category 1 Category 2 C

gory cific

4 Category 3

Categ Spec

Category

[ Varma et al. ‘07, Frome et al. ‘07, Weinberger et al. ‘08 Nilsback et al. ’08]

• Monolithic:

Monolithic:

– Less powerful… there is no “perfect” metric Can generalize to new categories – Can generalize to new categories

• Per category:

– More powerful – Do we really need thousands of metrics? – Have to train for new categories

• Would like to explore space between two

Would like to explore space between two extremes

• Idea:
• Idea:

– Group categories together L f i il i i f h – Learn a few similarity metrics, one for each group

• Learn a few good similarity metrics

Learn a few good similarity metrics

Labeled Dataset Query Image Similarity Metric

Monolithic

Category 1

MuSL

Category 2 C

gory cific

4 Category 3

Categ Spec

Category

• Need some framework to work with

Need some framework to work with…

• Boosting has many advantages:

F t l ti – Feature selection – Easy implementation – Performs well

• Training data:

Training data:

Images Category Labels

• Generate pairs:

– Sample negative pairs Sa p e ega e pa s ( ) 0 ( ) 1 ( , ), 0 ( , ), 1

• Train similarity metric/classifier:

Train similarity metric/classifier:

• Choose

to be binary ‐‐ i e Choose to be binary i.e.

• = L1 distance over binary vectors

ffi i t t t (XOR d ) – efficient to compute (XOR and sum)

• For convenience:

[Shakhnarovich et al. ’05, Fergus et al. ‘08]

• Given some objective function

Given some objective function

• Boosting = gradient ascent in function space
• Gradient = example weights for boosting

Gradient = example weights for boosting

chosen weak classifier current strong classifier

• ther weak classifiers

function space [Friedman ’01, Mason et al. ‘00]

• Goal: train

and recover Goal: train and recover mapping

• At runtime
• At runtime

– To compute similarity of query image to use use

y 2 Category 1 ategory 3 Category Category 4 C

• Run pre‐processing to group categories (i e k‐

Run pre processing to group categories (i.e. k means), then train as usual

• Drawbacks:
• Drawbacks:

– Hacky / not elegant N i l i i f d b l – Not optimal: pre‐processing not informed by class confusions, etc.

• How can we train & group simultaneously?

• Definitions:

Definitions:

Sigmoid Function Parameter

• Definitions:

Definitions:

• Definitions:

Definitions:

How well works with category

• Objective function:

Objective function:

• Each category “assigned” to classifier

• Replace max with differentiable approx

Replace max with differentiable approx. where is a scalar parameter where is a scalar parameter

• Each training pair has

weights Each training pair has weights

• Intuition:

Intuition:

Approximation of Difficulty of pair (like regular boosting)

2 x 10

• 4

6 x 10

• 5

w1

i

Difficult Pair Assigned to Easy Pair Assigned to

1 1.5 2 3 4 5 6

i

w2

i

w3

i

10 20 30 0.5 10 20 30 1 2 (b ti it ti ) (b ti it ti ) (boosting iteration) (boosting iteration)

for for for ‐ Compute weights ‐ Train on weighted pairs end end A i Assign

• Created dataset with many heterogeneous

Created dataset with many heterogeneous categories

0 8 0.75 0.8 cy 0.7 Accurac MuSL+retrain MuSL k-means Rand 5 10 15 20 0.65

K (n mber of classifiers)

Rand Monolithic Per Cat

Merged categories from:

• Caltech 101 [Griffin et al.]
• Oxford Flowers [Nilsback et al.]

UIUC T t [L b ik t l ]

K (number of classifiers)

• UIUC Textures [Lazebnik et al.]

MuSL M ns k‐mea k

Training more metrics overfits!

• Studied categorization performance vs

Studied categorization performance vs number of learned metrics

• Presented boosting algorithm to
• Presented boosting algorithm to

simultaneously group categories and train metrics metrics

• Observed overfitting behavior for novel

i categories

• Supported by

Supported by

– NSF CAREER Grant #0448615 NSF IGERT Grant DGE 0333451 – NSF IGERT Grant DGE‐0333451 – ONR MURI Grant #N00014‐08‐1‐0638 UCSD FWG id P j t (NSF I f t t G t – UCSD FWGrid Project (NSF Infrastructure Grant

• no. EIA‐0303622)

• Train similarity metric/classifier:

Train similarity metric/classifier:

• Let

Let then then

Matrix:

1 2 3 1 2 3 1 2 3 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20

Caltech Flowers Textures Caltech Flowers Textures Caltech Flowers Textures

‐ High value ‐ Low value