Boris Babenko, Steve Branson, Serge Belongie University of - PowerPoint PPT Presentation

Boris Babenko, Steve Branson, Serge Belongie University of California, San Diego ICCV 2009, Kyoto, Japan

• Recognizing multiple categories Recognizing multiple categories – Need meaningful similarity metric / feature space

• Recognizing multiple categories Recognizing multiple categories – Need meaningful similarity metric / feature space • Idea use training data to learn metric • Idea: use training data to learn metric – Goes by many names: • metric learning i l i • cue combination/weighting • kernel combination/learning • kernel combination/learning • feature selection

• Learn a single global similarity metric Learn a single global similarity metric Query Image Similarity Metric Labeled Dataset Monolithic Category 1 C Category 2 Category 3 [ Jones et al. ‘03, 4 Ch Chopra et al. ‘05, t l ‘05 Category Goldberger et al. ‘05, Shakhnarovich et al. ‘05 Torralba et al. ‘08]

• Learn similarity metric for each category (1 ‐ vs ‐ all) Learn similarity metric for each category (1 vs all) Query Image Similarity Metric Labeled Dataset Monolithic Category 1 C Category 2 Category 3 gory cific Categ 4 Spec Category [ Varma et al. ‘07, Frome et al. ‘07, Weinberger et al. ‘08 Nilsback et al. ’08]

• Monolithic : Monolithic : – Less powerful… there is no “perfect” metric – Can generalize to new categories Can generalize to new categories • Per category : – More powerful – Do we really need thousands of metrics? – Have to train for new categories

• Would like to explore space between two Would like to explore space between two extremes • Idea: • Idea: – Group categories together – Learn a few similarity metrics, one for each group L f i il i i f h

• Learn a few good similarity metrics Learn a few good similarity metrics Query Image Similarity Metric Labeled Dataset Monolithic Category 1 C Category 2 MuSL Category 3 gory cific Categ 4 Spec Category

• Need some framework to work with Need some framework to work with… • Boosting has many advantages: – Feature selection F t l ti – Easy implementation – Performs well

• Training data: Training data: Images Category Labels • Generate pairs: – Sample negative pairs Sa p e ega e pa s ( ( , ), 1 ) 1 ( ( , ), 0 ) 0

• Train similarity metric/classifier: Train similarity metric/classifier:

• Choose Choose to be binary to be binary ‐‐ i e i.e. • = L1 distance over binary vectors – efficient to compute (XOR and sum) ffi i t t t (XOR d ) • For convenience: [Shakhnarovich et al. ’05, Fergus et al. ‘08]

• Given some objective function Given some objective function • Boosting = gradient ascent in function space • Gradient = example weights for boosting Gradient = example weights for boosting chosen weak current strong classifier classifier other weak classifiers function space [Friedman ’01, Mason et al. ‘00]

• Goal : train Goal : train and recover and recover mapping • At runtime • At runtime – To compute similarity of query image to use use y 2 Category 1 ategory 3 Category Category 4 C

• Run pre ‐ processing to group categories (i e k ‐ Run pre processing to group categories (i.e. k means), then train as usual • Drawbacks: • Drawbacks: – Hacky / not elegant – Not optimal: pre ‐ processing not informed by class N i l i i f d b l confusions, etc. • How can we train & group simultaneously ?

• Definitions: Definitions: Sigmoid Function Parameter

• Definitions: Definitions:

• Definitions: Definitions: How well works with category

• Objective function: Objective function: • Each category “assigned” to classifier

• Replace max with differentiable approx Replace max with differentiable approx. where where is a scalar parameter is a scalar parameter

• Each training pair has Each training pair has weights weights

• Intuition: Intuition: Approximation of Difficulty of pair (like regular boosting)

Difficult Pair Easy Pair Assigned to Assigned to -4 -5 x 10 x 10 w 1 i i 6 6 2 2 w 2 5 i 1.5 w 3 4 i 1 3 2 0.5 1 10 20 30 10 20 30 (b (boosting iteration) ti it ti ) (boosting iteration) (b ti it ti )

for for for ‐ Compute weights ‐ Train on weighted pairs end end A i Assign

• Created dataset with many heterogeneous Created dataset with many heterogeneous categories 0 8 0.8 0.75 cy Accurac MuSL+retrain MuSL 0.7 k-means Rand Rand Merged categories from: Monolithic • Caltech 101 [Griffin et al.] Per Cat 0.65 • Oxford Flowers [Nilsback et al.] 5 10 15 20 • UIUC Textures [Lazebnik et al.] UIUC T t [L b ik t l ] K (n mber of classifiers) K (number of classifiers)

MuSL M k ‐ mea k ns

Training more metrics overfits!

• Studied categorization performance vs Studied categorization performance vs number of learned metrics • Presented boosting algorithm to • Presented boosting algorithm to simultaneously group categories and train metrics metrics • Observed overfitting behavior for novel categories i

• Supported by Supported by – NSF CAREER Grant #0448615 – NSF IGERT Grant DGE ‐ 0333451 NSF IGERT Grant DGE 0333451 – ONR MURI Grant #N00014 ‐ 08 ‐ 1 ‐ 0638 – UCSD FWGrid Project (NSF Infrastructure Grant UCSD FWG id P j t (NSF I f t t G t no. EIA ‐ 0303622)

• Train similarity metric/classifier: Train similarity metric/classifier: • Let Let then then

Matrix: 1 1 1 2 2 2 3 3 3 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20 2 4 6 8 10 12 14 16 18 20 Caltech Caltech Caltech Flowers Textures Flowers Textures Flowers Textures ‐ High value ‐ Low value