Reranking with Contextual dissimilarity measures from - - PowerPoint PPT Presentation

α-divergences Contextual dissimilarity measures Experiments

Reranking with Contextual dissimilarity measures from representational Bregman k-means

VISAPP 2010 Olivier Schwander Frank Nielsen {schwander,nielsen}@lix.polytechnique.fr

´ ENS Cachan – ´ Ecole Polytechnique – Sony CSL

Angers, May 21st 2010

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments

α-divergences Definition and properties Representational Bregman divergences Clustering Contextual dissimilarity measures Definition Retrieval process Reranking with contexts Experiments Experimental setup Results

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments

Introduction

Content-Based Image Retrieval

◮ Query by example paradigm ◮ Input: a query image ◮ Output: a rank list, with images sorted by similarity

Challenges

◮ Size of databases (millions or billions of entries) ◮ Speed, memory problems ◮ Similarity is subjective (Personalization ? User interaction ?

Contextual research ?)

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments

Motivation: hierarchical contexts

Are we searching for

◮ an animal ? ◮ a mammifer ? ◮ a cat ? ◮ a Siamese cat ? ◮ Felix ?

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments

Motivation: region contexts

Are we searching for

◮ sea ? ◮ forest ? ◮ moutains ? ◮ all the three ?

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

α-divergences

One parameter

◮ α ∈ R

On positive arrays (unnormalized discrete probabilities)

Dα(pq) =                       

• 4

1−α2

• 1−α

2 pi + 1+α 2 qi − p

1−α 2

i

q

1+α 2

i

• if α = ±1

pi log pi

qi + qi − pi = KL(pq)

if α = −1 pi log qi

pi + pi − qi = KL(qp)

if α = 1

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Properties

Invariance to reparametrization (Cencov)

◮ Levi-Civita connexion (Riemannian geometry) ◮ α-connexion

Information monotonicity

◮ Merging bins gives lower distance between histograms

Canonical divergences for constant curvature spaces

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Bregman divergences

One parameter

◮ F : Rd → R ◮ Convex and differentiable

BF(pq) = F(p) − F(q) − p − q, ∇F(q)

Generalization of many usual distances

◮ Machine learning ◮ Computer vision ◮ Information geometry

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Some examples of Bregman-divergences

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Representational Bregman divergence

Separable Bregman divergence

◮ BF(pq) = d i=0 BF(piqi)

Representation function k

◮ Continuous, monotonously increasing ◮ Possibly non-linear ◮ Change of the coordinate system xi = k(si)

Induced generator

◮ U(x) = d i=1 U(xi) = d i=1 U(k(si)) = F(s)

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Representational α-divergences

Uα(x) = 2 1 + α 1 − α 2 x

• 2

1−α

kα(x) = 2 2 − αx

1−α 2

Bregman divergence

◮ BU,k = BU◦k = BF

Warnings

◮ Not strictly convex (only U is) ◮ Not symmetrical

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

α-centroids

Definition

cR = arg min

c∈X n

• i=1

BU, k(pic) cL = arg min

c∈X n

• i=1

BU, k(cpi)

Closed-form formulas

cR = n−

2 1−α

• p

1−α 2

i

• 2

1−α

cL = n−

2 1+α

• p

1+α 2

i

• 2

1+α Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition and properties Representational Bregman divergences Clustering

Clustering with α-divergences

k-means, classical Lloyd algorithm

◮ rely on Bregman k-means, Banerjee 2005 ◮ Bregman k-means on k representation (not the same k)

Assignment step

◮ with α-divergences

Relocation step

◮ with α-centroids

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition Retrieval process Reranking with contexts

Contextual dissimilarity measures (Perronnin 2009)

Φf (ω; q, p, u) = f (q, ωp + (1 − ω)u) csf (q, p|u) = arg min

0≤ω≤1 Φg(ω; q, p, u)

p u q q⊥ q⋆

Approximate q with a mixture of q and u

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition Retrieval process Reranking with contexts

Building contexts

Hierarchical contexts (Perronnin 2009)

Rank list 1st context 2nd context 3rd context 4th context

Partition contexts

Rank list 1st context 2nd context 3rd context 4th context

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition Retrieval process Reranking with contexts

Simple Content-Based Image Retriebal system

Query by example

◮ Query image ◮ Rank list

With global descriptors

◮ GIST descriptors ◮ No bag-of-word

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition Retrieval process Reranking with contexts

Reranking with a single context

Given a short list

◮ Take a subset of the short list (from clustering or by

truncating)

◮ Estimate the context: centroid of the points ◮ For all entry csf (q, pi|uk) ◮ Rerank according the new scores

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Definition Retrieval process Reranking with contexts

Reranking with multiple contexts

Given a short list

◮ Build a set of contexts ◮ For each context, for all entry csf (q, pi|uk) ◮ Average the different scores ◮ Rerank according the new scores

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Comparisons

Baseline

◮ GIST descriptors ◮ α-divergences ◮ Kullback-Leibler

divergence

Contextual Measure of Dissimilarity

◮ GIST descriptors ◮ α-divergences ◮ Kullback-Leibler

divergence

No bag-of-words, and global descriptors

◮ focus on dissimilarity measure

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Dataset

Holidays dataset

◮ from INRIA ◮ J´

egou et al. 2008

Details

◮ 1500 images ◮ 500 classes ◮ 3 images by classes

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Evaluation

Mean average precision (mAP)

◮ Average of the precisions at the point of each relevant

document in the rank list

Warning

◮ Scores not directly comparable with Perronin one’s ◮ No use of a bag-of-word ◮ GIST descriptors

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Results: influence of α

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Results: various divergences

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Concluding remarks

Contexts

◮ Can improve mAP ◮ Not really “contexts” ◮ Input parameter ? ◮ Interactive visualization of

the rank list ?

α-divergence

◮ k-means ◮ Well chosen divergence

improves results

◮ α-divergences and

underlying geometry are interesting

Some drawbacks

◮ Reranking is slow ◮ Need to tune the α

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures

α-divergences Contextual dissimilarity measures Experiments Experimental setup Results

Thanks Any questions ?

◮ The dual Voronoi diagrams with respect to representational

Bregman, Nielsen and Nock, ISVD 2009

◮ A family of contextual measures of similarity between

distributions application to image retrieval, Perronnin, Liu, Renders, CVPR 2009

◮ Clustering with Bregman divergences, Banerjee, Merugu,

Dhillon, Ghosh., JMRL 2005

◮ Evaluation of GIST descriptors for web-scale image search,

Douze, J´ egou, Singh, Amsaleg, Schmid, ICIVR 2009

Olivier Schwander, Frank Nielsen Reranking with Contextual Dissimilarity Measures