Content-based Image Retrieval by Indexing Random Subwindows with - - PowerPoint PPT Presentation

Method Results Conclusions

Content-based Image Retrieval by Indexing Random Subwindows with Randomized Trees

Rapha¨ el Mar´ ee, Pierre Geurts, Louis Wehenkel

GIGA Bioinformatics Platform

• Dept. EE & CS (Montefiore Institute)

University of Li` ege Belgium

ACCV, 22th November 2007, Tokyo

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 1

Method Results Conclusions

Content-Based Image Retrieval (CBIR)

Goal

Given a reference database of unlabeled images, retrieve images similar to a new query image based only on visual content.

Challenges

To be robust to uncontrolled conditions To be fast (efficient indexing structures) and accurate (rich image descriptions) To avoid tedious manual adaptation specific to a task

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 2

Method Results Conclusions

Content-Based Image Retrieval (CBIR)

Goal

Given a reference database of unlabeled images, retrieve images similar to a new query image based only on visual content.

Challenges

To be robust to uncontrolled conditions To be fast (efficient indexing structures) and accurate (rich image descriptions) To avoid tedious manual adaptation specific to a task

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 3

Method Results Conclusions

Starting point: our method at CVPR05

Image classification with labeled training images and single class prediction Fast method

Random subwindow extraction Extremely randomized decision trees [Geurts et al. 2006]

Good accuracy results on various tasks

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 4

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

This work: extension for CBIR

Overview Detector: random subwindows Descriptor: subwindow raw pixel values Indexing subwindows: totally randomized trees Image similarity measure: derived from similarity measure between subwindows defined by trees

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 5

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Extraction of Random Subwindows

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 6

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Indexing subwindows with one Totally Randomized Tree

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 7

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Indexing subwindows with an Ensemble of T Trees

Parameters

T: the number of totally randomized trees nmin: the minimum node size, stop-spliting of a node if #node < nmin

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 8

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Similarity between two subwindows (one tree)

A tree T defines a similarity between two subwindows s and s′ : kT (s, s′) = 1

NL

if s and s′ reach the same leaf L containing NL subwindows,

• therwise

Two subwindows are very similar if they fall in a same leaf that has a very small subset of training subwindows

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 9

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Similarity between two subwindows (ensemble of T trees)

The similarity induced by an ensemble of T trees is defined by: kens(s, s′) = 1 T

T

• t=1

kTt(s, s′) (1) Two subwindows are similar if they are considered similar by a large proportion of the trees

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 10

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Similarity between two images

We derive a similarity between two images I and I ′ by: k(I, I ′) = 1 |S(I)||S(I ′)|

• s∈S(I),s′∈S(I ′)

kens(s, s′) (2) The similarity between two images is thus the average similarity between all pairs of their subwindows (2) is estimated by extracting at random from each image an a priori fixed number of subwindows

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 11

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Similarities between IQ and all reference images...

... are obtained by propagating subwindows from IQ, and by incrementing, for each subwindow s of IQ, each tree T , and each reference image (IR), the similarity k(IQ, IR) by the proportion of subwindows of IR in the leaf reached by s in the tree T , and by normalizing the resulting score.

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 12

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Propagation of one subwindow into trees

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 13

Method Results Conclusions Random Subwindows Totally Randomized Trees Similarity measure defined by trees

Extensions

Model recycling: Given a large set of unlabeled images we can build an ensemble of trees on these images, and then use this model to compare new images from another set. Incremental mode: It is possible to incorporate the subwindows of a new image into an existing indexing structure by propagating and recording their leaf counts. If a leaf happens to contain more than nmin subwindows, split the node.

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 14

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

ZuBuD (1/3): images of 201 buildings

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 15

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

ZuBuD (2/3): results

Protocol

1005 unlabeled reference images (640 × 480) 115 labeled test images (320 × 240) Recognition rate of the first ranked image

Results Dataset ls/ts us OM05 OM02 ZuBuD 1005/115 96.52% 93% to 98.2% 100% (with 10 trees, 1000 subwindows per image, nmin = 2 ie. fully developed trees)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 16

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

ZuBuD (3/3): query − → top 10 retrieved images

− → − →

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 17

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

IRMA (1/3): X-Ray images (from http://irma-project.org/)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 18

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

IRMA (2/3): Results

Protocol

9000 unlabeled reference images (approx. 512 × 512) 1000 labeled test images (57 classes) Recognition rate of the first ranked image

Results Dataset ls/ts us na¨ ıve NN KDGN07 IRMA 9000/1000 85.4% 29.7% 63.2% 87.4% (with 10 trees, 1000 subwindows per image, nmin = 2 ie. fully developed trees)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 19

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

IRMA (3/3): query − → top 5 retrieved images

− → − → − →

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 20

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

UkBench (1/2): images of 2550 “objects”

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 21

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

UkBench (2/2): results

Protocol

10200 unlabeled reference images (640 × 480) Same images for test (labeled) Recognition rate of the top-4 ranked images

(Number of correct images in first 4 retrieved images /40800) ∗ 100%

Results Dataset ls=ts us NS06 PCISZ07 UkBench 10200 75.25% 76.75% to 82.35% 86.25% (with 10 trees, 1000 subwindows per image, nmin = 4)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 22

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

META (1/2): images from various sources

Sources: LabelMe Set1-16, Caltech-256, Aardvark to Zorro, CEA CLIC, Pascal Visual Object Challenge 2007, Natural Scenes A. Oliva, Flowers, WANG, Xerox6, Butterflies, Birds.

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 23

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

META (2/2): results

Protocol

205763 unlabeled reference images 10200 UkBench labeled test images Recognition rate of the top-4 ranked images

(Number of correct images in first 4 retrieved images /40800) ∗ 100%

Results Dataset ls/ts us NS06 META/UkBench 205763/10200 66.74 % 54% to 79 % (with 10 trees, 50 subwindows per META image, 1000 subwindows per UkBench image, nmin = 2 ie. fully developed trees)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 24

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

Number of subwindows per training image: more is better

70% 75% 80% 85% 90% 95% 100% 100 200 300 400 500 600 700 800 900 1000 Recognition Nls ZuBuD: Influence of nb. training subwindows (T=10, Nts=1000, nmin=1) 83% 84% 85% 86% 100 200 300 400 500 600 700 800 900 1000 Recognition Nls IRMA: Influence of nb. training subwindows (T=10, Nts=1000, nmin=1) 60% 65% 70% 250 500 750 1000 1250 1500 Recognition Nls UKBench: Influence of nb. training subwindows (T=10, Nts=100, nmin=15) 60% 61% 62% 63% 64% 65% 66% 67% 10 20 30 40 50 Recognition Nls META: Influence of nb. training subwindows (T=10, Nts=1000, nmin=1)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 25

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

Number of trees T: more is better

90% 95% 100% 5 10 15 20 Recognition T ZuBuD: Influence nb. trees T (1000 subwindows per image, nmin=1) 75% 80% 85% 90% 5 10 Recognition T IRMA: Influence of nb. trees (1000 subwindows per image, nmin=1) 65% 70% 75% 80% 5 10 15 20 Recognition T UKBench: Influence of nb. trees (1000 subwindows per image, nmin=15) 55% 60% 65% 70% 5 10 Recognition nmin META: Influence of nb. trees (Nls=50, Nts=1000, nmin=1)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 26

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

Tree depth (minimum node size nmin): deeper is better

85% 90% 95% 100% 100 200 300 400 500 600 700 800 900 1000 Recognition nmin ZuBuD: Influence of nmin stop splitting (T=10, 1000 subwindows per image) 80% 85% 90% 100 200 300 400 500 600 700 800 900 1000 Recognition nmin IRMA: Influence of nmin stop splitting (T=10, 1000 subwindows per image) 55% 60% 65% 70% 75% 80% 100 200 300 400 500 600 700 800 900 1000 Recognition nmin UKBench: Influence of nmin stop splitting (T=10, 1000 subwindows per image) 55% 60% 65% 70% 100 200 300 400 500 600 700 800 900 1000 Recognition nmin META: Influence of nmin stop splitting (T=10, Nls=50, Nts =1000)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 27

Method Results Conclusions ZuBuD, IRMA, UKBench META: model recycling Influence of parameters

Number of subwindows per query image: more is better

65% 70% 75% 80% 85% 90% 95% 100% 100 200 300 400 500 600 700 800 900 1000 Recognition Nts ZuBuD: Influence of nb. query subwindows (T=10, Nls=1000, nmin=1) 60% 65% 70% 75% 80% 85% 90% 100 200 300 400 500 600 700 800 900 1000 Recognition Nts IRMA: Influence of nb. query subwindows (T=10, Nls=1000, nmin=1) 50% 55% 60% 65% 70% 75% 80% 250 500 750 1000 1250 1500 Recognition Nts UKBench: Influence of nb. query subwindows (T=10, Nls=1500, nmin=15) 40% 45% 50% 55% 60% 65% 70% 100 200 300 400 500 600 700 800 900 1000 Recognition Nts META: Influence of nb. query subwindows (T=10, Nls=50, nmin=1)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 28

Method Results Conclusions Summary Prospects

Summary

A simple method that yields quite good results on various tasks...

Unlabeled reference images Extraction of random subwindows Description by raw pixel values Indexing with totally randomized trees Image similarity derived from trees

... and has some nice practical properties

Only a few parameters Fast indexing, fast prediction (parallelization also possible) Model recycling, incremental mode (Implementation in Java, check http://www.montefiore.ulg.ac.be/~maree/)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 29

Method Results Conclusions Summary Prospects

Prospects

Applications

Tackle even more challenging visual tasks Deal with bigger databases (Flickr hits two billion images) Image near-duplicate detection Indexing of other types of data (e.g. audio)

Method

Combination with features/descriptors Mechanisms like relevance feedback, sub-image retrieval, . . .

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 30

Method Results Conclusions Summary Prospects

Acknowledgments

Vincent Botta for figures Walloon Region European Regional Development Fund National Fund for Scientific Research IRMA database courtesy of TM Lehmann and T. Deselaers (RWTH Aachen, Germany)

Mar´ ee et al. Indexing Random Subwindows with Randomized Trees 31