Image search through browsing using NN k networks Daniel Heesch, - - PowerPoint PPT Presentation

image search through browsing using nn k networks
SMART_READER_LITE
LIVE PREVIEW

Image search through browsing using NN k networks Daniel Heesch, - - PowerPoint PPT Presentation

Image search through browsing using NN k networks Daniel Heesch, Marcus Pickering, Stefan Rger, Alexei Yavlinsky TRECVID 2003 Overview Image and Collection Preprocessing Search and Relevance Feedback Temporal Browsing and NN k


slide-1
SLIDE 1

TRECVID 2003

Image search through browsing using NNk networks

Daniel Heesch, Marcus Pickering, Stefan Rüger, Alexei Yavlinsky

slide-2
SLIDE 2

Overview

  • Image and Collection Preprocessing
  • Search and Relevance Feedback
  • Temporal Browsing and NNk Browsing
  • TVID Results
slide-3
SLIDE 3

Preprocessing

  • Use only common keyframes + LIMSI transcript
  • Removal of bottom 51 lines
slide-4
SLIDE 4

11 Primitive Features

  • 4 Colour

– global HSV, centre HSV, marginal RGB colour moments, colour structure descriptor

  • 2 Structure

– convolution map features on grey image

  • 3 Texture

– simple features on image tiles

  • 1 Annotation

– Bag of stemmed-words (tf-idf)

  • 1 Localisation

– Thumbnail of grey image

slide-5
SLIDE 5

44x27 Thumbnail: Ad detection

  • average pixel difference between two thumbnails
slide-6
SLIDE 6

Distance of topic Q to image i given feature f

  • distf: Manhatten
  • KNN distance

– positive examples (set Q) – negative examples (set N, random)

slide-7
SLIDE 7

Fusion of features

  • Convex combination
  • w is the “plasticity” of our retrieval system
slide-8
SLIDE 8

Relevance Feedback

slide-9
SLIDE 9

Relevance Feedback

  • Minimize

with respect to w and convexity constraint.

slide-10
SLIDE 10

Browsing

  • Hierarchical (not yet)
  • In ranked list (not shown)
  • Temporal
  • Lateral
slide-11
SLIDE 11

Temporal Browsing

  • Movement along a sequence of shots
slide-12
SLIDE 12

Temporal Browsing

  • Movement along a sequence of shots
slide-13
SLIDE 13

Temporal Browsing

  • Movement along a sequence of shots
slide-14
SLIDE 14

Temporal Browsing

  • Movement along a sequence of shots
  • Q: Add to query panel
  • A: Add to assembly panel
slide-15
SLIDE 15

Assembly panel

slide-16
SLIDE 16

Pruning Panel

slide-17
SLIDE 17

Lateral Browsing

  • Images as vertices in a directed graph
  • Instantiate arc (i,j) iff there is a feature

combination w such that j is closest to i

  • NNk network
slide-18
SLIDE 18

NNk Network construction

  • For each image
  • for each w determine nearest neighbour and

compute corresponding proportion of weight space (= edge weight)

  • store adjacent images and edge weights
slide-19
SLIDE 19

Sampling the weight space

slide-20
SLIDE 20

Rationale

  • exposure of semantic richness
  • user decides which image meaning is the

correct one

  • network precomputed interactive
  • supports search without query formulation
slide-21
SLIDE 21

Properties

  • small average distance between any two

vertices (three nodes for 32,000 images)

  • high clustering coefficient: an image´s

neighbours are likely to be neighbours themselves

  • vertex degrees follow power-law distribution

scale-free small-world graph

slide-22
SLIDE 22

Browsing interface

  • Initial display:

query-by-example retrieval result OR high connectivity nodes (hubs)

  • Clicking on an image moves it into the

center and displays all adjacent nodes in the network

slide-23
SLIDE 23
slide-24
SLIDE 24
slide-25
SLIDE 25
slide-26
SLIDE 26
slide-27
SLIDE 27
slide-28
SLIDE 28
slide-29
SLIDE 29
slide-30
SLIDE 30
slide-31
SLIDE 31

Observations

  • Browsing can help to explore visual similarity
  • Some task are impossible with browsing alone:

find video shots with Senator Mark Sounder

  • Browsing can be a fun activity
slide-32
SLIDE 32

Interactive runs

IV III II I Browsing Relevance Feedback Search Runs

slide-33
SLIDE 33

Experimental design

  • 4 subjects, 4 runs square lattice design

I IV III II S4 II I IV III S3 III II I IV S2 IV III II I S1 T19-25 T13-18 T7-12 T1-6

slide-34
SLIDE 34
slide-35
SLIDE 35

Results

RANK

(out of 36)

MAP 0.46 Best 0.19 Median 0.18 Mean 27 0.13 B 8 0.23 S + B 4 0.26 S + RF 5 0.26 S + RF + B

slide-36
SLIDE 36

Conclusions

  • Competitive system: Three out of four runs

among the top 8 (of 36)

  • “Search by browsing‘‘ a viable alternative to

traditional search by example for visual topics