CERTH at MediaEval Placing Task 2013 Giorgos Kordopatis 1 , Symeon - - PowerPoint PPT Presentation

CERTH at MediaEval Placing Task 2013

Giorgos Kordopatis1, Symeon Papadopoulos 2, Eleftherios Spyromitros-Xioufis 2, Andreas L. Symeonidis 2 and Yiannis Kompatsiaris 2

1 Electrical Engineering Dept., Aristotle University of Thessaloniki, Greece 2 Information Technologies Institute, CERTH, Thessaloniki, Greece

Summary of our participation

• 2 tag-based runs

– Build upon the method of (Van Laere et al., ICMR ‘11) and make use

• f a two-level LDA scheme to filter non-(geo)-discriminative tags
• 2 visual runs

– A nearest neighbor scheme using VLAD+SURF features

(Spyromitros-Xioufis et al., WIAMIS ‘12) and an efficient indexing

method for very fast retrieval

• 1 hybrid run

– Uses a simple fallback scheme to combine a tag-based and a visual run

• No external gazetteers or Internet data!

2

Placing using tags – Training (1/3)

• Removal of noisy and irrelevant tags, e.g. “geolat=… ”

3

Filtering Spatial Clustering & local LDAs Global LDA & BoEW Assignment in Areas Similarity Search

Training Prediction

Placing using tags – Training (2/3)

• 𝑙-means clustering is applied on the lat-long values of the training

images to create 𝑙=5000 cluster-areas

• A local topic model is created for the images of each area using

LDA (100/20)

4

Filtering Spatial Clustering & local LDAs Global LDA & BoEW Assignment in Areas Similarity Search

Training Prediction

Placing using tags – Training (3/3)

• A global topic model is created for all training

images using LDA (500/50)

• For each global topic, a topic-area histogram is

computed by counting its frequency per area

• Top terms of the highest entropy topics are

added in the bag-of-excluded words (BoEW)

5

Filtering Spatial Clustering & local LDAs Global LDA & BoEW Assignment in Areas Similarity Search

Training Prediction

• Noisy or BoEW tags are filtered and Jaccard similarity between the

image’s tags and each local topic is computed

• Each test image 𝑈𝑗 is assigned to the most similar area:

– Area containing the highest similarity topic (tmax) – Area with the highest average (across topics) similarity (tmean) 6

Filtering Spatial Clustering & local LDAs Global LDA & BoEW Assignment in Areas Similarity Search

Training Prediction

Placing using tags – Prediction (1/2)

• Having assigned the image to an area:

– 𝑙 most similar images within the area are identified – their center of gravity is used as location estimate 7

Filtering Spatial Clustering & local LDAs Global LDA & BoEW Assignment in Areas Similarity Search

Training Prediction

Placing using tags – Prediction (2/2)

Placing using visual features

• We extract optimized VLAD+SURF vectors (Jegou and Chum, ECCV

2012) for all training images

• Search efficiently with Product Quantization (Jegou et al., TPAMI

2011)

– 14 bytes per image! – 13 hours for querying the index with the full test set!

• 1st variant – vnn: Assign the location of the most similar image
• 2nd variant – vclust: Top 𝑙 =20 nearest neighbors are

considered and an incremental spatial clustering algorithm is applied

8

Runs and Results

Measure tmax vnn tmean vclust hyb

acc(1km) 10.26 0.60 7.82 0.76 10.37 acc(10km) 23.53 0.99 19.86 1.16 23.70 Median error 651 6715 1028 6691 681 9

• Tag-based runs: tmax and tmean
• Visual runs: vnn and vclust
• Hybrid run: hyb, if there is at least one clean tag, tmax is

used, else vnn is used

Future work

• A more thorough analysis on the different sources of

error for the proposed scheme

– # of geographical areas – # of topics and terms per topic used in the local and the global LDAs – type and quality of visual features

• Extension to include additional metadata and external

resources, e.g. the author of an image as an indicator of the image location

10

The end

This work was supported by the SocialSensor FP7 project

11