Consuming videos with the ForkBrowser Consuming videos with the - - PowerPoint PPT Presentation

Consuming videos with the ForkBrowser Consuming videos with the ForkBrowser

Ork de Rooij, Cees G.M. Snoek, Marcel Worring Ork de Rooij, Cees G.M. Snoek, Marcel Worring

2 2 Consuming Videos With The ForkBrowser

What is search?: What is search?: enter query…

enter query…

The Problem: Search in video The Problem: Search in video

Query by keyword Query by concept Query by example

Query by region selection Automated Concept suggestion

Query component ‘mixer’

Language filter

… and search

Leads to: RANKING

  browse results browse results

3 3 Consuming Videos With The ForkBrowser

Mixing dimensions: CrossBrowser Mixing dimensions: CrossBrowser

Combination of query results and time Combination of query results and time

time results

• Easy to understand
• Time is very important
• Very fast browsing

4 4 Consuming Videos With The ForkBrowser

Mixing dimensions: RotorBrowser Mixing dimensions: RotorBrowser

Allows combinations based on query on demand Allows combinations based on query on demand

time Initial query result

• Doesn’t require the

user to think of options.

• Allows dataset exporation
• Browsing ‘outside query’ very

important for some topics

• Query screen visited only
• once

5 5 Consuming Videos With The ForkBrowser

Mixing dimensions Mixing dimensions

Easy to understand Easy to understand Time is very important Time is very important Very fast browsing Very fast browsing Doesn’t require the user to think of options. Allows dataset exporation Browsing ‘outside query’ very important for some types of query Limits visits to a “query screen”

A hybrid between both browsers is required

Typically a user wants to explore a dataset fast and easy, without difficult query screens.

6 6 Consuming Videos With The ForkBrowser

The ForkBrowser The ForkBrowser

Combination of fixed set of query methods Combination of fixed set of query methods time results

any query method

history

any query method

7 7 Consuming Videos With The ForkBrowser

The ForkBrowser The ForkBrowser

Combines Combines

 Fast browsing through results

Fast browsing through results

 Assignable dimensions, e.g.

Assignable dimensions, e.g.

Visual similarity Visual similarity semantic similarity semantic similarity

 User doesn’t have to revisit query screen

User doesn’t have to revisit query screen

 Animations on demand

Animations on demand

8 8 Consuming Videos With The ForkBrowser

The ForkBrowser The ForkBrowser

9 9 Consuming Videos With The ForkBrowser

Experiments Experiments

TRECVID 2007 Interactive Search TRECVID 2007 Interactive Search We compare: We compare:



Run with CrossBrowser (UVA_MM_1) Run with CrossBrowser (UVA_MM_1)



Run with ForkBrowser (UVA_MM_2) Run with ForkBrowser (UVA_MM_2)



Evaluation metrics try to minimalize effect of comparing expert users Evaluation metrics try to minimalize effect of comparing expert users

Set up: Set up:



Seed: Seed:

Automatic search results Automatic search results Query by concept, keyword and example Query by concept, keyword and example



Extra ‘tines’ in ForkBrowser: Extra ‘tines’ in ForkBrowser:

Weibull and Gabor visual similarity features Weibull and Gabor visual similarity features

What do we want to know? What do we want to know?



Is browsing using multiple dimensions useful? Is browsing using multiple dimensions useful?



Does a fixed layout lead to faster browsing and better results? Does a fixed layout lead to faster browsing and better results?

10 10 Consuming Videos With The ForkBrowser

Is browsing multiple dimensions useful? Is browsing multiple dimensions useful?

 Evaluate effectiveness of having multiple

Evaluate effectiveness of having multiple dimensions dimensions

11 11 Consuming Videos With The ForkBrowser

Query method usage per topic Query method usage per topic

train in motion train in motion Helped by: Helped by: animation animation time time

0205 0212 0209 0213 0214

boat moving past Helped by: animation time 3+ people at table Helped by: concept time visual similarity woman talking Helped by: visual similarity concept large crowd Helped by: concept

Different topics have different search strategies

12 12 Consuming Videos With The ForkBrowser

Unique results per browser Unique results per browser

50 100 150 200 250 300

Graph shows the number of (correct) shots retrieved Grey: shots found by both browser Red: shots found only by the CrossBrowser Green: shots found only by the ForkBrowser Both browsers find different results

13 13 Consuming Videos With The ForkBrowser

Movement vs Average Precision Movement vs Average Precision

1000 2000 3000 4000 5000 6000 7000 # of moves 0.0 0.2 0.4 0.6 0.8 1.0 A v e r a g e P r e c is io n

0197 0198 0199 0200 0201 0202 0203 0204 0205 0206 0207 0208 0209 0210 0211 0212 0213 0214 0215 0216 0217 0218 0219 0220 0197 0198 0199 0200 0201 0202 0203 0204 0205 0206 0207 0208 0209 0210 0211 0212 0213 0214 0215 0216 0217 0218 0219 0220

Movement to Average Precision ratio

Clear difference between # of movement actions required

14 14 Consuming Videos With The ForkBrowser

Conclusions Conclusions

Evaluation: Evaluation:

 Different combinations of query dimensions are beneficial

Different combinations of query dimensions are beneficial for individual topics for individual topics

 ForkBrowser requires less interaction steps from the user

ForkBrowser requires less interaction steps from the user for the same average precision for the same average precision

 Both browsers find different unique results

Both browsers find different unique results

15 15 Consuming Videos With The ForkBrowser

Any questions? Any questions?