TREC 2003 Video Retrieval Evaluation Overview Coordinators: Alan - - PowerPoint PPT Presentation

TREC 2003 Video Retrieval Evaluation

Overview

Coordinators: Alan Smeaton

Centre for Digital Video Processing Dublin City University

Wessel Kraaij

Department of Data Interpretation Information Systems Division TNO TPD

NIST: Paul Over

Retrieval Group Information Access Division Information Technology Laboratory National Institute of Standards and Technology

• Problem:

n Rapidly growing quantities of digital video n Increasing research in content-based retrieval from digital video n But no common basis for evaluation/comparison of approaches

• Approach:

n Find as much video data as possible and make it available to the community of researchers n Use the data to build an open, metrics-based evaluation in the Cranfield/TREC tradition n Invite participation and see what happens…

Origins

• Promote progress in content-based retrieval

from large amounts of digital video

• Answer some questions:

n How can systems achieve such retrieval (in collaboration with a human)? n How can one reliably benchmark such systems?

Goals

• Evolution… 2001

q TREC 2001 Video retrieval track q Data: 11 hrs (OpenVideo, NIST)

• 2 Tasks:

n Shot boundary determination n Search

• Fully automatic
• Interactive
• Participating groups: 12

• Evolution… 2002

q TREC 2002 Video retrieval track q Data: 73 hrs (Prelinger Archive)

• 3 Tasks:

n Shot boundary determination n High

• level feature extraction (10)

n Search (manual and interactive)

• Participating groups: 17
• New:

n Common shot reference defines unit of retrieval n Common key frames n Shared features, ASR output provided by LIMSI

• TRECVID Workshop
• Data: 133 hrs (1998 ABC/CNN news + C-SPAN)
• 4 Tasks:

n Shot boundary determination n High-level feature extraction (17) n Story segmentation and classification n Search (manual and interactive)

• Participating groups: 24
• New:

n Common annotation effort n Advisory committee

Evolution… 2003

• John Eakins

(University of Northumbria at Newcastle)

• Peter Enser

(University of Brighton)

• Alex Hauptmann

(CMU)

• Annemieke de Jong

(Netherlands Institute for Sound & Vision)

• Michael Lew

(Leiden Insitute of Advanced Computer Science)

• Georges Quenot

(CLIPS-IMAG Laboratory)

• John Smith

(IBM)

• Richard Wright

(BBC)

Advisory committee

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

• Shot Boundary Detection task
• SBD is an enabling function for almost all content-based
• perations on digital video, so its important;
• (Still) not a new problem, but a challenge because of

gradual transitions and false positives caused by photo flashes, rapid camera movement, object movement, etc.;

• Task is to identify transitions and determine whether each is

“cut”, “dissolve”, “fadeout/in” or “other”;

• TRECVID2003 dataset is slightly (10%) larger than 2002 but

has many more (78%) shot transitions;

• Shot Boundary Detection task
• Manually created ground truth of 3,734 transitions (thanks

again to Jonathan Lasko) with 70.7% hard cuts, 20.2% dissolves, 3.1% fades and 5.9% other … very similar ratios to 2002;

• Up to 10 submissions per group, measured using precision

and recall, with a bit of flexibility for matching gradual transitions;

• Most participating groups use their 10 submissions to

“tweak” some parameter;

• Accenture Technology Laboratories (US) X

X Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X

X X

• Univ. of Iowa (US) X

X

• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

14 Groups in Shot Boundary Detection

• What do the results look like ?

• Evaluation Measures

Precision = Recall = Frame Precision = Frame Recall =

# Transitions Correctly Reported # Transitions Reported # Transitions Correctly Reported # Transitions in Reference # Frames Correctly Reported in Detected Transitions # Frames reported in Detected Transitions # Frames Correctly Reported in Detected Transitions # Frames in Reference Data for Detected Transitions

• Recall and precision for cuts
• !

"

• #$$

#%& '( )*"' % %+ %,

• Recall and precision for cuts (zoomed)
• !

"

• #$$

#%& '( )*"' % %+ %,

• … and for Gradual Transitions …

• Recall and precision for gradual transitions
• !

"

• #$$

#%& '( )*"' % %+ %,

• Frame-recall & -precision for GTs
• !

"

• #$$

#%& '( )*"' % %+ %,

So, who did what ? The approaches….

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

• ./0(1-2

!"" # $%&' (%)% $&' * +*),)'!! '-. /'!! '-!")%!'!! '-!" $$)-& &)".&!,'!# &' *' '!"')'!!"' ))/",-""'!*!"'! *'!"'!'!#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

*32 0 "!'*"'! /'&'! $! '!/&'&% $'))/ $$ !""'"'! "' ) # 1 % "'!!" /'&)')"'''!# 2) '!)%" "'"'! ) & #

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

%&-02 % "10$$&, "! $' !% '!*)%'!! "'!!")%&' * '')'-# 2"$'& &)"'!* '!) & # 3 &" 4'!*,)4 "'!/&&&-" .) "' ) #

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

!*2 5&$% ) '')'- *'! ))'!/'!"/$ !"%% . /)) '')'- ,/!$ 6%% # 3! '')'-'!" '!&' ' &' ' '!"'% !"3 # !)%" )/-"% $%'! # 7 !'!))/#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

"(.2 8 "10%'" - 7 !'!))/

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

'4(/( 2 )%&' * '')'-"9! /'&! ! '')'- & &)"# 1 !" &/'!*" 7 & &)"' # 3"$! '$)$$&#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

#$$2 5% .$$ &!"":73 )&'*&'! "'! % '!*'! &! "")'4)- !" )%'!!!"&'!!"'! # 5"' ) ."*"%)&!*'!* '% '!*''-"# 1$''"'!)4'!*/'$ .!" ) & # 0% '$ !"" . )'!7#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

#%(&2 2"$'& &)"'!* !&* '!! '-"'! ,/!"9! # !)%" '!$! '!/&'& $% !'!! '! ,/!! %' #

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

'( %&-02 3),))%&' *"'! %"' !'!%'-' % "" % # 53 .&"%!)'! )%''!'* & "%'!&3./'& $'' !'!*,9 "%'!*& 3/&'&!"' &' #

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(%&-02 *3 # 8 '!*'!*/'!"/;< . % %! =0*'! ))'! &/'!"//'&:>%!" %!# 0 "! '')'-!" "$'& &)"'!*# 2'!!!#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

%&-0(5("'2 ,'!'!$$& ? &!* '!'*)%'!!# *-))&' *"'! # 55%'!# ,'!"./'&"$'& &)"'!*#

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

%&-0(5(-( 2 )%&' *'! '! /'& %, $)'!*'"$ # &' *' $$')'! & ,%!" .))/",-'!*'!" $' !% '!*,'!)% &' *# 7 $ '!*",%$&!* '!'))%'!'!;) & <# 2) "'!"! ''!-$ #

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

%&-0(5(+2 $' !"9! , "! ,'!*),))%&' * $')&%,!') &%,!') , "!$')@$') 1,) ')'!*!""""* "'! !"&!0)!!"'&'$"% ,'!'! & # 7 !'!))/

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

%&-0(5(#2 "') '),)&' '

• !

"

• #$$

#%& '( )*"' % %+ %,

Recall and precision for cuts (zoomed)

• !

"

• #$$

#%& '( )*"' % %+ %,

Gradual Transitions

• !

"

• #$$

#%& '( )*"' % %+ %,

Frame-recall & -precision for GTs

• Observations
• Most techniques are based on frame-frame

comparisons, some with sliding windows;

• Comparisons are based on colour and on

luminance, mostly;

• Some use adaptive thresholding, some don’t;
• Most operate on decoded video stream;
• Some have special treatment of motion during

GTs, of flashes, of camera wipes;

• Performances are getting better;

• Story segmentation and news typing
• Identify the individual news items in a news show
• New task in TRECVID, has been studied in ASR/IR

community (TDT)

• Hope to show the gain of using video features

1. Segmentation task

n Identify story boundaries in CNN and ABC news shows n Ground truth based on TDT 2 annotations n Evaluation based on precision & recall, boundaries have to be within +/- 5 seconds interval around ground truth boundaries

2. News classification task

n Annotate stories as either news or non-news n Evaluation based on percentage of correctly identified news story footage

• Dublin City University (Irl)

Fudan Univ. (China) IBM Research (US) KDDI (JP) National Univ. Singapore (Sing.) StreamSage (US)

• Univ. of Central Florida (US)
• Univ. of Iowa (US)

8 Participating Groups

• Story segmentation: recall and precision

by condition

• (6 7
• 6 7 7
• ((

(8-.

• Story segmentation: recall and precision

by system and condition (1-4)

• $%

" #$$ 9%

• '/

% %(+

• 1

1 1 1 1 1 1 1 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 2 3 3 2 4 4 4 4 4 4

Conditions: 1: V+A 2: V+A+ASR 3: ASR 4: Other

• Segmentation, within system (F)

0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1

NUS IBM Fudan Iowa DCU StreamSage

F (beta=1)

AV AV+ASR ASR

• Story classsification: news recall and precision

by condition

• (6 7

(6 7 7

• ((

(8-. 3(:(+:

• Story classsification: news recall and precision

by condition - zoomed

• (6 7
• 6 7 7

( (( (8-. 3(:(+:

• Story classifcation: news recall and precision

by system

• "

9% % %(+ 3(:(+:

• "

9% % %(+ 3(:(+:

• Story classifcation: news recall and precision

by system and condition (1-4) zoomed

1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 4 4 4 3 3 3 3 3 3 3

Conditions: 1: V+A 2: V+A+ASR 3: ASR 4: Other

• Classification, within system (F)

0,6 0,65 0,7 0,75 0,8 0,85 0,9 0,95 1

NUS IBM Fudan Iowa

F value

AV AV+ASR ASR OTHER

• $1(-0(%&-0(;<

%&(;.< "(.(;%< #$$(;=<( 9-(%&(/4(;/<

• '/ ;%<(

%&(5(-( ((;%<( %&(5(+(;%<((((((((((((((((((((((((((((

3%$&")'!

%&-0

1*!'! A2!&"'!, "!)% '!* !"&%' ' A')"'!, "!B A21 *!'!% '!*'! ')'!* A%), "!":! ) '' / ) '''! A3::@:! % '!*% '. ')!" $&$$'! %

• $1(-0(%&-0(;<

%&(;.< "(.(;%< #$$(;=<( 9-(%&(/4(;/<

• '/ ;%<(

%&(5(-( ((;%<( %&(5(+(;%<((((((((((((((((((((((((((((

3%$&")'!

#$$

1*!'! 2)) & ) ''" 2CD. 7DD::2E.% '!*%"' 6'!'!! '-.)Ł 1: 1%, (%!)-%), " *!'! ') '''!,%!"' . % '!*&% / & , !"&,%!"-!"'"1: ) '''! 1:F1F1.F1:1 !":1F1

• $1(-0(%&-0(;<

%&(;.< "(.(;%< #$$(;=<( 9-(%&(/4(;/<

• '/ ;%<(

%&(5(-( ((;%<( %&(5(+(;%<((((((((((((((((((((((((((((

3%$&")'!

• '/ ;>($%<

21!)- *!'!%! &&" ?

• )')&'!'!*"'!$'))-

&! *! '!')'!* 8 &" !"$')'!* )' %$& &!!%! $''!"%'!) %

• $1(-0(%&-0(;<

%&(;.< "(.(;%< #$$(;=<( 9-(%&(/4(;/<

• '/ ;%<(

%&(5(-( ((;%<( %&(5(+(;%<((((((((((((((((((((((((((((

3%$&")'!

%&-0(5(-(

,'!"1*!'!!" ) '''!? 1-,%!"' 4",-,)!4

• E!* -Ł

!/ . & -Ł !!!/ :*"9!!!!/ ' !)% '!? -)!*&' !* %!/ ) '''!

• $1(-0(%&-0(;<

%&(;.< "(.(;%< #$$(;=<( 9-(%&(/4(;/<

• '/ ;%<(

%&(5(-( ((;%<( %&(5(+(;%<((((((((((((((((((((((((((((

3%$&")'!

$1(-0(%&-0 "(. 9-(%&-0(/4 %&-0(5(+ $ !'! ))/+

• Observations
• Video provides strong clues for story segmentation and

even more for classification, best runs are either type 1 or 2

• AV runs generally have a higher precision
• Combination of AV and ASR gives a small gain for

segmentation

• Most approaches are generic
• Are the combination methods optimal?
• Are the ASR segmentation runs state of the art?

• FE Task definition
• Goal: Build benchmark for detection methods of high-level

features

• Secondary goal: feature-indexing can help search and

navigation

• New: common feature annotation

n Helps (a.o.) to standardize training resources across sites n Category A: sites work with just the common development data and common annotations n Category B: sites work with just the common development data and any annotation set n Category C: other

• FE evaluation
• Each feature is assumed to be binary: absent or

present for each shot

• Find shots that contain a certain feature, rank them

according to confidence measure, submit the top 2000

• Submissions are pooled
• Evaluate performance quality by measuring the

average precision of each feature detection method

• Accenture Technology Laboratories (US)

Carnegie Mellon Univ. (US) CLIPS-IMAG (FR) CWI Amsterdam / Univ. of Twente (NL) Fudan Univ. (China) IBM Research (US) Imperial College London (UK) Institut Eurecom (FR)

• Univ. of Central Florida (US)
• Univ. Oulu/VTT (FI)

10 Participating Groups

• 17 Features
• 11. Indoors
• 12. News subject face – not a news show person
• 13. People – at least three humans
• 14. Building – walled structure with roof
• 15. Road
• 16. Vegetation – living vegetation in its natural env.
• 17. Animal
• 18. Female speech – woman speaking (visible, audible)
• 19. Car/truck/bus – exterior of ..

• 17 Features
• 20. Aircraft
• 21. News subject monologue – uninterrupted
• 22. Non-studio setting
• 23. Sporting event
• 24. Weather news
• 25. Zoom in
• 26. Physical violence – between people / objects
• 27. Madeleine Albright – visible

• 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

Accenture Technology Laboratories (US) X X X Carnegie Mellon Univ. (US) X X X X X X X X X X X X X X X X X CLIPS-IMAG (FR) X CWI Amsterdam / Univ. of Twente (NL) X X X X X X X X X X X X X X Fudan Univ. (China) X X X X X X X X X X X X X X X X X IBM Research (US) X X X X X X X X X X X X X X X X X Imperial College London (UK) X Institut Eurecom (FR) X X X X X X X X X X X X X X X

• Univ. of Central Florida (US)

X X

• Univ. Oulu/VTT (FI) X X X X X X X X X X X X

X X X 6 6 6 6 6 7 6 6 6 6 4 7 6 8 3 6 6

Who worked on which features

3%$

p e

• p

l e i n d

• r

s N e w s f a c e v e g e t a t i

• n

b u i l d i n g r

• a

d c a r a n i m a l F e m a l e s p e e c h Z

• m

i n S p

• r

t i n g e v e n t W e a t h e r n e w s N

• n

s t u d i

• a

i r c r a f t N e w s m

• n
• P

h y s i c a l v i

• l

e n c e P e r s

• n

X

• AvgP by feature (all runs)

Middle half

• f the data

Median

• AvgP by feature (top 10 runs)
• &/(4

1 2 3 4 5 6 7 8 9 10 Median

Median ->

• AvgP by feature (top 5 runs by per feature)
• &/(4

?"*"8"8 ?"*"8 ?"*"8% ?"*$?3 ?"*$?3 ?"*@?" ?"*?"8 ?"*?@ ?"*9?9 ?"*89?A "?% "?% "? ?@?0 "? ?@?0 "? ?@?0 ?% ?% ?% ?% ?%

Female speech Zoom News subject monologue

• AvgP by feature (top 5 runs by per feature)

zoomed: Hard features

• &/(4

?"*"8"8 ?"*"8 ?"*"8% ?"*$?3 ?"*$?3 ?"*@?" ?"*?"8 ?"*?@ ?"*9?9 ?"*89?A "?% "?% "? ?@?0 "? ?@?0 "? ?@?0 ?% ?% ?% ?% ?%

M.A. aircraft Female speech vegetation violence Non-studio Car/truck animal road building people News face indoors

• AvgP by feature (top 5 runs per feature)

zoomed: Easy features

• &/(4

?"*"8"8 ?"*"8 ?"*"8% ?"*$?3 ?"*$?3 ?"*@?" ?"*?"8 ?"*?@ ?"*9?9 ?"*89?A "?% "?% "? ?@?0 "? ?@?0 "? ?@?0 ?% ?% ?% ?% ?%

weather zoom sports News subject monologue

• 0.1

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500

Total number of true shots Average precision

• Avg. precision

vs total number true for each feature

Medians Maximums

weather Non-studio

• 10

20 30 40 50 60 70 80 90

A_IBM-BOF_2 B_CMU09_9 B_Fudan_FE_Sys9_9 C_CMU06_6 C_Imperial-1_1 B_Fudan_FE_Sys6_6 C_Imperial-2_2 C_CMU03_3 A_CMU10_10 B_Fudan_FE_Sys1_1 A_ATL-Research_1 B_UCFVISION_1 A_IBM-DMF64_G_4 A_KNN_6 A_GMM_LSA_3 A_CWI_ML_3 A_GMM_5 Number of unique, true shots

33 of 60 runs contributed one or more unique, true shots

• A

_ I B M

• B

O F _ 2 A _ I B M

• M

L P _ B O R _ 3 B _ C M U 9 _ 9 B _ C M U 8 _ 8 B _ F u d a n _ F E _ S y s 9 _ 9 A _ I B M

• D

F 1 7 _ G _ 9 C _ C M U 6 _ 6 B _ C M U 7 _ 7 C _ I m p e r i a l

• 1

_ 1 C _ C M U 2 _ 2 B _ F u d a n _ F E _ S y s 6 _ 6 C _ C M U 4 _ 4 C _ I m p e r i a l

• 2

_ 2 A _ I B M

• B

O U _ 1 C _ C M U 3 _ 3 B _ F I _ O U _ M T 6 _ 2 A _ C M U 1 _ 1 A _ I B M

• M

L P _ E F C _ 7 B _ F u d a n _ F E _ S y s 1 _ 1 B _ F I _ O U _ M T 4 _ 6 A _ A T L

• R

e s e a r c h _ 1 A _ G M M _ L S A _ N E G _ 2 B _ U C F V I S I O N _ 1 B _ F I _ O U _ M T 3 _ 5 A _ I B M

• D

M F 6 4 _ G _ 4 A _ G M M _ N E G _ 4 A _ K N N _ 6 C _ C M U 1 _ 1 A _ G M M _ L S A _ 3 A _ K N N _ L S A _ 1 A _ C W I _ M L _ 3 B _ F I _ O U _ M T 1 _ 1 A _ G M M _ 5

1 1

• u

t d

• r

s 1 2 n e w s u b j e c t f a c e 1 3 p e

• p

l e 1 4 b u i l d i n g 1 5 r

• a

d 1 6 v e g e t a t i

• n

1 7 a n i m a l 1 8 f e m a l e s p e e c h 1 9 c a r / t r u c k / b u s 2 a i r c r a f t 2 1 n e w s s u b j e c t m

• n
• l
• q

u e 2 2 n

• n
• s

t u d i

• s

e t t i n g 2 3 s p

• r

t i n g e v e n t 2 4 w e a t h e r n e w s 2 5 z

• m

i n 2 6 p h y s i c a l v i

• l

e n c e 2 7 M a d e l e i n e A r l b r i g h t 1 1 6 1 1 6 5 3 8 1 2 1 1 1 1 9 17 22 32 19 17 10 40 28 31 37 38 23 2 6 4 11 1 1 4 2 2 1 6 6 4 12 7 2 1 1 2 1 1 4 6 3 8 2 16 1 3 2 1 1 23 21 1 1 5 10 6 3 48 19 8 14 30 3 32 4 18 12 1 29 12 16 22 17 1 8 15 15 10 7 5 28 6 2 4 5 3 2 36 7 546 48 3 10 20 30 40

Number of unique true shots

True shots contributed uniquely by run for a feature

• ATL

CMU Oulu Fudan IBM ICL Eurecom UCF CWI

1 1

• u

t d

• r

s 1 2 n e w s u b j e c t f a c e 1 3 p e

• p

l e 1 4 b u i l d i n g 1 5 r

• a

d 1 6 v e g e t a t i

• n

1 7 a n i m a l 1 8 f e m a l e s p e e c h 1 9 c a r / t r u c k / b u s 2 a i r c r a f t 2 1 n e w s s u b j e c t m

• n
• l
• q

u e 2 2 n

• n
• s

t u d i

• s

e t t i n g 2 3 s p

• r

t i n g e v e n t 2 4 w e a t h e r n e w s 2 5 z

• m

i n 2 6 p h y s i c a l v i

• l

e n c e 2 7 M a d e l e i n e A r l b r i g h t

1 12 24 11 73 5 5 242 52 129 1 3 3 2 1 54 149 40 309 282 36 56 32 2 3 18 2 6 6 16 44 10 7 22 18 1 4 17 1 4 7 12 8 6 6 21 12 12 15 25 26 2 5 2 55 19 21 34 7 55 54 13 32 29 52 30 120 122 37 23 28 10 77 116 9 17 37 6 21 39 19 25 50 75 100 125 150 175 200 225 250 275 300 325 350

Number of unique true shots

True shots contributed uniquely for a feature by a participating group

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

• ./0(

1-?

7$) A14'!!"'!.%! F& AG)!*&GH)"' ',%'! H$ ''!)- 5)1$& A2%"', "*!""'!H H'!*)'$

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

/((%&-0?

2))% 7 !'!))/

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

*3?

%?:2 C//%)",)'!"$ !) & !'!'!*:")'!2),'*& A1$4"'!;% '")< A:2' $,,)-!'!"'!!& $"'!* &

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

B('- '(>(%&-0(5( +-? %

F4'!*&-$& ' ?5%'! II(%-,- $) 3!'$,,')' '')") ; % " & 4<."''" '!$'),)4 4 $)&!!" . !4&4- , "!& )'4)'&"&&-*!&(%- $)

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

%&-0? ))%

1!% ?*'".)&' *. "*"''!.%.J.2"0 *'!.F&?%H). 1:.3::.:! D,9 ? A?1&!'"! A2!')?*'!/'&J A2'?"!' 2%"'?) $&?:5.7'&. E7

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

"(.?

2))% 7 !'!))/

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

'4(/( ?

5%?*'! 0 "!* "% '!*)% %!"J

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

• -- @'?

2$$)-E1 % J- *!"'!*'! *'! )% "% '!*J! )% K'' "%",-E1 8 !/% $3::!"J "

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

%&-0(5(-(

% F&!/ A )&' * '')'- ! %"' '!* A4! ?))!!!& &

• ./0(1-(;%<(

/((%&(;%< *3(;<( B('- '(>(%&(5(+- ;9< %&(;.< "(.(;%< '4(/( (;%#<

• -- @' ;<(

%&(5(-( ((;%<( %&(8>6(;<

3%$&")'!

%&-0(5(8 >(6?

"% % '!*? A:'! A$)))* A"**"'! A1))/))%"'% ;% " %" .&')!' . $. !)*% A5%% '!, "!0" %! '!*

• Observations
• Some feature detectors had quite good results
• Are features well chosen for search ?
• Is detection quality good enough?
• Which combination methods work well? Which

don’t?

• TRECVID2003: Search Task
• Search, summarisation, linking, etc. are the

ultimate operations on digital video and SBD, features, segmentation, are all enablers for this;

• TRECVID search is an extension of its text-only

analogue where systems, including a human in the loop, are presented with a topic and are to return up to 1,000 shots which meet the need;

• Note the unit of retrieval is the shot, not the news

story;

• Two search modes … manual and interactive, and

we’re not yet able for full automatic;

• Search Types: Interactive and Manual

• Search Types: Interactive and Manual
• Topics are MM and the interactions between text,

image, video, audio, are complex and understanding how exemplars represent information need, is not really understood;

• This task really benefitted from the ASR donated

by Jean-Luc Gauvain of LIMSI which is (anecdotally) very accurate;

• One baseline run based on ASR-only was required
• f every manual system;

• Topics
• We can’t achieve the ideal of topics from real users

searching our dataset;

• NIST created topics based on a number of basic

search types: generic/specific and person/thing/event where there are multiple relevant shots coming from more than one video;

• Videos were viewed by NIST personnel (sound off),

notes taken on content, and candidates emerged and were chosen;

• 25 Topics [total relevant found]

100.Find shots with aerial views containing both one or more buildings and one or more roads [87] 101.Find shots of a basket being made - the basketball passes down through the hoop and net [104] 102.Find shots from behind the pitcher in a baseball game as he throws a ball that the batter swings at [183] 103.Find shots of Yasser Arafat [33] 104.Find shots of an airplane taking off [44] 105.Find shots of a helicopter in flight or on the ground [52] 106.Find shots of the Tomb of the Unknown Soldier at Arlington National Cemetery [31] 107.Find shots of a rocket or missile taking off. Simulations are acceptable [62] 108.Find shots of the Mercedes logo (star) [34]

• 25 Topics

109.Find shots of one or more tanks [16] 110.Find shots of a person diving into some water [13] 111.Find shots with a locomotive (and attached railroad cars if any) approaching the viewer [13] 112.Find shots showing flames [228] 113.Find more shots with one or more snow-covered mountain peaks or ridges. Some sky must be visible behind them. [62] 114.Find shots of Osama Bin Laden [26] 115.Find shots of one or more roads with lots of vehicles [106] 116.Find shots of the Sphinx [12] 117.Find shots of one or more groups of people, a crowd, walking in an urban environment (for example with streets, traffic, and/or buildings) [665]

• 25 Topics

118.Find shots of Congressman Mark Souder [6] 119.Find shots of Morgan Freeman [18] 120.Find shots of a graphic of Dow Jones Industrial Average showing a rise for one day. The number of points risen that day must be visible. (Manual only) [47] 121.Find shots of a mug or cup of coffee. [95] 122.Find shots of one or more cats. At least part of both ears, both eyes, and the mouth must be visible. The body can be in any

• position. [122]

123.Find shots of Pope John Paul II [45] 124.Find shots of the front of the White House in the daytime with the fountain running [10]

• Evaluation
• Groups allowed to submit up to 10 runs and 37

interactive and 38 manual runs were submitted from 11 groups;

• All submissions were pooled and judged by NIST

assessors to variable depths depending on “hit rate” of finding relevant shots;

• Evaluation was trec_eval;

• Results

q Absolute performance figures must be taken in their context, so don’t believe the numbers … read the papers !

• We tried to level the field by standardising on time

spent (15 min.) and thought of introducing a reference system at each site, but TRECVID not yet mature enough for that;

• Also, submitted runs do not necessarily

correspond to 1 user, but can be aggregates of multiple users, 2+ groups did this;

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI)

X X

Shots Stories Features Search

24 Participating Groups

• 1

2 3 4 5 6 7 8 9 10 I _ A _ 1 _ D C U T r e c 1 2 a _ 5 I _ A _ 1 _ D C U T r e c 1 2 b _ 2 I _ B _ 1 _ I C L

• 2

_ 2 I _ B _ 2 _ L L 1 1 _ I r n d _ 3 I _ C _ 1 _ I u V f l 1 _ 1 M _ A _ 2 _ F u d a n _ S e a r c h _ R u n 8 _ 8 M _ A _ 2 _ I B M

• 3

_ 3 I _ A _ 2 _ I B M

• 4

_ 4 I _ B _ 2 _ O U M T _ I 3 V _ 3 M _ A _ 2 _ F u d a n _ S e a r c h _ R u n 6 _ 6 M _ B _ 2 _ L L 1 1 _ c

• m

b i n e d _ 5 M _ C _ 2 _ C M U 3 _ 4 I _ A _ 2 _ I B M

• 5

_ 5 I _ B _ 2 _ O U M T _ I 4 V T _ 4 M _ A _ 2 _ I B M

• 6

_ 6 M _ B _ 1 _ I C L

• _

5 I _ B _ 2 _ O U M T _ I 1 V _ 1 I _ B _ 2 _ O U M T _ I 2 V T _ 2 I _ C _ 2 _ C M U 2 _ 2 I _ C _ 2 _ C M U 1 _ 1

Number of unique, relevant shots

20 of 75 runs contributed 1+ unique, relv. shots

• C

M U D C U F u d a n I B M I C L I U L

• w

l a n d s O u l u

1 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 1 1 1 1 1 1 2 1 1 3 1 1 4 1 1 5 1 1 6 1 1 7 1 1 8 1 1 9 1 2 1 2 1 1 2 2 1 2 3 1 2 4

1 1 2 2 10 1 8 13 2 3 15 3 1 12 26 1 2 2 2 6 3 1 2 4 1 1 2 1 1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30

Number of unique true shots

Relevant shots contributed uniquely for a topic by a participating group

• Manual runs - top 10 (of 38)

(with mean human effort / topic)

• C_2_CMU03_5 (15 mins)

C_2_CMU03_3 (15 mins) C_2_CMU03_4 (15 mins) C_2_CMU03_7 (15 mins) C_2_CMU03_6 (15 mins) C_2_CMU03_8 (15 mins) A_1_CMU03_9 (15 mins) A_2_IBM-3_3 (15 mins) B_2_LL11_dynbestiASRNo Anchor_9 (3.5 mins) B_1_LL11_ASR_10 (0.5 mins)

• Interactive runs - top 10 (of 36)

(with mean elapsed time)

• C_2_CMU01 (15 mins)

C_2_CMU02 (15 mins) A_1_DCUTrec12a_5 (7 mins) B_1_ICL-4 (9.9 mins) B_1_ICL-2 (10.9 mins) B_2_LL11_Inaive (14.3 mins) B_1_ICL-3 (10.6 mins) B_2_OUMT_I2VT (12.1 mins) A_1_DCUTrec12b_2 (7 mins) B_2_LL11_Itxt (14.2 mins)

• 100

101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124

Topic number Average precision Interactive median Manual median Interactive max Manual max

Avgerage precision by topic

Y a s s e r A r a f a t U n k n

• w

n S

• l

d i e r T

• m

b O s a m a b i n L a d e n S p h i n x M a r k S

• u

d e r

• 0.1

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 5 1 1 5 2 2 5 3 3 5 4 4 5 5 5 5 6 6 5

Number of relevant found Average precision

Average precision (interactive max) vs number relevant shots found

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(/((%&-02 !'? - L $)' $' !*'!"''"%) . & 21..D/'& -,"'!* 4- .)-%%!"% !).')'!* , "!% #!&%!% " '$" '!/'&' ' %)' '! !",/ '!*# :!%)?%)'$)')*! )%.%.21.D!" % . ,'!"'!"'!/- .'!)*' $ %"5!"M')N# 7 !'!))/ * %) ;*'!<#

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(+ (;B(C(%(+-<2 *'!*'!'!%)'$)")'' ? %! $= &$'$)# ,'!"'!") = # ,'! ' - @% 9%"*! # ,%')")!*%*")& &# 7$%'!* &4-'!"# !',&!!%)!" ,'!'!@' %),&! ) 7 !'!))/

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

($1(-0(%&-02 ''!5O &)P'!'!' '!*/'& % .'! &."'!*$' # / - ''! /21 &!)- !"21$)% (%-'* &4-# 0&&" &)),/ '!*.% !))" 21@'* &,)!.5))/",- $!"'!*!"@'*# 2'/ '% % "!",!'" 6'*# 7 !'!))/

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

( %&-02 :!%) &% '!*"'!$$& !"&!,'!'! ? 21 )% &' * %)'$)%;)% &' ."*. %! %< M $') &N/&% ) $$$'$'.&%! *.*!) &% .%)'$) % .'!;!",9<. )%@%.)% *'! #

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

("(.2 '!"1$4!%!')!" !!, "&!'(% '!!%)'! 1% "%'!"$&!' &!'(% !"1% '! %)'$) &%!'! .!4'!* & , "! ),), # 2) "'"%))-%'%)'$)$) !!, ";/&'&' ,-!"M!%)N< !"% '!!!, "!"1, " ')'!/'*&'!*# 7 !'!))/

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

('4(/( 2 8 "216)/)))%@%. "' *"'!*'*)'4)-!'! !/ '4# 5% '!)%"*),))%.)% !.)% %%" '$ . 30)% ! .$')*- &%,!') .!)%'!') .'!. '* &! !"%!''-.21# ')4 .&%,!') !"' $)-. 5,-% !&%,!') ."B !%).'!'%! . %) *" 7 !'!))/

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

( (%&-02 8 "21!",%') - %!" '!' &!"(%-$! '! $)% '" &,/ '!*# !' &/'& %,9"'!*)) $' .'! ,%% "!)-'! # 5%%/4' '!)%" &, "! ' %)% #

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

( >%&-0(5('- '2

!' &/'&*%$ % ;'! $' B<.% '!*,'!'!? :8"!"% "'"M!$ NE121 4-/" 21

• ')"!' . & &! !QQ-

&,/ )$) # D!)-$)%! %,'" 8 " - !))'!$ *'! .!" )')-,'!" % R %$% $$'*! %,' '!#

M0 N;$$'<,9')- )",- %,''!*& %)/&& & / )",-&%

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(9-(%&-0(5(/42 / - '), "!21!"% '!* F" !"/,$!"&'*'!) (%-.7D1**'!*(%-# 5') & -, "! &% # 8 '*6'"&'!*!4 '!'!* & # !'!'%! % '/ $ & !" 4 )!! %) &/4"'$!%) '!'.% 5# 7 !'!))/

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(%&-0(5(9-.((;<2 $21!)-.% !)-. 21H% .'!'!' & 4# 5% ?***" %) *%$ % % "'!'! 4# 21/ E:1.,'!'!/ 21# & .&"'!*$'

• '!'! $$'#

1&,/ &"!!" -," 4-H21.) $ !"$ (% '!!'!)- '

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(%&-0(5(9-.((;<2 %) +! ' ')"'!'! 7' '!.,% ' ')"'!'!)) /&% !)-/ ) &!&& /+$%*!''!%-B E*',')'-'!'4!$ &. * '!% # :%&)%'!% R $$'!!" ' '!# 1&)$%)$'! !%% ! '!' &#

• Accenture Technology Laboratories (US) X X

Carnegie Mellon Univ. (US) X X CLIPS-IMAG (FR) X X CWI Amsterdam / Univ. of Twente (NL) X X Dublin City University (Irl) X X Fudan Univ. (China) X X X X FX-Pal (US) X IBM Research (US) X X X X Imperial College London (UK) X X X Indiana University (US) X Institut Eurecom (FR) X KDDI (JP) X X KU Leuven (BE) X Mediamill/U Amsterdam (NL) X National Univ. Singapore (Sing.) X X Ramon Llull Univ. (ES) X RMIT University (Aus) X StreamSage (US) X

• Univ. of Bremen (D)

X

• Univ. of Central Florida (US) X X X
• Univ. of Iowa (US) X X
• Univ. of Kansas (US) X
• Univ. of North Carolina (US)

X

• Univ. Oulu/VTT (FI) X X

Shots Stories Features Search

24 Participating Groups

(%&-0(5(8>62 & '!')% @$) & ,/ '!*!" & '')'-, "!' %) ;)%."* %%.'!<.!$%) ;% % <!")') ;21< '')'-# :!%)%! 7 ),'!'! % !"'* $'# !'%! +$$). - . '! $$'.;<,/ ,-' %)% !)-!";,<,/ ,-' %)% $)% 21+ %)'!"' ! '*!''! "'!#

• Observations
• Lots of variation, interesting shot browsing

interfaces, mixture of interactive & manual;

• Approximately as much use of donated features as

TV2002;

• A lot more participation, more runs, better at the

upper end … quite respectable curves !

• Nearly a dozen groups can now complete the

search task and the demos are impressive;

• Make notebook papers, presentations, and feedback on

plans for TRECVID 2004 available on the website in December

• Make final papers available on the website by mid
• March 2004
• Plans: probably complete 2
• y

r plan

n Add 80 hours of new test data from same news sources n Repeat 2003 tasks with some improvements

• More information as it develops at:

www

• n

lpir.nist.gov/projects/trecvid

Plans