TREC 2006 Video Retrieval Evaluation Introductions Paul Over* - - PowerPoint PPT Presentation

SLIDE 1

TREC 2006 Video Retrieval Evaluation

Introductions

Paul Over* Wessel Kraaij (TNO ICT) Tzveta Ianeva* Alan Smeaton (DCU) Lori Buckland*

* Retrieval Group Information Access Division Information Technology Laboratory NIST

SLIDE 2

• 13. Nov 2006

TRECVID 2006 2



Promote progress in content-based retrieval from large amounts of digital video –



combine multiple errorful sources of evidence



achieve greater effectiveness, speed, usability



Model an analyst interested in finding video of certain people, things, places, events, and combinations thereof



work on what video is “of” rather than what it is “about”



similar to needs of commercial video producers searching an archive for video to reuse rather than reshoot

Goals

SLIDE 3

• 13. Nov 2006

TRECVID 2006 3



Focus on relatively high-level functionality – near that of an end-user application like interactive search



Confront systems with unfiltered data and realistic queries



Measure against human abilities



Supplement with focus on supporting automatic components



Automatic search, High-level feature detection



Shot boundary determination



Integrate and profit from advances in low-level functionality, more narrowly tested



face recognition, text extraction, object recognition, etc

Goals

SLIDE 4

• 13. Nov 2006

TRECVID 2006 4



Answer some questions:



How can systems achieve such retrieval (in collaboration with a human)?



usefulness of generic features



which features most useful?



how/when to combine?



human & system collaboration



who does what?



what is the optimal interface?



How can one reliably benchmark such systems?

Goals

SLIDE 5

• 13. Nov 2006

TRECVID 2006 5

Evolution: data, tasks, participants,...

20 40 60 80 100 120 140 160 180

Hours of development data Hours of test data

10 20 30 40 50 60 70

2001 2002 2003 2004 2005 2006 Applied Finished

NIST

Prelinger archive ABC, CNN, C-Span ABC, CNN

Shot boundaries Shots Shots Shots Shots Shots Search Search Search Search Search Search Features Features Features Features Features Stories Stories BBC rushes BBC rushes Camera motion

Tasks: Data: Participants: English, Chinese, Arabic TV news 10 17 46 39 17 (so far) Peer-reviewed papers: English, Chinese, Arabic TV news

2001 2002 2003 2004 2005 2006

SLIDE 6

• 13. Nov 2006

TRECVID 2006 6



Data:



159 hrs (Nov/Dec.’05 news in Arabic, Chinese, and English)



50 hrs of BBC rushes



3 evaluated tasks (on news data)



Shot boundary determination



High-level feature extraction (39 submitted, 20 evaluated)



Search (automatic, manually-assisted, interactive)



Base scenario: an English-only searcher looking for video in Arabic, Chinese, and/or English



1 exploratory task (on BBC rushes)



Identify and remove redundancy



Organize/present according to useful features



Devise a practical, informative evaluation scheme

Evolution… 2006

SLIDE 7

• 13. Nov 2006

TRECVID 2006 7

More about the 2006 data: News

TR TRECV CVID- D-use se L Lang ng P Prog

• gram

am Hou

• urs

tv tv5 t tv6 6 En Eng NBC BC N NIGH GHTLY LYNEW EWS 9 9.0 tv tv5 t tv6 6 En Eng CNN NN L LIVE VEFRO ROM 14 14.5 tv6 6 En Eng CNN NN C COOP OPER R 8 8.3 tv6 6 En Eng MSN SN N NEWS WSLIV IVE 14 14.5 5

• - 4

46.3 .3 tv tv5 tv tv6 Chi hi C CCTV TV4 DA DAILY LY_NE NEWS S 9. 9.2 tv tv6 Chi hi P PHOE OENIX IX GO GOODM DMORN RNCN N 7. 7.5 tv tv6 Chi hi N NTDT DTV EC ECONF NFRNT NT 7. 7.8 tv tv6 Chi hi N NTDT DTV FO FOCUS USINT NT 5. 5.2

• - 2

29.7 .7 tv tv5 tv tv6 Ara ra L LBC C LB LBCNA NAHAR AR 35. 5.3 tv tv5 tv tv6 Ara ra L LBC C LB LBCNE NEWS 39. 9.5 tv tv6 Ara ra A ALH H HU HURRA RA_NE NEWS S 7. 7.8

• - 8

82.6 .6

• 15

158.6 .6 ho hours rs

Arabic Chinese English

(Some test data programs not represented in the training data) More Arabic than in 2005

SLIDE 8

• 13. Nov 2006

TRECVID 2006 8

More about the data: additional resources



BBN: ASR/MT (GALE system) on Chinese and Arabic



University of Amsterdam: MediaMill challenge data



Columbia U./CMU/IBM - LCSOM: 449 features

SLIDE 9

• 13. Nov 2006

TRECVID 2006 9

More about the data: BBC rushes



50 hours of “French Experience” video; unedited.



Some characteristics:



Mostly just natural sound (including crew noise)



Sometimes with an on-screen host



Lots of redundancy



very loooong shots (e.g.,... sun rising over several minutes)



multiple takes of actor/participant trying to get lines right



more interviews than in 2005



Potential gold mine for reuse after the original production is complete, BUT inaccessible.



Question: What sorts of things, large or small, can software do to help a searcher, unfamiliar with the material, efficiently find out what is there?

SLIDE 10

• 13. Nov 2006

TRECVID 2006 10

Evaluated tasks: 54 finishers

Acce centu ture e Tec echno nolog

• gy L

Labs bs USA SA

• --
• --
• RU

AIIA IA La Labor

• rato

tory Gree eece e S SB -

• - -
• - -
• AT&T

&T La Labs s - R Rese searc rch USA A S SB -

• - S

SE R RU Beij ijing ng Ji Jiaot

• tong

ng U. U. Chin ina

• - -
• - S

SE -

• Bilk

lkent nt U. U. Turk rkey y

• - F

FE S SE -

• Carn

rnegi gie M Mell llon n U. . USA A

• - F

FE S SE -

• Chin

inese se Ac Acade demy y of f Sci cienc nces s (CA CAS/M /MCG) G) Chin ina

• - -
• - -
• - R

RU Chin inese se Ac Acade demy y of f Sci cienc nces s (CA CAS/J /JDL) L) Chin ina S SB -

• - -
• - -
• Chin

inese se U.

• U. of
• f Ho

Hong g Kon

• ng

Chin ina S SB -

• - S

SE -

• City

ty Un Unive versi sity y of f Hon

• ng K

Kong ng (C (City tyUHK HK) Chin ina S SB F FE S SE -

• CLIP

IPS-I

• IMAG

AG Fran ance e S SB F FE S SE -

• Colu

lumbi bia U U. USA A

• - F

FE S SE -

• COST

ST292 92 (w (www. w.cos

• st29

292.o .org) g) *** ** SB B FE E SE E RU Curt rtin n U. . of f Tec echno nolog

• gy

Aus ustra ralia ia SB B --

• --
• RU

U DFKI KI Gm GmbH H Ger erman any

• --
• --
• RU

Doku kuz E Eylu lul U U. Tur urkey ey SB B --

• --
• --

Dubl blin n Cit ity U U. Irel eland nd

• - -
• - S

SE R RU Flor

• rida

da In Inter ernat ation

• nal

l U. USA A S SB -

• - -
• - -
• Fuda

dan U U. Chi hina a

• FE

E SE E --

SLIDE 11

• 13. Nov 2006

TRECVID 2006 11

2006: Extended teams



COST292



LABRI, Bordeaux



Delft University of Technology, Netherlands



Bilkent University



Dublin City University



National Technical University of Athens



Queen Mary, University of London



ITI, Thessaloniki, Greece



University of Belgrade



University of Zilina



University of Bristol

SLIDE 12

• 13. Nov 2006

TRECVID 2006 12

Evaluated tasks: 54 finishers (cont.)

FX Palo Alto Laboratory Inc USA SB FE SE -- Helsinki U. of Technology Finland SB FE SE -- Huazhong U. of Science and Technology China SB -- -- -- IBM T. J. Watson Research Center USA -- FE SE RU Imperial College London / Johns Hopkins U. UK/USA -- FE SE -- Indian Institute of Technology at Bombay India SB -- -- -- NUS / I2R Singapore -- FE SE -- IIT / NCSR Demokritos Greece SB -- -- -- Institut EURECOM France -- FE -- RU Joanneum Research Forschungsgesellschaft Austria -- -- -- RU KDDI/Tokushima U./Tokyo U. of Technology Japan SB FE -- -- Kspace (kspace.qmul.net) *** -- FE SE – Laboratory ETIS Greece SB -- -- -- LIP6 - Laboratoire d'Informatique de Paris 6 France -- FE -- -- Mediamill / U. of Amsterdam the Netherlands -- FE SE -- Microsoft Research Asia China -- FE -- -- Motorola Multimedia Research Laboratory USA SB -- -- -- National Taiwan U. Taiwan -- FE -- --

SLIDE 13

• 13. Nov 2006

TRECVID 2006 13

2006: Extended teams



K-SPACE



Queen Mary University of London



Koblenz University



Joanneum Research Forschungsgesellschaft mbH



Informatics and Telematics Institute



Dublin City University



Centrum voor Wiskunde en Informatica



Groupe des Ecoles des Telecommunications



Institut National de l'Audiovisuel



Institut Eurecom



University of Glasgow



German Research Centre for Artificial Intelligence (DFKI/LT)



Technische University Berlin



Ecole Polytechnique Federale de Lausanne



University of Economics, Prague

SLIDE 14

• 13. Nov 2006

TRECVID 2006 14

Evaluated tasks: 54 finishers (cont.)

NII/ISM Japan -- FE -- -- RMIT U. School of CS&IT Australia SB -- SE -- Tokyo Institute of Technology Japan SB FE -- -- Tsinghua U. China SB FE SE RU

• U. of Bremen TZI Germany -- FE -- --
• U. of California at Berkeley USA -- FE -- --
• U. of Central Florida USA -- FE SE --
• U. of Electro-Communications Japan -- FE -- --
• U. of Glasgow / U. of Sheffield

UK UK

• - FE

FE SE SE --

• U. of Iowa USA -- FE SE --
• U. of Marburg Germany SB -- -- RU
• U. of Modena and Reggio Emilia Italy SB -- -- --
• U. of Ottawa / Carleton U. Canada SB -- -- --
• U. of Oxford UK -- FE SE --
• U. of Sao Paolo

Br Brazi zil SB SB --

• - --
• - --
• U. Rey Juan Carlos Spain SB -- SE RU

Zhejiang U. China SB FE SE --

SLIDE 15

• 13. Nov 2006

TRECVID 2006 15

BBC rushes task: 12 finishers

Ac Accen entur ure T Tecn cnolo logy y Lab abs / / Si Sider erean an So Softw tware re, I Inc. c. US USA AT AT&T T Lab abs R Rese searc rch US USA Ch Chine nese e Aca cadem emy o

• f S

Scie ience ces ( (CAS AS/MC MCG) ) Ch China na CO COST2 T292 2 … Cu Curti tin U Univ iv. Au Austr trali lia DF DFKI I Kai aiser ersla laute tern n Ge Germa many Du Dubli lin C City ty Un Univ.

• v. /

/ Uni niv. . Rey ey Ju Juan n Car arlos

• s

Ir Irela land d / S Spai ain IB IBM US USA In Insti titut ut Eu Eurec ecom m Fr Franc nce Jo Joann nneum um Re Resea earch ch Au Austr tria Ma Marbu burg g Uni niv. . Ge Germa many Ts Tsing nghua ua Un Univ. v. Ch China na

SLIDE 16

• 13. Nov 2006

TRECVID 2006 16

Special thanks…



DTO, NIST/ITL for funding



Christian Petersohn (Fraunhofer (Heinrich Hertz) Institute) for the master shot segmentation



DCU Centre for Digital Video Processing for formatting the master shot reference and selecting the key frames



BBN for donating ASR/MT output for Arabic and Chinese sources



• Univ. of Amsterdam and Univ. of Iowa for helping in

distribution of BBC data



• Univ. of Amsterdam for MediaMill challenge data



Columbia Univ, CMU, IBM for LSCOM features

SLIDE 17

• 13. Nov 2006

TRECVID 2006 17

Agenda: Day 1



Arranged by task



Time for discussion of approaches & evaluation



Monday



High-level features



Possible data for 2007



Lunch



Rushes



Demo/Poster previews



Demos & Posters



Workshop supper

SLIDE 18

• 13. Nov 2006

TRECVID 2006 18

Map: NIST Admin. Building, 1st Floor

Posters Demos Papers “Continental breakfast” & snacks at morning breaks Bus to/from Holiday Inn

(included in the notebook)

Snacks at afternoon breaks Lunch Email

SLIDE 19

• 13. Nov 2006

TRECVID 2006 19

Agenda: Day 2



Tuesday



Shot boundaries



Search



Lunch



TRECVID planning



Reuse of 2003-2006 broadcast news data?



No new distributions



Experiments with LSCOM features? MediaMill baseline?



Increased emphasis on event (object/person/+activity) topics and use of more of the video? (what about speed and volume goals?)



Sound & Vision



Tasks? development data? evaluation?



Rushes (BBC, perhaps Telemadrid)



Tasks? development data? evaluation?



Other items?

SLIDE 20

• 13. Nov 2006

TRECVID 2006 20

Reminders



If you are driving to NIST rather than taking the NIST bus, you don’t need to stop at the Visitor Center tomorrow.



Just show you conference badge and phot ID at the gate as you drive in.



Workshop lunches will be in the main cafeteria.



Choose what you want (except bottled or packaged foods)



Proceed to the cashier and present your ticket



The workshop supper is closeby at Growler’s in Gaithersburg . This is a casual restaurant.



One ticket is included with your registration



You can buy additional tickets at the registration desk



If you don’t plan to attend, please turn in your ticket at the registration desk so someone else can attend.



If you are giving a talk, please have your computer connected or presentation loaded before the session begins.