TRECVID-2006: Search Task Alan Smeaton Dublin City University - PowerPoint PPT Presentation

TRECVID-2006: Search Task Alan Smeaton Dublin City University & Tzveta Ianeva NIST

Search Task Definition  Goal: promote progress in content-based retrieval from digital video via open, metrics-based evaluation;  Given a test collection, a topic and a common shot boundary reference, return a ranked list of at most 1,000 shots which best satisfy the need;  NIST created more topics asking for general (vs. specific)  NIST created 10 of 24 topics to ask for video of events – encouraging exploration beyond one-keyframe-per-shot  Videos were viewed by NIST personnel, notes taken on content, and candidates emerging were chosen; TRECVID 2006 2

Search Task Definition  Per-search measures: average precision, elapsed time  Per-run measure: mean average precision (MAP)  Interactive search participants were asked to have their subjects complete pre, post-topic and post- search questionnaires;  Each result for a topic can come from only 1 user search; same searcher does not need to be used for all topics. TRECVID 2006 3

Search Task Definition  Bing Xiang, John Makhoul, and Ralph Weischedel at BBN for providing MT/ASR  Christian Petersohn (Fraunhofer Institute) for master shot reference  DCU team for formatting and selecting keyframes  MediaMill team for 101 features baseline results donation  CMU and IBM for 449 LSCOM features annotations TRECVID 2006 4

Data characteristics  TRECVid 2006 data is again (deliberately) text- noisy with video from English language, Arabic & Chinese broadcasts;  32.2% of the test video comes from programs not represented in the development data  Text is derived from speech recognition and then machine translation, thus poorer quality than with English-only sources but ASR/MT from “state-of- the-art” GALE system. TRECVID 2006 5

2006: Search task participants (26, up from 20) AT AT&T T Lab abs – – Re Resea earch ch US USA Be Beiji jing g Jia iaoto tong g U. China Ch na Bi Bilke kent t U. Tu Turke key Ca Carne negie ie Me Mello lon U U. USA US Ch Chine nese e U. . of f Hon ong K Kong ng Ch China na Ci City y Uni niver ersit ity o of H Hong ng Ko Kong China Ch na CLIPS CL PS-IM IMAG Franc Fr nce Co Colum umbia ia U. U. US USA Du Dubli lin C City ty U. U. Ir Irela land d Fu Fudan an U. U. Ch China na FX FX Pa Palo o Alt lto L Labo borat atory ry In Inc US USA He Helsi sinki ki U. U. of of Te Techn hnolo logy Fi Finla land IB IBM T T. J J. W Wats tson n Res esear arch h Cen enter er US USA Im Imper erial al Co Colle lege e Lon ondon on / / Joh ohns s Hop opkin ins U U. UK, U UK USA TRECVID 2006 6

2006: Search task participants (continued) NU NUS / / I2 I2R Si Singa gapor ore Media Me iamil ill / / U. U. of of Am Amste terda dam Ne Nethe herla lands ds RM RMIT T U. . Sch chool ol of of CS CS&IT IT Au Austr trali lia Ts Tsing nghua ua U. U. Ch China na U. U. of of Ce Centr tral l Flo lorid ida US USA U. U. of of Gl Glasg sgow w / U U. o of S Shef effie ield UK UK U. of U. of Io Iowa US USA U. U. of of Ox Oxfor ord UK UK U. Re U. Rey J Juan an Ca Carlo los Spain Sp in Zh Zheji jiang ng U. U. Ch China na CO COST2 T292 2 (ww www.c .cost st292 92.or org) Fr Franc nce, , Net ether erlan ands, s, UK UK, I Irel eland nd, G Gree eece, e, Tu Turke key, , Se Serbi bia a and d Mon onten enegr gro, , Slo lovak akia K- K-Spa pace e (ks kspac ace.q .qmul ul.ne net) UK, G UK Germ rmany ny, A Aust stria ia, G Gree eece, e, Irela Ir land, d, Ne Nethe herla lands ds, F Fran ance, e, Sw Switz tzerl rland nd, C Czec echia ia TRECVID 2006 7

Search Types: Automatic, Manual and Interactive Number of runs: 76 automatic 11 manually assisted 36 interactive TRECVID 2006 8

Everybody likes to search automatically, dislikes manually 100% 80% Interactive 60% Manual Fully automatic 40% 20% 0% 2004 2005 2006 TRECVID 2006 9

24 Topics [ number of image, video examples and relevant found] 173. Find shots with a view of one or more tall buildings (more than 4 stories) and the top story visible [3, 4, 142] 174. Find shots with one or more people leaving or entering a vehicle [0, 10, 675] 175. Find shots with one or more soldiers, police, or guards escorting a prisoner [0, 4, 204] 176. Find shots of a daytime demonstration or protest with at least part of one building visible [4, 4, 111] 177. Find shots of US Vice President Dick Cheney [3, 3, 393] 178. Find shots of Saddam Hussein with at least one other person's face at least partially visible [8, 0, 99] 179. Find shots of multiple people in uniform and in formation [3, 5, 191] 180. Find shots of US President George W. Bush, Jr. walking [0, 5, 197] TRECVID 2006 10

24 Topics [ number of image, video examples and relevant found] 181. Find shots of one or more soldiers or police with one or more weapons and military vehicles [2, 6, 128] 182. Find shots of water with one or more boats or ships [3, 5, 307] 183. Find shots with one or more emergency vehicles in motion (e.g., ambulance, police car, fire truck, etc.) [0, 4, 299] 184. Find shots of one or more people seated at a computer with display visible [3, 4, 440] 185. Find shots of one or more people reading a newspaper [3, 4, 201] 186. Find shots of a natural scene - with, for example, fields, trees, sky, lake, mountain, rocks, rivers, beach, ocean, grass, sunset, waterfall, animals, or people; but no buildings, no roads, no vehicles [2, 4, 523] 187. Find shots of one or more helicopters in flight [0, 6, 119] TRECVID 2006 11

24 Topics [ number of image, video examples and relevant found] 188. Find shots of something burning with flames visible [3, 5, 375] 189. Find shots of a group including at least four people dressed in suits, seated, and with at least one flag [3, 5, 446] 190. Find shots of at least one person and at least 10 books [3, 5, 295] 191. Find shots containing at least one adult person and at least one child [3, 6, 775] 192. Find shots of a greeting by at least one kiss on the cheek [0, 5, 98] 193. Find shots of one or more smokestacks, chimneys, or cooling towers with smoke or vapor coming out [3, 2, 60] 194. Find shots of Condoleezza Rice [3, 7, 122] 195. Find shots of one or more soccer goalposts [3, 4, 333] 196. Find shots of scenes with snow [3, 6, 692] TRECVID 2006 12

Some statistics  2006:  Number of shots in test collection: 79.484 79.484  ~9.1% ~9.1% relevant shots found: 7.225 7.225  2005  Number of shots in test collection: 45.765 45.765  ~18.3% relevant shots found: 8.395 ~18.3% 8.395  2004  Number of shots in test collection: 33.367 33.367  ~5.4% ~5.4% relevant shots found: 1.800 1.800  2003  Number of shots in test collection: 32.318 32.318  ~6.5% relevant shots found: 2.114 ~6.5% 2.114 TRECVID 2006 13

2006: 20 sites contributed one or more unique, relevant shots 70 Number of unique, relevant shots 60 50 40 30 20 10 0 G U a s m . g t g . 2 n e d y . M a . e U U U U n M o c d w o r n A n 9 r g a e B o l u o n 2 d a o a i w o d g M C o r r k f I p T p o a t K a n l u I r n l x o o i I S s o C e a d o h l f a B - O g a n F o t g S g K i u O L g s n s j i h n P n F f n l . e J C a m e o c a U a i o h I i S g L r l u g e s G A H . t Z n C e U T T n J f i l . o f e U j l f y o o i o C e e . C / U . R B . U f U m o l l . a a e U a i . n i s k U e r o e n e T p i i n t s l a i m i h l m N e I C H a / i d R e 2 M I TRECVID 2006 14

2006: Rel shots contrib. uniquely per topic by team 9 28 14 8 12 19 6 5 11 16 8 5 7 7 1 4 10 2 4 6 3 1 5 4 2 2 1 8 2 2 2 1 2 3 2 Number of 7 3 1 2 2 1 1 2 1 1 2 6 1 4 unique 1 2 2 2 1 1 2 1 2 1 1 1 true 2 3 1 4 1 2 7 1 1 1 1 2 1 1 1 3 1 2 5 1 shots 1 1 1 1 1 1 1 3 1 1 2 2 1 9 1 3 1 1 2 1 5 1 2 1 1 9 1 1 ( 1 1 3 1 2 1 3 1 1 3 0 9 ( 3 1 1 6 1 ) 0 1 8 1 1 1 ( 1 ) 9 7 g 3 1 n 1 7 t 8 n 1 o 1 1 2 3 ( 5 t e G 7 4 a k A 1 ) i U l 1 4 J i M 1 M 8 B g ( g 6 I C n . 5 1 n _ U 1 ) S o 1 i 2 j 1 i P K n 8 e 9 9 a ( I 2 n B L g 3 2 d o ) T e 1 C n u S d c 0 ( o F d 1 8 O n a H 2 1 p r Topic (total relevant) o 1 C o y L s 9 ) f f g 1 o . K x U e o 9 O 7 l M ( . g ) U o a 9 1 e f B n u a e l o 1 2 l h h I d o . s . c g i U 7 8 e C U r ( a e n o n w 7 1 ) T i l w i l a s F s 9 h o o 1 f T o C i g r o l I m 7 1 e a l ( s r . f 3 p r o a a . 5 ) U t a U C d m n l 9 G . 1 i U r e e g 3 I k n e 7 C r ( n t n a u s a 3 2 ) Group i u p s f m o i 0 J j a l e e A . g 4 H U y h ( e n . Z 1 ) U i R S 4 / . U f 2 m o ) a . e U 186, 191, 196 have 500+ T l a l i n m o a i t i a d N e M / R 2 I TRECVID 2006 15