TRECVID-2014 Semantic Indexing task: Overview Georges Qunot - - PowerPoint PPT Presentation

TRECVID-2014 Semantic Indexing task: Overview

Georges Quénot Laboratoire d'Informatique de Grenoble George Awad Dakota Consulting, Inc

Outline

• Task summary (Goals, Data, Run types, Metrics)
• Evaluation details
• Inferred average precision
• Participants
• Evaluation results
• Hits per concept
• Results per run
• Results per concept
• Significance tests
• Progress task results
• Localization subtask results
• Global Observations
• Issues

2

Semantic Indexing task

• Goal: Automatic assignment of semantic tags to video segments (shots)
• Secondary goals:
• Encourage generic (scalable) methods for detector development.
• Semantic annotation is important for filtering, categorization, searching and

browsing.

• Participants submitted four types of runs:
• Main run Includes results for 60 concepts, from which NIST evaluated 30
• Localization run includes results for 10 pixel-wise localized concepts from the 60

evaluated concepts in main runs.

• Progress run Includes results for 60 concept for 2 non-overlapping datasets,

from which 1 dataset will be evaluated the next year.

3

Semantic Indexing task (data)

• SIN testing dataset
• Main test set (IACC.2.B): 200 hours, with durations between 10 seconds and 6

minutes.

• Progress test set (IACC.2.C): 200 hours and non overlapping from IACC.2
• SIN development dataset
• (IACC.1.A, IACC.1.B, IACC.1.C & IACC.1.tv10.training): 800 hours, used from

2010 – 2012 with durations between 10 seconds to just longer than 3.5 minutes.

• Total shots:
• Much more than in previous TRECVID years, no composite shots
• Development: 549,434
• Test: IACC.2.A (112,677), IACC.2.B (106,913), IACC.2.C (113,161)
• Common annotation for 346 concepts coordinated by

LIG/LIF/Quaero from 2007-2013 made available.

4

Semantic Indexing task (Concepts)

• Selection of the 60 target concepts
• Were drawn from 500 concepts chosen from the TRECVID “high

level features” from 2005 to 2010 to favor cross-collection experiments Plus a selection of LSCOM concepts so that:

• we end up with a number of generic-specific relations among them

for promoting research on methods for indexing many concepts and using ontology relations between them.

• we cover a number of potential subtasks, e.g. “persons” or “actions”

(not really formalized)

• It is also expected that these concepts will be useful for the content-

based (instance) search task.

• Set of relations provided:
• 427 “implies” relations, e.g. “Actor implies Person”
• 559 “excludes” relations, e.g. “Daytime_Outdoor excludes

Nighttime”

5

Semantic Indexing task (training types)

• Six training types were allowed:
• A – used only IACC training data (42 runs)
• B – used only non-IACC training data (0 runs)
• C – used both IACC and non-IACC TRECVID (S&V and/or

Broadcast news) training data (0 runs)

• D – used both IACC and non-IACC non-TRECVID training

data(29 runs)

• E – used only training data collected automatically using only the

concepts’ name and definition (4 runs)

• F – used only training data collected automatically using a query

built manually from the concepts’ name and definition (0 runs)

6

Semantic Indexing task (training types)

• Stricter interpretation of type A since 2014:
• Use of components built using other training data (e.g. face detectors) was

considered as acceptable as long as this was not for directly training the SIN target concepts (no sample directly annotated with SIN concepts used)

• Generalization to the use of components like semantic descriptors trained on

external data (e.g. ImageNet) was similar in principle but too close to type D

• Partially re-trained deep networks are even closer
• Many runs submitted in 2013 and earlier as type A would be

now requalified as type D with the new interpretation (not a problem)

• Results are now presented in a single table and plot for types

A-D (the training type still appear un the run names)

7

30 Single concepts evaluated(1)

3 Airplane* 9 Basketball 10 Beach* 13 Bicycling 15 Boat_Ship* 17 Bridges* 19 Bus* 25 Chair* 27 Cheering 29 Classroom 31 Computers* 41 Demonstration_Or_Protest 59 Hand* 63 Highway 71 Instrumental_Musician* 80 Motorcycle* 83 News_Studio* 84 Nighttime 100 Running* 105 Singing* 112 Stadium 117 Telephones* 163 Baby* 261 Flags* 267 Forest* 274 George_Bush* 321 Lakes 359 Oceans 392 Quadruped* 434 Skier

• The 19 marked with “*” are a subset of those tested in 2013

8

10 Localization Concepts evaluated (2)

• [3] Airplane
• [15] Boat_ship
• [17] Bridges
• [19] Bus
• [25] Chair
• [59] Hand
• [80] Motorcycle
• [117] Telephones
• [261] Flags
• [392] Quadruped

9

Evaluation

• Task: Find shots that contain a certain concept, rank them

according to confidence measure, submit the top 2000.

• The 30 evaluated single concepts were chosen after

examining TRECVid 2013 60 evaluated concept scores across all runs and choosing the top 45 concepts with maximum score variation.

• Each feature assumed to be binary: absent or present for

each master reference shot

• NIST sampled ranked pools and judged top results from all

submissions

• Metrics: inferred average precision per concept
• Compared runs in terms of mean inferred average precision

across the 30 concept results for main runs.

10

Inferred average precision (infAP)

• Developed* by Emine Yilmaz and Javed A. Aslam at

Northeastern University

• Estimates average precision surprisingly well using a

surprisingly small sample of judgments from the usual submission pools

• More features can be judged with same effort
• Increased sensitivity to lower ranks
• Experiments on previous TRECVID years feature

submissions confirmed quality of the estimate in terms of actual scores and system ranking

* J.A. Aslam, V. Pavlu and E. Yilmaz, Statistical Method for System Evaluation Using Incomplete Judgments Proceedings of the 29th ACM SIGIR Conference, Seattle, 2006.

11

2014: mean extended Inferred average precision (xinfAP)

• 2 pools were created for each concept and sampled as:
• Top pool (ranks 1-200) sampled at 100%
• Bottom pool (ranks 201-2000) sampled at 11.1%
• Judgment process: one assessor per concept, watched

complete shot while listening to the audio.

• infAP was calculated using the judged and unjudged pool by

sample_eval

30 concepts 191,717 total judgments 12248 total hits 7938 Hits at ranks (1-100) 2869 Hits at ranks (101-200) 1441 Hits at ranks (201-2000)

12

2014 : 15 Finishers

CMU Carnegie Mellon U. CRCV_UCF University of Central Florida EURECOM EURECOM - Multimedia Communications FIU_UM Florida International U., U. of Miami Insightdcu Insight Centre for Data Analytics IRIM CEA-LIST, ETIS, EURECOM, INRIA, LABRI, LIF, LIG, LIMSI, LIP6, LIRIS, LISTIC ITI_CERTH Information Technologies Institute, Centre for Research and Technology Hellas LIG Laboratoire d'Informatique de Grenoble MediaMill

• U. of Amsterdam

OrangeBJ Orange Labs International Center Beijing PicSOM Aalto U. PKUSZ_ELMT Peking University Engineering Laboratory of 3D Media Technology TokyoTech-Waseda Tokyo Institute of Technology, Waseda University UEC U. of Electro-Communications VIREO City U. of Hong Kong 13

500 1000 1500 2000 2500 3000

14

Inferred frequency of hits varies by concept

1%** **from total test shots

Total true shots contributed uniquely by team

Team

• No. of

Shots Team

• No. of

shots Insightdcu 81 Mediamill 6 UEC 34 PKUSZ_ELMT 3 CMU 32 VIREO 2 EURECOM 24 LIG 1 OrangeBJ 22 ITI_CERTH 19 HFUT* 16 FIU_UM 15 NHKSTRL* 13 NII* 13 CRCV_UCF 11 Picsom 11 TokyoTech-Waseda 4

Fewer unique shots compared to TV2013 & TV2012

*shots submitted in 2013 in progress task

1 5

0.05 0.1 0.15 0.2 0.25 0.3 0.35

D_MediaMill.14_1 D_MediaMill.14_2 D_MediaMill.14_3 A_MediaMill.14_4 D_PicSOM.14_1 D_PicSOM.14_3 D_TokyoTech-Waseda.14_1 D_TokyoTech-Waseda.14_2 nist.baseline.14 D_PicSOM.14_2 D_LIG.14_3 D_LIG.14_4 A_TokyoTech-Waseda.14_3 A_TokyoTech-Waseda.14_4 D_LIG.14_2 D_IRIM.14_2 D_IRIM.14_1 D_LIG.14_1 D_IRIM.14_4 A_CMU.14_1 D_IRIM.14_3 A_CMU.14_3 D_OrangeBJ.14_4 A_CMU.14_2 A_CMU.14_4 D_VIREO.14_2 D_EURECOM.14_1 A_ITI_CERTH.14_1 D_OrangeBJ.14_2 A_ITI_CERTH.14_2 D_EURECOM.14_2 D_VIREO.14_1 A_PicSOM.14_4 A_ITI_CERTH.14_3 A_OrangeBJ.14_1 D_UEC.14_1 D_CRCV_UCF.14_3 A_EURECOM.14_3 D_CRCV_UCF.14_2 D_UEC.14_2 D_CRCV_UCF.14_1 D_OrangeBJ.14_3 A_EURECOM.14_4 D_CRCV_UCF.14_4 A_UEC.14_3 A_ITI_CERTH.14_4 E_insightdcu.14_1 E_insightdcu.14_2 A_insightdcu.14_1 E_CMU.14_1 E_CMU.14_2 A_PKUSZ_ELMT.14_2 A_PKUSZ_ELMT.14_1 A_FIU_UM.14_4

Main runs scores – 2014 submissions

Median = 0.217

Mean InfAP. NIST median baseline run

16 Type D runs Type A runs (only IACC for training) Type E runs (no annotation)

0.05 0.1 0.15 0.2 0.25 0.3 0.35

D_MediaMill.14_1 D_MediaMill.14_2 D_MediaMill.14_3 A_MediaMill.14_4 D_PicSOM.14_1 D_PicSOM.14_3 D_TokyoTech-Waseda.14_1 D_TokyoTech-Waseda.14_2 nist.baseline.14 D_PicSOM.14_2 D_LIG.14_3 D_LIG.14_4 A_TokyoTech-Waseda.14_3 A_TokyoTech-Waseda.14_4 D_LIG.14_2 D_IRIM.14_2 D_IRIM.14_1 D_LIG.14_1 D_IRIM.14_4 A_CMU.14_1 D_IRIM.14_3 A_LIG.13_3 A_LIG.13_1 A_CMU.14_3 A_IRIM.13_1 A_VideoSense.13_4 D_OrangeBJ.14_4 A_CMU.14_2 A_CMU.14_4 D_VIREO.14_2 D_EURECOM.14_1 A_inria.lear.13_8 A_inria.lear.13_5 A_axes.13_8 A_axes.13_5 A_inria.lear.13_2 A_axes.13_2 A_ITI_CERTH.14_1 A_IRIM.13_2 D_OrangeBJ.14_2 A_ITI_CERTH.14_2 D_EURECOM.14_2 D_VIREO.14_1 A_PicSOM.14_4 A_ITI_CERTH.14_3 A_OrangeBJ.14_1 D_UEC.14_1 D_CRCV_UCF.14_3 A_NII.13_1 A_EURECOM.14_3 A_NII.13_2 D_CRCV_UCF.14_2 D_UEC.14_2 D_CRCV_UCF.14_1 D_OrangeBJ.14_3 A_EURECOM.14_4 A_ITI_CERTH.13_6 A_ITI_CERTH.13_5 D_CRCV_UCF.14_4 A_UEC.14_3 A_NHKSTRL.13_3 A_insightdcu.13_1 A_ITI_CERTH.14_4 E_insightdcu.14_1 A_UEC.13_2 E_insightdcu.14_2 A_insightdcu.14_1 A_HFUT.13_2 A_EURECOM.13_1 A_EURECOM.13_2 E_CMU.14_1 E_CMU.14_2 A_PKUSZ_ELMT.14_2 A_PKUSZ_ELMT.14_1 A_FIU_UM.14_4 A_FIU_UM.14_3

Median = 0.206

Mean InfAP. NIST median baseline run

* Submitted runs in 2013 against 2014 testing data (Progress runs)

Higher median and max scores than 2013 

17

Main runs scores – Including progress

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Airplane* Basketball Beach* Bicycling Boat_ship* Bridge* Bus* Chair* Cheering Classroom Computers* Demonstration Hand* Highway Instrument_Musician* Motorcycle* News_Studio* Nighttime Running* Singing* Stadium Telephones* Baby* Flags* Forest* George_Bush* Lakes Oceans Quadruped* Skier

Series1 Series2 Series3 Series4 Series5 Series6 Series7 Series8 Series9 Series1 Median 18

Top 10 InfAP scores by concept (Main runs)

Inf AP.

* Common concept in TV2013

Most common concept medians are higher than TV13 medians

Statistical significant differences among top 10 Main runs (using randomization test, p < 0.05)

• Run name

(mean infAP) D_MediaMill.14_1 0.332 D_MediaMill.14_2 0.331 D_MediaMill.14_3 0.319 A_MediaMill.14_4 0.316 D_PicSOM.14_1 0.288 D_PicSOM.14_3 0.284 D_TokyoTech-Waseda.14_1 0.281 D_TokyoTech-Waseda.14_2 0.280 D_PicSOM.14_2 0.272 D_LIG.14_3 0.266 D_MediaMill.14_1 D_MediaMill.14_3 D_TokyoTech-Waseda.14_2 D_TokyoTech-Waseda.14_1 D_LIG.14_3 D_PicSOM.14_1 D_PicSOM.14_3 D_PicSOM.14_2 D_LIG.14_3 D_MediaMill.14_4 D_TokyoTech-Waseda.14_2 D_TokyoTech-Waseda.14_1 D_LIG.14_3 D_PicSOM.14_1 D_PicSOM.14_3 D_PicSOM.14_2 D_LIG.14_3 D_MediaMill.14_2 D_MediaMill.14_3 D_TokyoTech-Waseda.14_2 D_TokyoTech-Waseda.14_1 D_LIG.14_3 D_PicSOM.14_1 D_PicSOM.14_3 D_PicSOM.14_2 D_LIG.14_3 D_MediaMill.14_4 D_TokyoTech-Waseda.14_2 D_TokyoTech-Waseda.14_1 D_LIG.14_3 D_PicSOM.14_1 D_PicSOM.14_3 D_PicSOM.14_2 D_LIG.14_3

19

Progress subtask

• Measuring progress of 2013 vs 2014 systems on

IACC.2.B dataset.

• 2014 systems used same training data and

annotations as in 2013.

• Total 6 teams submitted progress runs against

IACC.2.B dataset.

20

Progress subtask: Comparing best runs in 2013 & 2014 by team

0.05 0.1 0.15 0.2 0.25 0.3 EURECOM IRIM ITI_CERTH LIG UEC insightdcu

Mean InfAP

2013_system 2014_system

Randomization tests show that 2014 systems are better than 2013 systems (except for insightdcu)

21

0.05 0.1 0.15 0.2 0.25 0.3 0.35

A_UvA-Robb A_UvA-Bran A_Quaero-2013-3 A_TokyoTechCanon A_TokyoTechCanon A_Quaero-2013-1 A_IRIM-2013-1 A_IRIM-2013-2 A_IRIM-2013-4 A_axes.2013v2 A_axes.lf.3.chan A_CMU_Bart A_PicSOM_M_1 A_FTRDBJ-M2 A_FTRDBJ-M3 A_NTT_DUT_1 A_ITI-CERTH A_vireo.Baseline+DNN A_Kitty.13A2 A_ITI-CERTH A_IBM_4 A_dcu_savasa A_NHKSTRL1 A_vireo.DNN A_NTT_DUT_2 A_dcu_savasa A_NTT_DUT_3 A_FIU-UM-1 A_FIU-UM-2 A_NHKSTRL2 A_sriaurora.UCF_CRCV4 A_HFUT A_JRS4 A_FHHI_Base_GCB_SA A_EURECOM_ECU A_JRS3 A_TOSCA1 A_MindLABOMF_3 A_MindLABOMF_1 A_EURECOM_C A_TOSCA4 A_JRS2 A_VideoSense-2013-1 A_sheffield A_sheffield

Median = 0.128

Mean InfAP.

Main runs scores 2013

0.05 0.1 0.15 0.2 0.25 0.3 0.35

D_MediaMill.14_1 D_MediaMill.14_2 D_MediaMill.14_3 A_MediaMill.14_4 D_PicSOM.14_1 D_PicSOM.14_3 D_TokyoTech-Waseda.14_1 D_TokyoTech-Waseda.14_2 D_PicSOM.14_2 D_LIG.14_3 D_LIG.14_4 A_TokyoTech-Waseda.14_3 A_TokyoTech-Waseda.14_4 D_LIG.14_2 D_IRIM.14_2 D_IRIM.14_1 D_LIG.14_1 D_IRIM.14_4 A_CMU.14_1 D_IRIM.14_3 A_LIG.13_3 A_LIG.13_1 A_CMU.14_3 A_IRIM.13_1 A_VideoSense.13_4 D_OrangeBJ.14_4 A_CMU.14_2 A_CMU.14_4 D_VIREO.14_2 D_EURECOM.14_1 A_axes.inria.lear.13_8 A_axes.inria.lear.13_5 A_axes.inria.lear.13_2 A_ITI_CERTH.14_1 A_IRIM.13_2 D_OrangeBJ.14_2 A_ITI_CERTH.14_2 D_EURECOM.14_2 D_VIREO.14_1 A_PicSOM.14_4 A_ITI_CERTH.14_3 A_OrangeBJ.14_1 D_UEC.14_1 D_CRCV_UCF.14_3 A_NII.13_1 A_EURECOM.14_3 A_NII.13_2 D_CRCV_UCF.14_2 D_UEC.14_2 D_CRCV_UCF.14_1 D_OrangeBJ.14_3 A_EURECOM.14_4 A_ITI_CERTH.13_6 A_ITI_CERTH.13_5 D_CRCV_UCF.14_4 A_UEC.14_3 A_NHKSTRL.13_3 A_insightdcu.13_1 A_ITI_CERTH.14_4 E_insightdcu.14_1 A_UEC.13_2 E_insightdcu.14_2 A_insightdcu.14_1 A_HFUT.13_2 A_EURECOM.13_1 A_EURECOM.13_2 E_CMU.14_1 E_CMU.14_2 A_PKUSZ_ELMT.14_2 A_PKUSZ_ELMT.14_1 A_FIU_UM.14_4 A_FIU_UM.14_3

Main runs scores 2014

Median = 0.206

Mean InfAP.

23

Progress subtask: Concepts improved vs weaken by team

5 10 15 20 25 30 35 EURECOM IRIM ITI_CERTH LIG UEC insightdcu

Concepts

Total_improved Total_weaken

Most 2014 concepts improved

24

Concept localization subtask

• Goal
• Make concept detection more precise in time and space

than current shot-level evaluation.

• Encourage more reusable concept detectors design that is

independent from the context.

• Task
• For each of the 10 concepts
• For each of the top 1000 main run shots in SIN run
• For each I-Frame within the shot that contains the target, return
• the x,y coordinates of the (UL,LR) vertices of a bounding rectangle

containing all of the target concept and as little more as possible.

• Systems were allowed to submit more than 1 bounding box per I-frame

but only the one with maximum fscore were judged.

25

NIST Evaluation framework

Concept exists in shot (TP) Concept not in shot (FP)

171k I-frames

Sampling (select every 3rd I-frame)

57k I-frames

Concept exists in I-frame (TP) Concept not in I-frame (FP)

Draw bounding box SIN human assessors Localization human assessors

Semantic Indexing Assessment phase Concept Localization Assessment phase

26

Evaluation metrics

• Temporal localization: precision, recall and fscore

based on the judged I-frames.

• Spatial localization: precision, recall and fscore

based

• n

the located pixels representing the concept.

• An average of precision, recall and fscore for

temporal and spatial localization across all I-frames for each concept and for each run.

27

Participants (Finishers)

• 16 teams applied, only 1 team submitted 4 runs!
• UvA (University Of Amsterdam)

28

Temporal localization results by run

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Mean per run across all concepts I-frame Fscore I-frame Precision I-frame Recall

29

Spatial Localization results by run

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Mean per run across all concepts Mean Pixel Fscore Mean Pixel Precision Mean Pixel Recall Spatial localization seems to be better than temporal (contrary to 2013 results). Hard to conclude as all runs come from 1 team

30

TP vs FP submitted I-frames by run

10 20 30 40 50 60 70 80 Mean TP I-frames per shot across all concepts Mean FP I-frames per shot across all concepts

31

How can systems find the right balance between TP vs FP I-frames ?

Mean I-frames per shot

Results per concept

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 F-score

Median 4 3 2 1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Mean F-score Median 4 3 2 1

Temporal localization Spatial localization Most concepts are better in spatial localization compared to temporal. However, 1 team runs are not enough to conclude!

32

Samples of good localization

GT Sys

33

Samples of less good localization

Sys GT

34

Results per concept across all runs

0.1 0.2 0.3 0.4 0.5 0.6 0.7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Recall per concept Precision per concept

Temporal localization

0.1 0.2 0.3 0.4 0.5 0.6 0.7

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Mean Recall per concept Mean precision per concept

Spatial localization

submitted bounding boxes approximate G.T boxes in size with some overlap. Systems are good in finding the real box sizes, not so much the real position.

Flags Hand Motorcycle Airplane

Submissions missed a lot of TP I-frames in general.

35

2014 Observations

• 2014 main task was harder than 2013 main task (different

data and different set of target concepts)

• Raw system scores have higher Max and Median compared

to TV2013, still relatively low.

• Most common concepts with TV2013 have higher median

scores.

• Most Progress systems improved significantly from 2013 to

2014.

• Significantly less participants (15 versus 26 for TV2013),

most of the loss is in the “long tail”, partly explaining why the median performance is higher even though the task is harder.

• Localization runs missed a lot of TP I-frames.
• Localization submitted boxes approximate true box sizes with

some overlap.

36

2014 Observations

• Approaches similar to TV 2013 with many innovations
• Improved bag of visual words, many dense and pyramidal
• Fisher vectors and similar (VLAD, VLAT, SV...)
• Use of several key frames per shot
• Use of audio features (MFCC+)
• Use of trajectory-based features
• Encoding of spatial information in Fisher vectors
• More semantic features
• Pseudo-relevance feedback
• More deep learning, co-training with ImageNet
• Use of hidden layers in deep convolutional networks
• Fast Local Area Independent Representation for localization
• Hard negative mining

37

SIN 2015

• Globally keep the task similar and of similar

scale, test on IACC.2.C

• Further explore the “no annotation” and

“localization” variants

• Sharing of data still proposed by IRIM
• Method for measuring progress over years

→ more progress submission are encouraged → we may accept late 2013-2014 progress submissions for a better progress analysis

• Collaborative annotation unchanged
• Feedback welcome

Extra slides for reference

Motivation for xinfAP and pooling strategy

• to make the evaluation more sensitive to shots returned

below the lowest rank (~100) previously pooled and judged

• to adjust the sampling to match the relative importance of

highest ranked items to average precision

• to exploit more infAP’s ability to estimate of AP well even

at sampling rates much below the 50% rate used in previous years

NIST median baseline run by NIST

• A median baseline run is created for each run type and

training category.

• Basic idea:
• For each feature, find the median rank of each submitted shot

calculated across all submitted runs in that run type and training category.

• The final shot median rank value is weighted by the ratio of all

submitted runs to number of runs that submitted that shot:

dX nsSubmitte NumberOfRu rOfRuns TotalNumbe rank Median ShotX

rank Median

* _

_



Sharing of data for TRECVID SIN

• Organized by the IRIM groups of CNRS GRD ISIS.
• IRIM proposes its data sharing organization for the

TRECVID SIN task. This comprises:

• a wiki with read-write access for all
• a data repository with read access for all and currently a write

access only via one of the organizers

• a small set of simple file formats
• a (quite) simple directory structure
• Shared data mostly consist in descriptors and

classification scores.

• Rewarding principle (same as for other contributions)
• share and be cited and evaluated
• use freely and cite

Sharing of data for TRECVID SIN

• Wiki (write access with trecvid active participant

login/password):

• http://mrim.imag.fr/trecvid/wiki
• http://mrim.imag.fr/trecvid/wiki/doku.php?id=sin_2013_task
• Associated data for SIN 2010-2015 (access to some

parts with IACC collection login/password):

• http://mrim.imag.fr/trecvid/sin12
• Related actions:
• Sharing of low-level descriptors by CMU for TRECVID 2003-

2004

• Mediamill challenge (101 concepts) using TRECVID 2005 data
• Sharing of detection scores by CU-Vireo on TRECVID 2008-

2010 data

• Possible extension to other TRECVID tasks, e.g. MED.
• Currently needs update, announced when finished