CLEF and P CLEF and P PROMISEs PROMISEs Nicola a Ferro - - PowerPoint PPT Presentation

CLEF and P CLEF and P

Nicola

Information Management Sys Information Management Sys Department of Infor University of y

PROMISEs PROMISEs

a Ferro

stems (IMS) Research Group stems (IMS) Research Group rmation Engineering Padua, Italy

CLEF “Cl i ” CLEF “Classic” The CLEF Initiative The CLEF Initiative From CLEF 2010 to Surrounding CLEF: P g CLEF 2012 PROMISE

CLEF “Classic”

1997 – First CLIR system evalua TREC and NTCIR

CLEF actually began life in 1997 as Retrieval (CLIR) within TREC. Mainl

2000-2009 – CLIR evaluation in 2000 2009 CLIR evaluation in track at TREC)

Fully multilingual multimodal informa Fully multilingual, multimodal informa query in any medium and any langua multilingual multimedia collection co d ti it i th t l t lik and presenting it in the style most lik

Funding

DELOS NoE under FP5 2000 – 2003 DELOS NoE under FP6 2004 – 2007 T bl CLEF d FP7 2008 2009 (

ation campaigns in US and Japan:

a track for Cross Language Information y, English centered tasks (EN -> X, X -> EN

Europe: CLEF (extension of CLIR Europe: CLEF (extension of CLIR

ation retrieval systems capable of processin ation retrieval systems capable of processin age finding relevant information from a ntaining documents in any language and for k l t b f l t th kely to be useful to the user 3 (http://delos-noe.isti.cnr.it/) ( p ) 7 (http://www.delos.info/) (htt // t bl l f /)

Stimulation of research act unexplored areas unexplored areas Study and implementation Study and implementation for diverse types of cross-l Creation of a large set of e multilingual information acc g Quantitative and qualitative ti i l practice in cross-language Creation of reusable test c Creation of reusable test c benchmarking Building of a strong, multid tivity in new, previously

• f evaluation methodologies
• f evaluation methodologies

anguage IR systems empirical data about cess from the user perspectiv p p e evidence with respect to be t d l t system development collections for system collections for system disciplinary research commun

Ch i Changes in:

users: they increasingly interac users: they increasingly interac

• rganizations: they need to ma

t t d ff i / content and offer services/acce

Growing dissatisfaction with cu Growing dissatisfaction with cu

Future evaluation campaigns systems that better meet user n Multilingual issues must be st Multilingual issues must be st perspective

The MLIA/CLIR user model mu emerging trends g g

ct with content and other users ct with content and other users anage multilingual (versioned) t it ss to it

urrent available technology urrent available technology

must foster development of needs udied also from communicative udied also from communicative

ust now be adapted to meet

CLEF must offer a new evaluat CLEF must offer a new evaluat

Methodology definition: devel d ib d d b h i f describe needs and behavior of tasking users; System building: assessing sy identified user needs, tasks, and , , Results assessment: measurin component performance includi component performance includi user satisfaction Community building: involving information science and user st communities, e.g enterprise sea cultural heritage and infotainme Validation of technology: gua

ion cycle impacting on: ion cycle impacting on:

loping models and metrics to f th lti lt l d lti f the new multicultural and multi- ystem conformity wrt the newly d models; ng all aspects of system & ng response times usability and ng response times, usability, and g other research domains, e.g. MT udies sectors, & application arch, legal, patent, educational, ent areas; ranteeing that the results obtained

multilingual and multimodal system testi investigation of the use of unstructured, semantically enriched data in informatio creation of reusable test collections for b exploration of new evaluation methodolo experimental data; discussion of results comparison of app ing, tuning and evaluation; semi-structured, highly-structured, and

• n access;

benchmarking;

• gies and innovative ways of using

proaches exchange of ideas and transf

he CLEF Initiative is struct a series of Evaluation L conduct evaluation of inf conduct evaluation of inf and workshops to discus evaluation activities; i d C f a peer-reviewed Confer issues, including issues, including

investigation continuing the experiments using multiling particular, but not only, data p y research in evaluation meth

tured in two main parts: Labs, i.e. laboratories to formation access systems formation access systems ss and pilot innovative b d f rence on a broad range of

e activities of the Evaluation Labs gual and multimodal data; in a resulting from CLEF activities; g hodologies and challenges.

Organiz zation

Comm

Steering Committee Chair Steering Committee Chair

Nicola Ferro, University of Padua, Italy

Deputy Steering Committee Chair for

Julio Gonzalo, National Distance Educatio

Deputy Steering Committee Chair for

Carol Peters, ISTI, National Council of Re

Members Members

Martin Braschler, Zurich University of App Khalid Choukri Evaluations and Languag Khalid Choukri, Evaluations and Languag Paul Clough, University of Sheffield, Unite Donna Harman, National Institute for Stan Jaana Kekäläinen, University of Tampere Emanuele Pianta, Centre for the Evaluatio (CELCT) Italy (CELCT), Italy Maarten de Rijke, University of Amsterdam

mittee

the Conference

• n University (UNED), Spain

the Evaluation Labs

search (CNR), Italy plied Sciences, Switzerland ge resources Distribution Agency (ELDA) France ge resources Distribution Agency (ELDA), France ed Kingdom ndards and Technology (NIST), USA gy ( ) , Finland

• n of Language and Communication Technologies

m UvA, The Netherlands

CLEF 2010

What and how innov What and how innov Change in the coord Change in the coord Funding Funding vate vate ination ination

EF 2010 as a bridge he future The community th k t the key to succe e

Scientific and Technol Scientific and Technol Multilingual and Multime

nd

• ps

nd

• ps

nd

• ps

Labs a

• rksho

Labs a

• rksho

Labs a

• rksho

L Wo L Wo L Wo

Confe

• gical Advancement in
• gical Advancement in

edia Information Systems

nd

• ps

nd

• ps

nd

• ps

Labs a

• rksho

Labs a

• rksho

Labs a

• rksho

L Wo L Wo L Wo

erence

Conference

Two days Two days Large program committe Keynote talks and panel Publication in Springer L p g

no more LNCS post-proceed

Labs

Two days (more space t Lab selection committee Online publication (with Lab organizers are resp Lab organizers are resp for post-conference pub ee ls LNCS

dings of the CLEF working notes

than in CLEF classic) e ISBN) in time for the conference ponsible for individual outcomes ponsible for individual outcomes blication (special issues, ...)

Honorary Chair Honorary Chair

Carol Peters, ISTI-CNR, Italy

General Chairs

Maristella Agosti, University of Maarten de Rijke, University of Maarten de Rijke, University of

P Ch i Program Chairs

Nicola Ferro, University of Pad Alan Smeaton, Dublin City Univ

Lab Chairs

Martin Braschler, Zurich Univer Donna Harman NIST USA Padua, Italy f Amsterdam, The Netherlands f Amsterdam, The Netherlands ua, Italy versity, Ireland rsity of Applied Sciences, Switzerland

12 papers (8 full papers and 4 12 papers (8 full papers and 4

• ut of 21 submissions (17 full pap

Two keynote talks

Norbert Fuhr , “IR Between Scienc Experimentation” Ricardo Baeza-Yates, “Retrieval E

Other Evaluation Initiatives

Ellen Voorhees for TREC; Noriko Ellen Voorhees for TREC; Noriko FIRE; Jaap Kamps for INEX; Pav

Two panels

D H N ik K d M Donna Harman, Noriko Kando, Mo Ladies of Experimental Evaluation

4 short papers) 4 short papers)

ers and 4 short papers) ce and Engineering, and the Role of Evaluation in Practice” Kando for NTCIR; Prasenjit Majumder Kando for NTCIR; Prasenjit Majumder vel Braslavski for ROMIP i L l C l P t “Th F

• unia Lalmas, Carol Peters, “The Four

n”

Benchmarking activities

CLEF IP: A benchmarking a CLEF-IP: A benchmarking a ImageCLEF: A benchmarkin PAN: A benchmarking activit RespubliQA: A benchmarkin RespubliQA: A benchmarkin using multilingual political da WePS: A benchmarking activ WePS: A benchmarking activ

Workshops

CriES: A workshop aimed at for expertise in social media LogCLEF: A workshop aime studying search engine log f activity on intellectual property activity on intellectual property ng activity on image retrieval ty on plagiarism detection ng activity on question answering ng activity on question answering ata vity on web people search vity on web people search t exploring the evaluation of search a. ed at exploring methodologies for files

CLEF 2011

General chairs General chairs

Julio Gonzalo National Dist Julio Gonzalo, National Dist Spain Maarten de Rijke University Maarten de Rijke, University

Program chairs

Jaana Kekäläinen, Universit Mounia Lalmas Yahoo! Res Mounia Lalmas, Yahoo! Res

Lab chairs

Paul Clough, University of S ance Education University (UNED ance Education University (UNED y of Amsterdam The Netherlands y of Amsterdam, The Netherlands ty of Tampere, Finland search Barcelona Spain search Barcelona, Spain Sheffield, United Kingdom

14 papers (10 full papers and 4 s 14 papers (10 full papers and 4 s

• ut of 23 submissions (19 full papers

Two keynote talks

“ Elaine Toms , “Would you trust your Evaluation in the 21st Century” Omar Alonso “Crowdsourcing for Inf Omar Alonso, Crowdsourcing for Inf Evaluation”

Community Sessions

Other Evaluation Initiatives: Ellen V Other Evaluation Initiatives: Ellen V Jaap Kamps for INEX; Gareth Jones CHORUS+ Session: Information Ret C O US Sess o

• at o

et Infrastructure Session: the Promise Analytics meets Information Retrieva Funding Opportunity Session: Inte Workprogramme 2011 - 2012, Strate

hort papers) hort papers)

s and 4 short papers) ? IR system to choose your date? Re-thinking formation Retrieval Experimentation and formation Retrieval Experimentation and Voorhees for TREC; Hideo Joho for NTCIR; Voorhees for TREC; Hideo Joho for NTCIR; s for MediaEval trieval from Scientific Multimedia Data t e a

• Sc e t c

u t ed a ata e evaluation infrastructure, where Visual al elligent Information Management - FP7 ICT egic Objective 4.4. - General information and

Works

Benchmarking activities

CLEF-IP: A benchmarking activity CLEF IP: A benchmarking activity ImageCLEF: A benchmarking acti PAN: A benchmarking activity on p PAN: A benchmarking activity on p QA4MRE: A benchmarking activity systems through Question Answer y g LogCLEF: A benchmarking activit identification, query classification,

Workshops/Pilots Workshops/Pilots

CHiC: A workshop aimed at movin l ti f lt l h it di it evaluation of cultural heritage digit and helping to shape a possible ro MusicCLEF: A pilot benchmarking MusicCLEF: A pilot benchmarking engines that are based both on au descriptions

shops

• n intellectual property
• n intellectual property

vity on image retrieval plagiarism detection plagiarism detection y on the evaluation of Machine Reading ring and Reading Comprehension Test. g g p y on Multilingual Log File Analysis: Langu success of a query ng towards a systematic and large-scale t l lib i d i f ti t tal libraries and information access system

• admap for it.

g activity on the evaluation of music search g activity on the evaluation of music search udio content and on multilingual textual

CLEF 2012

Worksh

Benchmarking activities g

CHiC: CA benchmarking activity Europeana collections (2011 wo p ( CLEF-IP: A benchmarking activit ImageCLEF: A benchmarking ac ImageCLEF: A benchmarking ac INEX: the well-known Initiative fo with CLEF t o target new synerg with CLEF t o target new synerg andsemi-structured information a PAN: A benchmarking activity on PAN: A benchmarking activity on

QA4MRE: A benchmarking activit systems through Question Answe y g RepLab: A benchmarking activity management

Workshops

eHealth: a workshop on new ev

the Louhi series of workshops on

hops

y for the cultural heritage domain based o rkshop) p) ty on intellectual property ctivity on image retrieval ctivity on image retrieval

• r Evaluation of XML retrieval joins effor

ies between multilingual multimodal ies between multilingual, multimodal access n plagiarism detection n plagiarism detection

y on the evaluation of Machine Reading ring and Reading Comprehension Test. g g p

• n microblog data for online reputation

valuation issues in the health domain, rela NLP in Health Informatics

General chairs

Tiziana Catarci, Sapienza U Djoerd Hiemstra, University j , y

Program chairs

A l P ñ N ti l Di Anselmo Peñas, National Di Giuseppe Santucci, Sapienz pp p

Lab chairs

J i K l S di h I t Jussi Karlgren, Swedish Inst Christa Womser-Hacker Un niversity of Rome, Italy

• f Twente, The Netherlands

, i t L i U i it S i istance Learning University, Spain za University of Rome, Italy y y tit t f C t S i S d titute of Computer Science, Swed niversity of Hildesheim Germany

Open Bid for CLEF 2013 Open Bid for CLEF 2013 All information available on All information available on Website (soon) CLEF is the community: take CLEF is the community: take your home! the new the new e CLEF at e CLEF at

Surroundi Surroundi PROM PROM ing CLEF: ing CLEF: MISE MISE

Regular Evaluation Activities

Evaluation tasks based on use cases New evaluation methods and metrics Large and multidisciplinary community Large and multidisciplinary community

Automation in the Evaluation Pro

Open evaluation infrastructure Open evaluation infrastructure Automate the evaluation activities Continuous experimentation Distributed evaluation protocols

Collaboration and Re-use of knowledge-base knowledge-base

Visual analytics Annotation and discussion support

Knowledge Transfer and Uptake

The CLEF Initiative Technology take-up group and Techno transfer day Researchers exchange program g p g Two summer schools Best practices

Post-proceedings in p g LNCS Tutorials ser

Thank

http // clef campa

k You

aign org/