CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro - - PowerPoint PPT Presentation
The CLEF Initiative Conference and Labs of the E valuation Forum I N I T I A T I V E http:/ /www.clef - initiative.eu/ CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro University of Padua, Italy Forum for Information Retrieval
I N I T I A T I V E
Conference and Labs of the E valuation Forum
http:/ /www.clef-initiative.eu/
Forum for Information Retrieval Evaluation (FIRE 2014) 7th December 2014, Bangalore, India
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E2
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V Emultilingual and multimodal system testing, tuning and evaluation; investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access; creation of reusable test collections for benchmarking; exploration of new evaluation methodologies and innovative ways of using experimental data; discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge.
4
The CLEF Initiative is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure.
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E5
Scientific and Technological Advancement in Multilingual and Multimedia Information Systems
Conference Labs and Workshops Labs and Workshops Labs and Workshops Labs and Workshops Labs and Workshops Labs and Workshops
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V EParticipation 30 60 90 120 150 180 210
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
6
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E7
Tracks/Labs 2 4 6 8 10
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V EConference 10 20 30 40
2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Submitted Accepted
8
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E5 10 15 20 25
2010 2011 2012 2013 2014
Experimental Collections Evaluation Methods Evaluation Measures Evaluation Infrastructures Language Tools and Resources Tools, Systems, Applications Multimodaliy Information Visualization for Evaluation Longitudinal Studies
9
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E10
NEWSREEL LifeCLEF CLEF-ER CLEF eHealth RepLab INEX CHiC MusicCLEF PAN CriES WePS Grid@CLEF CLEF-IP LogCLEF INFILE@CLEF VideoCLEF CLEF@SemEval CLEF@MorphoChallenge GeoCLEF WebCLEF ImageCLEF QA@CLEF CLEF SR iCLEF Domain Specific Ad Hoc 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E5000 10000 15000 20000 2000-2009 2010-2014 CLEF TREC
11
16,100 hits 15,800 hits
Google Scholar for “CLEF evaluation”: 31,900 hits
16,400 hits 13,500 hits
Google Scholar for “TREC evaluation”: 29,900 hits
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E12
Goals
Cluster stakeholders with multidisciplinary competencies and different objectives
academia, industry, education, societal
Facilitate medium/long term research in information access and use and its evaluation Increase, transfer, and apply expertise
Promoting information access and use evaluation Critical thinking about advancing information access and use from a technical, economic, and a societal perspective
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E13
Collections Infrastructure Education and Knowledge Transfer
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V ESteering Committee Chair
Nicola Ferro, University of Padua, Italy
Paolo Rosso,Universitat Politècnica de València, Spain
Martin Braschler, Zurich University of Applied Sciences, Switzerland
Khalid Choukri, Evaluations and Language resources Distribution Agency (ELDA), France Paul Clough, University of Sheffield, United Kingdom Julio Gonzalo, National Distance Education University (UNED), Spain Donna Harman, National Institute for Standards and Technology (NIST), USA Djoerd Hiemstra, University of Twente, The Netherlands Evangelos Kanoulas, University of Amsterdam, The Netherlands Henning Müller, University of Applied Sciences Western Switzerland (HES-SO), Switzerland Mihai Lupu, Vienna University of Technology, Austria Maarten de Rijke, University of Amsterdam UvA, The Netherlands Giuseppe Santucci, Sapienza University of Rome, Italy Alan Smeaton, Dublin City University, Ireland Christa Womser-Hacker, University of Hildesheim, Germany
Jaana Kekäläinen, University of Tampere, Finland Carol Peters, ISTI, National Council of Research (CNR), Italy Emanuele Pianta, Centre for the Evaluation of Language and Communication Technologies (CELCT), Italy
14
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V Ehttp://clef2014.clef-initiative.eu/
16
Keynotes
Ann Blandford University College London, UK Susan Dumais, Microsoft Research Lab, USA Fabio Ciravegna, University of Sheffield, UK
Conference
evaluation domain specific approaches alternative search tasks
Labs
CLEF eHealth ImageCLEF INEX LifeCLEF NEWSREEL PAN QA RepLab
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V EGeneral chairs
Paul Clough, University of Sheffield, UK Mark Sanderson, RMIT University Melbourne, Australia
Evangelos Kanoulas, University of Amsterdam, The Netherlands Mihai Lupu, Vienna University of Technology, Austria
Martin Halvey, University of Strathclyde, UK Wessel Kraaij, TNO (Netherlands Organisation for Applied Scientific Research) and Radboud University Nijmegen, The Netherlands
Elaine Toms, University of Sheffield, UK Allan Hanbury, Vienna University of Technology, Austria
17
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V E19
Information Access in any language or modality Analytics for Information Retrieval Evaluation Initiatives Evaluation Methodologies Technology Transfer Interactive Information Retrieval Evaluation Specific Application Domains
http://clef2015.clef-initiative.eu/
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V Ehttp://clef2015.clef-initiative.eu/
20
CLEF eHealth ImageCLEF LifeCLEF Living Labs for IR NEWSREEL PAN QA Social Book Search
Nicola Ferro CLEF: 15 Years of IR Evaluation in Europe FIRE 2014, 7 December 2014, Bangalore, India
I N I T I A T I V EGeneral chairs
Josiane Mothe, IRIT, Université de Toulouse, France Jacques Savoy, University of Neuchâtel, Switzerland
Jaap Kamps, University of Amsterdam, The Netherlands Karen Pinel-Sauvagnat, IRIT, Université de Toulouse, France
Gareth Jones, Dublin City University, Ireland Eric San Juan, Université d'Avignon et des Pays du Vaucluse, France
Linda Cappellato, University of Padua, Italy Nicola Ferro, University of Padua, Italy
21