CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro - PowerPoint PPT Presentation

The CLEF Initiative Conference and Labs of the E valuation Forum I N I T I A T I V E http:/ /www.clef - initiative.eu/ CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro � University of Padua, Italy Forum for Information Retrieval Evaluation (FIRE 2014) 7th December 2014, Bangalore, India

Outline I N I T I A T I V E CLEF Overview � CLEF 2014 Highlights � CLEF 2015 Ongoing Activities CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 2 FIRE 2014, 7 December 2014, Bangalore, India

CLEF Initiative

Mission I N I T I A T I V E The CLEF Initiative is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. multilingual and multimodal system testing, tuning and evaluation; investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access; creation of reusable test collections for benchmarking; exploration of new evaluation methodologies and innovative ways of using experimental data; discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge. CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 4 FIRE 2014, 7 December 2014, Bangalore, India

Approach I N I T I A T I V E Scientific and Technological Advancement in Multilingual and Multimedia Information Systems Workshops Workshops Workshops Workshops Workshops Workshops Labs and Labs and Labs and Labs and Labs and Labs and Conference CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 5 FIRE 2014, 7 December 2014, Bangalore, India

Participation I N I T I A T I V E 100% Voluntary Effort Based Participation 210 180 150 120 90 60 30 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 Mainly voluntary effort + project funding CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 6 FIRE 2014, 7 December 2014, Bangalore, India

Labs I N I T I A T I V E Tracks/Labs 10 8 6 4 2 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 7 FIRE 2014, 7 December 2014, Bangalore, India

Conference I N I T I A T I V E Submitted Accepted Conference 40 30 20 10 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 8 FIRE 2014, 7 December 2014, Bangalore, India

Conference Topics I N I T I A T I V E Experimental Collections Evaluation Methods Evaluation Measures Evaluation Infrastructures Language Tools and Resources Tools, Systems, Applications Multimodaliy Information Visualization for Evaluation Longitudinal Studies 25 20 15 10 5 0 2010 2011 2012 2013 2014 CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 9 FIRE 2014, 7 December 2014, Bangalore, India

Labs over the Years I N I T I A T I V E NEWSREEL LifeCLEF CLEF-ER CLEF eHealth RepLab INEX CHiC MusicCLEF PAN CriES WePS Grid@CLEF CLEF-IP LogCLEF INFILE@CLEF VideoCLEF CLEF@SemEval CLEF@MorphoChallenge GeoCLEF WebCLEF ImageCLEF QA@CLEF CLEF SR iCLEF Domain Specific Ad Hoc 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 10 FIRE 2014, 7 December 2014, Bangalore, India

Publication “Universe” I N I T I A T I V E Google Scholar for “CLEF evaluation”: 31,900 hits Google Scholar for “TREC evaluation”: 29,900 hits CLEF TREC 20000 15000 10000 16,100 hits 16,400 hits 15,800 hits 13,500 hits 5000 0 2000-2009 2010-2014 CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 11 FIRE 2014, 7 December 2014, Bangalore, India

CLEF Association - Scope I N I T I A T I V E Mission Promoting information access and use evaluation Mission Critical thinking about advancing information access and use from a technical, economic, and a societal perspective Goals Cluster stakeholders with multidisciplinary competencies and different objectives academia, industry, education, societal Facilitate medium/long term research in information access and use and its evaluation Increase, transfer, and apply expertise CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 12 FIRE 2014, 7 December 2014, Bangalore, India

CLEF Association - Pillar Activities I N I T I A T I V E Education and Knowledge Transfer Collections Infrastructure CLEF CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 13 FIRE 2014, 7 December 2014, Bangalore, India

Steering Committee I N I T I A T I V E Steering Committee Chair Evangelos Kanoulas , University of Amsterdam, The Nicola Ferro , University of Padua, Italy Netherlands � Henning Müller , University of Applied Sciences Western Deputy Steering Committee Chair for the Conference Switzerland (HES-SO), Switzerland Paolo Rosso ,Universitat Politècnica de València, Spain Mihai Lupu , Vienna University of Technology, Austria � Maarten de Rijke , University of Amsterdam UvA, The Deputy Steering Committee Chair for the Labs Netherlands Giuseppe Santucci , Sapienza University of Rome, Italy Martin Braschler , Zurich University of Applied Sciences, Switzerland Alan Smeaton , Dublin City University, Ireland � Christa Womser-Hacker , University of Hildesheim, Members Germany � Khalid Choukri , Evaluations and Language resources Past Members Distribution Agency (ELDA), France Paul Clough , University of Sheffield, United Kingdom Jaana Kekäläinen , University of Tampere, Finland Julio Gonzalo , National Distance Education University Carol Peters , ISTI, National Council of Research (CNR), (UNED), Spain Italy Donna Harman , National Institute for Standards and Emanuele Pianta , Centre for the Evaluation of Language Technology (NIST), USA and Communication Technologies (CELCT), Italy Djoerd Hiemstra , University of Twente, The Netherlands CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 14 FIRE 2014, 7 December 2014, Bangalore, India

CLEF 2014

CLEF 2014: 15-18 September, Sheffield I N I T I A T I V E http://clef2014.clef-initiative.eu/ Keynotes Ann Blandford University College London, UK Susan Dumais, Microsoft Research Lab, USA Fabio Ciravegna, University of Sheffield, UK Conference evaluation domain specific approaches alternative search tasks other evaluation initiatives Labs CLEF eHealth ImageCLEF INEX LifeCLEF NEWSREEL PAN QA RepLab CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 16 FIRE 2014, 7 December 2014, Bangalore, India

CLEF 2014 Organization I N I T I A T I V E General chairs Paul Clough, University of Sheffield, UK Mark Sanderson, RMIT University Melbourne, Australia � Program chairs Evangelos Kanoulas, University of Amsterdam, The Netherlands Mihai Lupu, Vienna University of Technology, Austria � Lab chairs Martin Halvey, University of Strathclyde, UK Wessel Kraaij, TNO (Netherlands Organisation for Applied Scientific Research) and Radboud University Nijmegen, The Netherlands � Proceedings Chairs Elaine Toms, University of Sheffield, UK Allan Hanbury, Vienna University of Technology, Austria CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 17 FIRE 2014, 7 December 2014, Bangalore, India

CLEF 2015

CLEF 2015: 8-11 September, Toulouse I N I T I A T I V E http://clef2015.clef-initiative.eu/ Conference Submission Deadline April 12, 2015 Information Access in any language or modality Analytics for Information Retrieval Evaluation Initiatives Evaluation Methodologies Technology Transfer Interactive Information Retrieval Evaluation Specific Application Domains CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 19 FIRE 2014, 7 December 2014, Bangalore, India

CLEF 2015: 8-11 September, Toulouse I N I T I A T I V E http://clef2015.clef-initiative.eu/ Evaluation Cycle: November 2014 - May 2015 Registrations Open CLEF eHealth NEWSREEL ImageCLEF PAN LifeCLEF QA Living Labs for IR 
 Social Book Search CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 20 FIRE 2014, 7 December 2014, Bangalore, India

CLEF 2015 Organization I N I T I A T I V E General chairs Josiane Mothe, IRIT, Université de Toulouse, France Jacques Savoy, University of Neuchâtel, Switzerland � Program chairs Jaap Kamps, University of Amsterdam, The Netherlands Karen Pinel-Sauvagnat, IRIT, Université de Toulouse, France � Lab chairs Gareth Jones, Dublin City University, Ireland Eric San Juan, Université d'Avignon et des Pays du Vaucluse, France � Proceedings Chairs Linda Cappellato, University of Padua, Italy Nicola Ferro, University of Padua, Italy CLEF: 15 Years of IR Evaluation in Europe Nicola Ferro 21 FIRE 2014, 7 December 2014, Bangalore, India