The CLEF Association Conference and Labs of the E valuation Forum AS S O C IAT I O N http:/ /www.clef - initiative.eu/association CLEF 20 th Anniversary Nicola Ferro @frrncl University of Padua, Italy 10 th Conference and Labs of the Evaluation Forum (CLEF 2019) 9 th September 2019, Lugano, Switzerland
CLIR: The Grand Challenge AS S O C IAT I O N AAAI 1997 Spring Symposium: Fully multilingual and multimodal information retrieval systems [Doug Oard and David Hull] capable of processing a query in any medium and any language finding relevant information from a multilingual multimedia collection containing documents in any language and form and presenting it in the style most likely to be useful to the user Nicola Ferro CLEF 20th Anniversary � 2 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Carol Peters: When Everything Began AS S O C IAT I O N Nicola Ferro CLEF 20th Anniversary � 3 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Classic CLEF: Years+ of History AS S O C IAT I O N 1997 – First CLIR system evaluation campaigns in US and Japan: TREC and NTCIR CLEF actually began life in 1997 as a track for Cross Language Information Retrieval (CLIR) within TREC. Mainly, English centered tasks (EN -> X, X -> EN). 2000-2009 – CLIR evaluation in Europe: CLEF (extension of CLIR track at TREC) Fully multilingual, multimodal information retrieval systems capable of processing a query in any medium and any language finding relevant information from a multilingual multimedia collection containing documents in any language and form, and presenting it in the style most likely to be useful to the user Nicola Ferro CLEF 20th Anniversary � 4 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Classic CLEF: Achievements AS S O C IAT I O N Stimulation of research activity in new, previously unexplored areas Study and implementation of evaluation methodologies for diverse types of cross-language IR systems Creation of a large set of empirical data about multilingual information access from the user perspective Quantitative and qualitative evidence with respect to best practice in cross-language system development Creation of reusable test collections for system benchmarking Building of a strong, multidisciplinary research community Nicola Ferro CLEF 20th Anniversary � 5 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Classic CLEF: Achievements AS S O C IAT I O N Stimulation of research activity in new, previously unexplored areas R I Study and implementation of evaluation methodologies for l a u g n diverse types of cross-language IR systems i l n i t a l u e M p Creation of a large set of empirical data about multilingual o r u E s information access from the user perspective r e o g f a u g Quantitative and qualitative evidence with respect to best n a l practice in cross-language system development Creation of reusable test collections for system benchmarking Building of a strong, multidisciplinary research community Nicola Ferro CLEF 20th Anniversary � 5 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Where To Go Next? AS S O C IAT I O N The direction depends on the community From Classic CLEF to the CLEF Initiative Nicola Ferro CLEF 20th Anniversary � 6 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Mission AS S O C IAT I O N The CLEF Initiative is a self-organized body whose main mission is to promote research, innovation, and development of information access systems with an emphasis on multilingual and multimodal information with various levels of structure. multilingual and multimodal system testing, tuning and evaluation; investigation of the use of unstructured, semi-structured, highly-structured, and semantically enriched data in information access; creation of reusable test collections for benchmarking; exploration of new evaluation methodologies and innovative ways of using experimental data; discussion of results, comparison of approaches, exchange of ideas, and transfer of knowledge. Nicola Ferro CLEF 20th Anniversary � 7 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Approach AS S O C IAT I O N Scientific and Technological Advancement in Multilingual and Multimodal Information Systems Workshops Workshops Workshops Workshops Workshops Workshops Labs and Labs and Labs and Labs and Labs and Labs and Conference Nicola Ferro CLEF 20th Anniversary � 8 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Participation: Attendees AS S O C IAT I O N 100% Voluntary Effort Based Participation 210 180 150 120 90 60 30 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Mainly voluntary effort + project funding Nicola Ferro CLEF 20th Anniversary � 9 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Labs AS S O C IAT I O N Tracks/Labs 12 10 7 5 2 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 Nicola Ferro CLEF 20th Anniversary � 10 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Conference AS S O C IAT I O N Submitted Accepted Conference 70 60 50 40 30 20 10 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Nicola Ferro CLEF 20th Anniversary � 11 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Conference Topics AS S O C IAT I O N Experimental Collections Evaluation Methods Evaluation Measures Evaluation Infrastructures Language Processing and Resources Tools, Systems, Applications Multimodality Information Visualization for Evaluation Longitudinal Studies 50 40 30 20 10 0 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Nicola Ferro CLEF 20th Anniversary � 12 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Labs over the Years AS S O C IAT I O N 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Extracting Protests from News (ProtestNews) ProtestNews Identification and Verification of Political Claims (CheckThat!) CheckThat! Reproducibility (CENTRE@CLEF) CENTRE@CLEF Personalised Information Retrieval (PIR-CLEF) PIR-CLEF Early Risk Prediction on the Internet (eRisk) eRisk Multimodal Spatial Role Labeling (MSRL) MSRL Dynamic Search for Complex Tasks (CLEF DynSE) CLEF DynSE Microblog Cultural Contextualization (MC2) MC2 Social Book Search (SBS) SBS Living Labs (LL4IR) LL4IR News Recommendation Evaluation (NewsREEL) NewsREEL Biodiversity Identification and Prediction (LifeCLEF) LifeCLEF Entity Recognition (CLEF-ER) CLEF-ER CLEF eHealth CLEF eHealth Online Reputation Management (RepLab) RepLab Retrieval on Structured Datasets (INEX) INEX Cultural Heritage in CLEF (CHiC) CHiC Music Information Retrieval (MusiCLEF) MusiCLEF Digital Text Forensics and Stylometry (PAN) PAN Cross-lingual Expert Search (CriES) CriES Web People Search (WEPS) WEPS Component-based Evaluation (Grid@CLEF) Grid@CLEF Intellectual Property in the Patent Domain (CLEF-IP) CLEF-IP Log File Analysis (LogCLEF) LogCLEF Multilingual Information Filtering (INFILE) INFILE Cross-Language Video Retrieval (VideoCLEF) VideoCLEF CLEF@SemEval CLEF@SemEval CLEF@MorphoChallenge CLEF@MorphoChallenge Geographical Retrieval (GeoCLEF) GeoCLEF Multilingual Web Search (WebCLEF) WebCLEF Multimedia Retrieval (ImageCLEF) ImageCLEF Question Answering (QA@CLEF) QA@CLEF Spoken Document/Speech Retrieval (CLEF SR) CLEF SR Interactive Cross-Language IR (iCLEF) iCLEF Domain Specific Cross-Language IR (DS) DS Multilingual Text Retrieval (Ad-hoc) Ad-hoc 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 Nicola Ferro CLEF 20th Anniversary � 13 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Publication “Universe” AS S O C IAT I O N Google Scholar Metrics for “Cross-Language Evaluation Forum” 60 54 52 50 45 38 40 37 35 30 32 30 20 10 h5-index h5-median 0 2016 2017 2018 2019 Google Scholar for “ CLEF evaluation ” 53,400 hits Nicola Ferro CLEF 20th Anniversary � 14 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Publication “Universe” AS S O C IAT I O N Nicola Ferro CLEF 20th Anniversary � 15 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
20th Anniversary Book AS S O C IAT I O N Foreword by Donna Harman Part I – Experimental Evaluation and CLEF Part II – Evaluation Infrastructures Part III – Multilingual and Multimedia Information Retrieval Part IV – Retrieval in New Domains Part V – Beyond Retrieval Part VI – Impact and Future Challenges Nicola Ferro CLEF 20th Anniversary � 16 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
20th Anniversary Book AS S O C IAT I O N Foreword by Donna Harman Part I – Experimental Evaluation and CLEF Part II – Evaluation Infrastructures Part III – Multilingual and Multimedia Information Retrieval Part IV – Retrieval in New Domains Part V – Beyond Retrieval Part VI – Impact and Future Challenges Nicola Ferro CLEF 20th Anniversary � 16 @frrncl CLEF 2019, 9 September 2019, Lugano, Switzerland
Recommend
More recommend
