PROMISE ������� Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation Overall Project Presentation Coordinator: Nicola Ferro University of Padua, Italy December 2010 Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191 http://www.promise-noe.eu/
!"#$%&' Overview Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 2 December 2010
Fact Sheet !"#$%&' • PROMISE : Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation • Network of Excellence : Grant Agreement 258191 • Duration : 1 st September 2010 - 31 st August 2013 (36 months) • Effort : 314 man/months • Budget : 3,529,357 euros (EC contribution) • Consortium : 10 partners • University of Padua (UNIPD), Italy • Swedish Institute of Computer Science (SICS), Sweden • University of Amsterdam (UvA), The Netherlands • Sapienza University of Rome (ROMA1), Italy • University of Applied Sciences Western Switzerland (HES-SO), Switzerland • Information Retrieval Facility (IRF), Austria • Zurich University of Applied Sciences (ZHAW), Switzerland • Humboldt-Universität zu Berlin (UBER), Germany • Evaluations and Language resources Distribution Agency (ELDA), France • Centre for the Evaluation of Language Communication Technologies (CELCT), Italy • Web site : http://www.promise-noe.eu/ Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 3 December 2010
Goal !"#$%&' • To advance the experimental evaluation of complex multimedia and multilingual information systems in order to support the decision making process of individuals, commercial entities, and communities who develop, employ, and improve such complex systems • PROMISE will provide a virtual and open laboratory for conducting participative research and experimentation in which it will be possible to carry out, advance and bring automation into the evaluation and benchmarking of complex multimedia and multilingual information systems, by facilitating management and offering access, curation, preservation, re-use, analysis, visualization, and mining of the collected experimental data. Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 4 December 2010
Use Cases: Unlocking Culture !"#$%&' Mona Lisa ! La Gioconda ! La Joconde ! "#$% &'(%! It deals with information access to cultural heritage material held in large-scale digital libraries comprising libraries, archives, museum, and audio-visual archives. Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 5 December 2010
Use Cases: Search For Innovation !"#$%&' It deals with patent search and its peculiar requirements to seek out standardized method and framework for evaluating different tools for the intellectual property. Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 6 December 2010
Use Cases: Search For Innovation !"#$%&' ! It deals with patent search and its peculiar requirements to seek out standardized method and framework for evaluating different tools for the intellectual property. Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 6 December 2010
Use Cases: Visual Clinical Decision Support !"#$%&' It deals with visual information connected with text in the radiology domain in order to provide retrieval and access mechanisms able to jointly exploit textual and visual features. Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 7 December 2010
Use Cases: The Web !"#$%&' Translate Flickr ! Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 8 December 2010
Use Cases: The Web !"#$%&' Translate Flickr ! Translate Flickr ! Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 8 December 2010
Matching Different Needs !"#$%&' !"#$%&' Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 9 December 2010
Approach !"#$%&' • Regular Evaluation Activities • Evaluation tasks based on use cases • New evaluation methods and metrics • Open evaluation infrastructure • Large and multidisciplinary community • Automation in the Evaluation Process • Automate the evaluation activities • Continuous experimentation • Distributed evaluation protocols • Collaboration and Re-use of knowledge-base • Visual analytics • Annotation and discussion support • Knowledge Transfer and Uptake • The CLEF Initiative • Technology take-up group • Technology transfer day • Researchers exchange program • Two summer schools • Best practices • Three brainstorming workshops (at least) and tutorials Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 10 December 2010
Expected Impact !"#$%&' Effort Volume of Data Usage 1 2 3 Year Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 11 December 2010
Exploitation: Virtual Institute !"#$%&' Scientific and Technological Advancement in Multilingual and Multimedia Information Systems Evaluation Tasks Evaluation Tasks Evaluation Tasks Evaluation Tasks Evaluation Tasks Evaluation Tasks Use Cases and Use Cases and Use Cases and Use Cases and Use Cases and Use Cases and Collaborative and Open Evaluation Infrastructure Acquired Knowledge-base Evaluation Methodologies and Metrics Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 12 December 2010
Work Packages !"#$%&' WP2 WP3 WP5 Stakeholders Involvement Collaboration and Evaluation Infrastructure and Technology Transfer Knowledge Sharing WP7 WP4 Dissemination, IPR, Evaluation Methods and Resources and Metrichs WP6 Evaluation Activities Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 13 December 2010
WP Leaders !"#$%&' • WP1: Nicola Ferro, UNIPD • WP2: Jussi Karlgren, SICS • WP3: Maristella Agosti, UNIPD • WP4: Maarten de Rijke, UvA • WP5: Giuseppe Santucci, ROMA1 • WP6: Henning Müller, HES-SO • WP7: Khalid Choukri, ELDA Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 14 December 2010
Advisory Board !"#$%&' • Ricardo Baeza-Yates, Yahoo! Research, Barcelona, Spain • Makx Dekkers, Independent Consultant, Luxembourg • Donna Harman, Scientist Emeritus, National Institute of Standards and Technology (NIST), USA • Kalervo Järvelin, University of Tampere, Finland • Peter Pirolli, Palo Alto Research Center (PARC), USA Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 15 December 2010
!"#$%&' The CLEF Initiative Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 16 December 2010
CLEF “Classic” Objectives !"#$%&' • Promote research and stimulate development of multilingual IR systems for European (and non- European) languages, through • Creation of evaluation infrastructure and organisation of regular evaluation campaigns for system testing • Building of an MLIA/CLIR research community • Construction of publicly available test-suites • Major Goal • To anticipate the emerging needs of the R&D community, and • Encourage development of truly multilingual, multimodal systems Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 17 December 2010
CLEF “Classic”: Ten Years+ of History !"#$%&' • 1997 – First CLIR system evaluation campaigns in US and Japan: TREC and NTCIR • CLEF actually began life in 1997 as a track for Cross Language Information Retrieval (CLIR) within TREC. Mainly, English centered tasks (EN -> X, X -> EN). • 2000-2009 – CLIR evaluation in Europe: CLEF (extension of CLIR track at TREC) • Fully multilingual, multimodal information retrieval systems capable of processing a query in any medium and any language finding relevant information from a multilingual multimedia collection containing documents in any language and form, and presenting it in the style most likely to be useful to the user • Funding • DELOS NoE under FP5 2000 – 2003 (http://delos-noe.isti.cnr.it/) • DELOS NoE under FP6 2004 – 2007 (http://www.delos.info/) • TrebleCLEF under FP7 2008-2009 (http://www.trebleclef.eu/) Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 18 December 2010
CLEF “Classic” Achievements !"#$%&' • Stimulation of research activity in new, previously unexplored areas • Study and implementation of evaluation methodologies for diverse types of cross-language IR systems • Creation of a large set of empirical data about multilingual information access from the user perspective • Quantitative and qualitative evidence with respect to best practice in cross-language system development • Creation of reusable test collections for system benchmarking • Building of a strong, multidisciplinary research community Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 19 December 2010
The CLEF Initiative: Vision !"#$%&' CLEF 2010 as a bridge to the future Overall Project Presentation PROMISE NoE (FP7, grant agreement no. 258191) 20 December 2010
Recommend
More recommend
