A Proposal to Extend and Enrich the A Proposal to Extend and Enrich - PowerPoint PPT Presentation

First International Workshop on Evaluating Information Access (E First International Workshop on Evaluating Information Access (EVIA 2007) VIA 2007) Tokyo, May 15, 2007 Tokyo, May 15, 2007 A Proposal to Extend and Enrich the A Proposal to Extend and Enrich the Scientific Data Curation Curation of Evaluation Campaigns of Evaluation Campaigns Scientific Data Maristella Agosti, Giorgio Maria Di Nunzio, Nicola Ferro Information Management S ystems (IMS ) Research Group Information Management S ystems (IMS ) Research Group Department of Information Engineering Department of Information Engineering University of Padua, Italy University of Padua, Italy

Outline Outline � Background on experimental evaluation � Background on experimental evaluation � Motivations and obj ectives � Motivations and obj ectives � Discussion on the current methodology � Discussion on the current methodology � Possible extensions to the current methodology � Possible extensions to the current methodology � A concrete example: the DIRECT system � A concrete example: the DIRECT system st Int. Workshop on Evaluating Information Access 1 st 2 1 Int. Workshop on Evaluating Information Access 2 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007

Experimental Evaluation Background Experimental Evaluation Background � The � The Cranfield Cranfield evaluation methodology is a very well understood evaluation methodology is a very well understood paradigm paradigm � its main focus is on � its main focus is on experiment comparability experiment comparability and and performance performance evaluation evaluation � S � S everal successful evaluation initiatives (TREC, CLEF, NTCIR, … … ) everal successful evaluation initiatives (TREC, CLEF, NTCIR, ) have adopted this paradigm have adopted this paradigm � they have produced a � they have produced a huge amount of data , promoted the research promoted the research in in huge amount of data , the IR field, and favoured the creation of cross cross- -disciplinary communities disciplinary communities the IR field, and favoured the creation of � S � S teve Robertson, in his keynote at ECIR 2007, pointed out teve Robertson, in his keynote at ECIR 2007, pointed out � the IR � the IR field field has has a a long long tradition tradition in in evaluation evaluation and and “ “ t t he tradition that he tradition that began with Cranfield Cranfield is is alive and kicking alive and kicking , half a century later , half a century later” ” began with � we need to understand � we need to understand “ “ how and when to how and when to push its boundaries push its boundaries , how and , how and when to transcend it without throwing it away transcend it without throwing it away ” ” when to � such � such evaluation initiatives evaluation initiatives are an " are an " extremely valuable infrastructure extremely valuable infrastructure for for the [IR] field". the [IR] field". st Int. Workshop on Evaluating Information Access 1 st 3 1 Int. Workshop on Evaluating Information Access 3 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007

Motivation Motivation The experimental evaluation is a scientific activity and, as such, we have to realise that its outcomes are very valuable scientific data . � Comparable experiments � Comparable experiments � Performance measurements concerning the experiments � Performance measurements concerning the experiments � Descriptive statistics about a collection of experiment � Descriptive statistics about a collection of experiment � Hypothesis tests for in � Hypothesis tests for in- -depth analysis of the experiments depth analysis of the experiments st Int. Workshop on Evaluating Information Access 1 st 4 1 Int. Workshop on Evaluating Information Access 4 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007

Objectives Objectives Information Data Curation Hierarcy Investigate whether these scientific data are Investigate whether these scientific data are � properly modelled � properly modelled � effectively managed and archived � effectively managed and archived � carefully � External carefully curated curated and enriched and enriched Stakeholders in the current evaluation methodology in the current evaluation methodology st Int. Workshop on Evaluating Information Access 1 st 5 1 Int. Workshop on Evaluating Information Access 5 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007

Information Hierarchy Information Hierarchy � experimental collections � experimental collections and the and the experiments experiments are are data data , since they are the raw, basic , since they are the raw, basic elements needed for any further investigation elements needed for any further investigation � performance measurements � performance measurements are are information information , since they are the result of , since they are the result of computations and processing on the data, computations and processing on the data, � descriptive statistics � descriptive statistics and the and the hypothesis tests hypothesis tests are are knowledge knowledge , since they are a further , since they are a further elaboration of the information carried by the performance measurements elaboration of the information carried by the performance measur ements � theories, models, algorithms, and techniques � theories, models, algorithms, and techniques are are wisdom wisdom , since they provide , since they provide interpretation, explanation, and formalization of the content of the previous levels. the previous levels. interpretation, explanation, and formalization of the content of st Int. Workshop on Evaluating Information Access 1 st 6 1 Int. Workshop on Evaluating Information Access 6 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007

Data Curation Curation Data � S � S cientific data should be archived, cientific data should be archived, preserved preserved , , maintained over the time maintained over the time , , and made and made easily accessible easily accessible to interested users; to interested users; � Re � Re- -use of data for new research use of data for new research � Retention of expensive or difficult to generate data � Retention of expensive or difficult to generate data � Their � Their lineage lineage should be tracked since it allows us to j udge the quality and should be tracked since it allows us to j udge the quality and applicability of information for a given use; applicability of information for a given use; � Validation of published research results � Validation of published research results � S � S cientific data should be enriched cientific data should be enriched progressively adding further analyses and progressively adding further analyses and interpretations on them; interpretations on them; � Enhancement of existing data available for research proj ects � Enhancement of existing data available for research proj ects � It should be possible to � It should be possible to cite cite scientific data and their further elaboration scientific data and their further elaboration � cross � cross- -dissemination of scientific results to research communities and dissemination of scientific results to research communities and industrial industrial partners partners st Int. Workshop on Evaluating Information Access 1 st 7 1 Int. Workshop on Evaluating Information Access 7 M. Agosti M. Agosti, G.M. Di , G.M. Di Nunzio Nunzio, N. Ferro , N. Ferro EVIA 2007, Tokyo, May 15, 2007 EVIA 2007, Tokyo, May 15, 2007