[PPT] - CLEF: Ongoing Activities and Plans for the Future Maristella Agosti PowerPoint Presentation

SLIDE 1

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting – – Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007

Maristella Agosti Giorgio Di Nunzio Nicola Ferro

University of Padua Italy

{agosti, dinunzio, ferro}@dei.unipd.it

Carol Peters

IS TI-CNR, Area di Ricerca Italy

carol.peters@isti.cnr.it

CLEF: Ongoing Activities and Plans for the Future

SLIDE 2

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 2 2 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Objectives CLEF Objectives

Promote research and stimulate development of multilingual

Promote research and stimulate development of multilingual IR systems for European (and non IR systems for European (and non-

European) languages,

European) languages, through through

Creation of evaluation infrastructure and organisation of regula

Creation of evaluation infrastructure and organisation of regular r evaluation campaigns for system testing evaluation campaigns for system testing

Building of an MLIA/ CLIR research community

Building of an MLIA/ CLIR research community

Construction of publicly available test

Construction of publicly available test -

suites

suites

Major Goal

Major Goal

To anticipate the emerging needs of the R&D community, and

To anticipate the emerging needs of the R&D community, and

Encourage development of truly multilingual, multimodal systems

Encourage development of truly multilingual, multimodal systems

SLIDE 3

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 3 3 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF CLEF History History

1997

1997 – – First CLIR system evaluation campaigns in US and Japan: First CLIR system evaluation campaigns in US and Japan: Text REtrieval Conference (TREC) and NTCIR TREC) and NTCIR

CLEF actually began life in 1997 as a track for Cross Language Information

Retrieval (CLIR) within TREC. Mainly, English centered tasks (EN -> X, X -> EN).

2000

2000 – – CLIR evaluation in Europe: CLEF (extension of CLIR track at TRE CLIR evaluation in Europe: CLEF (extension of CLIR track at TREC) C)

Fully multilingual, multimodal information retrieval systems

Fully multilingual, multimodal information retrieval systems

capable of processing a query in any medium and any language

capable of processing a query in any medium and any language

finding relevant information from a multilingual multimedia coll

finding relevant information from a multilingual multimedia collection containing ection containing documents in any language and form, documents in any language and form,

and presenting it in the style most likely to be useful to the u

and presenting it in the style most likely to be useful to the user ser

The European Network of Excellence on Digital Libraries (DELOS

) The European Network of Excellence on Digital Libraries (DELOS ) has has supported CLEF since 2000 (however, this support will end this y supported CLEF since 2000 (however, this support will end this year 2007) ear 2007)

DELOS

under FP5 2000 DELOS under FP5 2000 – – 2003 ( 2003 (http:/ / delos http:/ / delos-

noe.isti.cnr.it

noe.isti.cnr.it / ) / )

DELOS

under FP6 2004 DELOS under FP6 2004 – – 2007 (http:/ / 2007 (http:/ / www.delos.info www.delos.info/ ) / )

SLIDE 4

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 4 4 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Technical Infrastructure History CLEF Technical Infrastructure History

1997

1997 – – 1999: CLEF as CLIR Track at TREC 1999: CLEF as CLIR Track at TREC

same infrastructure developed at NIS

T same infrastructure developed at NIS T

2000

2000 – – 2004: CLEF as CLIR stand 2004: CLEF as CLIR stand-

alone European initiative

alone European initiative

Martin

Martin Braschler Braschler developed and managed the CLEF infrastructure, developed and managed the CLEF infrastructure, starting from the TREC one starting from the TREC one

2005

2005 – – 2007: CLEF as CLIR stand 2007: CLEF as CLIR stand-

alone European initiative

alone European initiative

University of Padua designed and developed a new infrastructure,

University of Padua designed and developed a new infrastructure, applying concepts and approaches of the digital library field applying concepts and approaches of the digital library field

SLIDE 5

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 5 5 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF CLEF Organisation Organisation

CLEF is mainly a voluntary activity: the following Institutions

CLEF is mainly a voluntary activity: the following Institutions contributed to contributed to the the organisation

rganisation of the different tracks of the CLEF 2006 campaign:
f the different tracks of the CLEF 2006 campaign:
Centre

Centre for the for the Evaluation Evaluation of

f Human

Human Language Language and and Multimodal Multimodal Communication Communication Technologies (CELCT), Technologies (CELCT), Trento, Trento, Italy Italy

Centro per la Ricerca S

cientifica e Tecnologica, Centro per la Ricerca S cientifica e Tecnologica, Istituto Trentino di Cultura, Trento, Istituto Trentino di Cultura, Trento, Italy Italy

College of

College of Information Information S tudies S tudies and and Institute Institute for for Advanced Advanced Computer Computer S tudies S tudies, , U.

U. Maryland, US

A Maryland, US A

Dept

Dept . of Computer . of Computer S cience S cience, , U.

U. Indonesia

Indonesia

Depts
Depts. of Computer

. of Computer S cience S cience & & Medical Medical Informatics Informatics, , RWTH RWTH Aachen Aachen U., U., Germany Germany

Dept

Dept . of Computer S cience and . of Computer S cience and Information Information S ystems S ystems, , U.

U. Limerick

Limerick, , Ireland Ireland

Dept

Dept . of Computer . of Computer S cience S cience and and Information Information Engineering Engineering, , National National U.

U. Taiwan

Taiwan

Dept

Dept . of . of Information Information Engineering Engineering, , U.

U. Padua,

Padua, Italy Italy

Dept

Dept . of . of Information Information S ci, S ci, U.

U. Hildesheim,

Hildesheim, Germany Germany

Dept

Dept . of . of Information Information S tudies S tudies, , U.

U. S

heffield, UK S heffield, UK

Evaluations and Language Resources

Evaluations and Language Resources Distribution Distribution Agency Agency S arl S arl, Paris, , Paris, France France

German

German Research Research Centre Centre for for Artificial Artificial Intelligence, Intelligence, DFKI, DFKI, S aarbr S aarbrü

ücken

cken, , Germany Germany

Information

Information and and Language Language Processing Processing S ystems S ystems, , U. U. Amsterdam, Amsterdam, Netherlands Netherlands

IZ Bonn,

IZ Bonn, Germany Germany

Inst

Inst . . Information Information Technology Technology, , Hyderabad Hyderabad, India , India

LS

I LS I-

UNED, Madrid,

UNED, Madrid, S pain S pain

Linguateca

Linguateca, , S intef S intef , Oslo, , Oslo, Norway Norway

Linguistic

Linguistic Modelling Modelling Lab., Lab., Bulgarian Bulgarian Acad Acad S ci S ci

NIS

T, US A NIS T, US A

Biomedial

Biomedial Informatics Informatics, Oregon , Oregon Health Health and and S cience S cience University, US A University, US A

Research Computing

Research Computing Cente Cente, , Moscow Moscow S tate S tate U. U.

Research Institute for Linguistics,

Research Institute for Linguistics, Hungarian Hungarian Academy Academy of

f S

ciences S ciences

S

chool S chool of Computer

f Computer S

cience S cience and and Mathematics Mathematics, , Victoria U., Australia Victoria U., Australia

S

chool S chool of Computing, DCU,

f Computing, DCU, Ireland

Ireland

UC Data

UC Data Archive Archive and and S chool S chool of

f Information

Information Management and Management and S ystems S ystems, UC , UC Berkeley Berkeley, US A , US A

University "

University "Alexandru Alexandru Ioan Ioan Cuza Cuza", IAS I, Romania ", IAS I, Romania

U.
U. Hospitals

Hospitals and U and U.of .of Geneva Geneva, , S witzerland S witzerland

SLIDE 6

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 6 6 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Track CLEF Track Evolution Evolution

CLEF 2000 CLEF 2000

mono

mono-

, bi

, bi-

and multilingual text document retrieval (

and multilingual text document retrieval (Ad Ad-

Hoc

Hoc) )

mono

mono-

and cross

and cross-

language tasks on structured scientific data (

language tasks on structured scientific data (Domain Domain-

S

pecific S pecific) )

CLEF 2001 CLEF 2001 Added Added

interactive cross

interactive cross-

language retrieval (

language retrieval (iCLEF iCLEF) )

CLEF 2002 CLEF 2002 Added Added

cross

cross-

language spoken document retrieval (

language spoken document retrieval (CL CL-

SR

SR) )

CLEF 2003 CLEF 2003 Added Added

multiple language question answering (

multiple language question answering (QA@ CLEF QA@ CLEF) )

cross

cross-

language retrieval in image collections (

language retrieval in image collections (ImageCLEF ImageCLEF) )

CLEF 2005 CLEF 2005 Added Added

multilingual retrieval of Web documents (

multilingual retrieval of Web documents (WebCLEF WebCLEF) )

cross

cross-

language geographical retrieval (

language geographical retrieval (GeoCLEF GeoCLEF) )

SLIDE 7

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 7 7 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF 2000 CLEF 2000-

2006

2006 Participation Participation

CLEF 2000-2006 Participation

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 2000 2001 2002 2003 2004 2005 2006 Oceania South America North America Asia Europe

CLEF 2006: over 100 groups registered; 90 actually submitted runs for 1 or more tasks. Most groups from academia; just 9 research groups from industry.

SLIDE 8

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 8 8 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF CLEF Shift Shift in in Focus Focus

CLEF 2000-2006 Tracks

5 10 15 20 25 30 35 40

2000 2001 2002 2003 2004 2005 2006

Years

Participating Groups AdHoc DomSpec iCLEF CL-SR QA@CLEF ImageCLEF WebClef GeoClef

Breakdown of groups per track in 2006: Ad Hoc 25; Domain-Specific 4; iCLEF 3; QAatCLEF 37; ImageCLEF 25; CL-SR 6; WebCLEF 8; GeoCLEF 17

SLIDE 9

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 9 9 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Tracks: CLIR Text Retrieval CLEF Tracks: CLIR Text Retrieval

Ad

Ad-

hoc track (from 2000)

hoc track (from 2000)

mono

mono-

, bi

, bi-

and multilingual system performance tested

and multilingual system performance tested

bilingual task testing non

bilingual task testing non-

European languages (Amharic, Chinese, Hindi,

European languages (Amharic, Chinese, Hindi, Indonesian, Oromo, Telugu) against English target collection Indonesian, Oromo, Telugu) against English target collection

“

“ multi multi-

8 two

8 two-

years

years-

on / merging
n / merging”

” checked improvement in system checked improvement in system performance over time (CLEF 2003 performance over time (CLEF 2003

2005)

2005) and focused on problem of and focused on problem of merging results form different collections/ languages merging results form different collections/ languages

“

“ robust robust ” ” task emphasized importance of stable performance over task emphasized importance of stable performance over languages instead of high average performance languages instead of high average performance

Domain

Domain-

specific track (from 2000)

specific track (from 2000)

cross

cross-

language retrieval on German and Russian databases of social

language retrieval on German and Russian databases of social sciences sciences

SLIDE 10

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 10 10 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF CLEF Tracks Tracks: : Ad Ad-

Hoc

Hoc Tasks Tasks

Monolingual Monolingual Bilingual Bilingual Multilingual Multilingual CLEF 2000 CLEF 2000

DE;FR;IT DE;FR;IT X X→

→EN

EN X X→

→DE;EN;FR;IT

DE;EN;FR;IT

CLEF 2001 CLEF 2001

DE; ES ; FR; IT; NL DE; ES ; FR; IT; NL X X→

→EN, X

EN, X→

→NL

NL X X→

→DE;EN;ES

;FR;IT DE;EN;ES ;FR;IT

CLEF 2002 CLEF 2002

DE; ES ; FI; FR DE; ES ; FI; FR IT; NL; S V IT; NL; S V X X→

→DE;ES

;FI;FR;IT;NL;S V DE;ES ;FI;FR;IT;NL;S V X X→

→EN (newcomer)

EN (newcomer) X X→

→DE;EN;ES

;FR;IT DE;EN;ES ;FR;IT

CLEF 2003 CLEF 2003

DE; ES ; FI; FR DE; ES ; FI; FR IT; NL; RU; S V IT; NL; RU; S V IT IT→

→ES; DE

ES; DE→

→IT FR

IT FR→

→NL;

NL; FI FI→

→DE X

DE X→

→RU; X

RU; X→

→EN

EN X X→

→DE;EN;ES

;FR DE;EN;ES ;FR X X→

→DE;EN;ES

;FI FR;IT;NL;S V DE;EN;ES ;FI FR;IT;NL;S V

CLEF 2004 CLEF 2004

FI; FR; RU; PT FI; FR; RU; PT ES / FR/ IT/ RU ES / FR/ IT/ RU→

→FI

FI DE/ FI/ NL/ SV DE/ FI/ NL/ SV→

→FR

FR X X→

→RU;X

RU;X→

→EN

EN X X→

→FI;FR;RU;PT

FI;FR;RU;PT

CLEF 2005 CLEF 2005

BG; FR; HU; PT BG; FR; HU; PT X X→

→ BG;FR;HU;PT

BG;FR;HU;PT X X →

→EN

EN Multi8 2yrson Multi8 2yrson Multi8 merge Multi8 merge

CLEF 2006 CLEF 2006

BG; FR; HU; PT BG; FR; HU; PT X X→

→ BG;FR;HU;PT

BG;FR;HU;PT X X →

→EN

EN Robust Robust

CLEF 2007 CLEF 2007

BG; CZ; HU BG; CZ; HU X X→

→ BG;CS

;HU BG;CS ;HU X X →

→EN

EN (None) (None)

SLIDE 11

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 11 11 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF CLEF Tracks Tracks: : Ad Ad-

Hoc

Hoc Results Results Over the Over the Years Years

TREC

TREC-

6, 1997:

6, 1997:

EN

EN→

→FR: 49%

f best

FR: 49%

f best monolingual

monolingual French French system system

EN

EN→

→DE: 64%

f best

DE: 64%

f best monolingual

monolingual German German system system

CLEF 2002:

CLEF 2002:

EN

EN→

→FR: 83,4%

f best

FR: 83,4%

f best monolingual

monolingual French French system system

EN

EN→

→DE: 85,6%

f best

DE: 85,6%

f best monolingual

monolingual German German system system

CLEF 2003

CLEF 2003 enforced enforced the the use use of

f “

“ unusual unusual” ” language language pairs pairs: :

IT

IT→

→ES

: 83%

f best

ES : 83%

f best monolingual

monolingual S panish S panish IR system IR system

DE

DE→

→IT: 87%

f best

IT: 87%

f best monolingual

monolingual Italian Italian IR system IR system

FR

FR→

→NL: 82%

f best

NL: 82%

f best monolingual

monolingual Dutch Dutch IR system IR system

CLEF 2005

CLEF 2005

X

X→

→FR: 85%

f best

FR: 85%

f best monolingual

monolingual French French IR system IR system

X

X→

→PT: 88%

f best

PT: 88%

f best monolingual

monolingual Portuguese Portuguese IR system IR system

X

X→

→BG: 74%

f best

BG: 74%

f best monolingual

monolingual Bulgarian Bulgarian IR system IR system

X

X→

→HU: 73%

f best

HU: 73%

f best monolingual

monolingual Hungarian Hungarian IR system IR system

CLEF 2006

CLEF 2006 results results of

f “

“ estabilished estabilished” ” languages languages: :

X

X→

→FR: 94%

f best

FR: 94%

f best monolingual

monolingual French French IR system IR system

X

X→

→PT: 91%

f best

PT: 91%

f best monolingual

monolingual Portuguese Portuguese IR system IR system

X

X→

→BG: 52%

f best

BG: 52%

f best monolingual

monolingual Bulgarian Bulgarian IR system IR system

X

X→

→HU: 51%

f best

HU: 51%

f best monolingual

monolingual Hungarian Hungarian IR system IR system

nly one

participant for CLIR BG and HU

SLIDE 12

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 12 12 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Tracks: CLEF Tracks: Cross Cross-

Language Information Extraction (1/2)

Language Information Extraction (1/2)

Multilingual Question Answering

Multilingual Question Answering – – QA@ CLEF (from 2003) QA@ CLEF (from 2003)

mono

mono-

and cross

and cross-

language QA systems: target collections for

language QA systems: target collections for BG/ DE/ EN/ ES / FR/ IT/ NL/ PT BG/ DE/ EN/ ES / FR/ IT/ NL/ PT

pilot tasks: QA using

pilot tasks: QA using Wikipedia Wikipedia; answer validation exercise; ; answer validation exercise; “ “ time time-

constrained

constrained” ” exercise exercise

cross

cross-

language QA is around 60%
f monolingual QA

language QA is around 60%

f monolingual QA
Cross

Cross-

language Geographic Retrieval

language Geographic Retrieval -

GeoCLEF

GeoCLEF (from 2005) (from 2005)

Aim: to evaluate retrieval of multilingual documents with an emp

Aim: to evaluate retrieval of multilingual documents with an emphasis on hasis on geographic search geographic search

Many documents contains geo

Many documents contains geo-

references expressed in multiple languages

references expressed in multiple languages

Collections:

Collections:

Document languages: English, German, S

panish, Portuguese news Document languages: English, German, S panish, Portuguese news

Topics: English, German, Portuguese and S

panish Topics: English, German, Portuguese and S panish

Monolingual and bilingual ad

Monolingual and bilingual ad-

hoc retrieval tasks

hoc retrieval tasks

SLIDE 13

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 13 13 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Tracks: CLEF Tracks: Cross Cross-

Language Information Extraction (2/2)

Language Information Extraction (2/2)

Web track

Web track– – WebCLEF WebCLEF (from 2005) (from 2005)

Aim: evaluate systems that address multilingual information need

Aim: evaluate systems that address multilingual information needs on the s on the web web

Three tasks organized: mixed monolingual, multilingual, and bili

Three tasks organized: mixed monolingual, multilingual, and bilingual ngual English to S panish, with homepage and named page finding queries English to S panish, with homepage and named page finding queries for all for all tasks tasks

SLIDE 14

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 14 14 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Tracks: CLEF Tracks: Cross Cross-

Language Multimedia Retrieval (1/2)

Language Multimedia Retrieval (1/2)

Cross

Cross-

Language S

peech Retrieval (CL Language S peech Retrieval (CL-

S

R) (from 2002) S R) (from 2002)

2002

2002-

2004: evaluation of CLIR systems on noisy automatic transcripts

2004: evaluation of CLIR systems on noisy automatic transcripts of

f

spoken EN documents spoken EN documents

Written queries in ES

/ FR/ DE/ IT/ NL Written queries in ES / FR/ DE/ IT/ NL

retrieval with/ without known story boundaries

retrieval with/ without known story boundaries

use of multiple automatic transcriptions

use of multiple automatic transcriptions

from 2004: encourage development of technologies to facilitate a

from 2004: encourage development of technologies to facilitate access to ccess to spontaneous speech spontaneous speech

Mono

Mono-

and cross

and cross-

language retrieval systems are assessed on the

language retrieval systems are assessed on the Malach Malach collection of oral interviews ( collection of oral interviews (S hoah S hoah archives: English & Czech collections) archives: English & Czech collections)

Topics prepared in 5 languages (CZ, EN, FR, DE, ES

) Topics prepared in 5 languages (CZ, EN, FR, DE, ES )

A thesaurus, an in

A thesaurus, an in-

domain expansion collection, and a word lattice are

domain expansion collection, and a word lattice are available available

SLIDE 15

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 15 15 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Tracks: CLEF Tracks: Cross Cross-

Language Multimedia Retrieval (2/2)

Language Multimedia Retrieval (2/2)

ImageCLEF

ImageCLEF (from 2003) (from 2003)

Aim

Aim

To promote the evaluation of image retrieval and unite both imag

To promote the evaluation of image retrieval and unite both image and text e and text retrieval communities retrieval communities

To investigate how visual and textual features can best be comb

To investigate how visual and textual features can best be combined for ined for retrieval retrieval

2 main

2 main subtracks subtracks: photographic and medical image retrieval (ad hoc and : photographic and medical image retrieval (ad hoc and annotation tasks) annotation tasks)

Interactive CLIR

Interactive CLIR – – iCLEF iCLEF (from 2001) (from 2001)

Cross

Cross-

Language IR from a user

Language IR from a user-

inclusive perspective

inclusive perspective

Interactive document selection/ query formulation

Interactive document selection/ query formulation

How can interaction with user help a QA system

How can interaction with user help a QA system

How can interaction with user help an image retrieval system

How can interaction with user help an image retrieval system

task based on

task based on Flickr Flickr database: images with textual comments, captions, and database: images with textual comments, captions, and titles in many languages by using the so titles in many languages by using the so-

called

called folksonomy folksonomy

SLIDE 16

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 16 16 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Test Collections CLEF Test Collections

Ad Hoc,

Ad Hoc, QA@ CLEF QA@ CLEF, , iCLEF iCLEF, , GeoCLEF GeoCLEF

CLEF

CLEF multilingual multilingual comparable comparable corpus corpus of more

f more than

than 2M news 2M news docs docs in 13 in 13 languages languages: :

BG, CZ, DE, EN, ES

, FI, FR, HU, IT, NL, RU, SV, PT BG, CZ, DE, EN, ES , FI, FR, HU, IT, NL, RU, SV, PT

Domain

Domain-

S

pecific S pecific

GIRT

GIRT-

4 social

4 social science science database in EN and DE: more database in EN and DE: more that that 300,000 300,000 docs docs

Russian

Russian S

cial

S

cial S

cience S cience Corpus & IS IS S database: Corpus & IS IS S database: almost almost 250,000 250,000 docs docs

ImageCLEF

ImageCLEF

IAPR TC

IAPR TC-

12 database: 25,000

12 database: 25,000 phot os phot os with with captions captions in EN, DE, ES in EN, DE, ES

CasImage

CasImage radiological radiological db db with with case notes in FR and EN: 9,000 case notes in FR and EN: 9,000 images images

PEIR 33,000

PEIR 33,000 images images, MIR 2,000, , MIR 2,000, PathoPic PathoPic 9,000 9,000

IRMA EN/ DE

IRMA EN/ DE collection collection for for medical medical image image annotation annotation: 10,000 : 10,000 images images

Cross

Cross-

Language

Language S peech S peech Retrieval Retrieval

Malach

Malach collection collection of

f spontaneous

spontaneous conversational conversational speech speech derived derived from from the the S hoah S hoah archives archives: : in in English English ( (ca ca 750 750 hours hours); ); Czech Czech ( (ca ca 500 500 hours hours) )

WebCLEF

WebCLEF

EuroGOV

EuroGOV, more , more than than 3.35 M 3.35 M webpages webpages crawled crawled from from European European governmental governmental sites sites

SLIDE 17

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 17 17 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF Overall Results CLEF Overall Results

S

timulation of research activity in new, previously unexplored S timulation of research activity in new, previously unexplored areas, such as cross areas, such as cross-

language question answering, image and

language question answering, image and geographic information retrieval geographic information retrieval

S

tudy and implementation of evaluation methodologies for S tudy and implementation of evaluation methodologies for diverse types of cross diverse types of cross-

language IR systems

language IR systems

Documented improvement in system performance for cross

Documented improvement in system performance for cross-

language text retrieval systems

language text retrieval systems

Creation of a large set of empirical data about multilingual

Creation of a large set of empirical data about multilingual information access from the user perspective information access from the user perspective

Quantitative and qualitative evidence with respect to best

Quantitative and qualitative evidence with respect to best practice in cross practice in cross-

language system development

language system development

Creation of important, reusable test collections for system

Creation of important, reusable test collections for system benchmarking benchmarking

Building of a strong, multidisciplinary research community

Building of a strong, multidisciplinary research community

SLIDE 18

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 18 18 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Plans for the Future Plans for the Future

Challenge in CLEF 2007 is to stimulate transfer of research

Challenge in CLEF 2007 is to stimulate transfer of research results to market place results to market place

S

ame tracks as in 2006 but new tasks will focus on user S ame tracks as in 2006 but new tasks will focus on user-

riented activities (interactive and real
riented activities (interactive and real-
time exercises) and on

time exercises) and on multilingual web searching. multilingual web searching.

Tasks will be activated in collaboration with 2 important

Tasks will be activated in collaboration with 2 important international international NLPevaluation NLPevaluation campaigns (S emEval campaigns (S emEval-

2007 &

2007 & MorphChallenge MorphChallenge 2007) 2007)

Demos of operational systems planned for the workshop

Demos of operational systems planned for the workshop

An investigation into ways and forms to continue CLEF after

An investigation into ways and forms to continue CLEF after 2007 is now underway. External support for overall coordination 2007 is now underway. External support for overall coordination is essential if CLEF is to continue after 2007. is essential if CLEF is to continue after 2007.

SLIDE 19

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 19 19 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

We warmly thank all the participants and We warmly thank all the participants and

rganizers of CLEF who, with their effort and
rganizers of CLEF who, with their effort and

co co-

operation, make CLEF a successful initiative
peration, make CLEF a successful initiative

SLIDE 20

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 20 20 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

For further information see: For further information see:

http://www.clef http://www.clef-

campaign.org

campaign.org

r contact:
r contact:

Carol Peters Carol Peters -

IS

TI IS TI-

CNR

CNR E E-

mail:

mail: carol.peters@isti.cnr.it

carol.peters@isti.cnr.it

SLIDE 21

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 21 21 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT: How to Manage CLEF Scientific Data DIRECT: How to Manage CLEF Scientific Data

SLIDE 22

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 22 22 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

CLEF: CLEF: Need Need for for a a Digital Digital Library Library for for Scientific Scientific Data Data

The experimental evaluation is a scientific activity and, as suc

The experimental evaluation is a scientific activity and, as such, we have to h, we have to realise realise that its outcomes are very valuable scientific data. that its outcomes are very valuable scientific data.

Are these scientific data

Are these scientific data

properly

properly modelled modelled, ,

effectively managed and archived,

effectively managed and archived,

carefully

carefully curated curated and enriched and enriched in the current evaluation methodology? in the current evaluation methodology?

In this context, Digital Library S

ystems ( In this context, Digital Library S ystems (DLS s DLS s) represent the natural choice ) represent the natural choice for managing, making accessible, citing, for managing, making accessible, citing, curating curating, enriching, and preserving , enriching, and preserving all the information resources produced during an evaluation camp all the information resources produced during an evaluation campaign. aign.

SLIDE 23

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 23 23 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Test Test Collections Collections and and Scientific Scientific Data Data

S

cientific data, their enrichment and interpretation are essenti S cientific data, their enrichment and interpretation are essential al components of scientific research. components of scientific research.

The

The Cranfield Cranfield methodology traces out how these scientific data have to be methodology traces out how these scientific data have to be produced, while the statistical analysis of experiments provide produced, while the statistical analysis of experiments provide the means the means for further elaborating and interpreting the experimental result for further elaborating and interpreting the experimental result s. s.

Nevertheless, current methodologies do not require any particula

Nevertheless, current methodologies do not require any particular r coordination or synchronization between the basic scientific dat coordination or synchronization between the basic scientific dat a and the a and the analyses on them, which are treated as almost separated items. analyses on them, which are treated as almost separated items.

Data

E x p e r i m e n t s a n d E x p e r i m e n t a l C

l

l e c t i

n

s

Information Knowledge Wisdom

M e a s u r e s S t a t i s t i c s P a p e r s

SLIDE 24

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 24 24 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Key Points for a New Approach to Design and Key Points for a New Approach to Design and Develop an Evaluation Infrastructure Develop an Evaluation Infrastructure

Introduce a

Introduce a conceptual model conceptual model

it makes clear what are the

it makes clear what are the entities entities entailed by the information space of an entailed by the information space of an evaluation campaign, their evaluation campaign, their features features, and their , and their relationships relationships

logical models

logical models can be derived from it to can be derived from it to manage manage and and preserve preserve the experimental the experimental data data

commonly agreed

commonly agreed data formats data formats for for exchanging information exchanging information can be derived from it can be derived from it

Develop common

Develop common metadata formats metadata formats

they provide meaning to the data, and thereby enable their

they provide meaning to the data, and thereby enable their sharing sharing and and re re-

use

use

they allow to keep track of the

they allow to keep track of the lineage lineage of the managed information

f the managed information
Adopt a

Adopt a unique identification unique identification and and resolution resolution mechanism mechanism

it allows for explicit

it allows for explicit citation citation and and easy access easy access to the scientific data and it supports to the scientific data and it supports the the enrichement enrichement of the scientific data

f the scientific data
Digital

Digital Object Object Identifiers Identifiers (DOI) (DOI) are are valuable valuable possibility possibility since since they they have have been been applied applied to to scientific scientific data and data and allow allow metadata metadata to to be be associated associated with with identified identified

bj ects
bj ects
Give to

Give to organizations

rganizations responsible for evaluation initiatives an

responsible for evaluation initiatives an active role active role in in developing a comprehensive strategy for long developing a comprehensive strategy for long-

lived digital data collections

lived digital data collections

SLIDE 25

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 25 25 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Designing Designing the DLS the DLS

The design of the scientific DLS

is built around five main areas The design of the scientific DLS is built around five main areas of

f modelling

modelling: :

evaluation campaign:

evaluation campaign:

deals with the different aspects of an evaluation forum;

deals with the different aspects of an evaluation forum;

collection:

collection:

concerns the different collections made available by an evaluati

concerns the different collections made available by an evaluation forum;

n forum;
experiments:

experiments:

regards the experiments submitted by the participants and the ev

regards the experiments submitted by the participants and the evaluation metrics aluation metrics computed on these experiments; computed on these experiments;

pool/relevance assessment:

pool/relevance assessment:

is about the pooling method where a set of experiments is pooled

is about the pooling method where a set of experiments is pooled and the documents and the documents retrieved in those experiments are assessed with respect to the retrieved in those experiments are assessed with respect to the topics of the track the topics of the track the experiments belongs to; experiments belongs to;

statistical analysis:

models the different aspects concerning the statistical analysis of the experimental results.

SLIDE 26

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 26 26 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Conceptual Conceptual Model Model

STRUCTURE USE

(0, N) (1, N)

CONSISTOF

(1, N) (1, 1)

SUBSCRIBE

(0, N) (1, 1) (1, N) (0, N)

HASTOPIC

(1, N) (0, N) (1, N)

COMPOSE

(1, N)

ORGANIZE

(0, N) (0, N) (1, N) (1, 1) (1, N) (1, N) (1, N) (1, N) (0, N) (0, N) (1, N) (1, 1) (0, N)

HASFIELD Mandatory COLLECTION ID SUBMIT Timesamp (TS) Priority

(1, N)

USEFIELD

(0, N) (0, N)

Language Parser PARTICIPATE CAMPAIGN Acronym ID Status

(0, N)

USER ID Pwd ContactMail FullName UserType Country Language TRACK ID Description ClosingDate MetricStatus SubmissionStatus TASK ID MaxExperiments Description COLLECTIONFILE ID

(1, 1) (1, 1)

Mime

TOPICCONTENT Content

Mime

ID Language Position CONTAIN COMPOSE TOPICFIELD ID TOPIC ID ITEM Rank Score DOCUMENT ID Content EXPERIMENT ID Handle Type Description SourceLanguage QueryConstruction TopicField HASBLOB

(0, 1) (1, 1)

EXPERIMENTBLOB FileName SubmittedFile FileSize

SLIDE 27

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 27 27 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT: Running Systems Screenshots DIRECT: Running Systems Screenshots

Assessor Tasks

Assessor Tasks

topic creation

topic creation

relevance assessments

relevance assessments

Participant Tasks

Participant Tasks

topics download

topics download

experiment submission

experiment submission

performance measurements and statistical analyses visualization

performance measurements and statistical analyses visualization

SLIDE 28

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 28 28 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (1/6) Interface (1/6)

SLIDE 29

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 29 29 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (2/6) Interface (2/6)

SLIDE 30

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 30 30 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (3/6) Interface (3/6)

SLIDE 31

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 31 31 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (4/6) Interface (4/6)

SLIDE 32

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 32 32 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (5/6) Interface (5/6)

SLIDE 33

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 33 33 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Topic Topic Creation Creation Interface (6/6) Interface (6/6)

SLIDE 34

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 34 34 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Assessor Assessor Interface (1/2) Interface (1/2)

SLIDE 35

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 35 35 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT Assessor Interface (2/2) DIRECT Assessor Interface (2/2)

SLIDE 36

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 36 36 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Participant Participant Interface (1/4) Interface (1/4)

SLIDE 37

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 37 37 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT DIRECT Participant Participant Interface (2/4) Interface (2/4)

Topic Identifiers are DOI

SLIDE 38

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 38 38 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT Participant Interface (3/4) DIRECT Participant Interface (3/4)

SLIDE 39

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 39 39 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

DIRECT Participant Interface (4/4) DIRECT Participant Interface (4/4)

SLIDE 40

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 40 40 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Statistical Statistical Analyses Analyses

SLIDE 41

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 41 41 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

… …and and More More Statistical Statistical Analyses Analyses

In order to improve the current way of comparing bilingual and

In order to improve the current way of comparing bilingual and monolingual retrieval and provide better methods and tools for monolingual retrieval and provide better methods and tools for assessing the performances we need: assessing the performances we need:

more sophisticated analysis techniques to assess performances;

more sophisticated analysis techniques to assess performances;

effective methods and tools for communicating the results.

effective methods and tools for communicating the results.

Given the average performance for each single topic of the

monolingual and bilingual task, we want to study the distribution

f these

performances and employ different statistical tests to verify the following conditions:

the distributions of the performances are similar. the variances of the two distribution are similar. the mean of the two distributions are different.

SLIDE 42

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 42 42 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Mono Mono-

vs

vs Bili Bili Comparison Comparison (1/2) (1/2)

SLIDE 43

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 43 43 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

Mono Mono-

vs

vs Bili Bili Comparison Comparison (2/2) (2/2)

SLIDE 44

The NTCIR Workshop 6 Meeting The NTCIR Workshop 6 Meeting Tokyo, May 15 Tokyo, May 15-

18, 2007

18, 2007 44 44 M.

M. Agosti

Agosti, G.M. Di Nunzio, N. Ferro, C. Peters , G.M. Di Nunzio, N. Ferro, C. Peters

For further information see: For further information see:

http:// http://direct.dei.unipd.it direct.dei.unipd.it

r contact:
r contact:

E E-

mail: