PROMISE Participative Research labOratory for Multimedia and - - PowerPoint PPT Presentation

▶
promise
SMART_READER_LITE
LIVE PREVIEW

PROMISE Participative Research labOratory for Multimedia and - - PowerPoint PPT Presentation

Aug 06, 2023 377 likes •798 views

PROMISE Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation Overall Project Presentation Coordinator: Nicola Ferro University of Padua, Italy December 2010 Network of

slide-1
SLIDE 1

Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191 http://www.promise-noe.eu/

PROMISE

Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation

  • Coordinator:

Nicola Ferro University of Padua, Italy

December 2010

Overall Project Presentation

slide-2
SLIDE 2

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Overview

2

slide-3
SLIDE 3

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Fact Sheet

  • PROMISE: Participative Research labOratory for Multimedia and Multilingual

Information Systems Evaluation

  • Network of Excellence: Grant Agreement 258191
  • Duration: 1st September 2010 - 31st August 2013 (36 months)
  • Effort: 314 man/months
  • Budget: 3,529,357 euros (EC contribution)
  • Consortium: 10 partners
  • University of Padua (UNIPD), Italy
  • Swedish Institute of Computer Science (SICS), Sweden
  • University of Amsterdam (UvA), The Netherlands
  • Sapienza University of Rome (ROMA1), Italy
  • University of Applied Sciences Western Switzerland (HES-SO), Switzerland
  • Information Retrieval Facility (IRF), Austria
  • Zurich University of Applied Sciences (ZHAW), Switzerland
  • Humboldt-Universität zu Berlin (UBER), Germany
  • Evaluations and Language resources Distribution Agency (ELDA), France
  • Centre for the Evaluation of Language Communication Technologies (CELCT), Italy
  • Web site: http://www.promise-noe.eu/

3

slide-4
SLIDE 4

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Goal

  • To advance the experimental evaluation of complex

multimedia and multilingual information systems in order to support the decision making process of individuals, commercial entities, and communities who develop, employ, and improve such complex systems

  • PROMISE will provide a virtual and open laboratory for

conducting participative research and experimentation in which it will be possible to carry out, advance and bring automation into the evaluation and benchmarking of complex multimedia and multilingual information systems, by facilitating management and offering access, curation, preservation, re-use, analysis, visualization, and mining of the collected experimental data.

4

slide-5
SLIDE 5

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Use Cases: Unlocking Culture

5

La Joconde! La Gioconda! Mona Lisa! "#$% &'(%!

It deals with information access to cultural heritage material held in large-scale digital libraries comprising libraries, archives, museum, and audio-visual archives.

slide-6
SLIDE 6

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

It deals with patent search and its peculiar requirements to seek out standardized method and framework for evaluating different tools for the intellectual property.

Use Cases: Search For Innovation

6

slide-7
SLIDE 7

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

It deals with patent search and its peculiar requirements to seek out standardized method and framework for evaluating different tools for the intellectual property.

Use Cases: Search For Innovation

6

!

slide-8
SLIDE 8

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Use Cases: Visual Clinical Decision Support

7

It deals with visual information connected with text in the radiology domain in order to provide retrieval and access mechanisms able to jointly exploit textual and visual features.

slide-9
SLIDE 9

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Use Cases: The Web

8

!

Translate Flickr

slide-10
SLIDE 10

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Use Cases: The Web

8

!

Translate Flickr

!

Translate Flickr

slide-11
SLIDE 11

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Matching Different Needs

9

!"#$%&'

slide-12
SLIDE 12

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Approach

  • Regular Evaluation Activities
  • Evaluation tasks based on use cases
  • New evaluation methods and metrics
  • Open evaluation infrastructure
  • Large and multidisciplinary community
  • Automation in the Evaluation Process
  • Automate the evaluation activities
  • Continuous experimentation
  • Distributed evaluation protocols
  • Collaboration and Re-use of knowledge-base
  • Visual analytics
  • Annotation and discussion support
  • Knowledge Transfer and Uptake
  • The CLEF Initiative
  • Technology take-up group
  • Technology transfer day
  • Researchers exchange program
  • Two summer schools
  • Best practices
  • Three brainstorming workshops (at least) and

tutorials

10

slide-13
SLIDE 13

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Expected Impact

11

1 2 3 Year

Volume of Data Usage Effort

slide-14
SLIDE 14

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Exploitation: Virtual Institute

Collaborative and Open Evaluation Infrastructure Acquired Knowledge-base Evaluation Methodologies and Metrics Use Cases and Evaluation Tasks Use Cases and Evaluation Tasks Use Cases and Evaluation Tasks Use Cases and Evaluation Tasks Use Cases and Evaluation Tasks Use Cases and Evaluation Tasks Scientific and Technological Advancement in Multilingual and Multimedia Information Systems

12

slide-15
SLIDE 15

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Work Packages

13

Stakeholders Involvement and Technology Transfer

WP2

Evaluation Infrastructure

WP3

Evaluation Activities

WP6

Collaboration and Knowledge Sharing

WP5

Evaluation Methods and Metrichs

WP4

Dissemination, IPR, and Resources

WP7

slide-16
SLIDE 16

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

WP Leaders

  • WP1: Nicola Ferro, UNIPD
  • WP2: Jussi Karlgren, SICS
  • WP3: Maristella Agosti, UNIPD
  • WP4: Maarten de Rijke, UvA
  • WP5: Giuseppe Santucci, ROMA1
  • WP6: Henning Müller, HES-SO
  • WP7: Khalid Choukri, ELDA

14

slide-17
SLIDE 17

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Advisory Board

  • Ricardo Baeza-Yates, Yahoo! Research,

Barcelona, Spain

  • Makx Dekkers, Independent Consultant,

Luxembourg

  • Donna Harman, Scientist Emeritus, National

Institute of Standards and Technology (NIST), USA

  • Kalervo Järvelin, University of Tampere, Finland
  • Peter Pirolli, Palo Alto Research Center (PARC),

USA

15

slide-18
SLIDE 18

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

The CLEF Initiative

16

slide-19
SLIDE 19

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF “Classic” Objectives

  • Promote research and stimulate development of

multilingual IR systems for European (and non- European) languages, through

  • Creation of evaluation infrastructure and organisation of

regular evaluation campaigns for system testing

  • Building of an MLIA/CLIR research community
  • Construction of publicly available test-suites
  • Major Goal
  • To anticipate the emerging needs of the R&D community, and
  • Encourage development of truly multilingual, multimodal

systems

17

slide-20
SLIDE 20

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF “Classic”: Ten Years+ of History

  • 1997 – First CLIR system evaluation campaigns in US and Japan: TREC

and NTCIR

  • CLEF actually began life in 1997 as a track for Cross Language Information

Retrieval (CLIR) within TREC. Mainly, English centered tasks (EN -> X, X -> EN).

  • 2000-2009 – CLIR evaluation in Europe: CLEF (extension of CLIR track at

TREC)

  • Fully multilingual, multimodal information retrieval systems capable of processing a

query in any medium and any language finding relevant information from a multilingual multimedia collection containing documents in any language and form, and presenting it in the style most likely to be useful to the user

  • Funding
  • DELOS NoE under FP5 2000 – 2003 (http://delos-noe.isti.cnr.it/)
  • DELOS NoE under FP6 2004 – 2007 (http://www.delos.info/)
  • TrebleCLEF under FP7 2008-2009 (http://www.trebleclef.eu/)

18

slide-21
SLIDE 21

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF “Classic” Achievements

  • Stimulation of research activity in new, previously

unexplored areas

  • Study and implementation of evaluation methodologies

for diverse types of cross-language IR systems

  • Creation of a large set of empirical data about multilingual

information access from the user perspective

  • Quantitative and qualitative evidence with respect to best

practice in cross-language system development

  • Creation of reusable test collections for system

benchmarking

  • Building of a strong, multidisciplinary research community

19

slide-22
SLIDE 22

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

The CLEF Initiative: Vision

20

CLEF 2010 as a bridge to the future

slide-23
SLIDE 23

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Approach

21

Scientific and Technological Advancement in! Multilingual and Multimedia Information Systems! Conference! Labs and Workshops! Labs and Workshops! Labs and Workshops! Labs and Workshops! Labs and Workshops! Labs and Workshops!

slide-24
SLIDE 24

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Conference and Labs/Workshops

  • Conference
  • Two days
  • Large program committee
  • Keynote talks and panels
  • Publication in Springer LNCS
  • Labs
  • Two days (more space than in CLEF classic)
  • Lab selection committee
  • Online publication (with ISBN) in time for the conference
  • Lab organizers are responsible for individual outcomes

for post-conference publication (special issues, ...)

22

slide-25
SLIDE 25

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF 2010 Conference Topics

  • Novel methodologies for the design of evaluation tasks, especially user-centric ones;
  • Analysis of the impact of multilingual/multicultural/multimodal differences in interface and search

design;

  • Assessing multilinguality and multimodality in relevant application communities, e.g. digital libraries,

intellectual property, medical, music, video, and social media.

  • Alternative methods for improving and automating ground-truth creation, for example crowd-

sourcing or clicklog-based;

  • Prediction of success and satisfaction rates;
  • Task-oriented metrics of success and failure;
  • Evaluation of technology vs testing of scientific theories;
  • Innovative and easy to communicate techniques for analysing the experimental results, including

statistical analyses, data mining, and information visualization;

  • Alternatives for and comparison of item-based, list-based, set-based, and session-based evaluation;
  • Simulation (of queries, sessions, users) and information retrieval;
  • Infrastructures for bringing automation and collaboration in the evaluation process;
  • Component-based evaluation approaches;
  • Evaluation and analysis using private or anonymized test data;
  • Living laboratories and evaluating live systems.

23

slide-26
SLIDE 26

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF 2010 Labs and Workshops

  • Benchmarking activities
  • CLEF-IP: A benchmarking activity on intellectual property.
  • ImageCLEF: A benchmarking activity on image retrieval.
  • PAN: A benchmarking activity on plagiarism detection.
  • RespubliQA: A benchmarking activity on question answering using

multilingual political data.

  • WePS: A benchmarking activity on web people search.
  • Workshops
  • CriES: A workshop aimed at exploring the evaluation of searching

for expertise in social media.

  • LogCLEF: A workshop aimed at exploring methodologies for

studying search engine log files.

24

New New New

slide-27
SLIDE 27

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

More Information on CLEF 2010

25

http://www.clef2010.org/

slide-28
SLIDE 28

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF 2011: 19-22 September 2011, Amsterdam

26

http://www.clef2011.org/

slide-29
SLIDE 29

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

CLEF 2012 and Beyond

27

Open bid for hosting CLEF from 2012 onwards

slide-30
SLIDE 30

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

The Evaluation Infrastructure

28

slide-31
SLIDE 31

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Objectives

  • Manage and provide access to the

scientific data produced during evaluation activities

  • Support the organization and running of

evaluation campaigns

  • Increase the automation in the evaluation

process

  • Provide component-based evaluation
  • Foster the usage of the managed scientific

data

29

slide-32
SLIDE 32

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Requirements and Use Cases

  • Keep the users at the

center of the design

  • Drive us in pursuing the
  • bjectives
  • Shape the actual

functionalities

30

slide-33
SLIDE 33

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Scientific Data Management

  • Data modelling
  • Metadata modelling
  • description
  • administration
  • rights management
  • Unique identification
  • citation
  • re-use
  • Provenance

31

slide-34
SLIDE 34

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191) 32

!"#$%&'(") *++(++,")

  • $"./&0$%1)

!"#$%&'(") *++(++,") 2&+&1,"3)

  • $"./&0$%1)

$%4) !"#$%&'(") 2&+&1,"3)

  • $"./&0$%1)

$%4) !"#$%&'(") 2&+&1,"3)

  • $"./&0$%1)

$%4) !"#$%&'(")

  • "(0$"$.,%),5)

6,/78(%1+) 9"($.,%) ,5):,0&/+) ;<0("&8(%1) =7>8&++&,%) 9"($.,%),5)

  • ,,?+)

@(?(A$%/() *++(++8(%1)

  • ("5,"8$%/()

B($+7"(+) =/&(%.C/)

  • ",47/.,%)

6$1$) D % 5 , " 8 $ . , % ) E % , F ? ( 4 # ( ) G&+4,8)

=1$.+./$?) *%$?H+(+)

Evaluation Campaigns Support

slide-35
SLIDE 35

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Automation Increase

33

  • Automatic evaluation of a system
  • increase in the number of experiment
  • more detailed information
  • reduction of the effort to evaluate a system
  • more comparability
  • Programmatic access
  • increase in the number of experiment
  • new usages and new applications driven by the

community

  • evaluation as part of the development process,

like unit testing

  • pportunistic use by third parties with

confidentiality requirements

  • Improve ground-truth creation
  • reduction of human effort
  • pportunity for more user-centred evaluation
slide-36
SLIDE 36

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Component-based Evaluation

  • Need for deeper understanding of component-to-

component interaction

  • a step beyond black-box evaluation
  • Different architectural approaches
  • programming frameworks (UIMA-like)
  • asynchronous, loosely-coupled protocols (Grid@CLEF-like)
  • Web services
  • Extremely challenging
  • computational resources
  • amount of data produces
  • shift in the evaluation paradigm

34

slide-37
SLIDE 37

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Improved Usage

  • Improved collaboration
  • annotation support
  • Improved interpretation
  • visual analytics

35

slide-38
SLIDE 38

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Further Information

36

  • PROMISE Web Site
  • http://www.promise-noe.eu/
  • PROMISE on YouTube
  • http://www.youtube.com/promiseNoE
  • PROMISE on Flickr
  • http://www.flickr.com/photos/promisenoe/
  • PROMISE on Bibsonomy
  • http://www.bibsonomy.org/user/promisenoe
  • CLEF 2010
  • http://www.clef2010.org/
  • CLEF 2011
  • http://www.clef2011.org/
slide-39
SLIDE 39

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Contact

  • Dr. Nicola Ferro

Department of Information Engineering University of Padua Via Gradenigo 6/b 35131 Padova, Italy e-mail: ferro@dei.unipd.it Tel: +39 049 827 7939 Fax: +39 049 827 7799

37

slide-40
SLIDE 40

!"#$%&'

Overall Project Presentation December 2010

PROMISE NoE (FP7, grant agreement no. 258191)

Thank You

38