Reproducible IR needs an (IR) (Graph) Query Language Chris Kamphuis - - PowerPoint PPT Presentation

▶
reproducible ir needs an ir graph query language
SMART_READER_LITE
LIVE PREVIEW

Reproducible IR needs an (IR) (Graph) Query Language Chris Kamphuis - - PowerPoint PPT Presentation

Oct 06, 2022 312 likes •469 views

Reproducible IR needs an (IR) (Graph) Query Language Chris Kamphuis and Arjen P. de Vries Problem Different implementations of the same ranking function can produce very different effectiveness scores Problem Different implementations of the

slide-1
SLIDE 1

Reproducible IR needs an (IR) (Graph) Query Language

Chris Kamphuis and Arjen P. de Vries

slide-2
SLIDE 2

Problem

Different implementations of the same ranking function can produce very different effectiveness scores

slide-3
SLIDE 3

Problem

Different implementations of the same ranking function can produce very different effectiveness scores System MAP P@5 Indri 0.246 0.304 MonetDB and VectorWise 0.225 0.276 Lucene 0.216 0.265 Terrier 0.215 0.272

Effectiveness scores BM25 ClueWeb121

1 Mühleisen et al. (2014)

slide-4
SLIDE 4

Problem

Different implementations of the same ranking function can produce very different effectiveness scores System MAP@1000 ATIRE 0.2902 Lucene 0.3029 MG4J 0.2994 Terrier 0.2687

Effectiveness scores BM25 .GOV22

2 Arguello et al. (2015)

slide-5
SLIDE 5

Problem

Different implementations of the same ranking function can produce very different effectiveness scores System AP P@30 NDCG@20 Anserini 0.2531 0.3102 0.4240 ATIRE 0.2184 0.3199 0.4211 ielab 0.1826 0.2605 0.3477 Indri 0.2338 0.2995 0.4041 OldDog 0.2434 0.2985 0.4002 PISA 0.2534 0.3120 0.4221 Terrier 0.2363 0.2977 0.4049

Effectiveness scores BM25 Robust043

3 Clancy et al. (2019)

slide-6
SLIDE 6

Reasons for differences

Investigating why results differ is not easy

slide-7
SLIDE 7

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
slide-8
SLIDE 8

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
  • Different hyperparameter settings?
slide-9
SLIDE 9

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
  • Different hyperparameter settings?
  • Different function for IDF?
slide-10
SLIDE 10

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
  • Different hyperparameter settings?
  • Different function for IDF?
  • Should documents without at least one keyword match be scored?
slide-11
SLIDE 11

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
  • Different hyperparameter settings?
  • Different function for IDF?
  • Should documents without at least one keyword match be scored?
  • Wrong implementation?
slide-12
SLIDE 12

Reasons for differences

Investigating why results differ is not easy

  • Different preprocessing?
  • Different hyperparameter settings?
  • Different function for IDF?
  • Should documents without at least one keyword match be scored?
  • Wrong implementation?

Components Data management, processing, algorithms are all build on top of each

  • ther!
slide-13
SLIDE 13

Use a database

Split data management from query processing By representing the data in a database

  • Easier to see differences in document representation
  • Ranking functions need to be expressed precisely
slide-14
SLIDE 14

Use a database

A relational database has limitations When adding meta-data, entity information etc. the relational model is inconvenient for documents.

slide-15
SLIDE 15

Use a database

A graph database to represent more complex data Solution: Use a graph database where expressing queries that deal with more complex data structures are more easily expressed.