Combining Probabilistic and Translation- Based Models for - - PowerPoint PPT Presentation

▶
combining probabilistic and translation based models for
SMART_READER_LITE
LIVE PREVIEW

Combining Probabilistic and Translation- Based Models for - - PowerPoint PPT Presentation

Nov 15, 2022 315 likes •713 views

Combining Probabilistic and Translation- Based Models for Information Retrieval based on Word Sense Annotations Elisabeth Wolf, Delphine Bernhard, Iryna Gurevych Ubiquitous Knowledge Processing (UKP) Lab Prof. Dr. Iryna Gurevych Fachbereich

slide-1
SLIDE 1

Combining Probabilistic and Translation- Based Models for Information Retrieval based on Word Sense Annotations

Elisabeth Wolf, Delphine Bernhard, Iryna Gurevych Ubiquitous Knowledge Processing (UKP) Lab

  • Prof. Dr. Iryna Gurevych

Fachbereich Informatik Technische Universität Darmstadt

slide-2
SLIDE 2

UKP Motivation: monolingual task

UBC NUS

Comb

  • 1. Increase precision of WSD

Heuristic-based combinations

  • f both annotations

I N D E X I N G

  • 2. Apply translation-based model

+ combination with probabilistic m.

1. ….. 2. ….. 3. ….. 1. ….. 2. ….. 3. ….. 1. ….. 2. ….. 3. …..

Reranking of retrieved documents R E T R I E V A L

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-3
SLIDE 3
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-4
SLIDE 4
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-5
SLIDE 5
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-6
SLIDE 6
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-7
SLIDE 7
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

  • 1. Increase precision of WSD
slide-8
SLIDE 8
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest

0.82 + 0.32 = 1.14

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

0.18 + 0.21 = 0.39

slide-9
SLIDE 9
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="00735486-n"/> <SYNSET SCORE="0.21" CODE="03857483-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest

0.82 + 0.32 = 1.14

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

0.18 + 0.21 = 0.39 CombBest

slide-10
SLIDE 10
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="0111222-n "/> <SYNSET SCORE="0.21" CODE=„0333444-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest
  • CombBest+

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-11
SLIDE 11
  • 1. Increase precision of WSD
  • Four different index types:

<SYNSET SCORE="0.82" CODE="00735486-n"/> <SYNSET SCORE="0.18" CODE="03857483-n"/> <SYNSET SCORE="0.32" CODE="0111222-n "/> <SYNSET SCORE="0.21" CODE=„0333444-n"/> <SYNSET SCORE="0.47" CODE="01252343-n"/>

UBC NUS

  • UBCBest
  • NUSBest
  • CombBest
  • CombBest+

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

  • Terrier, version 2.1
  • Multi field indices: token, lemma, sense (UBCBest, NUSBest, CombBest, CombBest+)
slide-12
SLIDE 12
  • 2. Combination of Retrieval Models

Monolingual translation-based model (TM):

  • Motivation:
  • address the lexical gap problem
  • learn translation probabilities between terms trained on

parallel dataset: dictionary and encyclopedic definitions

  • „the translation probability reflects the association

between query term and document term”

  • Usage:
  • trained model recently successfully applied by

Bernhard&Gurevych (2009) for answer finding

  • trained on token

Divergence From Randomness BM25 model (DFR_BM25): Probabilistic model + Query expansion: Translation-based model: Kullback-Leibler model (KL):

  • 10 terms out of 3 top ranked docs

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-13
SLIDE 13
  • 2. Combination of Retrieval Models

Monolingual translation-based model (TM):

  • Motivation:
  • address the lexical gap problem
  • learn translation probabilities between terms trained on

parallel dataset: dictionary and encyclopedic definitions

  • „the translation probability reflects the association

between query term and document term”

  • Usage:
  • trained model recently successfully applied by

Bernhard&Gurevych (2009) for answer finding

  • trained on token

Divergence From Randomness BM25 model (DFR_BM25): Probabilistic model + Query expansion: Translation-based model: Kullback-Leibler model (KL):

  • 10 terms out of 3 top ranked docs

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-14
SLIDE 14
  • 2. Combination of Retrieval Models

Monolingual translation-based model (TM):

  • Motivation:
  • address the lexical gap problem
  • learn translation probabilities between terms trained on

parallel dataset: dictionary and encyclopedic definitions

  • „the translation probability reflects the association

between query term and document term”

  • Usage:
  • trained model recently successfully applied by

Bernhard&Gurevych (2009) for answer finding

  • trained on token

Divergence From Randomness BM25 model (DFR_BM25): Probabilistic model + Query expansion: Translation-based model: Kullback-Leibler model (KL):

  • 10 terms out of 3 top ranked docs

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-15
SLIDE 15
  • Hypothesis: probabilistic and translation-based models retrieve different sets of

relevant documents

  • 2. Combination of Retrieval Models

DFR_BM25 + KL

1. ….. 2. ….. 3. ….. 1. ….. 2. ….. 3. …..

TM

1. ….. 2. ….. 3. …..

token lemma sense token

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-16
SLIDE 16
  • Hypothesis: probabilistic and translation-based models retrieve different sets of

relevant documents A) normalization: rnorm(i) = (rorig(i) – rmin) / (rmax – rmin) B) CombSUM by Fox&Shaw(1994): rcomb(i) = SUM(Individual rnorm(i))

  • 2. Combination of Retrieval Models

DFR_BM25 + KL

1. ….. 2. ….. 3. ….. 1. ….. 2. ….. 3. …..

TM

1. ….. 2. ….. 3. …..

token lemma sense token

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-17
SLIDE 17
  • Retrieval based on indexed senses (DFR_BM25 +KL):
  • CombBest+ outperforms CombBest
  • Focus on „combined“ indices

Extrinsic evaluation: sense index types

Index type MAP (training) MAP (test) UBCBest 0.2514 0.2636 NUSBest 0.2930 0.3473 CombBest 0.2921 0.3313 CombBest+ 0.3011 0.3551

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-18
SLIDE 18
  • Retrieval based on indexed senses (DFR_BM25 +KL):
  • CombBest+ outperforms CombBest
  • Focus on „combined“ indices

Extrinsic evaluation: sense index types

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

Index type MAP (training) MAP (test) UBCBest 0.2514 0.2636 NUSBest 0.2930 0.3473 CombBest 0.2921 0.3313 CombBest+ 0.3011 0.3551

slide-19
SLIDE 19
  • Retrieval based on indexed senses (DFR_BM25 +KL):
  • CombBest+ outperforms CombBest
  • Focus on „combined“ indices

Extrinsic evaluation: sense index types

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

Index type MAP (training) MAP (test) UBCBest 0.2514 0.2636 NUSBest 0.2930 0.3473 CombBest 0.2921 0.3313 CombBest+ 0.3011 0.3551

slide-20
SLIDE 20
  • Retrieval based on indexed senses (DFR_BM25 +KL):
  • CombBest+ outperforms CombBest
  • Focus on „combined“ indices

Extrinsic evaluation: sense index types

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

Index type MAP (training) MAP (test) UBCBest 0.2514 0.2636 NUSBest 0.2930 0.3473 CombBest 0.2921 0.3313 CombBest+ 0.3011 0.3551

slide-21
SLIDE 21

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-22
SLIDE 22

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-23
SLIDE 23

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-24
SLIDE 24

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-25
SLIDE 25

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-26
SLIDE 26

Retrieval results on test data

  • highest MAP on lemmas
  • weights learnt on training data
  • improvement when combined with TM
  • in combination CombBest and

CombBest+ similar performance

  • in most cases no improvement when

utilizing word senses

  • best performance: combination of

lemma and token (translation)

* officially submitted

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-27
SLIDE 27

Discussion & Conclusions

  • Translation-based model
  • performance of stand-alone method lower than

probabilistic model

  • straightforward use
  • different tokenization scheme of collection and

translation-based model „public_transport“ vs. „public“ and „transport“

  • 61 out of 160 test topics contain at least one multi

word expression

  • combination always improves performance
  • promising for future work
  • train translation model with multi-word expression
  • train translation model on lemmas or senses

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-28
SLIDE 28

Discussion & Conclusions

  • Translation-based model
  • performance of stand-alone method lower than

probabilistic model

  • straightforward use
  • different tokenization scheme of collection and

translation-based model „public_transport“ vs. „public“ and „transport“

  • 61 out of 160 test topics contain at least one multi

word expression

  • combination always improves performance
  • promising for future work
  • train translation model with multi-word expression
  • train translation model on lemmas or senses

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-29
SLIDE 29

Discussion & Conclusions

  • Utilizing word sense annotations
  • heuristic-based combination of both sense

annotations

  • extrinsic evaluation shows improvement
  • however, overall no improvement:

how to use 65% accuracy annotated word senses to achieve improvements in IR ?

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-30
SLIDE 30

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

Thank you!

http://www.ukp.tu-darmstadt.de

Ubiquitous Knowledge Processing Lab

slide-31
SLIDE 31

References

  • Adam Berger and John Laerty. Information Retrieval as Statistical Translation. In Proceedings of the

1999 ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '99), pages 222-229, 1999.

  • Delphine Bernhard and Iryna Gurevych. Combining Lexical Semantic Resources with Question &

Answer Archives for Translation-Based Answer Finding. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 728-736, Suntec, Singapore, August 2009.

  • Edward A. Fox and Joseph A. Shaw. Combination of Multiple Searches. In Proceedings of the 2nd Text

REtrieval Conference (TREC-2), pages 243-252, 1994.

  • Iadh Ounis, Gianni Amati, Vassilis Plachouras, Ben He, Craig Macdonald, and Christina Lioma. Terrier:

A High Performance and Scalable Information Retrieval Platform. In Proceedings of ACM SIGIR'06 Workshop on Open Source Information Retrieval (OSIR 2006), 2006.

  • Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. Okapi

at TREC-4. In NIST Special Publication 500-236: The Fourth Text Retrieval Conference (TREC-4), pages 73-96, 1995.

  • Xiaobing Xue, Jiwoon Jeon, and W. Bruce Croft. Retrieval Models for Question and Answer Archives. In

SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 475{482, New York, NY, USA, 2008.

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-32
SLIDE 32

Index types: statistics

Index type # tokens # senses # tokens without sense UBCBest 40.7 Mil. 34.1 Mil. 6.6 Mil. NUSBest 34.5 Mil. 6.2 Mil. CombBest 31.7 Mil. 9.0 Mil. CombBest+ 35.1 Mil. 5.6 Mil.

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-33
SLIDE 33

Official results

Group Without WSD ukp 0.4509 reina 0.4452 uniba 0.4250 geneva 0.4171 know−center 0.4170 Group With WSD ukp 0.4500 uniba 0.4346 know-center 0.4222 reina 0.4123 jaen 0.3819

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-34
SLIDE 34

Official results: R-Precision

Group Without WSD reina 0.4354 ukp 0.4301 uniba* 0.4128 geneva* 0.4043 know-center* 0.4013 Group With WSD ukp 0.4243 uniba* 0.4153 know-center* 0.4061 reina 0.4024 jaen* 0.3651

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-35
SLIDE 35

Translation Models in IR

  • The translation probabilities have to be trained
  • Training data: parallel texts, aligned at the sentence level
  • Use of IBM translation models implemented in GIZA++

This is a very small house. C'est une maison minuscule.

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-36
SLIDE 36

Parallel dataset: dictionary and encyclopaedic definitions (LSR)

* based on slides of Delphine Bernhard

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-37
SLIDE 37

Translation Models in IR

  • Retrieval

sim(Q,D) = Πq in Q P(q|Md) P(q|Md) = Σw in D T(q|w) P(w|Md)

where T(q|w) is the probability that a query term q is the translation of a document term w

  • The translation probability reflects the association between query term and

document term

  • Translation models allow inexact matching to address the issues of synonymy

and polysemy

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab

slide-38
SLIDE 38
  • Retrieval based on indexed senses (DFR_BM25 +KL):
  • NUS WSD system seems to perform better
  • SemEval-2007 all-words WSD subtask: NUS 0.587; UBC 0.544
  • CombBest+ outperforms CombBest
  • Focus on „combined“ indices

Extrinsic evaluation: WSD combination

Index type MAP (training) MAP (test) UBCBest 0.2514 0.2636 NUSBest 0.2930 0.3473 CombBest 0.2921 0.3313 CombBest+ 0.3011 0.3551

02.10.09 | Computer Science Department | Ubiquitous Knowledge Processing Lab