ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on - - PDF document

▶

ictir 2016 slides the impact of fixed cost pooling

ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on - - PDF document

Nov 03, 2023 197 likes •731 views

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/308120269 ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias Presentation September 2016

slide-1

SLIDE 1 See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/308120269

ICTIR 2016 Slides - The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias

Presentation · September 2016 CITATIONS READS 38 1 author: Some of the authors of this publication are also working on these related projects: Abstracting Domain-Specific Information Retrieval and Evaluation (ADmIRE) View project Space-time mapping and modelling of soil properties in Mediterranean and Temperate areas View project Aldo Lipani University College London 57 PUBLICATIONS 223 CITATIONS SEE PROFILE All content following this page was uploaded by Aldo Lipani on 15 September 2016. The user has requested enhancement of the downloaded file.

slide-2

SLIDE 2

The Impact of   Fixed-Cost Pooling Strategies  

n Test Collection Bias

1Aldo Lipani, 2Guido Zuccon, 1Mihai Lupu, 3Bevan Koopman, 1Allan Hanbury 1Vienna University of Technology 2Queensland University of Technology 3Australian e-Health Research Centre

15 Sep 2016   ICTIR 2016 - Newark (DE)

slide-3

SLIDE 3

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures (P@10 and RBP) in presence

f a fixed-budget, which of the tested pooling strategies introduces

less pool bias?

slide-4

SLIDE 4

The Pooling Method

slide-5

SLIDE 5

The Pooling Method

slide-6

SLIDE 6

The Pooling Method

slide-7

SLIDE 7

The Pooling Method

slide-8

SLIDE 8

The Pooling Method

slide-9

SLIDE 9

The Pooling Method

slide-10

SLIDE 10

The Pooling Method

slide-11

SLIDE 11

The Pooling Method

slide-12

SLIDE 12

The Pooling Method

slide-13

SLIDE 13

The Pooling Method

slide-14

SLIDE 14

The Pooling Method

slide-15

SLIDE 15

What is the Pool Bias?

slide-16

SLIDE 16

What is the Pool Bias?

slide-17

SLIDE 17

BIASED

What is the Pool Bias?

slide-18

SLIDE 18

BIASED

It is the effect that documents that were not selected in the pool created from the original runs will never be considered relevant

What is the Pool Bias?

slide-19

SLIDE 19

Tested Pooling Strategies

slide-20

SLIDE 20

Tested Pooling Strategies

Standard pooling strategies:

slide-21

SLIDE 21

Tested Pooling Strategies

Standard pooling strategies:

slide-22

SLIDE 22

Tested Pooling Strategies

Standard pooling strategies:

Take@10

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

slide-23

SLIDE 23

Tested Pooling Strategies

Standard pooling strategies:

Take@10

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

slide-24

SLIDE 24

Tested Pooling Strategies

Standard pooling strategies:

slide-25

SLIDE 25

Tested Pooling Strategies

Standard pooling strategies: &

slide-26

SLIDE 26

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-27

SLIDE 27

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-28

SLIDE 28

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedA@10&0.8

slide-29

SLIDE 29

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedA@10&0.8

slide-30

SLIDE 30

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-31

SLIDE 31

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-32

SLIDE 32

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedB@10&0.8

slide-33

SLIDE 33

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedB@10&0.8

slide-34

SLIDE 34

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-35

SLIDE 35

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-36

SLIDE 36

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedC@10&0.8

slide-37

SLIDE 37

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

r4 r1 r2 r3 r5 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 ρ1 ρ2 ρ3 ρ4 ρ5 ρ6 ρ7 ρ8 ρ10 ρ9 1 2 3 4 5

RBPBasedC@10&0.8

slide-38

SLIDE 38

Tested Pooling Strategies

Standard pooling strategies: & RBP based pooling strategies*:

* A. Moffat, W. Webber, and J. Zobel. Strategic system comparisons via targeted relevance judgments.

slide-39

SLIDE 39

Methodology

slide-40

SLIDE 40

Test Collections

Ad Hoc 2-8 Web 9, Web 2001, Robust 2005 Genomics 2005 Legal 2006 Medical 2011 Microblog 2011

Methodology

slide-41

SLIDE 41

Test Collections

Ad Hoc 2-8 Web 9, Web 2001, Robust 2005 Genomics 2005 Legal 2006 Medical 2011 Microblog 2011

Domains

News Web Genomics Legal Medical Microblog

Methodology

slide-42

SLIDE 42

Test Collections

Ad Hoc 2-8 Web 9, Web 2001, Robust 2005 Genomics 2005 Legal 2006 Medical 2011 Microblog 2011

Domains

News Web Genomics Legal Medical Microblog We compared Standard Pooling strategies RBP Based Pooling strategies (with p=0.80 and p=0.73)

Methodology

slide-43

SLIDE 43

Test Collections

Ad Hoc 2-8 Web 9, Web 2001, Robust 2005 Genomics 2005 Legal 2006 Medical 2011 Microblog 2011

Domains

News Web Genomics Legal Medical Microblog We compared Standard Pooling strategies RBP Based Pooling strategies (with p=0.80 and p=0.73)

Metrics of Error

Mean Absolute Error System Rank Error SRE with statistical significance (p<0.05) MAE SRE SRE*

Methodology

slide-44

SLIDE 44

Summary of the Results for MAE

slide-45

SLIDE 45

Summary of the Results for MAE

P@10

1st 2nd 3rd

2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73

slide-46

SLIDE 46

Summary of the Results for MAE

P@10 RBP (p=0.80)

1st 2nd 3rd

2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73 2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73

slide-47

SLIDE 47

Summary of the Results for MAE

P@10 RBP (p=0.80)

1st 2nd 3rd

# Relevant Documents

2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73 2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73 2 4 6 8 10 12 14 16 T T+ A80 B80 C80 A73 B73 C73

slide-48

SLIDE 48

Conclusion

slide-49

SLIDE 49

Conclusion

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget, the best pooling strategy is

slide-50

SLIDE 50

Conclusion

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget, the best pooling strategy is

slide-51

SLIDE 51

Conclusion

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget, the best pooling strategy is However due to its limitations we recommend

slide-52

SLIDE 52

Conclusion

In the context of building a test collection for an IR task modelled by precision-based IR evaluation measures in presence of a fixed-budget, the best pooling strategy is However due to its limitations we recommend

slide-53

SLIDE 53

Thank you 

for you attention!

PoolBiasEstimators VisualPooling

View publication stats View publication stats