Applying the User-over-Ranking Hypothesis to Query Formulation - - PowerPoint PPT Presentation

Applying the User-over-Ranking Hypothesis to Query Formulation

Matthias Hagen Benno Stein

Bauhaus-Universit¨ at Weimar matthias.hagen@uni-weimar.de

ICTIR 2011 Bertinoro, Italy September 14, 2011

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 1

What is the User-over-Ranking hypothesis?

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 2

The User-over-Ranking Hypothesis

[Stein and Hagen, ECIR 2011]

Queries returning as many results as the user can consider increase retrieval performance.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 3

The User-over-Ranking Hypothesis

[Stein and Hagen, ECIR 2011]

Queries returning as many results as the user can consider increase retrieval performance.

Fine print: If ranking works: great! Use case is not some query like ebay. But more involved information needs, automatic systems, etc.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 3

Assumption 1: More keywords = more specific

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 4

Assumption 1: More keywords = more specific

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 4

Assumption 1: More keywords = more specific

Result list length Query length underspecific

• verspecific

Specificity of Queries

1 10 20 106

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 5

Assumption 2: User can arbitrarily specify information need

Result list length Query length underspecific

• verspecific

Specificity of Queries

1 10 20 106

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 6

Assumption 2: User can arbitrarily specify information need

Result list length Query length underspecific

• verspecific

Specificity of Queries

1 10 20 106

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 6

Assumption 3: User can consider about k results.

Result list length Query length underspecific

• verspecific

Specificity of Queries

1 10 20 106 Processing capacity k Result list length 10 100 103 104 k

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 7

Hypothesis: Specificity matches k = Optimum retrieval

Result list length Query length underspecific

• verspecific

Specificity of Queries

1 10 20 106 Processing capacity k Result list length 10 100 103 104 k

• ptimum retrieval

⇔ result set size = k

Probability for Retrieval Success

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 8

What is this hypothesis good for?

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 9

Query Formulation

Scenario

Given a set W of keywords Find a good query Q ⊆ W

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 10

Query Formulation

Scenario

Given a set W of keywords Find a good query Q ⊆ W

Previous approach

[Lee et al., CIKM 2009]

Learnt ranking function identifies the m best keywords from W . Based on: Known relevant documents Unrestricted index access Manually tuned m for each set W

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 10

Consider for instance . . . Known-Item Finding

Scenario

User accessed a document But did not store it

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 11

Consider for instance . . . Known-Item Finding

Scenario

User accessed a document But did not store it How can she find it again?

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 11

Consider for instance . . . Known-Item Finding

Scenario

User accessed a document But did not store it How can she find it again?

Solution

Remember some keywords

information retrieval query formulation web search search session user support search engine cost optimization

Query a search engine

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 11

But what query to formulate with the keywords?

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 12

Single keywords?

information retrieval / / / / / / / / query / / / / / / / / / / / / / / / / / / / / formulation / / / / / web / / / / / / / / / / / / search / / / / / / / / / / search / / / / / / / / / / / / / / session / / / / / / / user/ / / / / / / / / / / / / support / / / / / / / / / / search / / / / / / / / / / / / engine / / / / / / / cost/ / / / / / / / / / / / / / / / / / / / / /

• ptimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

Single keywords?

/ / / / / / / / / / / / / / / / / / information/ / / / / / / / / / / / / / / / / retrieval query formulation / / / / / web / / / / / / / / / / / / search / / / / / / / / / / search / / / / / / / / / / / / / / session / / / / / / / user/ / / / / / / / / / / / / support / / / / / / / / / / search / / / / / / / / / / / / engine / / / / / / / cost/ / / / / / / / / / / / / / / / / / / / / /

• ptimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

Single keywords?

/ / / / / / / / / / / / / / / / / / information/ / / / / / / / / / / / / / / / / retrieval / / / / / / / / query / / / / / / / / / / / / / / / / / / / / formulation web search / / / / / / / / / / search / / / / / / / / / / / / / / session / / / / / / / user/ / / / / / / / / / / / / support / / / / / / / / / / search / / / / / / / / / / / / engine / / / / / / / cost/ / / / / / / / / / / / / / / / / / / / / /

• ptimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

Single keywords? Underspecific!

/ / / / / / / / / / / / / / / / / / information/ / / / / / / / / / / / / / / / / retrieval / / / / / / / / query / / / / / / / / / / / / / / / / / / / / formulation web search / / / / / / / / / / search / / / / / / / / / / / / / / session / / / / / / / user/ / / / / / / / / / / / / support / / / / / / / / / / search / / / / / / / / / / / / engine / / / / / / / cost/ / / / / / / / / / / / / / / / / / / / / /

• ptimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

All keywords at once?

information retrieval query formulation web search search session user support search engine cost optimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

All keywords at once? Overspecific!

information retrieval query formulation web search search session user support search engine cost optimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 13

Remember the hypothesis . . . not too many results!

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 14

Solution: As many keywords as possible!

information retrieval query formulation web search search session / / / / / / / user/ / / / / / / / / / / / / support search engine cost optimization

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 15

“As many keywords as possible”-Query

Characteristics

Captures most of the remembered keywords Best possible description of the known-item Not too many results → user can check complete list

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 16

“As many keywords as possible”-Query

Characteristics

Captures most of the remembered keywords Best possible description of the known-item Not too many results → user can check complete list

Problem

Relevant documents not known No web index at user site Query size not known

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 16

“As many keywords as possible”-Query

Characteristics

Captures most of the remembered keywords Best possible description of the known-item Not too many results → user can check complete list

Problem

Relevant documents not known No web index at user site → Lee et al. not applicable Query size not known

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 16

We propose an approach for this scenario . . .

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 17

Problem Statement with Capacity

Promising Query

Given:

1

A set W of keywords

2

An upper bound k on the result list length

Find a largest query Q ⊆ W yielding at most k results

Optimization Problem!

Minimize the number of submitted web queries to find Q.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 18

Problem Statement with Capacity

Promising Query

Given:

1

A set W of keywords

2

An upper bound k on the result list length

Find a largest query Q ⊆ W yielding at most k results

Optimization Problem!

Minimize the number of submitted web queries to find Q.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 18

Baseline: Depth-First Search

w1, w2, w3, w4

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results]

(< 100)

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results]

(< 100)

w1 ∧ w3 ∧ w4 [0 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results]

(< 100)

w1 ∧ w3 ∧ w4 [0 results]

✘

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results]

(< 100)

w1 ∧ w3 ∧ w4 [0 results]

✘

w2 [10 000 results] w2 ∧ w3 [5 000 results] w2 ∧ w3 ∧ w4 [60 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results] w1 ∧ w3 ∧ w4 [0 results]

✘

w2 [10 000 results] w2 ∧ w3 [5 000 results] w2 ∧ w3 ∧ w4 [60 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline: Depth-First Search

w1, w2, w3, w4 w1 [10 000 results]

(> 100)

w1 ∧ w2 [0 results] w1 ∧ w3 [90 results] w1 ∧ w3 ∧ w4 [0 results]

✘

w2 [10 000 results] w2 ∧ w3 [5 000 results] w2 ∧ w3 ∧ w4 [60 results]

✘ ✘

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 19

Baseline’s Analysis

Major Drawback

All intermediate queries submitted. → Bad run time!

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 20

Baseline’s Analysis

Major Drawback

All intermediate queries submitted. → Bad run time!

Idea

Estimate the result list length before query submission.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 20

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search"

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

"information retrieval" + "web search"

16 % remain

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

"information retrieval" + "web search"

16 % remain

"query formulation" + "web search"

22 % remain

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

"information retrieval" + "web search"

16 % remain

"query formulation" + "web search"

22 % remain Our estimation scheme: avg(16 % , 22 %) = 19 % 87 100 · 0.19 = 16 500 results

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

"information retrieval" + "web search"

16 % remain

"query formulation" + "web search"

22 % remain Our estimation scheme: avg(16 % , 22 %) = 19 % 87 100 · 0.19 = 16 500 results Control: 35 700 results

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

Co-occurrence based Estimations

Estimate: "information retrieval" "query formulation" + "web search" Known:

"information retrieval" "query formulation"

87 100 results

"information retrieval" + "web search"

16 % remain

"query formulation" + "web search"

22 % remain Our estimation scheme: avg(16 % , 22 %) = 19 % 87 100 · 0.19 = 16 500 results Control: 35 700 results

Observation

Our scheme usually underestimates the real result list length.

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 21

“Informed” Baseline

w1, w2, w3, w4

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results] (< 100)

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results] (< 100)

w1 ∧ w3 ∧ w4

[30 estimated] Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results] (< 100)

w1 ∧ w3 ∧ w4

[30 estimated]

[0 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results] (< 100)

w1 ∧ w3 ∧ w4

[30 estimated]

[0 results]

✘

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results] (< 100)

w1 ∧ w3 ∧ w4

[30 estimated]

[0 results]

✘

[20 000 estimated]

w2 w2 ∧ w3 w2 ∧ w3 ∧ w4

[1 500 estimated] [40 estimated] Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results]

w1 ∧ w3 ∧ w4

[30 estimated]

[0 results]

✘

[20 000 estimated]

w2 w2 ∧ w3 w2 ∧ w3 ∧ w4

[1 500 estimated] [40 estimated]

[60 results]

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

“Informed” Baseline

w1, w2, w3, w4 w1 [15 000 estimated] w1 ∧ w2 [50 estimated]

[0 results]

w1 ∧ w3 [70 estimated]

[90 results]

w1 ∧ w3 ∧ w4

[30 estimated]

[0 results]

✘

[20 000 estimated]

w2 w2 ∧ w3 w2 ∧ w3 ∧ w4

[1 500 estimated] [40 estimated]

[60 results]

✘ ✘

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 22

Heuristic

Informed baseline + heuristic reordering of the keywords at each step

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 23

Experimental Setup

Corpus

775 papers on Computer Science (the known-items) 15 keywords extracted from each

System

Bing API as search engine Set k = 100 Measure number of submitted Web queries

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 24

Experimental Setup

Corpus

775 papers on Computer Science (the known-items) 15 keywords extracted from each

System

Bing API as search engine Set k = 100 Measure number of submitted Web queries

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 24

Experimental Results

Number of keywords 5 10 15 Promising query not possible 614 328 86 found 161 447 689 heuristic 10.39 24.93 53.78

• Avg. queries submitted

informed 10.36 27.01 108.78 baseline 11.81 30.94 116.22

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 25

Experimental Results

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Number of keywords Heuristic Informed Baseline

• Avg. ratio of submitted queries

0.5 0.6 0.7 0.8 1.1 1.0 0.9 0.4

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 25

Almost the end: The take-away messages!

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 26

What we have done

Results

User-over-Ranking

longer queries → fewer results

• ptimum retrieval performance

→ user capacity

Heuristic for promising queries Use cases:

Known-item finding Empty results lists Query sessions

Future Work

Co-occurrence source User study

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 27

What we have (not) done

Results

User-over-Ranking

longer queries → fewer results

• ptimum retrieval performance

→ user capacity

Heuristic for promising queries Use cases:

Known-item finding Empty results lists Query sessions

Future Work

Co-occurrence source User study

Matthias Hagen, Benno Stein Applying the User-over-Ranking Hypothesis to Query Formulation 27

What we have (not) done

Results

User-over-Ranking

longer queries → fewer results

• ptimum retrieval performance

→ user capacity

Heuristic for promising queries Use cases:

Known-item finding Empty results lists Query sessions

Future Work

Co-occurrence source User study

Thank you

• Matthias Hagen, Benno Stein

Applying the User-over-Ranking Hypothesis to Query Formulation 27