Simulation of Within-Session Query Variations using a Text - - PowerPoint PPT Presentation

Simulation of Within-Session Query Variations using a Text Segmentation Approach

Debasis Ganguly Johannes Leveling Gareth J.F. Jones

CNGL, School of Computing, Dublin City University, Ireland

Outline

Introduction to query reformulation Automatic generation of query reformulations Characteristics of the reformulations in terms of retrieval results Evaluation Conclusions and future work

Query Reformulation Types in IR

Specialization: more particular information need than in previous query

Example: “Mahatma Gandhi”  “Mahatma Gandhi non-violence movement”

Generalization: more general information than in previous query

Example: “Mahatma Gandhi assassination”  “Mahatma Gandhi life and works”

Query drift: move toward related but different information need

Example: “Mahatma Gandhi assassination”  “Gandhi film”

Motivation

Hypothesis: Automatic query reformulations can be used to simulate user query sessions Objective: Simulate query sessions in large quantities – Less time-consuming – Less expensive – No privacy issues – Independent from real data Simulated query sessions can help in

Session IR tasks: goal is to improve the IR effectiveness over an entire query session for a user Collaborative IR tasks: goal is to improve the IR effectiveness of a new user by utilizing user responses to related queries

Specialization

Initial query: “wildlife poaching” Very general search; no restrictions on particular animal species or locations

wildlife poaching Indian tigers African lions

After reading two documents, the user now knows that poaching is frequent for “African lions” and “Indian tigers” Adding these words make the query more specific

Generalization

Initial query: “osteoporosis” Specific information request (by using technical term); user may not be sure what it actually means

Osteoporosis

• steoporosis

After reading the document, the user knows that “osteoporosis” is a type of “bone disease” Substituting “osteoporosis” with the words “bone” and “disease” now means the user is interested in “bone diseases” in general instead of one particular bone disease

Document about bone diseases in general with a dedicated section on osteoporosis Bone Bone Bone Bone

Text Segmentation

Documents are composed of a series of densely discussed subtopics Text segmentation draws boundaries between topic shifts

Introduction – The search for life in space How the moon helped life evolve on earth The moon's chemical composition

Example from M. Hearst. CL. 1997.

Term Distribution

Perform text segmentation to get blocks of coherent text passages Terms densely distributed in a sub-topic are useful for specific reformulations Terms uniformly distributed throughout a document are useful for general reformulations

Term 1: dominant in topic 1 Term 2: dominant in topic 2 Term 3: general term topic 2 topic 1

Algorithm for Automatic Query Reformulation

Use top ranked documents from an initial retrieval step as external information for reformulations Categorize terms into two classes – specific and general by computing their distribution into the segments of the top ranked documents Generate candidate query reformulations

– Add the most specific terms from the most/least similar segments of documents to the original query to get a more specific/drifting query – Substitute original query terms with more general terms as

• btained from the pseudo-relevant set of documents
• Rank by score and select the best N variants

Term Scores

Specialization/drift score: combine

– term frequency in segment, – inverse segment frequency, and – idf

Generalization score: combine

– term frequency in document, – segment frequency, and – idf

• Combination in mixture model (see paper for details)

Result Set Characteristics

Specialization:

– Smaller set of relevant documents (queries are typically longer) – Top ranked documents for the original query become more general with respect to the specific reformulated query but are still relevant (overlap in top ranked documents)

Generalization:

– Larger set of relevant documents (queries are typically shorter) – Low overlap and high shift of top ranked documents retrieved in response to the original query

Evaluation Measures

Two measures:

Overlap of retrieved documents at cut-off 10, 20, 50 and 500: O(N) – Net perturbation of top m documents: 1/m Σk=1

m new_rank(dk)-k

p(N)

Expected observations:

– High overlap and low perturbation for specialized queries – Low overlap and high perturbation for general queries

Experiments

TREC disk 4+5 documents TREC-8 topics:

– Topic titles as initial queries for specific and drift reformulations – Topic description as initial queries for general reformulations

• Top 5 documents retrieved by LM (lambda=0.4)
• C99 algorithm for text segmentation
• Added at most 3 specific terms for specific/drift reformulations
• Retained at most 2 terms from for the general reformulations
• Generated query variants
• Judged query variants manually (by two assessors)

Results

Type Manual Assessment Result Set Measures Assessor-1 Assessor-2 O(10) O(20) O(50) O(500) p(5) Specific 39 (78%) 26 (52%) 39.0 38.1 42.7 44.7 367.9 General 39 (78%) 43 (86%) 22.4 22.5 24.5 32.2 2208.6 Drift 34 (68%) 35 (70%) 12.0 10.2 8.6 5.9 3853.3

Highest inter-assessor agreement for drift since a drift in information need is not subject to personal judgements Lowest inter-assessor agreement for specific reformulations since semantic specificity of added words can depend on personal judgement Specific and general reformulations which are associated with an increase in overlap percentage with increasing cut-off rank indicate that we get more “seen” documents further down the ranked list

Sample Output of Specific Reformulation

Specific reformulations Assessor 1 agrees Assessor 1 disagrees Assessor 2 agrees behavioural genetics chromosomes DNA genome cosmic events magnitude proton ion Assessor 2 disagrees N/A salvaging, shipwreck, treasure found aircraft Rotterdam

• Specific reformulations involve adding new words which ought to be

semantically related to the original keywords, and the degree of semantic closeness is often subject to personal judgments

• One of the assessors does not agree that adding the words “magnitude”,

“proton” and “ion” make the initial query “cosmic events” more specific

An Irish Perspective on Query Reformulation …

→ Elephants → Tigers

Wildlife poaching → Beer

?

Images from Flickr

… no, just a typo!

Wildlife poaching → Beer → Bear

→ Elephants → Tigers

Images from Flickr

Conclusions and Further Work

Our proposed method can be used to produce query reformulations with an average accuracy of 65%, 82% and 69% for the specialization, generalization and drift reformulation, respectively We introduced metrics such as the average percentage

• verlap at fixed number of documents and the average net

perturbation to quantify the retrieval result set changes Investigate relation between relevant documents for

• riginal query and relevant documents for query

reformulations

Any General or Specific Queries?

Specialization term scores

• tf (t, s) : term frequency of term t in a segment s
• |S|/sf(t) : how dominant is term t in segment s compared

to other segments of the same document

• idf(t) : how rare is term t in the collection

)) ( log( ). 1 ( ) ( | | ). , ( . ) , ( t idf a t sf S s t tf a s t

− + = φ

• Add ns terms with top φ(t,s) scores for more specific

query

Generalization term scores

• tf(t,d) : term frequency of term t in a document d

(instead of frequency in individual segments)

• sf(t)/|S| : segment frequency

(instead of inverse segment frequency)

• idf(t) : Inverse document frequency:

)) ( log( ). 1 ( | | ) ( ). , ( . ) ( t idf a S t sf d t tf a t

− + = ψ

• Select ng terms with top ψ(t) scores for more general

query