SLIDE 1
Search Snippet Evaluation
Mikhail Lebedev, Pavel Braslavski, Denis Savenkov
What is a snippet What is a snippet
#
Search Snippet Evaluation Mikhail Lebedev, Pavel Braslavski, Denis - - PowerPoint PPT Presentation
Search Snippet Evaluation Mikhail Lebedev, Pavel Braslavski, Denis - - PowerPoint PPT Presentation
Search Snippet Evaluation Mikhail Lebedev, Pavel Braslavski, Denis Savenkov CLEF 2011 CLEF 2011 What is a snippet What is a snippet # Why is it so important Why is it so important Search is ranking & representation Bad snippets spoil good
SLIDE 2
SLIDE 3
Why is it so important Why is it so important
Search is ranking & representation Bad snippets spoil good results Bad snippets spoil good results Overoptimized snippets lead to wrong clicks
SLIDE 4
When we use evaluation When we use evaluation
Compare with competitors Choose between different Choose between different algorithms g hi l i Machine learning
SLIDE 5
Evaluation approach Evaluation approach
User study Judgments Text‐based Clicks Text based metrics metrics
SLIDE 6
Evaluation approach Evaluation approach
User study Judgments Text‐based Clicks Text based metrics metrics
SLIDE 7
Eye tracking
SLIDE 8
Eye tracking Eye tracking
Different users, different strategies , g
Title is much more important than body hl h Highlighting is attractive U li k if i t t i User clicks even if snippet contains answer Media content may be ignored Media content may be ignored
SLIDE 9
Evaluation approach Evaluation approach
User study Judgments Text‐based Clicks Text based metrics metrics
SLIDE 10
Ideal snippet is: Ideal snippet is:
R d bl I f ti Readable Informative
SLIDE 11
Making ideal snippets Making ideal snippets
Machine learning Training set Training set Judgments Ab l t R l ti Absolute Relative
SLIDE 12
Relative assessment Relative assessment
SLIDE 13
Assessment issues Assessment issues
Assessors learn Snippet quality depends on ranking Snippet quality depends on ranking Assessment tool interface matters Pairs vs groups: time vs quality Pairs vs groups: time vs quality
SLIDE 14
Evaluation approach Evaluation approach
User study Judgments Text‐based Clicks Text based metrics metrics
SLIDE 15
Automated metrics Automated metrics
Highlighting Neatness Neatness Number of empty snippets Unique query words Unique query words
SLIDE 16
Evaluation approach Evaluation approach
User study Judgments Text‐based Clicks Text based metrics metrics
SLIDE 17
Click data Click data
Dwell time Abandonment Abandonment Inversions Time to first click Time to first click
SLIDE 18
Conclusions Conclusions
Different goals – different methods
User‐study: making assumptions Judgments: expensive quality Judgments: expensive quality Text based metrics: fast but rough Text‐based metrics: fast but rough Clicks: ranking influence
SLIDE 19