Incorporating Clicks, Attention and Satisfaction into a SERP - - PowerPoint PPT Presentation
Background Motivation Model & Metric Experimental Setup Results Summary Incorporating Clicks, Attention and Satisfaction into a SERP Evaluation Model Aleksandr Chuklin , Maarten de Rijke chuklin@google.com derijke@uva.nl
Background Motivation Model & Metric Experimental Setup Results Summary
Aleksandr Chuklin¶,§ Maarten de Rijke§ chuklin@google.com derijke@uva.nl
¶Google Research Europe §University of Amsterdam AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 1
Background Motivation Model & Metric Experimental Setup Results Summary
Main problem Combining relevance of individual SERP items (Rk) into a whole-page metric.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 3
Background Motivation Model & Metric Experimental Setup Results Summary
Examples
document 3 document 4 document 1 document 2 document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Examples Precision at N: P@N = 1 N
N
Rk
document 3 document 4 document 1 document 2 document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Examples Precision at N: P@N = 1 N
N
Rk Discounted Cumulative Gain (DCG): DCG@N =
N
1 log2 (1 + k) · Rk
document 3 document 4 document 1 document 2 document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Examples Precision at N: P@N = 1 N
N
Rk Discounted Cumulative Gain (DCG): DCG@N =
N
1 log2 (1 + k) · Rk Model-Based Metrics (Chuklin et al. 2013): Utility@N =
N
P(Ck = 1) · Rk
document 3 document 4 document 1 document 2 document 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 4
Background Motivation Model & Metric Experimental Setup Results Summary
Namely, improve this (Chuklin et al. 2013):
N
P(Ck = 1) · Rk
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 5
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 7
Background Motivation Model & Metric Experimental Setup Results Summary
9 1 3 5 6 7 8 4 2
1452
Image credits: F. Diaz, R.W. White, G. Buscher, and D. Liebling. Robust models of mouse movement on dynamic web search results pages. In CIKM, 2013. ACM Press AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 8
Background Motivation Model & Metric Experimental Setup Results Summary
High direct page utility (measured by DCG or ERR) leads to higher abandonment rate (SERPs with no clicks) direct page utility
Image credits: from A. Chuklin and P. Serdyukov. Good abandonments in factoid queries. In WWW, 2012. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 9
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
existing models mostly do not model non-trivial user attention patterns
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
existing models mostly do not model non-trivial user attention patterns existing models do not use explicit user satisfaction data
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 10
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 12
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 13
Background Motivation Model & Metric Experimental Setup Results Summary
Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
Examination assumption: click happens only when an item was examined and attractive: P(Ck = 1) = P(Ek = 1) · P(Ck = 1 | Ek = 1) N.B. Here we assume that P(Ck = 1 | Ek = 1) = α( Rk) where Rk comes from the raters and α is a logistic function.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 14
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 15
Background Motivation Model & Metric Experimental Setup Results Summary
Logistic regression model: P(Ek = 1) = ε( ϕk), where ϕk is a vector of features for SERP item k.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Logistic regression model: P(Ek = 1) = ε( ϕk), where ϕk is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be different from k) 1
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
Logistic regression model: P(Ek = 1) = ε( ϕk), where ϕk is a vector of features for SERP item k. Feature group Features # of features rank user-perceived rank of the SERP item (can be different from k) 1 CSS classes SERP item type (Web, News, Weather, Currency, Knowledge Panel, etc.) 10 geometry
umn (binary), width (w), height (h), w × h 5
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 16
Background Motivation Model & Metric Experimental Setup Results Summary
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 17
Background Motivation Model & Metric Experimental Setup Results Summary
in previous models, satisfaction comes only from clicked results;
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention;
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = σ(τ0 + U) =
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
in previous models, satisfaction comes only from clicked results; in our model it also comes from the SERP items that simply attracted attention; P(S = 1) = σ(τ0 + U) = σ
P(Ek = 1)ud( Dk) +
P(Ck = 1)ur( Rk)
Dk and Rk are ratings assigned by the raters for direct snippet relevance and result relevance respectively. ud and ur are linear functions of rating histograms.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 18
Background Motivation Model & Metric Experimental Setup Results Summary
Utility that determines the satisfaction probability: U =
P(Ek = 1)ud( Dk) +
P(Ck = 1)ur( Rk)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
Utility that determines the satisfaction probability: U =
P(Ek = 1)ud( Dk)
+
P(Ck = 1)ur( Rk)
has an additional term
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
Utility that determines the satisfaction probability: U =
P(Ek = 1)ud( Dk)
+
P(Ck = 1)ur( Rk)
has an additional term trained on mousing and satisfaction (in addition to clicks)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 19
Background Motivation Model & Metric Experimental Setup Results Summary
199 queries with explicit unambiguous feedback (satisfied / not satisfied);
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
199 queries with explicit unambiguous feedback (satisfied / not satisfied); 1,739 rated results
direct snippet relevance (D) result relevance (R)
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 21
Background Motivation Model & Metric Experimental Setup Results Summary
UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with fixed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a different and much bigger dataset.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Background Motivation Model & Metric Experimental Setup Results Summary
UBM model that agrees well with online team-draft experimental outcomes; PBM position-based model, a robust model with fewer parameters than UBM; random model that predicts click and satisfaction with fixed probabilities (learned from the data). uUBM from Chuklin et al. 2013. Similar to UBM, but parameters are trained on a different and much bigger dataset. CASnod is a stripped-down version that does not use (D) labels; CASnosat is a version of the CAS model that does not include the satisfaction term while optimizing the model; CASnoreg is a version of the CAS model that does not use regularization while
were trained with L2-regularization.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 22
Background Motivation Model & Metric Experimental Setup Results Summary
Correlation Between Metrics
Table: Correlation between metrics measured by average Pearson’s correlation coefficient. CASnosat CASnoreg CAS UBM PBM DCG uUBM CASnod 0.593 0.564 0.633 0.470 0.487 0.546 0.441 CASnosat 0.664 0.715 0.707 0.668 0.735 0.684 CASnoreg 0.974 0.363 0.379 0.417 0.341 CAS 0.377 0.394 0.440 0.360 UBM 0.814 0.972 0.882 PBM 0.906 0.965 DCG 0.943
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 24
Background Motivation Model & Metric Experimental Setup Results Summary
Metric Correlation with True Satisfaction
C A S n
C A S n
a t C A S n
e g C A S U B M P B M r a n d
D C G u U B M 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6
Pearson correlation coefficient between different model-based metrics and the user-reported satisfaction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 25
Background Motivation Model & Metric Experimental Setup Results Summary
Log-Likelihood of Click Prediction
C A S n
C A S n
a t C A S n
e g C A S U B M P B M r a n d
u U B M 4.5 4.0 3.5 3.0 2.5 2.0 1.5
Log-likelihood of the click data. Note that uUBM was trained on a totally different dataset.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 26
Background Motivation Model & Metric Experimental Setup Results Summary
A model-based metric needs to model satisfaction explicitly and use it for training.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite different from the previously used metrics, making it an interesting addition to TREC.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
A model-based metric needs to model satisfaction explicitly and use it for training. Direct snippet relevance (D) is essential for predicting satisfaction. The CAS metric is quite different from the previously used metrics, making it an interesting addition to TREC. When used as a model, CAS consistently predicts user satisfaction with a relatively small penalty in click prediction.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 28
Background Motivation Model & Metric Experimental Setup Results Summary
All content represents the opinion of the authors which is not necessarily shared or endorsed by their respective employers and/or sponsors. AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 29
Background Motivation Model & Metric Experimental Setup Results Summary
Log-Likelihood of Satisfaction Prediction
C A S n
C A S n
a t C A S n
e g C A S U B M P B M r a n d
u U B M 0.8 0.7 0.6 0.5 0.4 0.3 0.2
Log-likelihood of the satisfaction prediction. Some models have log-likelihood below −0.8, hence there are no boxes for them.
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 31
Background Motivation Model & Metric Experimental Setup Results Summary
CASrank is the model that only uses the rank to predict attention; CASnogeom only uses the rank and SERP item type information and does not use geometry; CASnoclass does not use the CSS class features (SERP item type). Pearson correlation with satisfaction
C A S r a n k C A S n
e
C A S n
l a s s C A S n
C A S 0.2 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6
Log-likelihood of clicks / satisfaction
CASrank CASnogeom CASnoclass CASnod CAS 2.5 2.4 2.3 2.2 2.1 2.0 1.9 1.8 1.7 C A S r a n k C A S n
e
C A S n
l a s s C A S n
C A S 0.65 0.60 0.55 0.50 0.45 0.40 0.35 0.30 0.25 0.20
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 32
Background Motivation Model & Metric Experimental Setup Results Summary
12% of the SERPs in our data are heterogeneous and our metric does well for them.
Table: Pearson correlation between utility of heterogeneous SERP and user-reported satisfaction. CAS UBM PBM random DCG uUBM 0.60 0.38
0.24
CASrank CASnogeom CASclass CASnod CASnosat CASnoreg 0.15
0.27
0.48 0.67
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 33
Background Motivation Model & Metric Experimental Setup Results Summary
Some raters were filtered out as spammers, but there was still some natural disagreement:
Table: Filtered out workers and agreement scores for remaining workers.
% of workers % of ratings Cohen’s Krippendorf’s label removed removed kappa alpha (D) 32% 27% 0.339 0.144 (R) 41% 29% 0.348 0.117
AC–MdR Incorporating Clicks, Attention and Satisfaction. . . 34