Towards Better Measurement of Attention and Satisfaction in Mobile - - PowerPoint PPT Presentation

Towards Better Measurement of Attention and Satisfaction in Mobile Search

Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam

Thanks!

Vidhya Navalpakkam Dale Webster Chih-Hung Hsieh

2

Mobile is popular!

• 25% of Web page visits come from mobile

[Statcounter.com, 2014]

• Mobile browsing grew five fold since 2010 (5%)

[Statcounter.com, 2014]

• One in every 5 search queries is issued from a

mobile device

[RKG Digital Marketing Report, 2013]

3

Our Study

Attention Measurement Satisfaction with Rich Results

Knowledge Graph Result

4

Satisfaction with Rich Results on Mobile: Background

• Long history of using clicks for measurement of search

satisfaction and result relevance

[Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006]

• Result relevance and implicit indicators (mouse cursor

hover, touch & swipe)

[Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013]

• Rich Answers do not require to click and mouse hovers

do not exist on mobile 

– What other implicit metrics can we use to infer result relevance/satisfaction without clicks/hovers?

5

User Study Design

• Two Factor (within Subject)

– Relevance – Presence

• 20 Search Tasks
• Users were asked to provide explicit

satisfaction score for each task (1-7 scale)

KG Relevant KG Not Relevant KG Present 5 Tasks 5 Tasks KG Absent 5 Tasks 5 Tasks

6

User Study Details

• Participants

– 24 users (diverse background, age, occupation)

• Mobile Eye Tracker Setup
• Calibration Directly
• n Phone Screen

7

Can Implicit User Metrics Indicate Answer Relevance?

• Page and Task metrics

– Time on SERP – Number of Scrolls – Time on Task

• Gaze Metrics

– Time on Rich Result (and %) – Total Time below Rich Result (and %)

• Viewport Metrics

– Time on Rich Result (and %) – Total Time below Rich Result (and %)

Knowledge Graph Result

8

KG is Not Relevant  More Scrolling

10

KG is Relevant  Faster Search (answer is found in KG without a click)

11

No Impact on User Satisfaction when KG is Not Relevant!

12

Gaze Metrics vs. KG Relevance

13

Relevant Not Relevant

More Time Below the KG Result

%Viewport Time Below

• vs. KG Relevance

15

5 10 15 20 25 30

Not Relevant Relevant

% Viewport Time Below KG

More time on results below Not Relevant KG

Satisfaction with Rich Results: Summary

• We can use Page and Viewport metrics to

infer KG relevance and satisfaction

• No impact on user satisfaction when Not

Relevant KG is shown

• Users view more results below the KG, when

it is Not Relevant

16

Our Study

Attention Measurement Satisfaction with Rich Results

Knowledge Graph Result

17

Attention Measurement in Search: Background

• Eye Tracking – accurate, but limited in scale

[Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008- 2010]

• Mouse Cursor Tracking – less accurate, but

scalable 

[Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013]

• Viewport Tracking – accurate (???), scalable

(on mobile)

18

Viewport Time Calculation: Primer

• Display Time = 10 sec
• ViewportTime(R1) = ?
• Coverage

– % of screen area occupied by the result (e.g. Coverage(KG) > Coverage(R1))

• Exposure

– % of result area visible on the screen (e.g. Exposure (R2) < 1.0)

• ViewportTime(R) = DisplayTime *

Coverage(R) * Exposure(R)

19

KG R1 R2

%Gaze Time %Viewport Time Viewport Time Gaze Time

Can we use Viewport Time to measure time spent on each result?

Pearson R = 0.57 Pearson R = 0.69

• ne search result

Correlation is high  can use Viewport Time to accurately measure time spent on individual search result at scale

20

Are attention patterns similar on desktop and mobile?

?

Granka et al., WWW 2004

?

21

Viewing Time vs. Result Position

Granka et al., WWW 2004

On desktop:

Why?

22

Short Scroll Effect

23

Short Scroll Effect

24

Short Scroll Effect

25

Do users have position preference when reading on a mobile phone?

26

Conclusions

• Viewport and Page metrics can be used to

measure Rich Answer Relevance and Satisfaction

• Viewport time provides accurate (R=0.69)

estimate on time spent on search result

• Users prefer to position content on top half of

the phone’s screen

27

Results Summary

Attention Measurement Satisfaction with Rich Results

28

%Gaze Time %Viewport Time

Viewport ≈ Gaze (on mobile) Pearson R = 0.69

Top half of the screen receives more Attention “Short-Scroll” effect

Desktop Mobile

Relevant Not Relevant

More results are viewed if Answer is Not Relevant No Impact on User Satisfaction when KG is Not Relevant!