Towards Better Measurement of Attention and Satisfaction in Mobile - PowerPoint PPT Presentation

Towards Better Measurement of Attention and Satisfaction in Mobile Search Dmitry Lagun , Chih-Hung Hsieh, Dale Webster, Vidhya Navalpakkam

Thanks! Vidhya Navalpakkam Chih-Hung Hsieh Dale Webster 2

Mobile is popular! • 25% of Web page visits come from mobile [Statcounter.com, 2014] • Mobile browsing grew five fold since 2010 (5%) [Statcounter.com, 2014] • One in every 5 search queries is issued from a mobile device [RKG Digital Marketing Report, 2013] 3

Our Study Attention Measurement Satisfaction with Rich Results Knowledge Graph Result 4

Satisfaction with Rich Results on Mobile: Background • Long history of using clicks for measurement of search satisfaction and result relevance [ Joachims et al., SIGIR 2005; Agichtein et al., SIGIR 2006] • Result relevance and implicit indicators (mouse cursor hover, touch & swipe) [ Huang et al., CHI 2011; Lagun et al., SIGIR 2011; Guo et al., SIGIR 2013] • Rich Answers do not require to click and mouse hovers do not exist on mobile  – What other implicit metrics can we use to infer result relevance/satisfaction without clicks/hovers? 5

User Study Design • Two Factor (within Subject) KG Not – Relevance KG Relevant Relevant – Presence 5 Tasks KG Present 5 Tasks 5 Tasks 5 Tasks • 20 Search Tasks KG Absent • Users were asked to provide explicit satisfaction score for each task (1-7 scale) 6

User Study Details • Participants – 24 users (diverse background, age, occupation) • Mobile Eye Tracker Setup • Calibration Directly on Phone Screen 7

Can Implicit User Metrics Indicate Answer Relevance? • Page and Task metrics – Time on SERP – Number of Scrolls – Time on Task Knowledge Graph Result • Gaze Metrics – Time on Rich Result (and %) – Total Time below Rich Result (and %) • Viewport Metrics – Time on Rich Result (and %) – Total Time below Rich Result (and %) 8

KG is Not Relevant  More Scrolling 10

KG is Relevant  Faster Search ( answer is found in KG without a click ) 11

No Impact on User Satisfaction when KG is Not Relevant! 12

Gaze Metrics vs. KG Relevance Not Relevant Relevant More Time Below the KG Result 13

%Viewport Time Below vs. KG Relevance More time on results below 30 Not Relevant KG % Viewport Time Below KG 25 20 15 10 5 0 Not Relevant Relevant 15

Satisfaction with Rich Results: Summary • We can use Page and Viewport metrics to infer KG relevance and satisfaction • No impact on user satisfaction when Not Relevant KG is shown • Users view more results below the KG, when it is Not Relevant 16

Our Study Attention Measurement Satisfaction with Rich Results Knowledge Graph Result 17

Attention Measurement in Search: Background • Eye Tracking – accurate, but limited in scale [ Granka et al., WWW 2004; Buscher et al., SIGIR, CHI 2008- 2010] • Mouse Cursor Tracking – less accurate, but scalable  [ Huang et al., CHI 2011, 2012; Lagun et al., SIGIR 2011; Guo et al., CHI 2010, WWW 2012; Navalpakkam et al., WWW 2013] • Viewport Tracking – accurate (???), scalable (on mobile) 18

Viewport Time Calculation: Primer • Display Time = 10 sec • ViewportTime(R1) = ? R1 • Coverage – % of screen area occupied by the result (e.g. Coverage(KG) > Coverage(R1 )) KG • Exposure – % of result area visible on the screen (e.g. Exposure (R2) < 1.0) • ViewportTime(R) = DisplayTime * R2 Coverage (R) * Exposure (R) 19

Can we use Viewport Time to measure time spent on each result? one search result Pearson R = 0.57 Pearson R = 0.69 %Viewport Time Viewport Time Gaze Time %Gaze Time Correlation is high  can use Viewport Time to accurately measure time spent on individual search result at scale 20

Are attention patterns similar on desktop and mobile? ? ? 21 Granka et al., WWW 2004

Viewing Time vs. Result Position Why? On desktop: 22 Granka et al., WWW 2004

Short Scroll Effect 23

Short Scroll Effect 24

Short Scroll Effect 25

Do users have position preference when reading on a mobile phone? 26

Conclusions • Viewport and Page metrics can be used to measure Rich Answer Relevance and Satisfaction • Viewport time provides accurate (R=0.69) estimate on time spent on search result • Users prefer to position content on top half of the phone’s screen 27

Results Summary Satisfaction with Attention Measurement Rich Results Pearson R = 0.69 %Viewport Time Viewport ≈ Gaze (on mobile) More results are viewed if %Gaze Time Answer is Not Top half of the Relevant screen receives more Attention Relevant Not Relevant No Impact on User Satisfaction “Short - Scroll” when KG is Not effect Relevant! Granka et al., WWW 2004 Mobile Desktop 28