The Impact of Screen Size on Data Quality Douglas Williams, Westat Aaron Maitland, Westat Andrew Mercer, Pew Research Center Roger Tourangeau, Westat
Acknowledgement The work reported here was supported by a grant from the National Science Foundation [SES-1261340, Mobile Devices for Survey Data Collection to RT]. The National Science Foundation is not responsible for the conclusions presented here. 2
Background • Past: Screen size has not been a concern – Some variation, but confident of laptop or desktop sizes • Present: Dramatic increase in the ownership of handheld/mobile devices – 64% of households have a mobile device - ACS (File and Ryan, 2014) – 64% (+9%) of adults own a smartphone (2015); 42% (+8%) own a tablet device (2014) (Pew Research Center) • Future: Will continue to increase – Growing diversity of mobile screens and resolutions 3
Relative Screen Sizes 10 popular mobile devices: 6 smartphone; 4 tablet
Background • What do we know? – Web • Placement or location of response affects selection (Tourangeau, Couper, and Conrad, 2004) – Early Mobile: • Visual elements outside field of view likely to be missed (Peytchev and Hill, 2010) – Recent Mobile: • Scale orientation affects item nonresponse (de Bruijne and Wijnat, 2014) • Primacy effects no different between web and mobile survey (Mavletova, 2013) – Most mobile research focused on response/breakoff rates 5
Overall Study Design • Generic Health Study: Health Attitudes & Lifestyle Survey (HALS) – In-person survey: 8 metro areas in U.S. – ABS sample – Three devices: Smartphone (iPhone 5s); Tablet (iPad Air); Laptop (Windows 7-based 15”) • Device randomly assigned to sampled address – Sampled respondent asked to complete survey on pre- assigned device – $20 for completing survey • Overall RR ~31% - no difference by assigned device: 513 total completed interviews 6
Relative Screen Sizes iPhone 5s; iPad Air; Windows 7 Laptop PC – Green denotes actual survey browser size for laptop
Study Design: Experiments • Exp. 1: 4 questions (2 w/13 options; 2 w/8 options) – Response order: reverse order of response options • Are primacy effects greater for smaller devices (screen size)? • Exp. 2: 10 total questions (5 scale options) – Question format: 1 item per page or grid with multiple items. • Are options outside viewable area (smartphone) selected less? 8
Mobile Screens
Laptop Screen
Results: Primacy • Proportion selecting from top half of list – Primacy effect for nearly all items/devices – Only significant main effect for device type: question B3 (p < 0.01)
Results: Primacy Cont’d • One item usually most selected in each list. – What is the magnitude of change when order changes? Most Question Selected Position S. Ph Tablet Laptop B1. Health condition/disease considered the biggest Obesity 3 / 11 31.9% 35.0% 46.3% problem in U.S.? B2. First choice for health Internet 7- mid 35.9% 50.0% 38.1% information source? B3. Source of health Doctor 1 / 8 75.8% 81.5% 76.8% information most reliable? B4. Most important behavior Freq. 1 / 8 40.0% 44.3% 45.8% to maintain healthy Exercise lifestyle? 12
Results: Position Change • Percentage decrease from first order to second order * Significantly different from laptop (p < 0.01) † Significantly different from tablet (p < 0.05)
Results: Grids • Design – 10 items: single item per page / two grids 5-items each – 5 response options rating severity of health conditions: ‘not serious at all’ to ‘extremely serious’ • Laptop/Tablet – all response options visible • Smartphone – only first three visible 14
Results: Grids Cont’d • Test – Repeated measures logistic model: – No significant difference for interaction of question format * device type; nor main effect for device type. First (two) Middle Last (two) Smartphone 21.0% 24.5% 54.5% Grid Tablet 25.3 19.9 54.8 Laptop 19.4 21.2 59.4 Smartphone 17.8 19.7 62.5 Single Tablet 18.9 21.0 60.1 Laptop 16.4 20.5 63.1 15
Conclusions • Primacy – Generally little evidence of differences by device type – some exceptions – Smartphone more affected by change in position – Some inconsistencies with tablet – Respondent expectations / Touch interface – further research • Grids – No significant difference, but further research needed. – Associated with breakoffs (Peytchev, 2009) 16
Thank you! douglaswilliams@westat.com 17
References File, Thom and Camille Ryan, “Computer and Internet Use in the United States: 2013,” American Community Survey Reports, ACS-28, U.S. Census Bureau, Washington, DC, 2014. Mavletova, Aigul. (2013). Data quality in PC and mobile Web surveys. Social Science Computer Review, 31, 725 ‐ 743. Pew Research Center, January, 2014, “E-Reading Rises as Device Ownership Jumps” Available at: http://pewinternet.org/Reports/2014/E-Reading-Update.aspx Pew Research Center, April, 2015, “The Smartphone Difference” Available at: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ Peytchev, Andy. (2009). Survey Breakoff. Public Opinion Quarterly, 73(1), 74-97. Peytchev, Andy, and Craig A. Hill. (2010). Experiments in Mobile Web Survey Design: Similarities to other Modes and Unique Considerations. Social Science Computer Review, 28, 319 ‐ 335. Tourangeau, Roger., Mick P. Couper, and Fred Conrad. (2004). Spacing, position, and order interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368-393. 18
Recommend
More recommend
