The Impact of Screen Size on Data Quality Douglas Williams, Westat - - PowerPoint PPT Presentation

The Impact of Screen Size on Data Quality

Douglas Williams, Westat Aaron Maitland, Westat Andrew Mercer, Pew Research Center Roger Tourangeau, Westat

The work reported here was supported by a grant from the National Science Foundation [SES-1261340, Mobile Devices for Survey Data Collection to RT]. The National Science Foundation is not responsible for the conclusions presented here.

2

Acknowledgement

• Past: Screen size has not been a concern

– Some variation, but confident of laptop or desktop sizes

• Present: Dramatic increase in the ownership of

handheld/mobile devices

– 64% of households have a mobile device - ACS (File and Ryan, 2014) – 64% (+9%) of adults own a smartphone (2015); 42% (+8%) own a tablet device (2014) (Pew Research Center)

• Future: Will continue to increase

– Growing diversity of mobile screens and resolutions

3

Background

Relative Screen Sizes

10 popular mobile devices: 6 smartphone; 4 tablet

• What do we know?

– Web

• Placement or location of response affects selection (Tourangeau,

Couper, and Conrad, 2004)

– Early Mobile:

• Visual elements outside field of view likely to be missed (Peytchev

and Hill, 2010)

– Recent Mobile:

• Scale orientation affects item nonresponse (de Bruijne and Wijnat,

2014)

• Primacy effects no different between web and mobile survey

(Mavletova, 2013)

– Most mobile research focused on response/breakoff rates

5

Background

• Generic Health Study: Health Attitudes & Lifestyle

Survey (HALS)

– In-person survey: 8 metro areas in U.S. – ABS sample – Three devices: Smartphone (iPhone 5s); Tablet (iPad Air); Laptop (Windows 7-based 15”)

• Device randomly assigned to sampled address

– Sampled respondent asked to complete survey on pre- assigned device – $20 for completing survey

• Overall RR ~31% - no difference by assigned

device: 513 total completed interviews

6

Overall Study Design

Relative Screen Sizes

iPhone 5s; iPad Air; Windows 7 Laptop PC – Green denotes actual survey browser size for laptop

• Exp. 1: 4 questions (2 w/13 options; 2 w/8 options)

– Response order: reverse order of response options

• Are primacy effects greater for smaller devices (screen size)?
• Exp. 2: 10 total questions (5 scale options)

– Question format: 1 item per page or grid with multiple items.

• Are options outside viewable area (smartphone) selected less?

8

Study Design: Experiments

Mobile Screens

Laptop Screen

Results: Primacy

• Proportion selecting from

top half of list

– Primacy effect for nearly all items/devices – Only significant main effect for device type: question B3 (p < 0.01)

• One item usually most selected in each list.

– What is the magnitude of change when order changes?

12

Results: Primacy Cont’d

Question Most Selected Position

• S. Ph

Tablet Laptop

• B1. Health condition/disease

considered the biggest problem in U.S.? Obesity 3 / 11 31.9% 35.0% 46.3%

• B2. First choice for health

information source? Internet 7- mid 35.9% 50.0% 38.1%

• B3. Source of health

information most reliable? Doctor 1 / 8 75.8% 81.5% 76.8%

• B4. Most important behavior

to maintain healthy lifestyle? Freq. Exercise 1 / 8 40.0% 44.3% 45.8%

Results: Position Change

* Significantly different from laptop (p < 0.01)

† Significantly different from tablet (p < 0.05)

• Percentage decrease from first order to second order

• Design

– 10 items: single item per page / two grids 5-items each – 5 response options rating severity of health conditions: ‘not serious at all’ to ‘extremely serious’

• Laptop/Tablet – all response options visible
• Smartphone – only first three visible

14

Results: Grids

• Test

– Repeated measures logistic model: – No significant difference for interaction of question format * device type; nor main effect for device type.

15

Results: Grids Cont’d

First (two) Middle Last (two) Grid Smartphone 21.0% 24.5% 54.5% Tablet 25.3 19.9 54.8 Laptop 19.4 21.2 59.4 Single Smartphone 17.8 19.7 62.5 Tablet 18.9 21.0 60.1 Laptop 16.4 20.5 63.1

• Primacy

– Generally little evidence of differences by device type – some exceptions – Smartphone more affected by change in position – Some inconsistencies with tablet – Respondent expectations / Touch interface – further research

• Grids

– No significant difference, but further research needed. – Associated with breakoffs (Peytchev, 2009)

16

Conclusions

Thank you!

douglaswilliams@westat.com

17

File, Thom and Camille Ryan, “Computer and Internet Use in the United States: 2013,” American Community Survey Reports, ACS-28, U.S. Census Bureau, Washington, DC, 2014. Mavletova, Aigul. (2013). Data quality in PC and mobile Web surveys. Social Science Computer Review, 31, 725‐743. Pew Research Center, January, 2014, “E-Reading Rises as Device Ownership Jumps” Available at: http://pewinternet.org/Reports/2014/E-Reading-Update.aspx Pew Research Center, April, 2015, “The Smartphone Difference” Available at: http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/ Peytchev, Andy. (2009). Survey Breakoff. Public Opinion Quarterly, 73(1), 74-97. Peytchev, Andy, and Craig A. Hill. (2010). Experiments in Mobile Web Survey Design: Similarities to other Modes and Unique Considerations. Social Science Computer Review, 28, 319‐335. Tourangeau, Roger., Mick P. Couper, and Fred Conrad. (2004). Spacing, position, and order interpretive heuristics for visual features of survey questions. Public Opinion Quarterly, 68(3), 368-393.

18

References