Mobile Web Survey Design: Scrolling versus Paging, SMS versus - - PowerPoint PPT Presentation

Mobile Web Survey Design: Scrolling versus Paging, SMS versus E-mail Invitations

1

DC-AAPOR & WSS Summer Conference Preview/Review 2014

July 22-23, 2014 Washington DC

Aigul Mavletova, NRU Higher School of Economics, Russia Mick P. Couper, Survey Research Center, University of Michigan

Paging vs. Scrolling Design

AAPOR suggestion:

Optimizing an online survey for completion in a mobile browser is something

• f

an art, but there are several notable conventions that can make it easier for respondents to

• complete. The overall layout should minimize the need for

2

• complete. The overall layout should minimize the need for

scrolling (either horizontally or vertically) to the extent possible. The number of questions per screen should generally be two or less to minimize scrolling.

Mobile Technologies for Conducting, Augmenting and Potentially Replacing Surveys: Report of the AAPOR Task Force on Emerging Technologies in Public Opinion Research (April, 2014)

Mobile web vs. PC web surveys

Higher

breakoff rates in mobile web surveys: 2-12 times higher.

• Longer

completion times: 1.5-3 times

3

Longer completion times: 1.5-3 times longer in mobile web surveys.

Paging vs. Scrolling Design

PC web surveys:

• No differences in response rates and breakoff rates.
• Simple skip logic: a scrolling version takes less time.

More complex skip logic: a scrolling version takes more time.

• A scrolling design increases item nonresponse rates.

4

• A scrolling design increases item nonresponse rates.

(Peytchev et al. 2006; Tourangeau et al. 2004; Vehovar et al., 2000) Mobile web surveys:

• McGeeney & Marlar (2013): higher breakoff rates in the paging

version (13% of mobile respondents; a 13-item survey).

SMS vs. E-mail

• E-mail produces higher response rates than an

SMS invitation in PC web surveys (Bosnjak et al. 2008).

• SMS significantly increases the percentage of

5

mobile web respondents (Crawford et al. 2013).

Hypotheses and Experimental Design

6

Hypotheses

Hypotheses

Scrolling vs. Paging Lower breakoff rate, longer completion times, and lower rate of item nonresponse are expected in the paging design.

7

Hypotheses

Hypotheses

Scrolling vs. Paging Lower breakoff rate, longer completion times, and lower rate of item nonresponse are expected in the paging design. SM S vs. SM S will result in lower participation rates.

8

SM S vs. E-mail SM S will result in lower participation rates. E-mail invitations are expected to increase the percentage of PC web respondents. SM S vs. E-mail Changing the mode of the reminder will improve participation rates compared to the conditions which have the invitation and reminder in the same mode.

Paging

Survey Design

Scrolling

Experimental Design

9

17 screens 2 screens

Paging

Survey Design

Scrolling

Experimental Design

Invitation Reminder

E-mail

Invitation- Reminder

10

17 screens 2 screens

SM S E-mail E-mail without URL SM S

+ + +

E-mail

+ +

Volunteer

• nline

access panel (Online Market Intelligence). 7 minutes survey. Respondents were encouraged to complete the survey via mobile devices.

Data Collection

11

via mobile devices. Fieldwork: 9-29th April, 2013, Russia. Software: Unipark. 2,110 respondents: 4, 000 invitations, participation rate=52.8% (1 invitation, 1 reminder)

17 questions: volunteer activities, practices

• f

helping

• ther

people, mobile web usage patterns, evaluation of the survey.

Questionnaire

12

No screening questions, quotas, and skipping logic. All questions were not obligatory.

Results: Scrolling vs. Paging

13

Participation Rates

53,8% 49,9% 51,8% 48,6%

49,0% 50,0% 51,0% 52,0% 53,0% 54,0% 55,0%

Scrolling Paging

14

46,0% 47,0% 48,0%

Participation rate Participation rate (mobile only) Paging

Number of completed interviews: 1,075 in the scrolling and 1,035 in the paging version

Sample Composition by Devices

Device % Smartphones 73.7% Feature phones 9.1% Tablets 3.3%

15

No difference between the scrolling and paging design

Tablets 3.3% PC 13.8% Total 2,110

Breakoff Rates

8,0% 6,0%* * 2,1%* * * 10,2% 3,3%* * 6,9%* * *

4,0% 6,0% 8,0% 10,0% 12,0%

Scrolling Paging

16

Chi-square (df=1), **p<0.01, ***p<0.001

2,1%* * *

0,0% 2,0%

Breakoff rate Breakoff rate on the first page Breakoff rate on the subsequent pages Paging

p=0.067

Breakoff Rate by Devices and Operating Systems

2,8% 5,5% 6,5% 30,3%

Tablets PC Smartphones Feature phones Device* * * Device* * *

Chi-square= 171.69*** (df=3)

17

0,0% 10,0% 20,0% 30,0% 40,0%

Operating systems* * *

Chi-square = 21.35*** (df=4)

2,2% 4,3% 5,4% 5,9% 12,8%

0,0% 2,0% 4,0% 6,0% 8,0% 10,0% 12,0% 14,0% 16,0%

iOS Bada Windows Phone Android Symbian

***p<0.001

Completion Times (minutes)

4,5* * * 4,9* * * 9,1* * * 6,6* * *

4 5 6 7 8 9 10

Scrolling Paging

18

t-test, ***p<0.001

1 2 3

Objective mean completion time Subjective evaluation of completion time Paging

Device

Scrolling Paging

Chi-square (df=1) Any technical problems while completing the

18.9% (203) 25.8% (267)

14.56* * *

Technical Difficulties

19

while completing the survey

(203) (267)

14.56* * *

***p<0.001

I tem Nonresponse

Device Scrolling Paging Statistics At least one item nonresponse 8.4% 9.9% Chi-square =1.40

20

(nonrespondents) (df=1) Overall item nonresponse 1.84% (SD=8.6%) 1.44% (SD=7.8%) t=1.12 (df=2,108)

SM S vs. E-mail

21

Participation Rates

52,7% 51,3%* * * 52,8% 43,7%* * *

30,0% 40,0% 50,0% 60,0%

SM S

22

Chi-square (df=1), ***p<0.001

0,0% 10,0% 20,0%

Participation Rate Participation Rate (mobile devices only) SM S E-mail

Participation Rates

60,9% 54,6% 53,2% 48,7% 44,1%

30,0% 40,0% 50,0% 60,0% 70,0%

23

Chi-square = 39.316 (df=4), p<0.001

0,0% 10,0% 20,0%

E-mail-SM S SM S-E-mail SM S-SM S SM S-E-mail without URL E-mail-E-mail

Participation Rate (Mobile Only)

53,5% 52,4% 52,1% 47,7% 32,8%

30,0% 40,0% 50,0% 60,0%

24

Chi-square = 62.039 (df=4), p<0.001

0,0% 10,0% 20,0%

E-mail-SM S SM S-SM S SM S-E-mail SM S-E-mail without URL E-mail-E-mail

PC Respondents

40,3% 27,0%

20,0% 25,0% 30,0% 35,0% 40,0% 45,0%

25

10,2% 3,8% 3,1%

0,0% 5,0% 10,0% 15,0% 20,0%

E-mail-E-mail E-mail-SM S SM S-E-mail SM S-E-mail without URL SM S-SM S

Progress of Participation Rates

27% 31% 33% 35% 36% 37% 53% 26% 27%

40% 50% 60%

SM S

Participation Rate

26

21% 27% 31% 11% 15% 19%21% 24% 26% 27%

0% 10% 20% 30%

1 2 3 4 5 6 7 8 9 10 SM S E-mail

Hours

Total

Main Findings

(1) Scrolling design makes the process

• f

completing the survey easier. (2) SMS invitation is more efficient compared to e- mail in encouraging respondents to complete the

27

mail in encouraging respondents to complete the survey via a mobile device.