AAPOR 2011 Membership Survey Preliminary Paradata Analysis Heather - - PowerPoint PPT Presentation

AAPOR 2011 Membership Survey Preliminary Paradata Analysis

Heather Hammer, Abt SRBI

American Association for Public Opinion Research 67th Annual Conference Presentation May 18, 2012

AAPOR 2011 Membership Survey Paradata Analysis

Background for the Analysis

• Was Senior Study Director at the Temple University Institute

for Survey Research (ISR) when the opportunity to conduct the 2011 AAPOR Membership Survey was announced

– ISR was a CASES shop – CASES recently added web survey capacity – Wanted to develop and pilot a unified case history coding

scheme for ISR’s multi-mode studies

– Proposed to add telephone outreach to the 2011 AAPOR

Membership Survey e-mail and mail nonresponse effort for those who did not complete a web or mail survey by the third e-mail reminder

2

AAPOR 2011 Membership Survey Paradata Analysis

Background for the Analysis

• Relied heavily on David Chearo and Martha Van Haitsma’s

Survey Practice paper titled “Standardized Attempt Codes for Unified Multi-Mode Case Histories” (February 2012)

• Developed unified incoming vs. outbound categories and

codes to record the mode, source, and type of outcome for a maximum of 13 outcomes

• Created a data entry screen for interviewers to capture the

telephone and other manual contact attempt data

3

AAPOR 2011 Membership Survey Paradata Analysis

Contact Attempt Modes

00 no attempt - action needed 01 in-person/CAPI 02 telephone/CATI/IVR 03 mail 04 web form 05 email 06 fax 07 text message 08 instant message 09 voice message

4

AAPOR 2011 Membership Survey Paradata Analysis

Contact Attempt Sources

1 selected respondent 2 eligible proxy 3 informant 4 interviewer 5 supervisor 6 tracer 7 system generated

5

AAPOR 2011 Membership Survey Paradata Analysis

Selected Outbound Contact with R

6

056 057 alternate mode requested/needed 058 alternate language requested/needed 059 060 appointment made/request for definite return contact 061 request callback preferred times/days or general 062 request for information or materials 063 suspend contact attempts until future date (e.g., R on vacation) 064 065 promise to return contact/complete survey 066 067 break-off (partial), follow up needed 068 069

AAPOR 2011 Membership Survey Paradata Analysis

Outbound Phone Noncontact Examples

050 continuous ring 051 no message left 052 no message possible 053 message left 054 line busy

7

AAPOR 2011 Membership Survey Paradata Analysis

Manual Data Entry Screen

8

AAPOR 2011 Membership Survey Paradata Analysis

Mode and Source Drop Down Lists

9

AAPOR 2011 Membership Survey Paradata Analysis

Data Collection (June 27 - August 25, 2011)

Outcome Number Number of Individuals in Sequence Contact Attempt Sequence Completed by Any Mode Mean Days Between t and t-1 2,844 Total Listed 1 2,771 E-mail Invitation Start 2 2,119 E-mail Reminder 1 482 8 3 1,862 E-mail Reminder 2 249 8 4 1,691 Mail Questionnaire 156 7 5 1,640 E-mail Reminder 3 12 2 6 1,348 Outbound Telephone 289 5 7 814 Outbound Telephone 189 12 8 50 Outbound Telephone 88 8 9 7 Outbound Telephone 35 5 10 Outbound Telephone 19 11 1 Outbound Telephone 8 12 Outbound Telephone 1 13 Outbound Telephone 4 End

10

AAPOR 2011 Membership Survey Paradata Analysis

Completed Surveys by Mode and Type

Type of Complete Completion Mode as Percent of All Completes Type as Percent of All Completes Type as Percent of Total Eligible Web Phone Mail Total Total

Current Members

1059 87.7% 3 0.2% 145 12.0% 1207 78.8% 1986 69.8%

Former Members

154 82.8% 1 0.5% 31 16.7% 186 12.1% 607 21.3%

Conference Attendees

111 80.4% 1 0.7% 26 18.8% 138 9.0% 251 8.8%

Total

1324 86.5% 5 0.3% 202 13.2% 1531 100.0% 2844 100.0%

11

AAPOR 2011 Membership Survey Paradata Analysis

Cumulative Completes by Mode

12

All Completed Surveys N=1,531 Web Completes N=1,324 Mail Completes N=202 Phone Completes N=5

Completed Survey Outcome Number Typical Sequence Outcome N %

• Cum. %

N %

• Cum. %

N %

• Cum. %

N %

• Cum. %

1 E-mail invite 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2 E-mail reminder 1 482 31.5 31.5 474 35.8 35.8 8 4.0 4.0 0.0 0.0 3 E-mail reminder 2 249 16.3 47.7 245 18.5 54.3 4 2.0 5.9 0.0 0.0 4 Mail questionnaire 155 10.1 57.9 152 11.5 65.8 3 1.5 7.4 0.0 0.0 5 E-mail reminder 3 12 0.8 58.7 12 0.9 66.7 0.0 7.4 0.0 0.0 6 Mostly outbound phone 289 18.9 77.5 241 18.2 84.9 47 23.2 30.7 1 20.0 20.0 7 Mostly outbound phone 189 12.3 89.9 116 8.8 93.7 72 35.6 66.3 1 20.0 40.0 8 Mixed outbound phone 88 5.7 95.6 37 2.8 96.5 48 23.8 90.1 3 60.0 100.0 9 Varies 35 2.3 97.9 23 1.7 98.2 12 5.9 96.0 0.0 10 Varies 19 1.2 99.2 15 1.1 99.3 4 2.0 98.0 0.0 11 Varies 8 0.5 99.7 7 0.5 99.8 1 0.5 98.5 0.0 12 Varies 1 0.1 99.7 0.0 99.8 1 0.5 99.0 0.0 13 Varies 4 0.3 100.0 2 0.3 100.0 2 1.0 100.0 0.0 Total 1,531 100.0 1,324 100.0 202 100.0 5 100.0

AAPOR 2011 Membership Survey Paradata Analysis

Cumulative Response Rates by Type

13

AAPOR 2011 Membership Survey Paradata Analysis

Phone Effort by Date for Outcomes 6-9

14

AAPOR 2011 Membership Survey Paradata Analysis

Targeted Nonrespondent Completes

• 41.3% of the 1,531 completed surveys, including 33.3% of

the 1,324 completed web surveys, 92.6% of the 202 completed mail surveys, and all five of the completed telephone surveys were done between the start of the telephone nonresponse effort at Outcome 6 and the end of data collection at Outcome 13

• Whereas two-thirds (66.7%) of all completed web surveys

were done by the time the telephone nonresponse effort was launched at outcome 6, only 7.4% of the completed mail surveys had been received, and no telephone surveys had been completed even though there was telephone contact with some individuals prior to Outcome 6

15

AAPOR 2011 Membership Survey Paradata Analysis

Question for the Paradata Analysis

• Did the telephone effort make a meaningful

contribution to the response rate?

16

AAPOR 2011 Membership Survey Paradata Analysis

Phone Targeted Nonrespondents

Type of Complete Completion Mode as Percent of All Targeted Nonrespondent Completes Type as Percent

• f All Targeted

Nonrespondent Completes Web Phone Mail Total

Current Members

336 71.3% 3 0.6% 132 28.0% 471 74.5%

Former Members

69 69.7% 1 1.0% 29 29.3% 99 15.7%

Conference Attendees

36 58.1% 1 1.6% 25 40.3% 62 9.8%

Total

441 69.8% 5 0.8% 186 29.4% 632 100.0%

17

AAPOR 2011 Membership Survey Paradata Analysis

Completed at t+1 after Phone at t

Type of Complete Completion Mode as Percent of Total Completes Type as Percent of Total Completes Web Mail Total

Current Members

115 58.7% 81 41.3% 196 75.7%

Former Members

14 43.8% 18 56.3% 32 12.3%

Conference Attendees

8 58.1% 23 40.3% 31 12.0%

Total

137 52.9% 122 47.1% 259 100.0%

18

p=0.002

The 259 web and mail surveys completed at the

• utcome

immediately following the

• utbound phone

contact or message comprise 41% of all 632 completes among phone targeted nonrespondents.

AAPOR 2011 Membership Survey Paradata Analysis

Paradata Analysis Answer to Question

• The paradata suggest that the telephone effort

boosted the response rate among targeted nonrespondents, however; there was no randomization, no comparison group, and this simple analysis did not model the cumulative effect of mailing the questionnaire, sending a third e-mail reminder, and the telephone effort.

– e.g., indirect effects are evident in patterns

where telephone contact at time t leads to sending the e-mail link again and a complete at

• utcome t+2

19

AAPOR 2011 Membership Survey Paradata Analysis

Cumulative Means by Date, Members

20

AAPOR 2011 Membership Survey Paradata Analysis

Cumulative Means, Former Members

21

AAPOR 2011 Membership Survey Paradata Analysis

Thank You

22

• The AAPOR Membership and Chapter Relations Committee
• Joe Murphy, RTI International
• Liz Hamel, The Henry J. Kaiser Family Foundation
• Chase H. Harrison, Harvard University Business School
• Dan Merkle, ABC News
• Andy Weiss, Abt SRBI
• Rene Bautista, NORC

AAPOR 2011 Membership Survey Paradata Analysis

For More Information

23

• AAPOR 2011 Membership Survey Final Report
• Available at www.aapor.org
• Member Connect > Membership Research

AAPOR 2011 Membership Survey Paradata Analysis 24