Connec Con necting: ting: Adapting Survey Methods to Reach a - - PowerPoint PPT Presentation

Con Connec necting: ting:

Pr Presentation ion at the 20 2014 AA AAPO POR Co Conference Anaheim, , Ca Cali lifornia ia

Adapting Survey Methods to Reach a Highly Mobile Sample

Melissa Dugger

May 16, 2014

2

Presentation Overview

• Study background
• Sample
• Data collection
• Locating
• Outcomes
• Lessons learned

3

A Vision, Strategy, and Evaluation

• San Francisco-based intermediary with strategic vision
• Strategy: social enterprise employment
• Evaluation culture
• Key interests: Employment, housing, criminal activity

4

Social Enterprises (1)

Organization Social Enterprises Business Line Annual Employment (approximate) Year Started Target Population Organization A Blue Skies Cafes 18 1986 Mental health disabilities Cleaning 23 2009 People’s Harvest Food processing 2013 Organization B Golden State Works Street cleaning 108 2011 Parolees Organization C Chrysalis Enterprises Staffing 500 1984 Formerly incarcerated, homeless Street cleaning

5

Social Enterprises (2)

Organization Social Enterprises Business Line Annual Employment (approximate) Year Started Target Population Organization D SF Solutions Lobby services 55 2007 Homeless Maintenance services 30 Organization E Back-to- Work Retail 36 2012 Any barrier Organization F Hope Builders Construction 12 2007 Young adults aged 18–28 who are not enrolled in school

• r participating the

labor market Organization G 360° Solutions Pest control 10 2012 Homeless

6

Data Collection

Study Component Interviewer Administration Start Date Completion Date Incentive Baseline survey Social enterprise staff member When participant enters the social enterprise 4/1/2012 3/31/2013 Study brochure and magnet with Mathematica’s information Exit survey Social enterprise staff member Six months after participant leaves the social enterprise 5/22/2013 8/29/2013 $20 gift card Follow-up survey Mathematica interviewer Approximately

• ne year after

participant leaves the social enterprise 9/3/2013 3/14/2014 $20 gift card

7

Follow-Up Survey

• Conducted approximately one year after the baseline survey
• Hard copy survey was administered by phone by Mathematica

phone interviewers

• Field locating started on November 15, 2013
• Data collection ended on March 14, 2014
• Hired four field locators

Group Start Phone Calls 1 September 4, 2013 2 October 2, 2013 3 November 2, 2013

8

MJS Target Population and Samples

Source: Mathematica Jobs Study database Target population (n = 718) Consent (85%) Work SE (87%) Sampled (95%) Not work SE (n=82) Baseline survey (n = 609) Exit survey (n = 527) Follow-up survey population (n = 582)

9

A Difficult Population to Follow

All Org A Org B Org C Org D Org E Org F Org G Average age 41 37 33 44 41 46 25 49 Never employed 25 46 16 25 29 26 23 No high school diploma 29 10 51 27 17 6 38 Ever convicted 69 25 100 71 61 20 62 20 Homeless 85 84 90 83 90 77 38 100 Work income 23 18 32 17 30 35 79 69

Note: Homeless means did not stay in owned or rented housing for entirety of year before baseline survey; work income is the percentage of income from work. All numbers percentages unless otherwise noted.

10

Targets by Organization

Number of Cases Expected Completes Organization A 41 23 Organization B 94 52 Organization C 356 196 Treatment 274 151 Control 82 45 Organization D 42 23 Organization E 31 17 Organization F 13 7 Organization G 5 3 Total 582 321

11

Two Tiers of Locating

Level 1: Traditional locating Level 2: Integration of organizations

12

Level 1: Traditional Locating (1)

• Obtained extensive contact information on baseline and exit surveys
• Identified and updated contact information with “holiday cards”
• Used locating resources, including Web searches and a search through

a database of public records

13

Level 1: Traditional Locating (2)

• Identified incarcerated individuals weekly and monitored release dates

to remove individuals temporarily or permanently from survey efforts

• Sent email to nonrespondents and refusal conversion letter

to respondents who had refused

• Deployed field locators
• Heavily monitored staff productivity and locating efforts

14

Level 1: Traditional Locating (3)

Locating Start Date Completion Rate When Started Completion Rate at the End Level 1 11/15/14 24% (141) 39% (231)

15

Level 1: Traditional Locating (4)

Expected Completes Total Completes to Date # of Completes Needed Organization A 23 19 4 Organization B 52 20 32 Organization C 196 152 44 Organization D 23 14 9 Organization E 17 17 Organization F 7 7 Organization G 3 2 1 Total 321 231 90

16

Level 2: Integrated Organizations in Locating (1)

• Hired organization staff as field locators
• Had organizations provide updated contact information for

nonrespondents, including current work status in the social enterprise

• Provided staff at organizations with incentives
• Integrated information about the follow-up survey into organization's

support services (for example, retention staff reminded participants verbally or with written materials about the study and provided phone numbers, phones, and on-site gift cards for completing the survey)

17

Level 2: Integrated Organizations in Locating (2)

Locating Start Date Completion Rate when Started Completion Rate at the End Level 1 11/15/14 24% (141) 39% (231) Level 2 01/28/14 39% (231) 44% (258)

18

Level 2: Integrated Organizations in Locating (3)

Expected Completes Total Completes to Date # of Completes Needed Organization A 23 19 4 Organization B 52 20 32 Organization C 196 152 44 Organization D 23 20 3 Organization E 17 17 Organization F 7 8

• 1

Organization G 3 4

• 1

Total 321 258 63

19

Assessment of Tier 1 & 2 Locating Effort

• 11 weeks in the field
• Current response rate 44 percent
• 3 weeks left in the field period
• 63 completes needed to reach target response rate
• Average of 9 completes per week

20

Assessment of Locating Effort

• Organizations use Facebook for outreach
• One social enterprise participant blogged about their experience

21

Can We Use Facebook to Reach Out to Participants to Complete the Follow-Up Survey?

22

Relevant Resources

• According to the Public Policy Institute of California, internet use

among Californians ages 35 to 54 is 88 percent

• In southern California, where there is a high number
• f nonrespondents, internet use was over 85 percent

23

Level 3: Using Social Media for Locating

• Ask organizations to post a reminder message about the follow-up

survey on their Facebook page

• Use Facebook to find additional information that might

facilitate locating

• Send personalized messages to nonrespondents

24

Facebook Profile

25

Facebook Protocol: Match Information

• First and last name
• Date of birth
• City and state
• Email address
• Family member
• Mention of social enterprise

26

Facebook Outcome

*We only considered cases with an interim status code for Facebook locating.

Organizations Number of Cases Name Email Unable to Locate Organization B 20 8 4 8 Organization C 127 36 23 68 TOTAL 147 71 76

27

Email or Name Match

4 23 8 36 8 68 10 20 30 40 50 60 70 80 Org B Org C Email Name Unable to locate Number of Cases

28

Email or Name and Other Information

12 59 3 6 10 20 30 40 50 60 70 Org B Org C Other Email or Name Number of Cases

29

Locating Start Dates and Completion Rates

Completion rate is: # completed surveys/# follow up sample

Locating Start Date Completion Rate When Started Completion Rate at the End Level 1 11/15/14 24% (141) 39% (231) Level 2 01/28/14 39% (231) 44% (258) Level 3 02/20/14 44% (258) 47% (274)

30

Completes by Organization

Expected Completes Number of Completes Organization A 23 21 Organization B 52 32 Organization C 196 170 Treatment 151 138 Control 45 32 Organization D 23 21 Organization E 17 18 Organization F 7 8 Organization G 3 4 Total 321 274

31

Response Rates

Response rate = # completed/(# surveyed - # incarcerated - # deceased)

Number Response Rates (%) Surveyed Completed Surveys Incarcerated Deceased Organization A 41 21 1 53 Organization B 94 32 20 1 44 Organization C 356 170 21 1 51 Treatment 274 138 16 1 54 Comparison 82 32 5 42 Organization D 42 21 2 53 Organization E 31 18 58 Organization F 13 8 1 1 73 Organization G 5 4 80 Total 582 274 45 3 51

32

Lessons Learned

• If you think you may want to implement social media into the locating

effort, gain consent and important contact information at the beginning

• f data collection
• Leverage the social media outreach from the organizations at the onset
• f data collection
• Don’t discount using social media for hard-to-reach populations

33

For More Information

• Melissa Dugger

MDugger@mathematica-mpr.com