Redesignin ing th the Americ ican Community Survey Recruit itment Messagin ing: From analy lysis to desig ign Jonathan Schreiner American Community Survey Office, U.S. Census Bureau DC-AAPOR/WSS Review-Preview Summer Conference June 12, 2019 – Washington, D.C. This presentation is released to inform interested parties of ongoing research and to encourage discussion of work in progress. Any views expressed on statistical, methodological, operational, or technical issues are those of the authors and not necessarily those of the U.S. Census Bureau.
ACS Self-Response Mail Contact Strategy Nonrespondents All Mailable Sample Addresses Nonrespondents First Fourth Second Third Fifth Mailing Mailing Mailing Mailing Mailing Letter Reminder Letter Paper Questionnaire Reminder Postcard Reminder Letter Instruction Card (internet) Letter FAQ Brochure Instruction Card (choice) Multilingual Brochure FAQ Brochure Return Envelope 2
Holistic Design • “Research has been successful in identifying specific design features that tend to have larger versus smaller effects [on response rates], but generally leaves unanswered how all elements of a design fit together .” (Dillman, Christian and Smyth 2014) 3
What if we start from “scratch”? Nonrespondents All Mailable Sample Addresses Nonrespondents First Fourth Second Third Fifth 14 18 4 7 Mailing Mailing Mailing Mailing Mailing Paper Questionnaire ? ? ? ? ? 4
Presentation Outline Step 1: Develop a “Strategic Framework” for messaging Step 2: Evaluate current materials Step 3: Design new materials Overview of next steps 5
Step 1: Develop a Strategic Framework for messaging 1 2 3 4 6 6
Step 1: Develop a Strategic Framework for messaging 7
Step 1: : Develop a Strategic Framework for messaging What do we know about the ACS STRATEGIC FRAMEWORK audience? What does the What do the literature tell experts think us about of ACS messaging? messaging? 8
Four Lessons Learned 1) Limit volume of messaging Oliver, Heimel & Schreiner (2017) “Strategic Framework for Messaging in the ACS Mail Materials” 9
Four Lessons Learned 1) Limit volume of messaging 2) Follow Plain Language guidelines Oliver, Heimel & Schreiner (2017) “Strategic Framework for Messaging in the ACS Mail Materials” 10
Four Lessons Learned 1) Limit volume of messaging 2) Follow Plain Language guidelines 3) Each mailing needs to have a clear purpose and highlight different reasons to participate Oliver, Heimel & Schreiner (2017) “Strategic Framework for Messaging in the ACS Mail Materials” 11
Four Lessons Learned 1) Limit volume of messaging 2) Follow Plain Language guidelines 3) Each mailing needs to have a clear purpose and highlight different reasons to participate 4) Leverage the Census Bureau brand Oliver, Heimel & Schreiner (2017) “Strategic Framework for Messaging in the ACS Mail Materials” 12
Step 2: : U Use the Strategic Framework report to Evaluate the Current ACS Materials 13
Lesson 1: Limit volume of messaging Mailing 1: 129 messages in these 5 mail pieces 14
Lesson 2: : Follow Plain Language guidelines Mail Piece FRE Score Interpretation Grade Level 10 th -12 th Mailing 1 Letter “fairly difficult to read” Multilingual Brochure “difficult to read” College FAQ Brochure “difficult to read” College Flesch Reading Ease Scores of ACS Mail Materials Source: U.S. Census Bureau, American Community Survey Office Schreiner, Oliver & Poehler (forthcoming) 15
Lesson 3: : Each mailing should have a clear purpose and highlight different reasons to participate Repetition across mail materials 1 st mailing letter 2nd mailing letter 3rd mailing letter 16
Lesson 3: : Each mailing should have a clear purpose and highlight different reasons to participate Repetition across mail materials 4 th mailing post card 5 th mailing letter 17
Lesson 4: : Leverage the Census Bureau brand Graphic Used Envelopes X Instruction Cards X FAQ Brochure X Multilingual Brochure X Letters X X Survey Form X Post Card X 18
Lesson 4: : Leverage the Census Bureau brand Four sponsors? 1 1) UNITED STATES DEPARTMENT OF COMMERCE 2) Economics and Statistics Administration 3) U.S. Census Bureau 4 3 2 4) OFFICE OF THE DIRECTOR 19
Lesson 4: : Leverage the Census Bureau brand 20
Step 3: : Apply Findings to Design New Materials To address the four lessons learned, ACS is testing two main changes the messaging strategy for our letters. 21
Change 1: Apply a Strategic Messaging Plan First Third Second Mailing Mailing Mailing Trust Benefits Burden Reduction • Fewer, more focused messages per mailing • Increased comprehension • Less repetition across mailings • More appeals to respond 22
Change 2: Redesign ACS letters • New Design • Reduce text • Focus messaging • Add white space • Use plain language • Improved branding 23
Version 1 • Clear branding • Use color and to draw attention • F-pattern headings to ease readability, call attention 24
Version 2 • Use color and icons to draw attention • Tag line to add additional message and branding 25
Version 3 • Color “Side bar” provides additional information • Text reduced and focused 26
Consistent design across mailings Letter 1 Letter 2 Letter 3 27
Next xt Steps for the Strategic Framework Inventory Develop Analyze results Develop Field revisions to Qualitative and Analyze and make Strategic Testing Current ACS mail evaluation recommendations Framework materials Messaging STEP 1 STEP 2 STEP 3 STEP 4 STEP 6 STEP 5 In progress 28
Thank you! Jon Schreiner jonathan.schreiner@census.gov (301) 763 9286 • Oliver, Heimel & Schreiner (2017) “Strategic Framework for Messaging in the ACS Mail Materials ” https://www.census.gov/library/working-papers/2017/acs/2017_Oliver_01.html • Dillman , Christian & Smyth (2014) “The Tailored Design Method” • Schreiner, Oliver & Poehler (forthcoming) “Assessment of Messaging in the 2018 American Community Survey Mail Contact Materials” 29
30
Appendices 31
Appendix A – • More complete list of lessons learned by SF area. 32
What do we know about the ACS audience • Some people distrust the government • ACS is unknown, but many know the Census Bureau • Messages that convey community-level benefits are viewed favorably • Messages about the confidentiality do not have high believability This slide summarizes results from the following Decennial Census and ACS mindset and motivation studies: Segmentation and Mindset Study (2007), Census Barriers, Attitudes, and Motivators Surveys I (2008), Census Barriers, Attitudes, and Motivators Surveys II (2010), Collaborative research with Reingold, Inc. (2013-2014), and Collaborative research with Gallup (2014). 33
What do the Experts think of f ACS Messaging? • Each contact needs a clear purpose • Messaging between mailings is distinct, but mutually supportive • Community-level benefits • Emphasize culturally relevant messages of empowerment • Attach ACS to the Census Bureau • Social norms 2016 NAS CNSTAT Workshop & Social and Behavioral Sciences Team 34
What does the Lit iterature tell us about Messaging? • Multiple mail contacts should feel like a single, continuous conversation between the Census Bureau and the ACS recipient • Each contact should highlight different reasons to participate • Limit the number of messages per contact • Establish trust early with messages that are believable • Convey the benefits of ACS data to the recipient’s community • Reduce respondent’s perceptions of burden • Use plain language to communicate This slide summarizes literature from a number of fields, including Communications, Behavioral Sciences, Marketing, Sociology, Survey Methodology, Psychology, and Plain Language. 35
Appendix B • Current Materials Coding Methodology 36
Coding Methodology • Developed a list of potential messages in ACS materials • Strategic Framework & Current Materials • 74 codes for messaging in 4 categories : • Trust • Benefits • Reduce Burden • Instructions & Information and Required Elements Codebook Example Code Theme 1.0 Trust: Recipients must believe that the survey is legitimate and that the messages are truthful 1.2 Confidentiality/Data Security 1.2.1 By law, Census must protect your data 1.2.2 Census employees face fines and prison if they violate law 37
Example of f a Coded Letter 4.5 (Required information) 1.1.1 (Sponsorship) 4.1.1 (Part of a random sample) A message from the Director, U.S. Census Bureau ... 2.3.7 (Highlight importance) Your household has been randomly selected to complete a very important national survey, the American Community Survey. The U.S. Census Bureau 2.2.2 (National level benefit) conducts this survey to give our country an up-to-date picture of how we live — our education, employment, housing, and more. Using the enclosed 4.4.1 (Call to action) instructions , please complete the survey online as soon as possible at: 8
Appendix C – Cut slides 39
Recommend
More recommend
