Lock Sampling or: Yes, Panels are Different - Now What? 2019 DC-AAPOR/WSS Review-Preview Summer Conference Jake Soffronoff Public Policy Analyst USPS Office of Inspector General Disclaimer: The author of this presentation is solely responsible for its content. The opinions expressed are the author’s own and do not represent the views of the United States Postal Service Office of Inspector General, the United States Postal Service, or any other organization.
Survey Research at USPS OIG Established a survey research shop from the ground up within the agency • I execute all phases of research, aside from sending respondents to our online platform • Panel companies only send respondents to OIG’s online platform – I dictate sample design and constantly monitor and manage field with panel PMs • Goal: “Comprehensible but Rigorous” online research JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
Why Try to Improve Estimates? Essential Fact: Everyone interacts with “the mail,” but not everyone interacts with it in the same way – e.g., those who shop online vs. not, those in rural vs urban areas, etc. Essential Question: How do I run national surveys online about the United States Postal Service that correct for the bias created by an “online - only” format as much as possible ? Approach: Incrementally combined experience, intuition, and logic to develop “Lock Sampling” approach, which has yielded encouraging results. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
Published Results Comparison: National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…? The personal USPS post office A personal USPS post office box registered to someone Weighted per each Your own personal USPS post box registered to someone who does not live in your project's respective office box else in your household household weights June 2017 June 2017 June 2017 June 2017 June 2017 June 2017 National USPS OIG National USPS OIG National USPS OIG Telephone Online Telephone Online Telephone Online Omnibus Survey Omnibus Survey Omnibus Survey Sample Size 1010 1923 Difference 1010 1923 Difference 1010 1923 Difference 12% 11% 8% 6% 3% 4% Yes -1 pt -2 pts +1 pt 87% 87% 90% 92% 96% 93% No 0 pts +2 pts -3 pts 0% 2% 1% 2% 1% 3% Not Sure/DK +2 pts +1 pt +2 pts 1% NA 0% NA 1% NA Refused (1 pt) (<1 pt) (1 pt) Source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix. ( ): Points missing due to answer option Full question text in presentation appendix. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE differences across modes.
Published Results Comparison: National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…? Weighted per each project's A USPS post office box registered to your USPS post office box registered to a employer business that you own/run respective weights June 2017 June 2017 National June 2017 National June 2017 Telephone USPS OIG Telephone USPS OIG Omnibus Online Survey Omnibus Online Survey Sample Size 1010 1923 Difference 1010 1923 Difference 3% 3% 2% 4% Yes 0 pts +2 pts 95% 93% 98% 94% No -2 pts -4 pts 0% 2% Not Sure/DK 2% 4% +2 pts +2 pts 0% NA 0% NA Refused (<1 pt) (<1 pt) Source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix. ( ): Points missing due to answer option Full question text in presentation appendix. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE differences across modes.
Published Results Comparison: National Telephone Omnibus vs. “Lock Sampled” Online Post Office Visitation PO Visitation Weighted per each project's respective weights March 2017 National May 2017 USPS OIG Telephone Omnibus Online Survey Difference Counts 1012 1316 6% 7% Past day +1 pt 21% 20% Past week -1 pt 20% 24% Past month +4 pts 15% 17% Past 3 months +2 pts 9% 10% Past 6 months +1 pt 9% 9% Past year 0 pts More than one year ago 15% 10% -5 pt 6% 1% Never -5 pt Not sure NA 3% (3 pts) 1% NA Visited, don't recall when (1 pt) 0% NA Don't know (<1 pt) — NA Refused ( — ) Source: United States Postal Service Office of Inspector General. (2017). Delivering Peace of Mind: Certified Mail and the U.S. Postal Service. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-18-002.pdf ( ): Points missing due to answer option Full question text in presentation appendix. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE differences across modes.
Key Assumptions 1. Online nonprobability panels are composed differently than the general population. • If they were not, I would be thrilled : none of this would be necessary. 2. The rows are where the magic happens: One can not weight what does not exist. 3. Quota sampling allows researchers to build robust models of the population that can help correct for panels’ weaknesses when implemented with patience. 4. Online biases are not only demographic, but are also behavioral/quasi-cultural due to inherent “onlinedness.” JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
My “Onlinedness” Definition The cultural manifestations of how those who use the internet interact with the world around them. These manifestations vary by the method and frequency of internet use. • Example: How often one shops online directly relates to how often, and in what forms, one interacts with the United States Postal Service. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
Without Lock Sampling: PM eCommerce Participation Hispanic Status x Nativity Past month eCommerce participant 108% US Born Hispanic 61% Compounding NOT past month eCommerce participant 37% Foreign Born Hispanic 8% TOTAL 82% NOT Hispanic 92% Complications TOTAL 82% • Limited space left for slow- Age by Gender Male Female responding groups. Male 18-24 36% Female 18-24 87% • Due to fast-responders filling Male 25-34 48% Female 25-34 79% quotas, 63% of “Not past month Male 35-44 62% Female 35-44 87% eCommerce participants” have yet Male 45-54 80% Female 45-54 98% to respond, and now when they do Male 55-64 92% Female 55-64 108% they can not be from the Middle Male 65-75 108% Female 65-75 108% Atlantic or New England regions, Male TOTAL 70% Female TOTAL 94% and can not be male aged 65+ or Subregion female 55+. Middle Atlantic 108% East South Central 56% • The problem compounds for other New England 108% South Atlantic 70% lagging groups. Here, 92% of East North Central 97% Pacific 78% Foreign Born Hispanic respondents West North Central 103% Mountain 90% face the same restrictions as above, West South Central 51% and also can not have ordered TOTAL 82% Table shows each quota group’s progress to reaching its proportionate sample size at a snapshot in time. To allow for cleanin g & online in the past month. quota interplay, in USPS OIG projects each quota group is capped at 108% before additional quota respondents are terminated. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
The BIG Problem Designing a quota sampling model that actively counteracts the observed deficiencies of your data source does not lend itself to easy data collection. JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE
Recommend
More recommend
