Lock Sampling or: Yes, Panels are Different - Now What? 2019 - - PowerPoint PPT Presentation

lock sampling
SMART_READER_LITE
LIVE PREVIEW

Lock Sampling or: Yes, Panels are Different - Now What? 2019 - - PowerPoint PPT Presentation

Lock Sampling or: Yes, Panels are Different - Now What? 2019 DC-AAPOR/WSS Review-Preview Summer Conference Jake Soffronoff Public Policy Analyst USPS Office of Inspector General Disclaimer: The author of this presentation is solely responsible


slide-1
SLIDE 1

Lock Sampling

  • r: Yes, Panels are Different - Now What?

2019 DC-AAPOR/WSS Review-Preview Summer Conference Jake Soffronoff

Public Policy Analyst USPS Office of Inspector General Disclaimer: The author of this presentation is solely responsible for its content. The opinions expressed are the author’s

  • wn and do not represent the views of the United States Postal Service Office of Inspector General, the United States

Postal Service, or any other organization.

slide-2
SLIDE 2

Survey Research at USPS OIG

Established a survey research shop from the ground up within the agency

  • I execute all phases of research, aside from sending respondents to
  • ur online platform
  • Panel companies only send respondents to OIG’s online platform – I dictate

sample design and constantly monitor and manage field with panel PMs

  • Goal: “Comprehensible but Rigorous” online research

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-3
SLIDE 3

Why Try to Improve Estimates?

Essential Fact: Everyone interacts with “the mail,” but not everyone interacts with it in the same way – e.g., those who shop

  • nline vs. not, those in rural vs urban areas, etc.

Essential Question: How do I run national surveys online about the United States Postal Service that correct for the bias created by an “online-

  • nly” format as much as possible?

Approach: Incrementally combined experience, intuition, and logic to develop “Lock Sampling” approach, which has yielded encouraging results.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-4
SLIDE 4

Published Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…?

Weighted per each project's respective weights

Your own personal USPS post

  • ffice box

A personal USPS post office box registered to someone else in your household The personal USPS post office box registered to someone who does not live in your household

June 2017 National Telephone Omnibus June 2017 USPS OIG Online Survey Difference June 2017 National Telephone Omnibus June 2017 USPS OIG Online Survey Difference June 2017 National Telephone Omnibus June 2017 USPS OIG Online Survey Difference Sample Size 1010 1923 1010 1923 1010 1923

Yes

12% 11%

  • 1 pt

8% 6%

  • 2 pts

3% 4%

+1 pt No

87% 87%

0 pts

90% 92%

+2 pts

96% 93%

  • 3 pts

Not Sure/DK

0% 2%

+2 pts

1% 2%

+1 pt

1% 3%

+2 pts Refused

1% NA

(1 pt)

0% NA

(<1 pt)

1% NA

(1 pt)

Source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix.

Full question text in presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-5
SLIDE 5

Weighted per each project's respective weights

A USPS post office box registered to your employer USPS post office box registered to a business that you own/run

June 2017 National Telephone Omnibus June 2017 USPS OIG Online Survey Difference June 2017 National Telephone Omnibus June 2017 USPS OIG Online Survey Difference Sample Size 1010 1923 1010 1923

Yes

3% 3%

0 pts

2% 4%

+2 pts No

95% 93%

  • 2 pts

98% 94%

  • 4 pts

Not Sure/DK

2% 4%

+2 pts

0% 2%

+2 pts Refused

0% NA

(<1 pt)

0% NA

(<1 pt)

Published Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…?

Full question text in presentation appendix.

Source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-6
SLIDE 6

Published Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Post Office Visitation

Source: United States Postal Service Office of Inspector General. (2017). Delivering Peace of Mind: Certified Mail and the U.S. Postal Service. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-18-002.pdf

Weighted per each project's respective weights

PO Visitation March 2017 National Telephone Omnibus May 2017 USPS OIG Online Survey

Difference

Counts 1012 1316

Past day

6% 7%

+1 pt Past week

21% 20%

  • 1 pt

Past month

20% 24%

+4 pts Past 3 months

15% 17%

+2 pts Past 6 months

9% 10%

+1 pt Past year

9% 9%

0 pts More than one year ago

15% 10%

  • 5 pt

Never

6% 1%

  • 5 pt

Not sure

NA 3%

(3 pts) Visited, don't recall when

1% NA

(1 pt) Don't know

0% NA

(<1 pt) Refused

— NA

(—)

Full question text in presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-7
SLIDE 7

Key Assumptions

  • 1. Online nonprobability panels are composed differently than the general

population.

  • If they were not, I would be thrilled: none of this would be necessary.
  • 2. The rows are where the magic happens: One can not weight what does not exist.
  • 3. Quota sampling allows researchers to build robust models of the population

that can help correct for panels’ weaknesses when implemented with patience.

  • 4. Online biases are not only demographic, but are also behavioral/quasi-cultural

due to inherent “onlinedness.”

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-8
SLIDE 8

My “Onlinedness” Definition

The cultural manifestations of how those who use the internet interact with the world around them. These manifestations vary by the method and frequency of internet use.

  • Example: How often one shops online directly relates to

how often, and in what forms, one interacts with the United States Postal Service.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-9
SLIDE 9

Without Lock Sampling:

Compounding Complications

  • Limited space left for slow-

responding groups.

  • Due to fast-responders filling

quotas, 63% of “Not past month eCommerce participants” have yet to respond, and now when they do they can not be from the Middle Atlantic or New England regions, and can not be male aged 65+ or female 55+.

  • The problem compounds for other

lagging groups. Here, 92% of Foreign Born Hispanic respondents face the same restrictions as above, and also can not have ordered

  • nline in the past month.

PM eCommerce Participation Hispanic Status x Nativity Past month eCommerce participant 108% US Born Hispanic 61% NOT past month eCommerce participant 37% Foreign Born Hispanic 8% TOTAL 82% NOT Hispanic 92% TOTAL 82% Age by Gender Male Female Male 18-24 36% Female 18-24 87% Male 25-34 48% Female 25-34 79% Male 35-44 62% Female 35-44 87% Male 45-54 80% Female 45-54 98% Male 55-64 92% Female 55-64 108% Male 65-75 108% Female 65-75 108% Male TOTAL 70% Female TOTAL 94% Subregion Middle Atlantic 108% East South Central 56% New England 108% South Atlantic 70% East North Central 97% Pacific 78% West North Central 103% Mountain 90% West South Central 51% TOTAL 82%

Table shows each quota group’s progress to reaching its proportionate sample size at a snapshot in time. To allow for cleaning & quota interplay, in USPS OIG projects each quota group is capped at 108% before additional quota respondents are terminated.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-10
SLIDE 10

The BIG Problem Designing a quota sampling model that actively counteracts the observed deficiencies of your data source does not lend itself to easy data collection.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-11
SLIDE 11

What is Lock Sampling?

A series of active measures taken to improve the final quality of data collected through online panels by: 1. Employing a collection of industry “best practices” to ensure that the data collected are clean. 2. Employing a sample design tailored to counteract panel deficiencies, including overrepresented respondent “onlinedness.” 3. Employing a “canal lock”-style field methodology that enforces the sample design by building in space and time for slow-responding groups to respond.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-12
SLIDE 12

Ensuring Clean Data

  • Respondents de-duped against related prior research.
  • Multiple active and passive “bot catchers” included in all

projects1.

  • In-field “please select this” traps with automatic termination.
  • Open-ended question at end of survey asking what the survey was about.
  • Data cleaned prior to weighting and analysis to remove

speeders, straightliners, and bots/inauthentic responders.

  • “Scoring-style” cleaning on multiple factors.

1: Full question text for example “bot catcher” questions included in the appendix of this presentation.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-13
SLIDE 13

Fairly Typical

Quotas & Weights to Actively Counteract Panel Deficiencies

Less Common, Actively Compensate For Panel Weaknesses

  • Age within gender
  • Nine geographic subregions
  • Race
  • Household online purchasing
  • Education level – including HS or less
  • Nativity within Hispanic/Latino status
  • Income
  • Employment status

Fairly Typical, Though Often Less Granular

Quotas and Weights Weights Only

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-14
SLIDE 14

Enforcing the Sample Design

Multi-phase approach — like the locks in a canal

Pace of field is determined by the eCommerce quotas, but other lagging groups are also allowed through “leaky caps.” Phases:

  • Open Field Phase: Attempt to achieve balance across standard demographic

quotas, while expecting hard-to-reach groups to lag behind.

  • “Leaky Cap” Phase: Lock out all but the lagging groups, who are accepted

regardless of other demographic characteristics.

  • Leads to demographic skews that are corrected during subsequent
  • pen field phases.
  • Repeat until full sample is achieved

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-15
SLIDE 15

Why Locks?

  • Locks ensure that the desired sampling units exist in the sample
  • Scale of “onlinedness” problem is too large to be handled through
  • versampling: Locks prevent irreparable damage
  • Locks ensure sample more closely matches weighting criteria,

minimizing the effect of weighting

  • Example: Project completed in 2018 was weighted on 44 items1, 35 of which were included in the quota design. Resulting

weights were between 0.35 – 3.76, with 62% of sample weighted up, 38% weighted down.

  • Interlaced field phases maintain focus on overall sample composition

while minimizing time effects

1: “Items” in this context counts each subcategory that was weighted for the project. For example, for purposes of this count each of 9 census subregions are counted as an “item.” Categories employed as weights for the project discussed here are: Age by Gender, Hispanic status by Nativity, Race, Subregion, Education, eCommerce Purchasing, Income, and Employment. Income and employment were weighted but not employed as quotas.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-16
SLIDE 16

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Open Field Phase: Opening typical of quota sample project – soft launch to check survey

programming, and pick up a portion of field while attempting general overall demographic balance.

slide-17
SLIDE 17

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 12% TOTAL 20%

slide-18
SLIDE 18

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 27% TOTAL 26%

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Leaky Cap Phase: A “leaky cap” locks out all but slowest-

responding/hard-to-reach groups, who are accepted regardless of other demographic characteristics.

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 12% TOTAL 20%

slide-19
SLIDE 19

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 27% TOTAL 26%

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Leaky Cap Phase: A “leaky cap” locks out all but slowest-

responding/hard-to-reach groups, who are accepted regardless of other demographic characteristics.

slide-20
SLIDE 20

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 27% TOTAL 26%

slide-21
SLIDE 21

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

PM eCommerce Participation Past month eCommerce participant 46% NOT past month eCommerce participant 32% TOTAL 40%

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Open Field Phase: Re-open field, attempt to correct

for skews created during leaky cap phase, gain ground

  • n overall field

PM eCommerce Participation Past month eCommerce participant 26% NOT past month eCommerce participant 27% TOTAL 26%

slide-22
SLIDE 22

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Open Field Phase: Re-open field, attempt to correct

for skews created during leaky cap phase, gain ground

  • n overall field

PM eCommerce Participation Past month eCommerce participant 46% NOT past month eCommerce participant 32% TOTAL 40%

slide-23
SLIDE 23

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

Lock Sampling: Multi-phase Approach to Field, like the Locks in a Canal

  • Repeat until complete: “Leaky Cap” and “Open Field”

phases repeated until full sample achieved, with the size of each field phase determined by the total n.

PM eCommerce Participation Past month eCommerce participant 108% NOT past month eCommerce participant 100% TOTAL 104%

slide-24
SLIDE 24

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

  • Data Cleaning: Remove straightliners, speeders,

bots/inauthentic respondents through “scoring system” cleaning

Lock Sampling: Multi-phase Approach to Field

PM eCommerce Participation Past month eCommerce participant 103% NOT past month eCommerce participant 92% TOTAL 98%

slide-25
SLIDE 25

Jake Soffronoff | Lock Sampling | DC-AAPOR/WSS Review-Preview Summer Conference

  • Data Cleaning & Weighting: Weight cleaned data to

population proportions

PM eCommerce Participation Past month eCommerce participant 100% NOT past month eCommerce participant 100% TOTAL 100%

Lock Sampling: Multi-phase Approach to Field

slide-26
SLIDE 26

Questions For Further Research

  • Is lock sampling robust in the face of additional testing?
  • Testing across multiple panel companies and field management teams, testing the results of full surveys instead of
  • ne-off questions, additional research into subgroup analyses, etc.
  • Are all of the aspects of lock sampling necessary to achieve comparable results, or are some steps

superfluous?

  • Are there additional steps that could be added to the lock sampling approach that would lead to better

results?

  • Is a past month eCommerce quota for controlling “onlinedness” robust across subject matter, or does

the control need to be subject-specific?

  • If modeled online samples can be made to yield similar results to probability methods, how should

confidence testing be handled?

  • Could lock sampling be implemented/automated by panel companies (i.e., on the sample supplier side)

to unburden overtaxed researchers while improving the quality of data provided?

  • Will enlightened panel companies ever make all of this unnecessary by adequately recruiting to correct

for the weak spots that inspire researcher angst?

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-27
SLIDE 27

Thank You

Lock Sampling

  • r: Yes, Panels are Different - Now What?

2019 DC-AAPOR/WSS Review-Preview Summer Conference

Jake Soffronoff

Public Policy Analyst USPS Office of Inspector General

slide-28
SLIDE 28

Appendix

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-29
SLIDE 29

Subgroup Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…?

Full question text in presentation appendix. Weighted per each project's respective weights

Your own personal USPS post office box A personal USPS post office box registered to someone else in your household

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference Sample Size

490 945 520 978 490 945 520 978

Yes

12% 14%

+2 pts

12% 8%

  • 4 pts

9% 6%

  • 3 pts

8% 5%

  • 3 pts

No

87% 85%

  • 2 pts

87% 90%

+3 pts

89% 92%

+3 pts

91% 92%

+1 pts NS/DK

0% 2%

+2 pts

0% 3%

+3 pts

2% 2%

0 pts

0% 2%

+2 pts Refused

0% NA

(<1 pt)

1% NA

(1 pt)

— NA

(—)

1% NA

(1 pt)

Subgroups

Subgroup analysis of source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC:

  • Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf

Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-30
SLIDE 30

Subgroup Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…?

Full question text in presentation appendix. Weighted per each project's respective weights

The personal USPS post office box registered to someone who does not live in your household A USPS post office box registered to your employer

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference Sample Size

490 945 520 978 490 945 520 978

Yes

3% 5%

+2 pts

3% 3%

0 pts

3% 4%

+1 pts

2% 2%

0 pts No

95% 92%

  • 3 pts

96% 95%

  • 1 pts

93% 93%

0 pts

96% 93%

  • 3 pts

NS/DK

2% 3%

+1 pts

0% 3%

+3 pts

3% 3%

0 pts

1% 4%

+3 pts Refused

0% NA

(<1 pt)

1% NA

(1 pt)

0% NA

(<1 pt)

1% NA

(1 pt)

Subgroups

Subgroup analysis of source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC:

  • Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf

Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-31
SLIDE 31

Subgroup Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Do you currently have access to…?

Full question text in presentation appendix. Weighted per each project's respective weights

USPS post office box registered to a business that you own/run

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference Sample Size

490 945 520 978

Yes

3% 6%

+3 pts

1% 2%

+1 pts No

96% 92%

  • 4 pts

99% 96%

  • 3 pts

NS/DK

0% 2%

+2 pts

0% 2%

+2 pts Refused

0% NA

(<1 pt)

1% NA

(1 pt)

Subgroups

Subgroup analysis of source: United States Postal Service Office of Inspector General. (2017). Billions Served: Foot Traffic at the Post Office. Washington, DC:

  • Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-17-012.pdf

Notes: Over the telephone, the question was asked in a national omnibus survey unrelated to postal activities, while online the question was a part of a survey entirely focused on postal activities. Consequently, the introduction text to the question differed slightly across modes in order to ensure that both sets of respondents had similar context when answering the question. In the online survey the question was only asked of those who had ever visited a U.S. Post Office (99% of the general public); tables here remove those not asked from the online base. Full question text for both modes is included in the presentation appendix.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

( ): Points missing due to answer option differences across modes.

slide-32
SLIDE 32

Subgroup Results Comparison:

National Telephone Omnibus vs. “Lock Sampled” Online Post Office Visitation

Subgroup analysis of source: United States Postal Service Office of Inspector General. (2017). Delivering Peace of Mind: Certified Mail and the U.S. Postal Service. Washington, DC: Author. https://www.uspsoig.gov/sites/default/files/document-library-files/2017/RARC-WP-18-002.pdf

Weighted per each project's respective weights

PO Visitation

Telephone: Male Online: Male

Difference

Telephone: Female Online: Female

Difference Counts

491 645 521 671

Past day

5% 7%

+2 pts

7% 7%

0 pts Past week

21% 20%

  • 1 pt

21% 19%

  • 2 pts

Past month

18% 25%

+7 pts

22% 23%

+1 pts Past 3 months

13% 16%

+3 pts

17% 18%

+1 pt Past 6 months

10% 10%

0 pts

8% 9%

+1 pts Past year

9% 8%

  • 1 pt

8% 10%

+2 pts More than one year ago

17% 10%

  • 7 pts

13% 10%

  • 3 pts

Never

7% 1%

  • 6 pts

5% 1%

  • 4 pts

Not sure

NA 3%

(3 pts)

NA 4%

(4 pts) Visited, don't recall when

0% NA

(<1 pt)

1% NA

(1 pt) Don't know

0% NA

(<1 pt)

0% NA

(<1 pt) Refused

— NA

(—)

— NA

(—)

Full question text in presentation appendix.

Subgroups

( ): Points missing due to answer option differences across modes.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-33
SLIDE 33

Online Purchasing:

Online Panel when allowed to fall out naturally in the sample vs. Telephone Omnibus1

Past Month vs. Not Past Month Household eCommerce Purchasing2 June 2016 Online Panel: Not Lock Sampled March 2017 Telephone Omnibus3

Sample Size 1,207 1,012

Past Month 81% 56% Not Past Month 19% 44%

1: Telephone results weighted per telephone omnibus contractor’s standard weighting procedures. Online survey quota sampled to U.S. Bureau of the Census population estimates on age, gender, nativity within ethnicity, race, and geographic subregion. Following cleaning to remove speeders and straightliners, online data were weighted to on all quota variables, as well as on education and employment status. Before reporting, data were also weighted on Past Month eCommerce purchasing, but the data shown here shows the “natural fall out” of the sample before that additional weight was applied. 2:Full question text for both modes provided in presentation appendix. Data netted as shown when implemented as a quota target/weight in OIG projects —“Past Month” net includes all responses within past month, while “Not Past Month” net includes all other data points, including don’t know/refused. 3: Since 2017, OIG has run this question annually at same time of year in a probability random digit dialed telephone omnibussurvey by an industry-recognized provider. These annually refreshed results are used to weight/provide quota targets on subsequent projects.

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-34
SLIDE 34

When, if ever, was the last time that you or someone in your household placed an order online for a physical product that was to be delivered to your home, or to the home of someone else? When answering, please think of the last time you ordered any product online, even if it has not yet been delivered. Was that order placed within the… (READ LIST)? 01 Past day 02 Past week 03 Past month 04 Past 3 months 05 Past 6 months 06 Past year 07 More than one year ago 08 Never 97 (DO NOT READ) Ordered online, don’t recall last time 98 (DO NOT READ) Don’t know 99 (DO NOT READ) Refused

eCommerce Participation: Phone Question Text

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-35
SLIDE 35

eCommerce Participation: Online Question Text

As originally asked for June 2016 online survey: When was the last time you or someone in your household ordered a product online and had it delivered to your home?

  • a. Within the past week
  • b. Within the past month but not the past week
  • c. Within the past 3 months but not the past month
  • d. Within the past 6 months but not the past 3 months
  • e. More than 6 months ago
  • f. I have not ordered a product online and had it delivered to my home

Question refined for all subsequent research, more closely matching telephone version of question: When, if ever, was the last time that you or someone in your household placed an order online for a physical product that was to be delivered to your home, or to the home of someone else? When answering, please think of the last time you ordered any product online, even if it has not yet been delivered. Was that order placed within the…

  • a. Past day
  • b. Past week
  • c. Past month
  • d. Past 3 months
  • e. Past 6 months
  • f. Past year
  • g. More than one year ago
  • h. Never

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-36
SLIDE 36

Post Office Box Access: Online Question Text

Note: Online question text does not include some additional contextual information included in the phone question text because that contextual information had been provided through previous questions in the USPS-only online survey.

A Post Office Box, also known as a PO Box or a Postal Box, is a numbered box in a US Post Office that is assigned to a person or organization, where mail for them is kept until collected. Do you currently have access to…? [Scale: Yes, No, Not Sure] a.…your own personal USPS post office box (registered to you) b.…a personal USPS post office box registered to someone else in your household (family member, roommate, spouse, etc.) c.…the personal USPS post office box registered to someone who does not live in your household d.…a USPS post office box registered to your employer e.…a USPS post office box registered to a business that you own/run

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-37
SLIDE 37

Post Office Box Access: Phone Question Text

Note: Telephone question text includes some additional contextual information that had been provided through previous questions in the USPS-only online survey, but which wasn’t available to respondents when the question was asked outside of that context over the phone.

A Post Office Box, also known as a P.O. Box or a Postal Box, is a mail delivery point that is offered by the United States Postal Service, usually for a fee. P.O. Boxes are numbered boxes located inside of U.S. Post Offices. P.O. Boxes provide their owners with an address for their mail that is separate from their home or business addresses. P.O. Boxes are located inside of United States Postal Service Post Offices, and so they are different from the mailbox you would have at your home, and are also different from a box that you could rent from another provider, such as U.P.S. or Fed-Ex. Do you currently have access to (INSERT ITEM)? 1 Yes 2 No 8 (DO NOT READ) Don’t know/Not Sure 9 (DO NOT READ) Refused a. your own personal U.S.P.S. post office box that is registered to you b. a personal U.S.P.S. post office box registered to someone else in your household, such as a family member, roommate, spouse, etc. c. the personal U.S.P.S. post office box registered to someone who does not live in your household d. a U.S.P.S. post office box registered to your employer e. a U.S.P.S. post office box registered to a business that you own or run

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-38
SLIDE 38

Post Office Visitation: Online Question Text

When, if ever, was the last time that you visited a post office and went inside, even if you did not make a purchase or complete any other business while there?

  • a. Past day
  • b. Past week
  • c. Past month
  • d. Past 3 months
  • e. Past 6 months
  • f. Past year
  • g. More than one year ago
  • h. Never
  • i. Not sure

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-39
SLIDE 39

When, if ever, was the last time that you visited a post office and went inside? When answering, please think of the last time you visited a post office and went inside, even if you did not complete any business while you were there. Was your last visit to a post office within the… (READ LIST)? 01 Past day 02 Past week 03 Past month 04 Past 3 months 05 Past 6 months 06 Past year 07 More than one year ago 08 Never 97 (DO NOT READ) Visited, don’t recall when 98 (DO NOT READ) Don’t know 99 (DO NOT READ) Refused

Post Office Visitation: Phone Question Text

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-40
SLIDE 40

Bot/Inauthentic Respondent Removal Question: Closed-Ended Terminate Example

Paying attention and reading the instructions carefully is critical, if you are paying attention please choose Orange below. [RANDOMIZE LIST]

  • a. Red [TERMINATE]
  • b. Orange
  • c. Blue [TERMINATE]
  • d. Silver [TERMINATE]
  • e. Green [TERMINATE]
  • f. Yellow [TERMINATE]

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE

slide-41
SLIDE 41

Bot/Inauthentic Respondent Screening Question: Open-Ended Question at End of Survey

In just a couple of words, what was this survey about? [Open-ended, 3 Character Min, 200 Character Max. Used as an Input for Data Cleaning]

JAKE SOFFRONOFF | LOCK SAMPLING | DC-AAPOR/WSS REVIEW-PREVIEW SUMMER CONFERENCE