How Do Weighting Targets Affect Pre- Election Poll Results? Kyley - - PowerPoint PPT Presentation

▶
how do weighting targets affect pre
SMART_READER_LITE
LIVE PREVIEW

How Do Weighting Targets Affect Pre- Election Poll Results? Kyley - - PowerPoint PPT Presentation

Sep 19, 2023 43 likes •288 views

How Do Weighting Targets Affect Pre- Election Poll Results? Kyley McGeeney Senior Director of Survey Methods Haley Tran PRIVILEGED AND CONFIDENTIAL 1 Background What were looking at and why Background Introduction Often need to

slide-1
SLIDE 1

1 PRIVILEGED AND CONFIDENTIAL

How Do Weighting Targets Affect Pre- Election Poll Results?

Kyley McGeeney Senior Director of Survey Methods Haley Tran

slide-2
SLIDE 2

Background

What we’re looking at and why

slide-3
SLIDE 3

Introduction

Background

  • Often need to interview likely voters (LVs)

– Pre-election polls – Non-election years for public affairs clients

  • How do we do it?

– Publicly-released phone surveys: interview gen pop  filter for analysis – Nonprobability web surveys: screen out respondents up front

  • If you screen on likely voters, what targets do you use to weight?

Slide 3

slide-4
SLIDE 4

Research Question

Background

  • If you screen on likely voters, what targets do you use to weight?

– American Community Survey (ACS)? – Current Population Survey (CPS) Voting and Registration Supplement? – Voterfile (VF)? – Exit Polls?

Slide 4

slide-5
SLIDE 5

Literature

Background

  • Exit Poll is biased, Voterfile and CPS pretty similar

– 2004 Exit Poll electorate younger, more minorities than VF, CPS (McDonald 2005) – Exit Poll: tendency to severely underrep. older, white voter w/o college degree

(Cohn 2017)

– Exit Poll 2012: younger, more educated, diverse than 2012 CPS (Cohn 2016) – Exit Poll 2014: more educated, younger than 2014 CPS (Pew 2016)

  • How does this bias affects weighted horserace estimates?

Slide 5

slide-6
SLIDE 6

Methods

How we conducted this research

slide-7
SLIDE 7

Weighting Design

Methods

  • Reweighted Nov 2016 pre-election poll using various targets

– ACS (for Gen Pop and filtered on RV/LV) – CPS Voting and Registration Supplement (RV, LV) – Voterfile (RV, LV) – Exit Poll (LV)

  • Weighted to targets that would have been available Nov 2016
  • Weighted to targets that later became available

Slide 7

Note: Data used include: ACS 2015 1-year estimate, CPS 2012 and 2016, Catalist Voterfile in 2012 and 2016, Exit Poll 2012 and 2016

slide-8
SLIDE 8

Targets Available in November 2016

Methods

  • 2012 CPS Voting and Registration Supplement

– RVs: who was registered in 2012 – LVs: who voted in 2012

  • Nov 2016 Voterfile data

– RVs: who was registered to vote – LVs: Voted in 2012 or registered since 2012

  • 2012 Exit Poll data

– LVs: who voted in 2012

Slide 8

slide-9
SLIDE 9

Targets Available After November 2016

Methods

  • 2016 CPS Voting and Registration Supplement

– RVs: who was registered in 2016 – LVs: who voted in 2016

  • 2017 Voterfile data

– RVs: who was registered to vote – LVs: Voted in 2016

  • 2016 Exit Poll data

– LVs: who voted in 2016

Slide 9

slide-10
SLIDE 10

Weighting Variables

Methods

  • ACS, CPS, VF

– Age x Gender – Gender x Education – Age x Education – Region – Race/Ethnicity x Education

Slide 10

  • Exit Poll

– Age – Gender – Race/Ethnicity x Education

(not available 2012)

– Race/Ethnicity – Education

slide-11
SLIDE 11

Data Collection

Methods

  • General population survey
  • Nonprobability web panel
  • Quotas for age x gender, region, education, race/ethnicity
  • Field dates: November 1-4, 2016
  • Total n = 803, RV = 734, LV = 702
  • Likely voter screen was single likelihood to vote question

Slide 11

slide-12
SLIDE 12

Analysis

Methods

  • Calculated poll error for each set of weights
  • Poll error = poll margin – actual margin

– E.g. (poll % Clinton - % Trump) – (actual % Clinton - % Trump)

  • Actual margin = 2.1

Slide 12

slide-13
SLIDE 13

Results

What we found

slide-14
SLIDE 14

ACS General Population Weights

Results

  • Weight total sample to ACS gen pop then filter to RV/LV
  • Did not work very well here and error increased with filtering
  • Selection bias and LV screen can play a part too

Slide 14 6.1 7.9 9.6 2 4 6 8 10 12 GP ACS RV ACS LV ACS

Poll Error Using the ACS Gen Pop Weights

(poll margin - actual margin)

slide-15
SLIDE 15

Targets Available in November 2016

Results

  • Most accurate

– Weighting RVs to CPS or voterfile RV targets

  • Least accurate

– Gen pop weights – Exit Poll

Slide 15

6.1 7.9 9.6 0.4

  • 3.2

0.5 1.6 6.7

  • 4
  • 2

2 4 6 8 10 GP RV LV RV LV RV LV LV ACS CPS VF Exit Poll

Poll Error Using Targets Available in Nov 2016

(poll margin - actual margin)

slide-16
SLIDE 16

Targets Available After November 2016

Results

  • Most accurate

– Weighting LVs to voterfile 2016 voter targets

  • Least accurate

– Gen pop weights – Exit Poll

Slide 16

6.1 7.9 9.6

  • 4
  • 2.8

1.7 0.4 6.4

  • 2

2 4 6 8 10 GP RV LV RV LV RV LV LV ACS CPS VF Exit Poll

Poll Error Using Targets Available After Nov 2016

(poll margin - actual margin)

slide-17
SLIDE 17

Conclusion

What we learned and how to use it

slide-18
SLIDE 18

Limitations

Conclusion

  • Nonprobability sample
  • LV screen = 1 question
  • Voterfile LV definition might be defined differently by others

Slide 18

slide-19
SLIDE 19

Summary

Conclusion

  • Do we need to interview gen pop sample and then filter? No

– Appears to be okay to screen out non-registered or non-likely voters and weight

  • What targets should we use in non-election years for likely voters?

– Voterfile targets for 2016 voters works well

  • What targets leading up to an election?

– CPS or voterfile RV targets

Slide 19

slide-20
SLIDE 20

Thank you!

kmcgeeney@ps-b.com

slide-21
SLIDE 21

Appendix

slide-22
SLIDE 22

References

Appendix

  • Cohn, N. (2018). Trump Losing College-Educated Whites? He Never Won Them in the First Place.

Retrieved from New York Times website: https://www.nytimes.com/2018/02/27/upshot/trump-losing- college-educated-whites-he-never-won-them-in-the-first-place.html

  • Cohn, N. (2016). There Are More White Voters Than People Think. That’s Good News for Trump. Retrieved from New

York Times website: https://www.nytimes.com/2016/06/10/upshot/there-are-more-white-voters-than- people-think-thats-good-news-for-trump.html?_r=0

  • Keeter, S. and R. Igielnik. (2016). Can Likely Voter Models Be Improved? Retrieved from Pew Research Center

webite: http://www.pewresearch.org/2016/01/07/comparing-the-results-of-different-likely-voter- models/

  • McDonald, M. (2005). The True Electorate: A Cross-Validation of Voter Registration Files and Election

Survey Demographics. Public Opinion Quarterly, 71(4), 588-602.

Slide 22

slide-23
SLIDE 23

Survey Questions

Appendix

  • Likely voters:

– How likely are you to vote in the upcoming presidential election on November 8, 2016?

1) Definitely will vote [LIKELY VOTER] 2) Probably will vote [LIKELY VOTER] 3) Might or might not vote 4) Probably will not vote 5) Definitely will not vote

  • Registered voters:

– Are you…?

1) Currently registered to vote 2) Not yet registered to vote 3) Don’t know Slide 23

  • Voter Behavior

– In the 2016 general election for President, do you plan to vote for ….. ?

1) Democrat Hillary Clinton 2) Republican Donald Trump 3) Libertarian Gary Johnson 4) I’m not sure about this

slide-24
SLIDE 24

Weight variables for each target

Appendix

  • 2015 ACS 1-year General Population

– Age x Gender – Gender x Education – Age x Education – Region – Race/Ethnicity x Education

  • 2012, 2016 CPS Registered/Likely Voter

– Age x Gender – Gender x Education – Age x Education – Region – Race/Ethnicity x Education

Slide 24

  • 2016 Catalist Voterfile/Pre-election Registered/Likely Voter

– Age x Gender – Gender x Education – Age x Education – Region – Race/Ethnicity x Education

  • 2016 Exit Poll Likely Voter

– Age – Gender – Race/Ethnicity x Education – Race/Ethnicity – Education

  • 2012 Exit Poll Likely Voter

– Age – Gender – Race/Ethnicity – Education