V0D 14 Nov 2018 Why 25% of Voter Polls Are Wrong V0D 2018 CTC 1 V0D 2018 CTC 2 Why 25% of Voter Polls 25% of Voter Polls are Wrong are Wrong! Polls are short surveys. by • Less demographic information. Milo Schield, Augsburg University • Fewer choices for answers. Fellow: American Statistical Association Polls typically involve sampling: sampling error. US Rep: International Statistical Literacy Project Voter polls are of wide-spread interest. December 2, 2018 25%* of voter polls in the final 21 days are wrong . Twin Cities Critical Thinking Club (CTC) Wrong: actual result outside 95% confidence interval Slides: www.StatLit.org/pdf/2018-Schield-CTC-Slides.pdf In 2016, 29%** of those in final 21 days were wrong. * https://fivethirtyeight.com/features/how-the-fivethirtyeight-senate-forecast-model-works/ ** https://fivethirtyeight.com/features/the-polls-are-all-right/ V0D 2018 CTC 3 V0D 2018 CTC 4 Random Sampling: Voter Polls: Non-Response Bias Before or After Surveys and polls often interview people by phone. For a given election, voting polls are either Plus: Cheap; random sample. • before the election (opinion polls) or Minus: Non-response bias. • after the election (exit polls). People who respond are not representative of the • Opinion polls forecast : use hypothetical questions entire population. • Exit polls explain : use factual questions The lower the response rate, Explanatory polls just tabulate – like sports statistics the greater the risk of bias. Forecast polls use models – like weather forecasts http://www.pewresearch.org/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/pm-05-15-2017_rddnonresponse-00-14/ V0D 2018 CTC 5 V0D 2018 CTC 6 Chance of Winning: Error in Forecast Polls: “It depends” Sampling and Forecast Election (forecast) polls are harder to interpret! Forecast polls have sampling error (theoretical): • Difference b/t sample and population statistic Consider two candidates in different contests • Entirely due to random sampling Both have 52% of their respective votes. • Decreases as sample size increases • Candidate #1 has 60% chance of winning. Forecast polls also have modeling error (empirical) • Candidate #2 has a 75% chance of winning. • Difference between the forecast and actual result • Due to different models of undecided Q. How can they have different chances of winning? • Independent of sample size A. It depends on the size of the polling error ! www.StatLit.org/pdf/2018-Schield-CTC-Slides.pdf Page 1
V0D 14 Nov 2018 Why 25% of Voter Polls Are Wrong V0D 2018 CTC 7 V0D 2018 CTC 8 Average Opinion Polling Error Opinion Polling Error 1972-2016 Bigger than Sampling Error . Average popular-vote polling error* in final 21 days: • 4 points: Presidential • 5.4 points: US Senate and Governor • 6.2 points: US House • 8.9 points: Primary * D-R margin error. https://fivethirtyeight.com/features/the-polls-are-all-right/ 95% sampling error by sample size: • 3.3 points: n = 900 • 3.0 points: n = 1,024 [Most common sample size] • 2.8 points: n = 1,225 V0D 2018 CTC 9 V0D 2018 CTC 10 Low Response Rates: Causes of Error Non-Response Bias ??? in Opinion Polls . Causes of error unique to opinion (forecast) polls: 1. Respondents undecided about whether to vote 2. Respondents undecided about who to vote for 3. Respondents change their mind before voting 4. Voter opinion polls must forecast • which undecided will vote • who undecided voters will vote for • which third-party voters will change their mind 5. Different polls allocate undecided differently 6. Polls allocate using demographic/historical data V0D 2018 CTC 11 V0D 2018 CTC 12 Main Cause of Polling Error: Conclusions Undecided Undecided, Third-Party & Other: last three Presidential elections To interpret polls, know the difference: 2016 ….. 1. between an opinion poll, an exit poll and a survey 2. between polling error and sampling error * Biggest peak (16%) To evaluate polls, know that * Slowest to decide 1. the chance of winning depends on polling error * Hardest to model 2. polling error is bigger than sampling error * Biggest problem 3. polling error exists because opinion polls are forecasts for forecasters 4. the size of polling error depends on the contest Drew Linzer tweet (11/5/2016) Q. How can pollsters educate readers on polling error? Permission granted. www.hot1045.net/decodedc/yes-undecided-voters-still-exist www.StatLit.org/pdf/2018-Schield-CTC-Slides.pdf Page 2
V0D 2018 CTC 1 Why 25% of Voter Polls are Wrong by Milo Schield, Augsburg University Fellow: American Statistical Association US Rep: International Statistical Literacy Project December 2, 2018 Twin Cities Critical Thinking Club (CTC) Slides: www.StatLit.org/pdf/2018-Schield-CTC-Slides.pdf
V0D 2018 CTC 2 25% of Voter Polls are Wrong! Polls are short surveys. • Less demographic information. • Fewer choices for answers. Polls typically involve sampling: sampling error. Voter polls are of wide-spread interest. 25%* of voter polls in the final 21 days are wrong . Wrong: actual result outside 95% confidence interval In 2016, 29%** of those in final 21 days were wrong. * https://fivethirtyeight.com/features/how-the-fivethirtyeight-senate-forecast-model-works/ ** https://fivethirtyeight.com/features/the-polls-are-all-right/
V0D 2018 CTC 3 Random Sampling: Non-Response Bias Surveys and polls often interview people by phone. Plus: Cheap; random sample. Minus: Non-response bias. People who respond are not representative of the entire population. The lower the response rate, the greater the risk of bias. http://www.pewresearch.org/2017/05/15/what-low-response-rates-mean-for-telephone-surveys/pm-05-15-2017_rddnonresponse-00-14/
V0D 2018 CTC 4 Voter Polls: Before or After For a given election, voting polls are either • before the election (opinion polls) or • after the election (exit polls). • Opinion polls forecast : use hypothetical questions • Exit polls explain : use factual questions Explanatory polls just tabulate – like sports statistics Forecast polls use models – like weather forecasts
V0D 2018 CTC 5 Chance of Winning: “It depends” Election (forecast) polls are harder to interpret! Consider two candidates in different contests Both have 52% of their respective votes. • Candidate #1 has 60% chance of winning. • Candidate #2 has a 75% chance of winning. Q. How can they have different chances of winning? A. It depends on the size of the polling error !
V0D 2018 CTC 6 Error in Forecast Polls: Sampling and Forecast Forecast polls have sampling error (theoretical): • Difference b/t sample and population statistic • Entirely due to random sampling • Decreases as sample size increases Forecast polls also have modeling error (empirical) • Difference between the forecast and actual result • Due to different models of undecided • Independent of sample size
V0D 2018 CTC 7 Average Opinion Polling Error 1972-2016 .
V0D 2018 CTC 8 Opinion Polling Error Bigger than Sampling Error Average popular-vote polling error* in final 21 days: • 4 points: Presidential • 5.4 points: US Senate and Governor • 6.2 points: US House • 8.9 points: Primary * D-R margin error. https://fivethirtyeight.com/features/the-polls-are-all-right/ 95% sampling error by sample size: • 3.3 points: n = 900 • 3.0 points: n = 1,024 [Most common sample size] • 2.8 points: n = 1,225
V0D 2018 CTC 9 Low Response Rates: Non-Response Bias ??? .
V0D 2018 CTC 10 Causes of Error in Opinion Polls Causes of error unique to opinion (forecast) polls: 1. Respondents undecided about whether to vote 2. Respondents undecided about who to vote for 3. Respondents change their mind before voting 4. Voter opinion polls must forecast • which undecided will vote • who undecided voters will vote for • which third-party voters will change their mind 5. Different polls allocate undecided differently 6. Polls allocate using demographic/historical data
V0D 2018 CTC 11 Main Cause of Polling Error: Undecided Undecided, Third-Party & Other: last three Presidential elections 2016 ….. * Biggest peak (16%) * Slowest to decide * Hardest to model * Biggest problem for forecasters Drew Linzer tweet (11/5/2016) Permission granted. www.hot1045.net/decodedc/yes-undecided-voters-still-exist
V0D 2018 CTC 12 Conclusions To interpret polls, know the difference: 1. between an opinion poll, an exit poll and a survey 2. between polling error and sampling error To evaluate polls, know that 1. the chance of winning depends on polling error 2. polling error is bigger than sampling error 3. polling error exists because opinion polls are forecasts 4. the size of polling error depends on the contest Q. How can pollsters educate readers on polling error?
Recommend
More recommend
