How Does Questionnaire Design Affect Party ID? Kyley McGeeney - - PowerPoint PPT Presentation

1 PRIVILEGED AND CONFIDENTIAL

How Does Questionnaire Design Affect Party ID?

Kyley McGeeney Senior Director of Survey Methods Jennifer Miller-Gonzalez, Haley Tran

Background

What we’re looking at and why

Introduction

Background

• Lots of research on order effects and context effects
• Researchers hypothesize placement of party ID (PID) affects results

– Some swear by asking up front, others at the end – Literature shows mixed results

Slide 3

Research Questions

Background

• How does asking PID in beginning vs. at the end affect results?
• Is this effect moderated at all by survey topic?

Slide 4

Methods

How we conducted this research

Experimental Design

Methods

• Used one wave of monthly online omnibus
• Fielded two separate questionnaires – political, corporate
• Split sampled each questionnaire – PID asked up front vs end
• Weighted each split sample separately to GP targets
• Looked at weighted leaned party (party + party lean)

Slide 6

Corporate Survey Political Survey ½ PID at beginning ½ PID at beginning ½ PID at end ½ PID at end

Party ID Questions

Methods

PARTY Generally, which party’s candidates or policies do you tend to align with the most?

/* RANDOM ROTATE CHOICES */

1. Democratic 2. Republican 3. Independent 4. Other /* SPECIFY */ /* DO NOT ROTATE */ 5. I don’t align with any party /* DO NOT ROTATE */

PARTY_LEAN (IF C3-5): Do you tend to…? /* RANDOM ROTATE CHOICES */

1. Lean Democrat 2. Lean Republican 3. Other /* DO NOT ROTATE */ 4. Neither /* DO NOT ROTATE */

Slide 7

Data Collection

Methods

• General population survey of US adults 18+
• Nonprobability web panel
• Quotas for age x gender, region, education, race/ethnicity
• Field dates: May 1-4, 2017
• Political survey n=1002
• Corporate survey n=2004

Slide 8

Results

What we found

Topic and Position Affect Party ID

Results

• Corporate higher % Ind. than Political; Beginning higher % Rep. than End
• Corporate: higher Rep. at start, higher Rep lean at End; Independent > Political Ind.

Slide 10

* F 36% 38% 37% 36% 37% 34% 37% 40% 7% 9% 7% 9% 7% 8% 8% 10% 22% 17% 19% 21% 20% 23% 18% 17% 6% 6% 5% 7% 5% 8% 7% 6% 29% 29% 31% 27% 32% 26% 31% 28% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Corporate Political Beginning End Beginning End Beginning End Topic Position Corporate Political

Party ID by Topic, Position, and Topic x Position

Democrat Lean Democrat No Lean Lean Republican Republican

n=2004 n=1002 n=1512 n=1494 n=1014 n=990 n=498 n=504

A B C D E F G H

*B *G,H *E *F *D

Conclusion

What we learned and how to use it

Summary

Conclusion

• Topic matters

– Corporate survey will likely have fewer partisans than a political survey – This could be break off or context effect

• Position matters

– Asking PID at the beginning results in more Reps than asking at the end – Asking PID at the end leads to a less partisan sample for Corporate than Political

• What to do with this?

– PSB typically asks PID at the beginning to get a clean read – Others don’t want to prime people to respond to survey through a partisan lens

Slide 12

Limitations

Conclusion

• Nonprobability sample
• Only tested two topics, perhaps others would yield different results
• Political survey likely underpowered

Slide 13

Thank you!

kmcgeeney@ps-b.com

Topic and Position Affect Party ID

Results

• Corporate higher % Independent than Political, especially at End
• Political higher % Dem/Dem lean at the End

Slide 15 A B C D E F G H

22% 17% 19% 21% 20% 23% 18% 17% 35% 36% 37% 34% 36% 34% 38% 34% 43% 47% 44% 45% 44% 43% 45% 49% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Corporate Political Beginning End Beginning End Beginning End Topic Position Corporate Political

Party ID by Topic, Position, and Topic x Position

Independent/No Lean Republican/Lean Republican Democrat/Lean Democrat

*G,H

n=2004 n=1002 n=1512 n=1494 n=1014 n=990 n=498 n=504

*B * F