How Does Questionnaire Design Affect Party ID? Kyley McGeeney - PowerPoint PPT Presentation

How Does Questionnaire Design Affect Party ID? Kyley McGeeney Senior Director of Survey Methods Jennifer Miller-Gonzalez, Haley Tran PRIVILEGED AND CONFIDENTIAL 1

Background What we’re looking at and why

Background Introduction • Lots of research on order effects and context effects • Researchers hypothesize placement of party ID (PID) affects results – Some swear by asking up front, others at the end – Literature shows mixed results Slide 3

Background Research Questions • How does asking PID in beginning vs. at the end affect results? • Is this effect moderated at all by survey topic? Slide 4

Methods How we conducted this research

Methods Experimental Design • Used one wave of monthly online omnibus • Fielded two separate questionnaires – political, corporate • Split sampled each questionnaire – PID asked up front vs end • Weighted each split sample separately to GP targets • Looked at weighted leaned party (party + party lean) Corporate Survey Political Survey ½ PID at beginning ½ PID at beginning ½ PID at end ½ PID at end Slide 6

Methods Party ID Questions PARTY Generally , which party’s candidates or policies do you tend to align with the most? /* RANDOM ROTATE CHOICES */ 1. Democratic 2. Republican 3. Independent 4. Other /* SPECIFY */ /* DO NOT ROTATE */ I don’t align with any party /* DO NOT ROTATE */ 5. PARTY_LEAN (IF C3-5): Do you tend to…? /* RANDOM ROTATE CHOICES */ 1. Lean Democrat 2. Lean Republican 3. Other /* DO NOT ROTATE */ 4. Neither /* DO NOT ROTATE */ Slide 7

Methods Data Collection • General population survey of US adults 18+ • Nonprobability web panel • Quotas for age x gender, region, education, race/ethnicity • Field dates: May 1-4, 2017 • Political survey n=1002 • Corporate survey n=2004 Slide 8

Results What we found

Results Topic and Position Affect Party ID • Corporate higher % Ind. than Political; Beginning higher % Rep. than End • Corporate: higher Rep. at start, higher Rep lean at End; Independent > Political Ind. Party ID by Topic, Position, and Topic x Position A B C D E F G H 100% 90% 27% 26% 28% 29% 29% 31% 31% 32% *F *D 80% 70% 8% 7% 6% *E 6% 6% 5% 5% 7% 60% 17% 17% 21% 22% 23% 19% 20% 18% *G,H 50% *B 10% 9% 40% 7% 9% 7% 8% 7% 8% * F 30% 20% 40% 38% 37% 37% 37% 36% 36% 34% 10% 0% Corporate Political Beginning End Beginning End Beginning End Topic Position Corporate Political Slide 10 n=2004 n=1002 n=1512 n=1494 n=1014 n=990 n=498 n=504 Democrat Lean Democrat No Lean Lean Republican Republican

Conclusion What we learned and how to use it

Conclusion Summary • Topic matters – Corporate survey will likely have fewer partisans than a political survey – This could be break off or context effect • Position matters – Asking PID at the beginning results in more Reps than asking at the end – Asking PID at the end leads to a less partisan sample for Corporate than Political • What to do with this? – PSB typically asks PID at the beginning to get a clean read – Others don’t want to prime people to respond to survey through a partisan lens Slide 12

Conclusion Limitations • Nonprobability sample • Only tested two topics, perhaps others would yield different results • Political survey likely underpowered Slide 13

Thank you! kmcgeeney@ps-b.com

Results Topic and Position Affect Party ID • Corporate higher % Independent than Political, especially at End • Political higher % Dem/Dem lean at the End Party ID by Topic, Position, and Topic x Position A B C D E F G H 100% 90% 80% 43% 43% 44% 44% 45% 45% 47% 49% * F 70% 60% 50% 40% 34% 35% 34% 36% 37% 38% 36% 34% 30% 20% 23% 22% 10% 21% *B 19% 20% *G,H 18% 17% 17% 0% Corporate Political Beginning End Beginning End Beginning End Topic Position Corporate Political n=2004 n=1002 n=1512 n=1494 n=1014 n=990 n=498 n=504 Independent/No Lean Republican/Lean Republican Democrat/Lean Democrat Slide 15