Interview Methodology for use with Spanish-Speaking Respondents - - PowerPoint PPT Presentation

Adaptation of Standard Cognitive Interview Methodology for use with Spanish-Speaking Respondents

Patricia Goerman and Ryan King, U.S. Census Bureau Presented at the 69th annual conference of the American Association for Public Opinion Research (AAPOR) Anaheim, CA: May 15-18, 2014

Outline of talk

• Cognitive interviewing (CI)
• Past research
• Research questions
• Study methods
• Findings
• Next steps

Cognitive interviewing (CI)

Definition:

• One-on-one interviews to evaluate whether

respondents interpret, comprehend and respond to survey questions as intended. Think aloud procedure Probes (concurrent or retrospective)

Cognitive interview probes

Meaning oriented

• “What does the term ‘foster child’ mean to you in this

question?” Process oriented

• “How did you arrive at/choose that answer?”

Paraphrasing

• “Can you tell me in your own words what that question

is asking?” Recall

• “How do you remember that you (saw a dentist 3

times) in the last year?”

Past research

• Monolingual cognitive interview (CI) methods:
• Presser et. al. 2004; Willis 2005; Beatty and Willis

2007; Blair and Conrad 2011

• Adaptation of CI Method for use across

languages/cultures:

• Dean et al. 2007; Willis and Zahnd 2007; Fitzgerald and

Miller 2009; Pan et al. 2010; Harkness et al 2010

Research questions

• Why have previous studies found that CI

techniques cause discomfort among non- English respondents?

• What specific interview techniques and

probes work best with Spanish-speaking CI respondents?

Methods

• 48 cognitive interviews
• Spanish speakers
• Segment of American Community Survey (ACS):

CATI/CAPI

• Types of question tested:
• Person level: age, sex, DOB, marital, Hispanic
• rigin/race, place of birth, year of entry, education,

ancestry, language spoken, English proficiency

• Household level: Type unit, year built, when moved,

number of acres, number rooms and bedrooms, plumbing, telephone, vehicles

Demographic characteristics of Spanish speakers (n=48)

National origin Gender 44 Mexican origin 42 Female 4 Some other origin 6 Male Education level Age 35 Less than HS Range: 24-74 13 HS diploma or more Mean: 46 Median: 44 Missing: 4

Type of interviews: Standard v. experimental

• Variation of introduction and probe wording
• Standard interviews
• Direct translation of typical U.S. English protocol

(procedure and wording)

• Experimental interviews
• Variation of introductory statements/conversation
• Different sample probes to begin
• Flexibility in probe wording

Pre-interview interactions: Both standard and experimental

• Discussions of the interview process prior to

beginning

• Reason for doing the interview
• Confidentiality
• Reasons for taping the interview

Pre-interview interactions: Experimental v. standard

Experimental N=28 Standard N=20 Information provided about Int. 36% 10% Family Background 36% 5% How learned Spanish 36% 5% Time spent abroad 11% 5% Discuss Job at Census 36% 0% Small talk between interviewer and respondent about: The respondent 71% 40% Interview location 61% 60% Census Bureau / surveys 39% 10% Interview city 18% 0%

Coding scheme/analysis

Yes/no codes: 1. Was probe administered? 2. Was probe administered as worded in protocol? 3. Did respondent understand when read as worded? 4. Was probe reworded? 5. Was probe understood when reworded? 6. Did probe cause discomfort? 7. Did respondent provide “useful” answer to probe? Descriptive codes: 1. Type probe rewording done? (description) 2. What type of discomfort did it cause? (description)

Inter-coder reliability

• Application of Kappa statistic
• Kappa Scores
• Total of 7 cases coded by both interviewer/coders
• Overall: 0.68 (n=892)
• Yes/No: 0.71 (n=784) (good)
• Type probe rewording: 0.50 (n=69) (fair-good)
• Discomfort: 0.38 (n=39) (poor)

Frequency of rewording by probe type and interview type

77% 86% 61% 90% 76% 21% 30% 41% 85% 26%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Meaning Oriented E=132, S=94 Paraphrasing E=92, S=94 Process Oriented E=44, S=61 Think Aloud E=48, S=47 Recall E=17, S=19 Experimental Standard

Usefulness of probes by interview type

77% 75% 91% 42% 100% 74% 70% 72% 43% 89%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Meaning Oriented E=132, S=94 Paraphrasing E=92, S=94 Process Oriented E=44, S=61 Think Aloud E=48, S=47 Recall E=17, S=19 Experimental Standard

Overall Discomfort

15% 53% 16% 6% 0% 18% 8% 17% 21% 31% 12% 2% 13% 21%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Total E=333 S=315 Pause E=27 S=11 Embarass E=8 S=16 Annoying E=3 S=6 Emotional E=0 S=1 Laughing E=9 S=7 Other E=4 S=11 Experimental Standard

Overall usefulness of probes by education level and treatment

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Meaning Oriented LTHS-E=99, LTHS-S=68, MTHS-E=33, MTHS-S=26 Paraphrasing LTHS-E=69, LTHS-S=64, MTHS-E=23, MTHS-S=30 Process Oriented LTHS-E=33, LTHS-S=40, MTHS-E=11, MTHS-S=21 Think Aloud LTHS-E=33, LTHS-S=32, MTHS-E=15, MTHS-S=15 Recall LTHS-E=14, LTHS-S=14, MTHS-E=3, MTHS-S=5

Less Than High School - Experimental Less Than High School - Standard More Than High School - Experimental More Than High School - Standard

Frequency of “successful” major probe rewording

82% 79% 91% 50% 100% 80% 100% 59% 35% 100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Meaning Oriented E=95, S=20 Paraphrasing E=14, S=2 Process Oriented E=22, S=17 Think Aloud E=36, S=31 Recall E=12, S=3 Experimental Standard

Probe rewording: Think aloud

• Original/scripted probes:
• ¿Podría pensar en voz alta cuando está

decidiendo cómo contestar la pregunta?

• Por favor dígame qué está pensando.
• Rewording:
• Y si quisiera hablar mientras piensa, me

gustaría.

• Me encantaría oír lo que está pensando

mientras mira…

Probe rewording: Paraphrase

• Original/scripted probe:
• ¿Podría decirme con sus propias palabras qué

información pide esa pregunta?

• Rewording:
• ¿Y qué información le esta pidiendo esta

pregunta? ¿Qué le parece?

• Si ud tuviera que preguntar eso a alguien,

cómo lo diría?

• ¿Qué le parece que quieren preguntar (o

saber)?

Next steps

• Additional analysis:
• Look at emergent probes in experimental cases
• Look at debriefing, people’s experience with

reading, literacy, education, etc.

• New research:
• Different national origin Spanish speakers
• Speakers of additional languages

Adaptation of Standard Cognitive Interview Methodology for use with Spanish-Speaking Respondents

Patricia Goerman and Ryan King, U.S. Census Bureau For more information: E-mail: Patricia.L.Goerman@census.gov

Disclaimer: This presentation is intended to inform people about research and to encourage discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau.