Cognitive Testing of Survey Translations: Does Respondent - - PowerPoint PPT Presentation

Cognitive Testing of Survey Translations: Does Respondent Translations: Does Respondent Language Proficiency Matter?

Patricia Goerman and Mikelyn Meyers, U.S. Census Bureau Hyunjoo Park and Mandy Sha RTI International Hyunjoo Park and Mandy Sha, RTI International Alisu Schoua Glusberg, RSS Presented at the 70th annual conference of the American Presented at the 70 annual conference of the American Association for Public Opinion Research (AAPOR) Hollywood, FL: May 14-17, 2015

1

Overview of the Issue Overview of the Issue

• Rule of thumb: survey translations should be

tested with monolingual respondents g p

• Assumptions:

M li l th i t d d

• Monolinguals are the intended users
• Bilinguals maybe more likely to understand “bad”
• r overly literal translations:
• For example:
• group home (hogar de grupo)
• home schooling (enseñanza en el hogar)

2

Why Does it Matter? Why Does it Matter?

• More costly and time consuming to restrict

testing to only monolingual respondents g y g p

• Difficulties with recruiting/interviewing

monolinguals monolinguals

• Distrust/lack of understanding of purpose
• Correlation with lower income status

3

Review of the Literature Review of the Literature

• S

P t ti ith li l bili l littl

• Survey Pretesting with monolinguals v. bilinguals -very little

empirical research

(Park, et al. 2014)

l l ff l h

• Monolinguals: Difficulty with cognitive interview process

(Park, et al. 2013; Pan et al. 2010; Goerman, 2006)

• Differences between monolinguals and bilinguals in cognitive

g g g processes

(Bialystok, 2007; Hernandez and Bates 2001; Marian et al. 2009)

• Monolingual status may be a correlate of other variables, e.g.,

Monolingual status may be a correlate of other variables, e.g., educational attainment

(Ridolfo and Schoua Glusberg, 2011)

• Important to include bilinguals in testing to account for differences
• Important to include bilinguals in testing to account for differences

in acculturation and other demographic characteristics

(Willis and Zahnd 2007)

4

Definitions Definitions

• Cognitive testing: One-on-one interviews to

evaluate whether respondents interpret, comprehend and respond to survey questions as intended.

• Usability testing: One-on-one interview to

study whether online questionnaire can be y q answered effectively, efficiently and with satisfaction by target respondents (Wang, satisfaction by target respondents (Wang, 2015)

5

2014 Census Test Spanish Internet Cognitive/Usability Testing

O i i f d

• Origin of present study
• Quick turnaround project: Test Spanish-language

Census 2014 test instrument with 10 Spanish speakers

• Recruitment goals
• Different national origins

g

• Internet experience
• Monolingual or Spanish-dominant, bilingual

g p , g respondents to best test the translation

6

Preliminary Findings/ Impressions Preliminary Findings/ Impressions

• Correlation between bilingual ability, education

level and internet/computer experience

• Accidental recruitment:
• Non-computer literate monolingual

Non computer literate, monolingual

• Computer savvy, fluent English speakers

I f d i i i h h d

• Issues of mode interacting with other respondent
• characteristics. How does language proficiency fit

in with other considerations?

7

Design of Empirical Research Design of Empirical Research

• Decennial Census testing across modes and

languages, large ongoing contract g g , g g g

• Double the number of respondents so that

half are monolingual half bilingual half are monolingual, half bilingual

• Spanish testing:
• Paper, internet, interviewer administered form
• Chinese Korean Vietnamese Russian Arabic
• Chinese, Korean, Vietnamese, Russian, Arabic
• Fillable forms: internet/paper, self administered mode

8

Research Questions for Overall Project

• Do monolingual and bilingual respondents

help to “uncover” same number and types of p yp issues?

• Are there differences by mode?
• Are there differences by mode?
• Are there differences by language?

9

Early Results: Spanish, Interviewer- Administered Instrument

• Respondents asked about speaking, reading

ability, dominant language in screening y, g g g

• 39 Spanish-speaking respondents

19 li l

• 19 monolingual
• 20 bilingual
• Coding of interview summaries (3 coders)
• Whether probe administered
• Whether probe administered
• Whether understood concept as intended

10

Respondent Characteristics

Demographic Characteristics Monolinguals Bilinguals Demographic Characteristics Monolinguals Bilinguals Gender Male 9 8 Female 10 12 18-34 6 9 35-44 3 6 Age 45-54 8 3 44-64 1 2 65+ 1 65+ 1 Education* Less than HS 10 HS grad/GED 7 16 College + 1 4 Less than 5 7 4 Years in US** 6-15 years 10 7 16+ 2 6

*Education level was missing for 1 monolingual respondent Education level was missing for 1 monolingual respondent **Years in US was missing for 3 bilingual respondents

11

Results: Concepts Monolinguals p g Misunderstood More Frequently

Concept Monolinguals Bilinguals % Misunderstood N % Misunderstood N Foster child 94% 18 53% 19 Military assignment 60% 10 25% 12 Afroamericano 47% 15 19% 16 Afroamericano 47% 15 19% 16 Own with mortgage 41% 17 15% 20 Live/stay somewhere else 21% 14 0% 15 y Group home 87% 15 80% 10 Owner/renter name 40% 15 36% 11 % Misunderstood is calculated as : # who misunderstood a # who were asked probe (N)

12

Concepts Bilinguals p g Misunderstood More Frequently

Monolinguals Bilinguals Concept % Misunderstood N % Misunderstood N Housemate/roommate 43% 14 73% 15 O f d l 0% 18 20% 20 Own free and clear 0% 18 20% 20 Seasonal home 50% 14 55% 11 Tenure question 16% 19 21% 19 Tenure question 16% 19 21% 19 Confidentiality 17% 6 21% 14 Indigena de las americas 50% 14 54% 13

13

Understanding of “Foster Child” by English-Speaking Ability

100% 37% 70% 80% 90% 90% 50% 60% 70% Speak English well / very well 63% 20% 30% 40% very well Speak English not well / not at all 10% 0% 10% Understood N = 10* Misunderstood N = 27 Understood, N = 10 Misunderstood, N = 27

*Data regarding understanding of this concept were unavailable for 2 respondents out of 39 NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno”

14

Summary of Findings Summary of Findings

ili l h f h bl

• Bilinguals caught most of the same problems as

monolinguals but often less frequently

• Depending on the size of a bilingual sample,

researchers may or may not see an issue

• One term was more problematic for bilinguals

than monolinguals (Free and clear). Testing with

• nly monolinguals may have masked this issue
• Including only bilinguals in interviewer

g y g administered (CAPI) testing may work as a cost saving measure but some issues maybe lost g y

15

Next Steps Next Steps

C l l i d li bili d d

• Calculate intercoder reliability and recode as necessary
• More analysis of nuances in the types of problems

id tifi d identified

• Analysis of additional modes in Spanish:

I (d ll d)

• Internet (data collected)
• Paper questionnaires (pending funding)

E i ti f li l bili l d t i

• Examination of monolingual v. bilingual respondents in

5 additional languages (pending funding)

• Add analysis of English speakers for comparison
• Add analysis of English speakers for comparison
• Look at “levels” of bilingualism

16

Cognitive Testing of Survey Translations: Does Respondent Language Proficiency Matter? Language Proficiency Matter?

Patricia Goerman and Mikelyn Meyers, U.S. Census Bureau H j P k d M d Sh RTI I t ti l Hyunjoo Park and Mandy Sha, RTI International Alisu Schoua Glusberg, RSS

For more information: E-mail: Patricia.L.Goerman@census.gov

Disclaimer: This presentation is intended to inform people about research and to encourage p f p p g

• discussion. The views expressed are those of the authors and not necessarily those
• f the U.S. Census Bureau.

17

Understanding of “Foster Child” by English-Speaking Ability

30% 30% 7% 80% 100% 26% 60% Speak English very well Speak English well 37% 60% 20% 40% Speak English not well Speak English not at all 10% 0% 0% Understood, N = 10 Misunderstood, N = 27

*Data regarding understanding of this concept were unavailable for 2 respondents out of 39 NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno”

18