Cognitive Testing of Survey Translations: Does Respondent Translations: Does Respondent Language Proficiency Matter? Patricia Goerman and Mikelyn Meyers, U.S. Census Bureau Hyunjoo Park and Mandy Sha RTI International Hyunjoo Park and Mandy Sha, RTI International Alisu Schoua Glusberg, RSS Presented at the 70 th annual conference of the American Presented at the 70 annual conference of the American Association for Public Opinion Research (AAPOR) Hollywood, FL: May 14-17, 2015 1
Overview of the Issue Overview of the Issue Rule of thumb: survey translations should be tested with monolingual respondents g p Assumptions: Monolinguals are the intended users M li l th i t d d Bilinguals maybe more likely to understand “bad” or overly literal translations: For example: group home (hogar de grupo) home schooling (enseñanza en el hogar) 2
Why Does it Matter? Why Does it Matter? More costly and time consuming to restrict testing to only monolingual respondents g y g p Difficulties with recruiting/interviewing monolinguals monolinguals Distrust/lack of understanding of purpose Correlation with lower income status 3
Review of the Literature Review of the Literature S Survey Pretesting with monolinguals v. bilinguals -very little P t ti ith li l bili l littl empirical research (Park, et al. 2014) Monolinguals: Difficulty with cognitive interview process l l ff l h ( Park, et al. 2013; Pan et al. 2010; Goerman, 2006 ) Differences between monolinguals and bilinguals in cognitive g g g processes (Bialystok, 2007; Hernandez and Bates 2001; Marian et al. 2009) Monolingual status may be a correlate of other variables, e.g., Monolingual status may be a correlate of other variables, e.g., educational attainment (Ridolfo and Schoua Glusberg, 2011) Important to include bilinguals in testing to account for differences Important to include bilinguals in testing to account for differences in acculturation and other demographic characteristics (Willis and Zahnd 2007) 4
Definitions Definitions Cognitive testing: One-on-one interviews to evaluate whether respondents interpret, comprehend and respond to survey questions as intended. Usability testing: One-on-one interview to study whether online questionnaire can be y q answered effectively, efficiently and with satisfaction by target respondents (Wang, satisfaction by target respondents (Wang, 2015) 5
2014 Census Test Spanish Internet Cognitive/Usability Testing Origin of present study O i i f d Quick turnaround project: Test Spanish-language Census 2014 test instrument with 10 Spanish speakers Recruitment goals Different national origins g Internet experience Monolingual or Spanish-dominant, bilingual g p , g respondents to best test the translation 6
Preliminary Findings/ Impressions Preliminary Findings/ Impressions Correlation between bilingual ability, education level and internet/computer experience Accidental recruitment: Non-computer literate monolingual Non computer literate, monolingual Computer savvy, fluent English speakers Issues of mode interacting with other respondent I f d i i i h h d characteristics. How does language proficiency fit in with other considerations? 7
Design of Empirical Research Design of Empirical Research Decennial Census testing across modes and languages, large ongoing contract g g , g g g Double the number of respondents so that half are monolingual half bilingual half are monolingual, half bilingual Spanish testing: Paper, internet, interviewer administered form Chinese Korean Vietnamese Russian Arabic Chinese, Korean, Vietnamese, Russian, Arabic Fillable forms: internet/paper, self administered mode 8
Research Questions for Overall Project Do monolingual and bilingual respondents help to “uncover” same number and types of p yp issues? Are there differences by mode? Are there differences by mode? Are there differences by language? 9
Early Results: Spanish, Interviewer- Administered Instrument Respondents asked about speaking, reading ability, dominant language in screening y, g g g 39 Spanish-speaking respondents 19 monolingual 19 li l 20 bilingual Coding of interview summaries (3 coders) Whether probe administered Whether probe administered Whether understood concept as intended 10
Respondent Characteristics Demographic Characteristics Demographic Characteristics Monolinguals Monolinguals Bilinguals Bilinguals Male 9 8 Gender Female 10 12 18-34 6 9 35-44 3 6 Age 45-54 8 3 44-64 1 2 65+ 65+ 1 1 0 0 Less than HS 10 0 HS grad/GED 7 16 Education* College + 1 4 Less than 5 7 4 Years in US** 6-15 years 10 7 16+ 2 6 *Education level was missing for 1 monolingual respondent Education level was missing for 1 monolingual respondent **Years in US was missing for 3 bilingual respondents 11
Results: Concepts Monolinguals p g Misunderstood More Frequently Monolinguals Bilinguals Concept % Misunderstood N % Misunderstood N Foster child 94% 18 53% 19 Military assignment 60% 10 25% 12 Afroamericano Afroamericano 47% 47% 15 15 19% 19% 16 16 Own with mortgage 41% 17 15% 20 Live/stay somewhere else y 21% 14 0% 15 Group home 87% 15 80% 10 Owner/renter name 40% 15 36% 11 % Misunderstood is calculated as : # who misunderstood a # who were asked probe (N) 12
Concepts Bilinguals p g Misunderstood More Frequently Monolinguals Bilinguals Concept % Misunderstood N % Misunderstood N Housemate/roommate 43% 14 73% 15 O Own free and clear f d l 0% 0% 18 18 20% 20% 20 20 Seasonal home 50% 14 55% 11 Tenure question Tenure question 16% 16% 19 19 21% 21% 19 19 Confidentiality 17% 6 21% 14 Indigena de las americas 50% 14 54% 13 13
Understanding of “Foster Child” by English-Speaking Ability 100% 90% 37% 80% 70% 70% 60% 90% Speak English well / 50% very well very well 40% Speak English not well / 63% 30% not at all 20% 10% 10% 0% Understood N = 10* Misunderstood N = 27 Understood, N = 10 Misunderstood, N = 27 *Data regarding understanding of this concept were unavailable for 2 respondents out of 39 NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno” 14
Summary of Findings Summary of Findings Bilinguals caught most of the same problems as ili l h f h bl monolinguals but often less frequently Depending on the size of a bilingual sample, researchers may or may not see an issue One term was more problematic for bilinguals than monolinguals (Free and clear). Testing with only monolinguals may have masked this issue Including only bilinguals in interviewer g y g administered (CAPI) testing may work as a cost saving measure but some issues maybe lost g y 15
Next Steps Next Steps Calculate intercoder reliability and recode as necessary C l l i d li bili d d More analysis of nuances in the types of problems id identified tifi d Analysis of additional modes in Spanish: Internet (data collected) I (d ll d) Paper questionnaires (pending funding) Examination of monolingual v. bilingual respondents in E i ti f li l bili l d t i 5 additional languages (pending funding) Add analysis of English speakers for comparison Add analysis of English speakers for comparison Look at “levels” of bilingualism 16
Cognitive Testing of Survey Translations: Does Respondent Language Proficiency Matter? Language Proficiency Matter? Patricia Goerman and Mikelyn Meyers, U.S. Census Bureau Hyunjoo Park and Mandy Sha, RTI International H j P k d M d Sh RTI I t ti l Alisu Schoua Glusberg, RSS For more information: E-mail: Patricia.L.Goerman@census.gov Disclaimer: This presentation is intended to inform people about research and to encourage p f p p g discussion. The views expressed are those of the authors and not necessarily those of the U.S. Census Bureau. 17
Understanding of “Foster Child” by English-Speaking Ability 100% 7% 30% 80% 30% Speak English very well 60% Speak English well 26% 40% 60% Speak English not well Speak English not at all 20% 37% 0% 10% 0% Understood, N = 10 Misunderstood, N = 27 *Data regarding understanding of this concept were unavailable for 2 respondents out of 39 NOTE: “Foster child” was translated as “Hijo(a) de crianza del programa Foster del gobierno” 18
Recommend
More recommend
