Assessing Translations: How Different Checking Procedures Compare Under Field Conditions Gijs van Houten, International Survey Methodologist Danielle Cuddington, Research Analyst Katie Simmons, Associate Director, Research Steve Schwarzer, Research Methodologist
Contents • Approac aches t to q questio ionn nnair aire t translat ation • Resear arch q questions ns Survey m methodo dology • • Tra ransl slation p processe sses • Analy lysi sis • Results ts • Conc nclusions a and nd discus ussion www.pewresearch.org 2 March 24, 2020
Approaches to questionnaire translation • Hig High-quality ty, , equiv ival alent nt translat ation i n is challeng nging ng • Back ck-trans anslat ation u n used t to b be most t com common on a approa oach • In r rec ecen ent y yea ears, mo move t towards comm mmittee a appr pproaches • Translation verification www.pewresearch.org 3 March 24, 2020
Research Questions • Do Does es t the e ver erification pr proces ess lea ead t to a differ erent f final translation t than t the e back- translat ation p n process? • Do Does es t the e final t translation pr provided b by the e ver erification pr process lea ead t to different results, e eit ither in in terms o of data ta q qua uality y or r subst bstantive re result lts? s? www.pewresearch.org 4 March 24, 2020
Global Attitudes Project • Nationall lly representative t telephon hone a and fa face-to-face i intervi views w ws with a adult lts s aged 1 18 and nd o over, c cond nducted annua nnually y sinc nce 2 2002 • Sp Sprin ing 20 2014 4 • 44 countries from March 17 to June 5, 2014, totaling 48,643 respondents • 75 languages and language versions Sp Sprin ing 2 2015 • • 40 countries from March 25 to May 27, 2015, totaling 45,435 respondents • 79 languages and language versions Both e editions condu duct cted d under the d dir irection o of Prin inceton Su Survey Research • Associ ciates Internat national nal • CATI, CA CAPI a and P d PAPI PI www.pewresearch.org 5 March 24, 2020
www.pewresearch.org 6 March 24, 2020
Prior to 2013: Back-translation • Bac ack-tran ansla lation Local vendor provided original translator and back-translator • Pew Research Center reviewed back-translation and provided feedback to • local vendor • Englis lish-cen entric Meaning can be lost in translation • www.pewresearch.org 7 March 24, 2020
Translation Process: 2013 and after • Translation verification with commit ittee approac ach Translator provided by local vendor, verifier provided by separate vendor • Pew Research Center reviews comments of verifier and consults with • translator and verifier to arrive at optimal solution • Verifying g tr trend nd que uest stions ns In 2015, verification was carried out for existing translations of 11 items • Trend translations were verified for 1 to 66 languages per item, resulting • in 313 verified language-item combinations • Verification results were treated conservatively, favoring comparability over time when translation issues were judged not to affect meaning www.pewresearch.org 8 March 24, 2020
Coding of verification outcome • Two indep depen enden ent c coder ders 89% correspondence • Consultation to arrive at agreement on 37 language-item combinations • • Discrepancies mainly related to approach to dealing with interviewer instructions and ‘do not read out’ passages No o issue ssue: no fla lags, s, mino nor sp spelling ng, p pun unctua uation n issu ssues • • 194 language-item combinations • Flag agged, no change: ge: prefer erential changes ges, minor g gramma mmar issues es • 111 language-item combinations • Flag agged, changed: unclear m mean aning, incorrect transla latio ions • 8 language-item combinations www.pewresearch.org 9 March 24, 2020
Analysis • Four t trend items ex examined Today, which ONE of the following do you think is the world's leading economic • power — the United States, China, Japan, OR countries of the EU? • Which comes closest to your view — China will eventually replace the U.S. as the world's leading superpower, China has already replaced the U.S. as the world's leading superpower, or China will never replace the U.S. as the world's leading superpower? • Do you think the government of China respects the personal freedoms of its people or don't you think so? • Do you think the government of the United States respects the personal freedoms of its people or don't you think so? All f ll four i r items w s were re field lded for 6 r 66 la languages s or r la language v versi sions ( s (N=264) • www.pewresearch.org 10 March 24, 2020
Changes in item non-response, by verification outcome Flagge gged, no no Flagge gged, No i issu ssue change ge change ge 2014 201 015 2014 201 015 2014 201 015 World’s leading 9.8% 6.7% 12.0% 8.1% 11.6% 13.1% economic power (U.S.) World’s leading 16.7% 14.2% 25.2% 18.0% 16.5% 14.3% superpower (U.S.) China respects personal 22.0% 16.7% 20.3% 15.9% 3.8% 3.4% freedoms U.S. respects personal 16.6% 11.8% 18.8% 13.3% 0% 0% freedoms www.pewresearch.org 11 March 24, 2020
Changes in correlations with related items, by verification outcome Flagge gged, no no Flagge gged, No i issu ssue change ge change ge 2014 201 015 2014 201 015 2014 201 015 U.S. favorability -.210 -.204 -.268 -.298 -.197 -.154 World’s leading economic power (U.S.) Confidence in Obama -.191 -.159 -.239 -.288 -.194 -.209 U.S. favorability -.169 -.161 -.068 -.095 .049 .085 World’s leading superpower (U.S.) Confidence in Obama -.157 -.123 -.054 -.082 .055 .170 China respects .383 a -.247 a China favorability .306 .315 .322 .330 personal freedoms .050 a -.054 a .310 .323 .355 .338 U.S. favorability U.S. respects personal freedoms .101 a -.148 a .336 .301 .411 .379 Confidence in Obama a N < 100 www.pewresearch.org 12 March 24, 2020
Conclusion and discussion • The e ver erification pr process i is time me inten ensive b but c can still b be e carried o out within t the e tight t time meframe of the pr e project • Verif ification revealed a a smal all number o of s serio ious is issues w wit ith e exis isting t tran anslatio ions Outcomes of t the v e ver erification pr proces ess d did n not h have t the e e expe pected i impa pact on i item em • no non-re resp sponse se and constr truct v t validity ty • In s n some c cases, th the unc uncovered issue ues m might h have r referred t to w wrong ng tr trans nslations but ut no not ne t necessarily “ y “bad que uestions” • Correlat atio ions ns i impac acted b by global al e event nts • Nature re o of t the i items ( s (e.g .g. . nominal, d , dichotomous) s) li limited possi sibi bili lity f for a r assessi sing imp mpact o t on distributi tions www.pewresearch.org 13 March 24, 2020
Questions or suggestions? Gijs van Houten Danielle Cuddington International Survey Methodologist Research Analyst GvanHouten@PewResearch.org DCuddington@PewResearch.org www.pewproject.org 14 March 24, 2020
Recommend
More recommend
