Assessing Translations: How Different Checking Procedures Compare - - PowerPoint PPT Presentation

▶
assessing translations how different checking procedures
SMART_READER_LITE
LIVE PREVIEW

Assessing Translations: How Different Checking Procedures Compare - - PowerPoint PPT Presentation

Nov 04, 2022 370 likes •533 views

Assessing Translations: How Different Checking Procedures Compare Under Field Conditions Gijs van Houten, International Survey Methodologist Danielle Cuddington, Research Analyst Katie Simmons, Associate Director, Research Steve Schwarzer,

slide-1
SLIDE 1

Assessing Translations: How Different Checking Procedures Compare Under Field Conditions

Gijs van Houten, International Survey Methodologist Danielle Cuddington, Research Analyst Katie Simmons, Associate Director, Research Steve Schwarzer, Research Methodologist

slide-2
SLIDE 2

March 24, 2020

2 www.pewresearch.org

Contents

  • Approac

aches t to q questio ionn nnair aire t translat ation

  • Resear

arch q questions ns

  • Survey m

methodo dology

  • Tra

ransl slation p processe sses

  • Analy

lysi sis

  • Results

ts

  • Conc

nclusions a and nd discus ussion

slide-3
SLIDE 3

March 24, 2020

3 www.pewresearch.org

Approaches to questionnaire translation

  • Hig

High-quality ty, , equiv ival alent nt translat ation i n is challeng nging ng

  • Back

ck-trans anslat ation u n used t to b be most t com common

  • n a

approa

  • ach
  • In r

rec ecen ent y yea ears, mo move t towards comm mmittee a appr pproaches

  • Translation verification
slide-4
SLIDE 4

March 24, 2020

4 www.pewresearch.org

Research Questions

  • Do

Does es t the e ver erification pr proces ess lea ead t to a differ erent f final translation t than t the e back- translat ation p n process?

  • Do

Does es t the e final t translation pr provided b by the e ver erification pr process lea ead t to different results, e eit ither in in terms o

  • f data

ta q qua uality y or r subst bstantive re result lts? s?

slide-5
SLIDE 5

March 24, 2020

5 www.pewresearch.org

Global Attitudes Project

  • Nationall

lly representative t telephon hone a and fa face-to-face i intervi views w ws with a adult lts s aged 1 18 and nd o

  • ver, c

cond nducted annua nnually y sinc nce 2 2002

  • Sp

Sprin ing 20 2014 4

  • 44 countries from March 17 to June 5, 2014, totaling 48,643 respondents
  • 75 languages and language versions
  • Sp

Sprin ing 2 2015

  • 40 countries from March 25 to May 27, 2015, totaling 45,435 respondents
  • 79 languages and language versions
  • Both e

editions condu duct cted d under the d dir irection o

  • f Prin

inceton Su Survey Research Associ ciates Internat national nal

  • CATI, CA

CAPI a and P d PAPI PI

slide-6
SLIDE 6

March 24, 2020

6 www.pewresearch.org

slide-7
SLIDE 7

March 24, 2020

7 www.pewresearch.org

Prior to 2013: Back-translation

  • Bac

ack-tran ansla lation

  • Local vendor provided original translator and back-translator
  • Pew Research Center reviewed back-translation and provided feedback to

local vendor

  • Englis

lish-cen entric

  • Meaning can be lost in translation
slide-8
SLIDE 8

March 24, 2020

8 www.pewresearch.org

Translation Process: 2013 and after

  • Translation verification with commit

ittee approac ach

  • Translator provided by local vendor, verifier provided by separate vendor
  • Pew Research Center reviews comments of verifier and consults with

translator and verifier to arrive at optimal solution

  • Verifying

g tr trend nd que uest stions ns

  • In 2015, verification was carried out for existing translations of 11 items
  • Trend translations were verified for 1 to 66 languages per item, resulting

in 313 verified language-item combinations

  • Verification results were treated conservatively, favoring comparability
  • ver time when translation issues were judged not to affect meaning
slide-9
SLIDE 9

March 24, 2020

9 www.pewresearch.org

Coding of verification outcome

  • Two indep

depen enden ent c coder ders

  • 89% correspondence
  • Consultation to arrive at agreement on 37 language-item combinations
  • Discrepancies mainly related to approach to dealing with interviewer

instructions and ‘do not read out’ passages

  • No
  • issue

ssue: no fla lags, s, mino nor sp spelling ng, p pun unctua uation n issu ssues

  • 194 language-item combinations
  • Flag

agged, no change: ge: prefer erential changes ges, minor g gramma mmar issues es

  • 111 language-item combinations
  • Flag

agged, changed: unclear m mean aning, incorrect transla latio ions

  • 8 language-item combinations
slide-10
SLIDE 10

March 24, 2020

10 www.pewresearch.org

Analysis

  • Four t

trend items ex examined

  • Today, which ONE of the following do you think is the world's leading economic

power — the United States, China, Japan, OR countries of the EU?

  • Which comes closest to your view — China will eventually replace the U.S. as the

world's leading superpower, China has already replaced the U.S. as the world's leading superpower, or China will never replace the U.S. as the world's leading superpower?

  • Do you think the government of China respects the personal freedoms of its people
  • r don't you think so?
  • Do you think the government of the United States respects the personal freedoms
  • f its people or don't you think so?
  • All f

ll four i r items w s were re field lded for 6 r 66 la languages s or r la language v versi sions ( s (N=264)

slide-11
SLIDE 11

March 24, 2020

11 www.pewresearch.org

Changes in item non-response, by verification outcome

No i issu ssue Flagge gged, no no change ge Flagge gged, change ge 2014 201 015 2014 201 015 2014 201 015 World’s leading economic power (U.S.) 9.8% 6.7% 12.0% 8.1% 11.6% 13.1% World’s leading superpower (U.S.) 16.7% 14.2% 25.2% 18.0% 16.5% 14.3% China respects personal freedoms 22.0% 16.7% 20.3% 15.9% 3.8% 3.4% U.S. respects personal freedoms 16.6% 11.8% 18.8% 13.3% 0% 0%

slide-12
SLIDE 12

March 24, 2020

12 www.pewresearch.org

Changes in correlations with related items, by verification outcome

No i issu ssue Flagge gged, no no change ge Flagge gged, change ge 2014 201 015 2014 201 015 2014 201 015 World’s leading economic power (U.S.) U.S. favorability

  • .210
  • .204
  • .268
  • .298
  • .197
  • .154

Confidence in Obama

  • .191
  • .159
  • .239
  • .288
  • .194
  • .209

World’s leading superpower (U.S.) U.S. favorability

  • .169
  • .161
  • .068
  • .095

.049 .085 Confidence in Obama

  • .157
  • .123
  • .054
  • .082

.055 .170 China respects personal freedoms China favorability .306 .315 .322 .330 .383a

  • .247a

U.S. respects personal freedoms U.S. favorability .310 .323 .355 .338 .050a

  • .054a

Confidence in Obama .336 .301 .411 .379 .101a

  • .148a

a N < 100

slide-13
SLIDE 13

March 24, 2020

13 www.pewresearch.org

Conclusion and discussion

  • The

e ver erification pr process i is time me inten ensive b but c can still b be e carried o

  • ut within t

the e tight t time meframe of the pr e project

  • Verif

ification revealed a a smal all number o

  • f s

serio ious is issues w wit ith e exis isting t tran anslatio ions

  • Outcomes of t

the v e ver erification pr proces ess d did n not h have t the e e expe pected i impa pact on i item em no non-re resp sponse se and constr truct v t validity ty

  • In s

n some c cases, th the unc uncovered issue ues m might h have r referred t to w wrong ng tr trans nslations but ut no not ne t necessarily “ y “bad que uestions”

  • Correlat

atio ions ns i impac acted b by global al e event nts

  • Nature

re o

  • f t

the i items ( s (e.g .g. . nominal, d , dichotomous) s) li limited possi sibi bili lity f for a r assessi sing imp mpact o t on distributi tions

slide-14
SLIDE 14

March 24, 2020

www.pewproject.org 14

Questions or suggestions?

Gijs van Houten

International Survey Methodologist GvanHouten@PewResearch.org

Danielle Cuddington

Research Analyst DCuddington@PewResearch.org