Evaluating Gender Bias in Machine Translation Gabriel Stanovsky, - - PowerPoint PPT Presentation

▶
evaluating gender bias in machine translation
SMART_READER_LITE
LIVE PREVIEW

Evaluating Gender Bias in Machine Translation Gabriel Stanovsky, - - PowerPoint PPT Presentation

Jan 22, 2023 102 likes •473 views

Evaluating Gender Bias in Machine Translation Gabriel Stanovsky, Noah Smith and Luke Zettlemoyer ACL 2019 Grammatical Gender Some languages encode grammatical gender (Spanish, Italian, Russian, ) doctor maestro doctora maestra

slide-1
SLIDE 1

Evaluating Gender Bias in Machine Translation

Gabriel Stanovsky, Noah Smith and Luke Zettlemoyer ACL 2019

slide-2
SLIDE 2
  • Some languages encode grammatical gender (Spanish, Italian, Russian, …)

Grammatical Gender

doctor doctora doctor maestra maestro

slide-3
SLIDE 3
  • Some languages encode grammatical gender (Spanish, Italian, Russian, …)
  • Other languages do not (English, Turkish, Basque, Finnish, …)

Grammatical Gender

doctor doctora maestra maestro doctor teacher

slide-4
SLIDE 4

Translating Gender

  • Variations in gender mechanisms prohibit one-to-one translations

The doctor asked the nurse to help her in the procedure. La doctora le pidió a la enfermera que le ayudara con el procedimiento.

slide-5
SLIDE 5

Is MT gender biased?

slide-6
SLIDE 6

Is MT gender biased?

slide-7
SLIDE 7

Is MT gender biased?

slide-8
SLIDE 8

Is MT gender biased?

slide-9
SLIDE 9

1. Can we quantitatively evaluate gender translation in MT?

Research Questions

slide-10
SLIDE 10

1. Can we quantitatively evaluate gender translation in MT? 2. How much does MT rely on gender stereotypes vs. meaningful context?

Research Questions

slide-11
SLIDE 11

1. Can we quantitatively evaluate gender translation in MT? 2. How much does MT rely on gender stereotypes vs. meaningful context?

3.

Can we reduce gender bias by rephrasing source texts?

Research Questions

slide-12
SLIDE 12

Research Questions

1. Can we quantitatively evaluate gender translation in MT? 2. How much does MT rely on gender stereotypes vs. meaningful context?

3.

Can we reduce gender bias by rephrasing source texts?

slide-13
SLIDE 13

English Source Texts

  • Winogender (Rudinger et al., 2018) & WinoBias (Zhao et al., 2018)

○ 3888 English sentences designed to test gender bias in coreference resolution ○ Following the Winograd schema

The doctor asked the nurse to help her in the procedure. The doctor asked the nurse to help him in the procedure.

slide-14
SLIDE 14

English Source Texts

  • Winogender (Rudinger et al., 2018) & WinoBias (Zhao et al., 2018)

○ 3888 English sentences designed to test gender bias in coreference resolution ○ Following the Winograd schema

  • Observation: These are very useful for evaluating gender bias in MT!

The doctor asked the nurse to help her in the procedure. The doctor asked the nurse to help him in the procedure.

slide-15
SLIDE 15

English Source Texts

  • Winogender (Rudinger et al., 2018) & WinoBias (Zhao et al., 2018)

○ 3888 English sentences designed to test gender bias in coreference resolution ○ Following the Winograd schema

  • Observation: These are very useful for evaluating gender bias in MT!

○ Equally split between stereotypical and non-stereotypical role assignments ○ Gold annotations for gender ○

The doctor asked the nurse to help her in the procedure. The doctor asked the nurse to help him in the procedure.

slide-16
SLIDE 16

Methodology: Automatic evaluation of gender bias

Input: MT model + target language Output: Accuracy score for gender translation

slide-17
SLIDE 17

Methodology: Automatic evaluation of gender bias

1. Translate the coreference bias datasets

○ To target languages with grammatical gender Input: MT model + target language Output: Accuracy score for gender translation The doctor asked the nurse to help her in the procedure.

slide-18
SLIDE 18

Methodology: Automatic evaluation of gender bias

1. Translate the coreference bias datasets

○ To target languages with grammatical gender Input: MT model + target language Output: Accuracy score for gender translation The doctor asked the nurse to help her in the procedure. La doctora le pidió a la enfermera que le ayudara con el procedimiento.

slide-19
SLIDE 19

Methodology: Automatic evaluation of gender bias

1. Translate the coreference bias datasets

○ To target languages with grammatical gender

2. Align between source and target

○ Using fast align (Dyer et al., 2013) Input: MT model + target language Output: Accuracy score for gender translation The doctor asked the nurse to help her in the procedure. La doctora le pidió a la enfermera que le ayudara con el procedimiento.

slide-20
SLIDE 20

Methodology: Automatic evaluation of gender bias

1. Translate the coreference bias datasets

○ To target languages with grammatical gender

2. Align between source and target

○ Using fast align (Dyer et al., 2013)

3. Identify gender in target language

○ Using off-the-shelf morphological analyzers or simple heuristics in the target languages Input: MT model + target language Output: Accuracy score for gender translation The doctor asked the nurse to help her in the procedure. La doctora le pidió a la enfermera que le ayudara con el procedimiento.

slide-21
SLIDE 21

Methodology: Automatic evaluation of gender bias

1. Translate the coreference bias datasets

○ To target languages with grammatical gender

2. Align between source and target

○ Using fast align (Dyer et al., 2013)

3. Identify gender in target language

○ Using off-the-shelf morphological analyzers or simple heuristics in the target languages Quality estimated at > 85% vs. 90% IAA Doesn’t require reference translations! Input: MT model + target language Output: Accuracy score for gender translation The doctor asked the nurse to help her in the procedure. La doctora le pidió a la enfermera que le ayudara con el procedimiento.

slide-22
SLIDE 22

Research Questions

1. How well does machine translation handle gender? 2. How much does MT rely on gender stereotypes vs. meaningful context?

3.

Can we reduce gender bias by rephrasing source texts?

slide-23
SLIDE 23

Results

Google Translate

Acc (%) Language male doctors & female nurses

slide-24
SLIDE 24

Results

Google Translate

Acc (%) Language male nurses & female doctors

slide-25
SLIDE 25

Results

Acc (%) Language

Google Translate

Gender bias gap

slide-26
SLIDE 26

Results

  • MT struggles with non-stereotypical roles across languages and systems
  • Often doing significantly worse than random coin-flip
  • Academic models (Ott et al., 2018; Edunov et al., 2018) exhibit similar behavior
slide-27
SLIDE 27

Examples

slide-28
SLIDE 28

Research Questions

1. How well does machine translation handle gender? 2. How much does MT rely on gender stereotypes vs. meaningful context?

3.

Can we reduce gender bias by rephrasing source texts?

slide-29
SLIDE 29

Do Gendered Adjectives Affect Translation?

  • Black-box injection of gendered adjectives (similar to Moryossef et al., 2019)

○ the pretty doctor asked the nurse to help her in the operation

○

the handsome nurse asked the doctor to help him in the operation

slide-30
SLIDE 30

Do Gendered Adjectives Affect Translation?

  • Black-box injection of gendered adjectives (similar to Moryossef et al., 2019)

○ the pretty doctor asked the nurse to help her in the operation

○

the handsome nurse asked the doctor to help him in the operation

  • Improved performance for most tested languages and models [mean +8.6%]

○ + 10% on Spanish and Russian

slide-31
SLIDE 31

Do Gendered Adjectives Affect Translation?

  • Black-box injection of gendered adjectives (similar to Moryossef et al., 2019)

○ the pretty doctor asked the nurse to help her in the operation

○

the handsome nurse asked the doctor to help him in the operation

  • Improved performance for most tested languages and models [mean +8.6%]

○ + 10% on Spanish and Russian

  • Requires oracle coreference resolution!

○ Attests to the relation between coreference resolution and MT

slide-32
SLIDE 32

Limitations & Future Work

  • Artificially-created dataset

○ Allows for controlled experiment ○ Yet, might introduce its own annotation biases

  • Medium-size

○ Easy to overfit - not good for training

slide-33
SLIDE 33

Limitations & Future Work

  • Artificially-created dataset

○ Allows for controlled experiment ○ Yet, might introduce its own annotation biases

  • Medium-size

○ Easy to overfit - not good for training

  • Future work

○ Collect naturally occurring samples on a large scale

slide-34
SLIDE 34

Conclusion

  • First quantitative automatic evaluation of gender bias in MT

○ 6 SOTA MT models on 8 diverse target languages ○ Doesn’t require reference translations

  • Significant gender bias found in all models in all tested languages
  • Code and data: https://github.com/gabrielStanovsky/mt_gender

○ Easily extensible with more languages and MT models

slide-35
SLIDE 35

Conclusion

  • First quantitative automatic evaluation of gender bias in MT

○ 6 SOTA MT models on 8 diverse target languages ○ Doesn’t require reference translations

  • Significant gender bias found in all models in all tested languages
  • Code and data: https://github.com/gabrielStanovsky/mt_gender

○ Easily extensible with more languages and MT models

Thanks for listening!

¡Gracias por su atención! Merci pour l'écoute! Grazie per aver ascoltato! Спасибо за внимание! Спасибі за слухання! !הבשקהה לע הדות !تﺎﺻﻧﻹا ﻰﻠﻋ ارﻛﺷ Danke fürs Zuhören! Come to the the Gender Bias Workshop! (Friday)