Automatic Estimation of Simultaneous Interpreter Performance Craig - - PowerPoint PPT Presentation

▶
automatic estimation of simultaneous interpreter
SMART_READER_LITE
LIVE PREVIEW

Automatic Estimation of Simultaneous Interpreter Performance Craig - - PowerPoint PPT Presentation

Nov 07, 2023 137 likes •484 views

Automatic Estimation of Simultaneous Interpreter Performance Craig Stewart 1 , Nikolai Vogler 1 , Junjie Hu 1 , Jordan Boyd-Graber 2 , Graham Neubig 1 1 Language Technologies Institute, Carnegie Mellon University 2 CS, iSchool, UMIACS, LSC,

slide-1
SLIDE 1

Automatic Estimation of Simultaneous Interpreter Performance

Craig Stewart1, Nikolai Vogler1, Junjie Hu1, Jordan Boyd-Graber2, Graham Neubig1

1Language Technologies Institute, Carnegie Mellon University 2CS, iSchool, UMIACS, LSC, University of Maryland

1

slide-2
SLIDE 2

Simultaneous Interpretation (SI)

Translation of the spoken word in real time

2

slide-3
SLIDE 3

Computer Assisted Interpretation (CAI)

3

slide-4
SLIDE 4

Computer Assisted Interpretation (CAI)

4

How do we ensure maximum utility with minimum distraction?

slide-5
SLIDE 5

Estimating Interpreter Performance

5

Don’t offer help when they don’t need it! Estimate how well the interpreter is doing

slide-6
SLIDE 6

Quality Estimation

6

We already do this in Machine Translation! Can we apply it to Simultaneous Interpretation?

slide-7
SLIDE 7

QuEst++ is an existing framework for QE (Specia et al., 2015)

7

Quality Estimation

slide-8
SLIDE 8

QuEst++ is an existing framework for QE (Specia et al., 2015)

8

Quality Estimation

Source/Target Sentence Pairs

slide-9
SLIDE 9

QuEst++ is an existing framework for QE (Specia et al., 2015)

9

Quality Estimation

Source/Target Sentence Pairs Extract Features

slide-10
SLIDE 10

QuEst++ is an existing framework for QE (Specia et al., 2015)

10

Quality Estimation

Source/Target Sentence Pairs Extract Features Reference Translations

slide-11
SLIDE 11

QuEst++ is an existing framework for QE (Specia et al., 2015)

11

Quality Estimation

Source/Target Sentence Pairs Extract Features Reference Translations Apply Evaluation Metric

slide-12
SLIDE 12

QuEst++ is an existing framework for QE (Specia et al., 2015)

12

Quality Estimation

Source/Target Sentence Pairs Extract Features Reference Translations Apply Evaluation Metric Learn Model

slide-13
SLIDE 13

QuEst++ is an existing framework for QE (Specia et al., 2015)

13

Quality Estimation

Source/Target Sentence Pairs Extract Features Reference Translations Apply Evaluation Metric Learn Model Score Predictions

slide-14
SLIDE 14

Method

14

slide-15
SLIDE 15

Method

15

Source/Target Sentence Pairs Extract Features Reference Translations Apply Evaluation Metric Learn Model Score Predictions

slide-16
SLIDE 16

Method

16

Source speaker/ Interpreter transcripts Extract Features Reference Translations Apply Evaluation Metric Learn Model Score Predictions

slide-17
SLIDE 17

Method

17

Source speaker/ Interpreter transcripts English-Japanese (3 interpreters) English-French English-Italian Reference Translations Apply Evaluation Metric Learn Model Score Predictions

slide-18
SLIDE 18

Method

18

Source speaker/ Interpreter transcripts Reference Translations Apply Evaluation Metric Learn Model Score Predictions QuEst++ baseline features

slide-19
SLIDE 19

Method

19

Source speaker/ Interpreter transcripts QuEst++ baseline features Reference Translations Apply Evaluation Metric Learn Model Score Predictions Features tailored to interpretation

slide-20
SLIDE 20

Method

20

Source speaker/ Interpreter transcripts QuEst++ baseline features Reference Translations Apply Evaluation Metric (METEOR) Learn Model Score Predictions Features tailored to interpretation

slide-21
SLIDE 21

Method

21

Source speaker/ Interpreter transcripts QuEst++ baseline features Reference Translations Apply Evaluation Metric (METEOR) Learn Model (Support Vector Regression) Score Predictions Features tailored to interpretation

slide-22
SLIDE 22

Method

22

Source speaker/ Interpreter transcripts QuEst++ baseline features Reference Translations Apply Evaluation Metric (METEOR) Learn Model (Support Vector Regression) Test using 10-fold cross-validation Features tailored to interpretation

slide-23
SLIDE 23

Baseline Features

  • Number of words
  • Average word length
  • Language model probability
  • N-gram frequency
  • Punctuation

23

slide-24
SLIDE 24

Features of interpretation

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound formal

reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?”

24

slide-25
SLIDE 25

Features of interpretation

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound

formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?”

25

slide-26
SLIDE 26

Features of interpretation

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound

formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?”

26

slide-27
SLIDE 27

Features of interpretation

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound

formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?”

27

slide-28
SLIDE 28

Features of interpretation

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound

formal reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) GLOSS: “Ehm the Parliament... after the elections... we’ll gi- will give way to a debate on the ehm on the case for example of President Mugabe instead of putting Brazil on the agenda?”

28

slide-29
SLIDE 29

SI Model Features

  • Pauses/hesitations/incomplete words
  • Non-specific words - is the interpreter avoiding specific terminology?
  • Cognates/loan words - if a word is almost identical in both languages an

interpreter shouldn’t struggle with it (unless it’s a ‘false friend’!)

29

slide-30
SLIDE 30

Results

30

slide-31
SLIDE 31

Results

31

slide-32
SLIDE 32

Analysis

SOURCE: “Will the Parliament grant President Dilma Rousseff, on the very first

  • ccasion after her groundbaking groundbreaking election and for no sound formal

reason, the kind of debate that we usually reserve for people like Mugabe? So, I ask you to remove Brazil from the agenda of the urgencies.” (48 words) INTERP: “Ehm il Parlamento... dopo le elezioni... darem- dar spazio a un dibattito sul ehm sul caso per esempio del presidente Mugabe invece di mettere il Brasile all’ordine del giorno?” (27 words) ACTUAL METEOR: 0.0789 BASELINE PREDICTION: 0.1271 SI MODEL PREDICTION: 0.0660

32

slide-33
SLIDE 33

Future Work

  • Evaluation Metric - finding a metric better aligned with the uniqueness of

strategies in SI

  • Live system integration - streamlining the system to provide instantaneous

feedback

  • ASR - evaluate the model on ASR output
  • Speech model - enhance the model using prosodic speech features

33

slide-34
SLIDE 34

Questions?

34

cas1@cs.cmu.edu github/craigastewart/qe_sim_interp