Neural Reranking Improves Subjective Quality of Machine Translation: - - PowerPoint PPT Presentation

▶
neural reranking improves subjective quality of machine
SMART_READER_LITE
LIVE PREVIEW

Neural Reranking Improves Subjective Quality of Machine Translation: - - PowerPoint PPT Presentation

Jan 19, 2024 503 likes •714 views

Neural Reranking Improves Subjective Quality of Machine Translation Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015 Graham Neubig, Makoto Morishita, Satoshi Nakamura Nara Institute of Science and

slide-1
SLIDE 1

1

Neural Reranking Improves Subjective Quality of Machine Translation

Neural Reranking Improves Subjective Quality of Machine Translation: NAIST at WAT 2015

Graham Neubig, Makoto Morishita, ○Satoshi Nakamura Nara Institute of Science and Technology (NAIST) 2015-10-16

slide-2
SLIDE 2

2

Neural Reranking Improves Subjective Quality of Machine Translation

Statistical Translation Frameworks

Symbolic Models Phrase-based MT [Koehn+ 03] Tree-to-String MT [Liu+ 06] Encoder-Decoder [Sutskever+ 14] Attentional [Bahdanau+ 15]

he has a cold 彼 は 風邪 を 引いている he 彼 は has 引いている a cold 風邪 を he 彼 は has 引いている a cold 風邪 を 彼 は 風邪 he has a cold

PRP VBZ DET NN VP NP S

引いている を

Continuous-space (Neural) Models

he has a cold <s> 彼 彼 は は 風邪 風邪 を

引いて いる

を <s>

引いて いる

he has a cold

g1,...,g4 a1 a2 a3 a4 hi-1 hi ri-1

P(ei|F,e1,...,ei-1)

slide-3
SLIDE 3

3

Neural Reranking Improves Subjective Quality of Machine Translation

Relative Merits/Demerits

  • Symbolic Models

✔ Inner workings well understood ✔ Better at translating low-frequency words

  • Continuous-space Models

✔ Easier to implement ✔ Produce more fluent output ✔ Probabilistic model – can score output of other systems!

slide-4
SLIDE 4

4

Neural Reranking Improves Subjective Quality of Machine Translation

Reranking with Neural MT Models

he has a cold

Input

T2S/ PBMT

N-best w/MT Features

  • 1. 彼は寒さを持っている t=-0.5 l=-5.6 | -6.1
  • 2. 彼は風邪を持っている t=-0.9 l=-5.8 | -6.7
  • 3. 彼は風邪を引いた

t=-1.5 l=-5.3 | -6.8

  • 4. 彼は風邪がある

t=-1.9 l=-5.4 | -7.3

Neural Model

Neural Features

nmt=-5.8 nmt=-5.5 nmt=-3.4 nmt=-5.2

  • 2. 彼は寒さを持っている t=-0.5 l=-5.6 nmt=-5.8 | -10.9
  • 3. 彼は風邪を持っている t=-0.9 l=-5.8 nmt=-5.5 | -11.2
  • 1. 彼は風邪を引いた

t=-1.5 l=-5.3 nmt=-3.4 | -9.2

  • 4. 彼は風邪がある

t=-1.9 l=-5.4 nmt=-5.2 | -12.5

Rescored/Reranked N-best

Reranking

slide-5
SLIDE 5

5

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Know About Reranking?

  • Reranking greatly improves BLEU score, even over

strong baseline systems:

Sutskever+ 2014 Alkhouli+ 2015

en-fr BLEU Base 33.3 Rerank 36.5 de-en BLEU ar-en BLEU Baseline 30.6 26.4 Reranked 32.3 27.0

slide-6
SLIDE 6

6

Neural Reranking Improves Subjective Quality of Machine Translation

What Don't We Know About Reranking?

  • Does reranking improve subjective impressions of

results?

  • What are the qualitative differences before/after

reranking with neural MT models?

slide-7
SLIDE 7

7

Neural Reranking Improves Subjective Quality of Machine Translation

Experiments

slide-8
SLIDE 8

8

Neural Reranking Improves Subjective Quality of Machine Translation

Experimental Setup

  • Data: ASPEC Scientific Abstracts
  • Japanese ↔ English, Chinese
  • Baseline: NAIST WAT2014 Tree-to-String System
  • Strong baseline achieving high scores
  • Implemented using Travatar (http://phontron.com/travatar)
  • Neural MT Model: Attentional model
  • Trained ~500k sent., 256 hidden nodes, 2 model ensemble
  • Use words occurring 3+ times (vocab 50,000~80,000)
  • Trained w/ lamtram (http://github.com/neubig/lamtram)
  • Automatic Evaluation: BLEU, RIBES
  • Manual Evaluation: WAT 2015 HUMAN Score
slide-9
SLIDE 9

9

Neural Reranking Improves Subjective Quality of Machine Translation

Results

en-ja ja-en zh-ja ja-zh 10 20 30 40 50 BLEU en-ja ja-en zh-ja ja-zh 70 75 80 85 90 Base Rerank RIBES +1.6 +2.8 +2.5 +1.5 +1.8 +2.7 +1.4 +1.8

Confirm what we know: Neural reranking helps automatic evaluation.

en-ja ja-en zh-ja ja-zh 10 20 30 40 50 60 70 Base Rerank HUMAN +12.5 +23.7 +10.0 +4.2

Show what we didn't know: Also help manual evaluation.

slide-10
SLIDE 10

10

Neural Reranking Improves Subjective Quality of Machine Translation

What is Getting Better?

  • Perform detailed categorization of the changes in

Japanese-English results:

  • 1. Is the sentence better/worse after ranking?
  • 2. What is the main error corrected: insertion, deletion,

substitution, reordering, or conjugation?

  • 3. What is the detailed subcategory?
slide-11
SLIDE 11

11

Neural Reranking Improves Subjective Quality of Machine Translation

Main Types of Errors Corrected/Caused

Type Improved Degraded % Impr. Reordering 55 9 86% Deletion 20 10 67% Insertion 19 2 90% Substitution 15 11 58% Conjugation 8 1 89% Total 117 33 78%

Overall improvements re-confirmed In particular fixing reordering, insertion, and conjugation errors

slide-12
SLIDE 12

12

Neural Reranking Improves Subjective Quality of Machine Translation

#1 Detailed Improvement Category: Phrasal Reordering (+26, -4)

Source Base Rerank Ref

症例2 においては、直腸がんの肝転移に対する化学療法中に、 発赤、硬結、皮膚潰ようを生じた。 In case 2, reddening, induration, and skin ulcer appeared during chemical therapy for liver metastasis of rectal cancer. In case 2, occurred during chemotherapy for liver metastasis of rectal cancer, flare, induration, skin ulcer. In case 2, the flare, induration, skin ulcer was produced during the chemotherapy for hepatic metastasis of rectal cancer.

slide-13
SLIDE 13

13

Neural Reranking Improves Subjective Quality of Machine Translation

#2 Detailed Improvement Category: Auxiliary Verb Ins./Del. (+15, -0)

Source Base Rerank Ref

これにより得られる支配方程式は壁面乱流のようなせん断乱流に も有用て゚ある。 Governing equation derived by this method is useful for turbulent shear flow like turbulent flow near wall. The governing equation is obtained by this is also useful for such as wall turbulence shear flow. The governing equation obtained by this is also useful for shear flow such as wall turbulence.

slide-14
SLIDE 14

14

Neural Reranking Improves Subjective Quality of Machine Translation

#3 Detailed Improvement Category: Coordinate Structures (+13, -2)

Source Base Rerank Ref

レーザー加工は高密度光束による局所的な加熱とアブレーション により行う。 Laser work is done by local heating and ablation with high density light flux. The laser processing is carried out by local heating by high- density luminous flux and ablation. The laser processing is carried out by local heating and ablation by high-density flux.

slide-15
SLIDE 15

15

Neural Reranking Improves Subjective Quality of Machine Translation

#4 Detailed Improvement Category: Verb Agreement (+6, 0)

Source Base Rerank Ref

ラングミュア‐ ブロジェット法や包接化にも触れた。 Langmuir-Blodgett method and inclusion compounds are mentioned. Langmuir-Blodgett method and inclusion is also discussed. Langmuir-Blodgett method and inclusion are also mentioned.

slide-16
SLIDE 16

16

Neural Reranking Improves Subjective Quality of Machine Translation

What Wasn't Helped: Terminology (+2, -4)

Source Base Rerank Ref

放射熱を利用する赤外線応用計測が応力解析に役立っている Infrared ray applied measurement using radiant heat is useful for stress analysis. The infrared application measurement using radiant heat is useful in the stress analysis. Infrared ray application measurement using radiation heat is useful for stress analysis.

slide-17
SLIDE 17

17

Neural Reranking Improves Subjective Quality of Machine Translation

Conclusion

slide-18
SLIDE 18

18

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Know Now?

  • Neural reranking improves subjective quality of

machine translation output.

  • Main gains are from grammatical factors, and not

lexical selection.

slide-19
SLIDE 19

19

Neural Reranking Improves Subjective Quality of Machine Translation

What Do We Still Not Know Yet?

  • How do neural translation models compare with neural

language models?

  • How does reranking compare with pure neural MT?
slide-20
SLIDE 20

20

Neural Reranking Improves Subjective Quality of Machine Translation

Thank You!