ListNet-based MT Rescoring Jan Niehues, Quoc Khanh Do, Alexandre - - PowerPoint PPT Presentation

▶

listnet based mt rescoring

ListNet-based MT Rescoring Jan Niehues, Quoc Khanh Do, Alexandre - - PowerPoint PPT Presentation

Feb 27, 2024 119 likes •384 views

ListNet-based MT Rescoring Jan Niehues, Quoc Khanh Do, Alexandre Allauzen and Alex Waibel KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS 0 2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics

slide-1

SLIDE 1

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

ListNet-based MT Rescoring

Jan Niehues, Quoc Khanh Do, Alexandre Allauzen and Alex Waibel

KIT – University of the State of Baden-Wuerttemberg and National Research Center of the Helmholtz Association

www.kit.edu

slide-2

SLIDE 2

Motivation

1

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Log-linear model is widely used in SMT

Use during decoding Use in MT rescoring

MT Rescoring

Easy and efficient way to integrate of complex models

Machine learning view

Ranking problem Promising approach: ListNet algorithm

Apply ListNet algorithm to SMT

slide-3

SLIDE 3

Related Work

2

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Optimization in Machine translation Minimum Error Rate Training (MERT) (Och, 2003)

Standard in most machine translation systems

MIRA (Watanabe et al., 2007; Chiang et al., 2008) PRO (Hopkins and May, 2011) Expected BLEU (Rosti et al, 2011; He and Deng, 2012) Ranking in machine learning ListNet algorithm (Cao et al., 2007) Overview over different ranking algorithms (Chen et al., 2009)

slide-4

SLIDE 4

Overview

3

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Motivation ListNet Algorithm MT Rescoring

MT specific problems

Evaluation

WMT IWSLT - TED

slide-5

SLIDE 5

ListNet -Ranking

4

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Input:

List Model score Metric for reference ranking

Hypothesis Model Metric A 7.4 24.4 B 7.8 24.2 C 7.2 24.5 D 7.1 24.1

slide-6

SLIDE 6

ListNet -Ranking

4

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Input:

List Model score Metric for reference ranking

Hypothesis Model Metric B 7.8 24.2 A 7.4 24.4 C 7.2 24.5 D 7.1 24.1 According to the model

slide-7

SLIDE 7

ListNet -Ranking

4

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Input:

List Model score Metric for reference ranking

Hypothesis Model Metric C 7.2 24.5 A 7.4 24.4 B 7.8 24.2 D 7.1 24.1 According to the metric

slide-8

SLIDE 8

ListNet -Ranking

4

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Input:

List Model score Metric for reference ranking

Hypothesis Model Metric B 7.8 24.2 A 7.4 24.4 C 7.2 24.5 D 7.1 24.1 ⇒ Hypothesis Model Metric C 7.2 24.5 A 7.4 24.4 B 7.8 24.2 D 7.1 24.1 Aim: Learn a model to rank like the metric

slide-9

SLIDE 9

ListNet - Idea

5

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Define a probability distribution over possible rankings Learn model that produces a distribution similar to the one defined by the metric Problem: large number of possible rankings Define a probability distribution associated to the model ranking based

n first ranked object

Ps(j) = exp(sj) ∑n

k=1 exp(sk)

(1)

slide-10

SLIDE 10

ListNet - Distribution

6

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

0.2 0.4 0.6 0.8 1 A B C D Probability Hypothesis Model Metric

Minimize cross-entropy difference

slide-11

SLIDE 11

Overview

7

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Motivation ListNet Algorithm MT Rescoring

MT specific problems

Evaluation

WMT IWSLT - TED

slide-12

SLIDE 12

MT Rescoring

8

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Use ListNet to rescore N-Best list

Train log-linear model

Input:

N-Best list Additional features

Learn new weights for log-linear model

slide-13

SLIDE 13

Model

9

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Define probability distribution associated to the model ranking Ps(j) = exp(sj) ∑n

k=1 exp(sk)

(2) Problem:

Many scores are small probabilities Log-probabilities are very small negative values exp(s) calculation may be erroneous

Feature normalization:

Linear transform all features to the range [−1, 1]

Score normalization:

Linear transform the final score of the model to the range [−r, r]

slide-14

SLIDE 14

Metric

10

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Define probability distribution associated to the reference ranking Reference ranking for every sentence needed Ranking induced by MT metric Sentence-wise MT metric

Metric: BLEU+1 (Liang et al. 2006) Smoothed version of BLEU score

Py(i)(x(i)

j

) = exp(BLEU(x(i)

j

)) ∑ni

j′=1 exp(BLEU(x(i) j′ )

(3)

slide-15

SLIDE 15

Training

11

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Minimize cross-entropy difference between model-based and BLEU+1-based probability distribution

Use ListNet algorithm to calculate derivation

Stochastic gradient descent

100,000 batches Batch size of 10

slide-16

SLIDE 16

Overview

12

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Motivation ListNet Algorithm MT Rescoring

MT specific problems

Evaluation

WMT IWSLT - TED

slide-17

SLIDE 17

Evaluation

13

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

WMT 2015 EN-DE

PBMT System Additional features based on neural network translation models

WMT 2015 DE-EN

PBMT System Additional features using RBM-based translation models and source DWL

TED 2014 EN-DE

Translation of TED talks

slide-18

SLIDE 18

WMT – English to German

14

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

20 20.5 21 21.5 22 Baseline NCE SOUL SOUL+NCE BLEU Feature ListNet PRO KBMira MERT No Resco.

slide-19

SLIDE 19

WMT – German to English

15

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

27 27.5 28 28.5 29 Baseline SDWL SDWL+RBMTM BLEU Feature ListNet PRO KBMira MERT No Resco.

slide-20

SLIDE 20

Convergence

16

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

13 13.2 13.4 13.6 13.8 14 14.2 14.4 14.6 14.8 15 200 400 600 800 1000 BLEU+1 Samples (x1000) Dev score

slide-21

SLIDE 21

Score normalization

17

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

19 19.5 20 20.5 21 21.5 22 0.1 1 10 100 BLEU Range Score Feature

slide-22

SLIDE 22

TED – English to German

18

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

22 22.5 23 23.5 24 24.5 25 Baseline extra Dev Data BLEU Feature ListNet PRO KBMira MERT No Resco.

slide-23

SLIDE 23

Conclusion

19

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Presented a new technique to train log-linear model

Scale to many features Consider whole list Technique can also be applied to more complex models

Evaluated using different tasks and languages

WMT English – German WMT German – English IWSLT –TED English – German

Translation quality improvements measured in BLEU score

Outperform MERT in all configurations Less prone to overfitting

slide-24

SLIDE 24

WMT – English to German

20

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Baseline NCE SOUL SOUL+NCE System Dev Test Dev Test Dev Test Dev Test Baseline 20.19 MERT 20.63 20.52 21.24 20.92 21.36 20.84 21.36 20.94 KB-MIRA 20.64 20.38 21.51 20.96 21.65 20.83 21.71 21.06 PRO 20.17 21.01 21.04 21.25 21.18 21.31 21.14 21.34 ListNet 19.95 20.98 21.00 21.51 21.02 21.54 21.14 21.63

slide-25

SLIDE 25

WMT – German to English

21

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Baseline SDWL SDWL+RBMTM System Dev Test Dev Test Dev Test Baseline 27.77 MERT 28.18 27.80 28.24 27.65 28.23 27.64 KB-MIRA 28.23 28.06 28.18 28.00 28.00 27.88 PRO 27.38 28.01 27.56 28.14 28.68 28.04 ListNet 28.00 27.87 27.89 28.18 27.94 28.28

slide-26

SLIDE 26

TED – English to German

22

2015-09-17 Jan Niehues - ListNet-based MT Rescoring KIT - Institute for Anthropomatics and Robotics and LIMSI-CNRS

Baseline extra Dev Data System Dev Test Dev Test Baseline 23.67 MERT 27.69 23.46 25.63 23.36 KB-MIRA 27.47 23.19 25.65 23.76 PRO 26.67 23.10 25.00 23.65 ListNet 27.37 23.51 25.49 24.08