What Kind of Language Is Hard to Language-Model? ACL 2019 Sabrina J. - - PowerPoint PPT Presentation
What Kind of Language Is Hard to Language-Model? ACL 2019 Sabrina J. Mielke and Ryan Cotterell, Kyle Gorman, Brian Roark, Jason Eisner Johns Hopkins University // City University of New York Graduate Center // Google sjmielke@jhu.edu Twitter:
ACL 2019 Sabrina J. Mielke and Ryan Cotterell, Kyle Gorman, Brian Roark, Jason Eisner
Johns Hopkins University // City University of New York Graduate Center // Google
sjmielke@jhu.edu
Twitter: @sjmielke – paper and thread pinned! 1
2
No.
2
No.
2
No.
German.
2
No.
German.
2
No.
German.
It depends.
2
No.
German.
It depends.
2
No.
German.
It depends.
Actually, rather technical factors.
2
No.
German.
It depends.
Actually, rather technical factors.
2
No.
German.
It depends.
Actually, rather technical factors.
It’s different, but not actually easier!
2
“Difficulty”
3
“Difficulty” Models and languages
3
“Difficulty” Models and languages What correlates with difficulty?
3
“Difficulty” Models and languages What correlates with difficulty? And... is Translationese really easier?
3
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
I love Florence!
0.03 ⇒ 5 bits
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
I love Florence!
0.03 ⇒ 5 bits de
Ich grüße meine Oma und die Familie dahein.
0.008 ⇒ 7 bits
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
I love Florence!
0.03 ⇒ 5 bits de
Ich grüße meine Oma und die Familie dahein.
0.008 ⇒ 7 bits nl
Alle mensen worden vrij en gelijk in waardigheid en rechten geboren.
0.0004 ⇒ 11 bits
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
I love Florence!
0.03 ⇒ 5 bits de
Ich grüße meine Oma und die Familie dahein.
0.008 ⇒ 7 bits nl
Alle mensen worden vrij en gelijk in waardigheid en rechten geboren.
0.0004 ⇒ 11 bits Issue 1: Different topics/styles/content
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
Resumption of the session.
0.013 ⇒ 6.5 bits de
Wiederaufnahme der Sitzung.
0.011 ⇒ 6.3 bits nl
Hervatting van de sessie.
0.012 ⇒ 6.4 bits Issue 1: Different topics/styles/content Solution: train and test on translations! Europarl: 21 languages share ~40M chars Bibles: 62 languages share ~4M chars
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
Resumption of the session.
0.013 ⇒ 6.5 bits de
Wiederaufnahme der Sitzung.
0.011 ⇒ 6.3 bits nl
Hervatting van de sessie.
0.012 ⇒ 6.4 bits Issue 1: Different topics/styles/content Solution: train and test on translations! Europarl: 21 languages share ~40M chars Bibles: 62 languages share ~4M chars
and this one takes a big ILP to solve, which is really fun
Gurobi
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
Resumption of the session.
0.013 ⇒ 6.5 bits de
Wiederaufnahme der Sitzung.
0.011 ⇒ 6.3 bits nl
Hervatting van de sessie.
0.012 ⇒ 6.4 bits Issue 1: Different topics/styles/content Solution: train and test on translations! Europarl: 21 languages share ~40M chars Bibles: 62 languages share ~4M chars
and this one takes a big ILP to solve, which is really fun
Gurobi
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
Resumption of the session.
0.013 ⇒ 6.5 bits de
Wiederaufnahme der Sitzung.
0.011 ⇒ 6.3 bits nl
Hervatting van de sessie.
0.012 ⇒ 6.4 bits Issue 1: Different topics/styles/content Solution: train and test on translations! Europarl: 21 languages share ~40M chars Bibles: 62 languages share ~4M chars
and this one takes a big ILP to solve, which is really fun
Gurobi
Issue 2: Comparing scores
4
Language models measure surprisal/information content (NLL; −log p(·)): p(·) ⇒ NLL en
Resumption of the session.
0.013 ⇒ 6.5 bits de
Wiederaufnahme der Sitzung.
0.011 ⇒ 6.3 bits nl
Hervatting van de sessie.
0.012 ⇒ 6.4 bits Issue 1: Different topics/styles/content Solution: train and test on translations! Europarl: 21 languages share ~40M chars Bibles: 62 languages share ~4M chars
and this one takes a big ILP to solve, which is really fun
Gurobi
Issue 2: Comparing scores Use total bits of an
Why?
4
5
Every UNK is “cheating” – morphologically rich languages have more UNKs, unfairly advantaging them.
5
Every UNK is “cheating” – morphologically rich languages have more UNKs, unfairly advantaging them.
5
Every UNK is “cheating” – morphologically rich languages have more UNKs, unfairly advantaging them.
Example: if puˇ ccz and Putschde are equally likely, they should be equally “difficult.”
5
Every UNK is “cheating” – morphologically rich languages have more UNKs, unfairly advantaging them.
Example: if puˇ ccz and Putschde are equally likely, they should be equally “difficult.” ⇒ just use overall bits (i.e., surprisal/NLL) of an aligned sentence
5
Every UNK is “cheating” – morphologically rich languages have more UNKs, unfairly advantaging them.
Example: if puˇ ccz and Putschde are equally likely, they should be equally “difficult.” ⇒ just use overall bits (i.e., surprisal/NLL) of an aligned sentence
[note: total easily obtainable from BPC or perplexity by multiplying with total chars/words]
5
For fully parallel corpora...
Resump- tion
session Wieder- aufnah- me der ... Възобн- овяване на се- ... The peace that ... Der gestern verein- ... Мирът, който беше ... Although we were not al- ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Jetzt ist die Zeit ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text
Image CC-BY Mike Grauer Jr / flickr
For fully parallel corpora...
Resump- tion
session Wieder- aufnah- me der ... Възобн- овяване на се- ... The peace that ... Der gestern verein- ... Мирът, който беше ... Although we were not al- ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Jetzt ist die Zeit ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y1,bg y2,en y2,de y2,bg y3,en y3,de y3,bg y4,en y4,de y4,bg
LM surprisals/NLLs
6
For fully parallel corpora... we can just sum everything up and compare – that is fair.
Resump- tion
session Wieder- aufnah- me der ... Възобн- овяване на се- ... The peace that ... Der gestern verein- ... Мирът, който беше ... Although we were not al- ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Jetzt ist die Zeit ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y1,bg y2,en y2,de y2,bg y3,en y3,de y3,bg y4,en y4,de y4,bg
LM surprisals/NLLs
⇒ ⇒ ⇒
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒ y2,de
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒ y2,de = n2
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒ y2,de = n2 · exp dde
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒ y2,de n2 · exp dde ∼
log-normal noise This is a probabilistic model we can perform inference in!
6
But what if there’s missing data? Or we want robustness?
Resump- tion
session Wieder- aufnah- me der ... The peace that ... Der gestern verein- ... Мирът, който беше ... Obwohl wir nicht ... Макар че не бяхме ... Now we can fi- nally ... Накрая всички можем ...
en de bg 1 2 3 4 aligned multi-text language model y1,en y1,de y2,en y2,de y2,bg y3,de y3,bg y4,en y4,bg
LM surprisals/NLLs
n1 n2 n3 n4 ⇒ ⇒ ⇒ ⇒ den dde dbg ⇒ ⇒ ⇒ y2,de n2 · exp dde ∼
log-normal noise This is a probabilistic model we can perform inference in!
n
q u i t e ,
r a c t u a l m
e l i s H E T E R O S C E D A S T I C yi j = ni · exp(dj) · exp(εi j) σ2
i = ln
ni
σ2−σ2
i
2
, σ2
i
6
“Difficulty” Models and languages What correlates with difficulty? And... is Translationese really easier?
7
Formerly state-of-the-art-ish AWD-LSTM (Merity et al., 2018) language models: char-RNNLM:
RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell
t h e c a t c h a s e d
8
Formerly state-of-the-art-ish AWD-LSTM (Merity et al., 2018) language models: char-RNNLM:
RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell
t h e c a t c h a s e d
BPE-RNNLM, few merges:
RNN cell RNN cell RNN cell RNN cell RNN cell
the ca@ @ t cha@ @ sed
8
Formerly state-of-the-art-ish AWD-LSTM (Merity et al., 2018) language models: char-RNNLM:
RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell RNN cell
t h e c a t c h a s e d
BPE-RNNLM, few merges:
RNN cell RNN cell RNN cell RNN cell RNN cell
the ca@ @ t cha@ @ sed
BPE-RNNLM, many merges:
RNN cell RNN cell RNN cell RNN cell
the cat cha@ @ sed
8
It depends on the language (total surprisal, given merges as a ratio of the vocabulary): ratio: ← lower is better 0.2 0.4 0.6 0.8 1
bg bg bg bg bg bg bg bg bg bg cs cs cs cs cs cs cs cs cs cs da da da da da da da da da da de de de de de de de de de el el el el el el el el el el en en en en en en en en en en es es es es es es es es es es et et et et et et et et et et fi fi fi fi fi fi fi fi fi fi fr fr fr fr fr fr fr fr fr fr hu hu hu hu hu hu hu hu hu hu it it it it it it it it it it lt lt lt lt lt lt lt lt lt lt lv lv lv lv lv lv lv lv lv lv nl nl nl nl nl nl nl nl nl nl pl pl pl pl pl pl pl pl pl pl pt pt pt pt pt pt pt pt pt pt ro ro ro ro ro ro ro ro ro ro sk sk sk sk sk sk sk sk sk sk sl sl sl sl sl sl sl sl sl sl sv sv sv sv sv sv sv sv sv sv
9
It depends on the language (total surprisal, given merges as a ratio of the vocabulary): ratio: ← lower is better 0.2 0.4 0.6 0.8 1
bg bg bg bg bg bg bg bg bg bg cs cs cs cs cs cs cs cs cs cs da da da da da da da da da da de de de de de de de de de el el el el el el el el el el en en en en en en en en en en es es es es es es es es es es et et et et et et et et et et fi fi fi fi fi fi fi fi fi fi fr fr fr fr fr fr fr fr fr fr hu hu hu hu hu hu hu hu hu hu it it it it it it it it it it lt lt lt lt lt lt lt lt lt lt lv lv lv lv lv lv lv lv lv lv nl nl nl nl nl nl nl nl nl nl pl pl pl pl pl pl pl pl pl pl pt pt pt pt pt pt pt pt pt pt ro ro ro ro ro ro ro ro ro ro sk sk sk sk sk sk sk sk sk sk sl sl sl sl sl sl sl sl sl sl sv sv sv sv sv sv sv sv sv sv
9
It depends on the language (total surprisal, given merges as a ratio of the vocabulary): ratio: ← lower is better 0.2 0.4 0.6 0.8 1
bg bg bg bg bg bg bg bg bg bg cs cs cs cs cs cs cs cs cs cs da da da da da da da da da da de de de de de de de de de el el el el el el el el el el en en en en en en en en en en es es es es es es es es es es et et et et et et et et et et fi fi fi fi fi fi fi fi fi fi fr fr fr fr fr fr fr fr fr fr hu hu hu hu hu hu hu hu hu hu it it it it it it it it it it lt lt lt lt lt lt lt lt lt lt lv lv lv lv lv lv lv lv lv lv nl nl nl nl nl nl nl nl nl nl pl pl pl pl pl pl pl pl pl pl pt pt pt pt pt pt pt pt pt pt ro ro ro ro ro ro ro ro ro ro sk sk sk sk sk sk sk sk sk sk sl sl sl sl sl sl sl sl sl sl sv sv sv sv sv sv sv sv sv sv
is this one going to be fine?
9
It depends on the language (total surprisal, given merges as a ratio of the vocabulary): ratio: ← lower is better 0.2 0.4 0.6 0.8 1
bg bg bg bg bg bg bg bg bg bg cs cs cs cs cs cs cs cs cs cs da da da da da da da da da da de de de de de de de de de el el el el el el el el el el en en en en en en en en en en es es es es es es es es es es et et et et et et et et et et fi fi fi fi fi fi fi fi fi fi fr fr fr fr fr fr fr fr fr fr hu hu hu hu hu hu hu hu hu hu it it it it it it it it it it lt lt lt lt lt lt lt lt lt lt lv lv lv lv lv lv lv lv lv lv nl nl nl nl nl nl nl nl nl nl pl pl pl pl pl pl pl pl pl pl pt pt pt pt pt pt pt pt pt pt ro ro ro ro ro ro ro ro ro ro sk sk sk sk sk sk sk sk sk sk sl sl sl sl sl sl sl sl sl sl sv sv sv sv sv sv sv sv sv sv
is this one going to be fine? Yeah: it doesn’t matter that much.
chars BPE (0.4) BPE (best)
bg cs da de el en es et fi fr hu it lt lv nl pl pt ro sk sl sv
9
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10 difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv
en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv
en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl vs.
harder easier de en deu eng −15 −10 −5 5 10 15 20 25 −15 −10 −5 5 10 15 20
easier with BPE easier with chars
de de de de de de de de de de de en en en en
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Bibles 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl vs.
harder easier de en deu eng bg cs da el fi fr hu it lt nl pt ro bul ces dan ell fin fra hun ita lit nld por ron −15 −10 −5 5 10 15 20 25 −15 −10 −5 5 10 15 20
easier with BPE easier with chars
cmn afr aln arb arz ayr ayr bba ben ben bqc bul bul cac cak ceb ceb ceb ces ces cnh cym dan deu deu deu deu deu deu deu deu deu deu deu ell eng eng eng eng epo fin fin fin fra fra fra fra fra fra fra fra fra fra fra guj gur hat hat hrv hun hun ind ind ita ita ita ita kek kek kjb lat lit mah mam mri mya nld nor nor plt poh por por por qub quh quy quz ron rus som tbz tcw tgl tlh tpi tpm ukr ukr vie vie vie wal wbm xho zom
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Bibles 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl vs.
harder easier de en deu eng bg cs da el fi fr hu it lt nl pt ro bul ces dan ell fin fra hun ita lit nld por ron −15 −10 −5 5 10 15 20 25 −15 −10 −5 5 10 15 20
easier with BPE easier with chars
cmn
afr aln arb arz ayr ayr bba ben ben bqc bul bul cac cak ceb ceb ceb ces ces cnh cym dan deu deu deu deu deu deu deu deu deu deu deu ell eng eng eng eng epo fin fin fin fra fra fra fra fra fra fra fra fra fra fra guj gur hat hat hrv hun hun ind ind ita ita ita ita kek kek kjb lat lit mah mam mri mya nld nor nor plt poh por por por qub quh quy quz ron rus som tbz tcw tgl tlh tpi tpm ukr ukr vie vie vie wal wbm xho zom
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Bibles 10
−4 −3 −2 −1 1 2 3 4 5 −8 −6 −4 −2 2 4 6 8 10
easier with BPE easier with chars
bg cs da el es et fi fr hu it lt lv nl pl pt ro sk sl sv en de
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Europarl vs.
harder easier de en deu eng bg cs da el fi fr hu it lt nl pt ro bul ces dan ell fin fra hun ita lit nld por ron −15 −10 −5 5 10 15 20 25 −15 −10 −5 5 10 15 20
easier with BPE easier with chars
cmn afr aln arb arz ayr ayr bba ben ben bqc bul bul cac cak ceb ceb ceb ces ces cnh cym dan deu deu deu deu deu deu deu deu deu deu deu ell eng eng eng eng epo fin fin fin fra fra fra fra fra fra fra fra fra fra fra guj gur hat hat hrv hun hun ind ind ita ita ita ita kek kek kjb lat lit mah mam mri mya nld nor nor plt poh por por por qub quh quy quz ron rus som tbz tcw tgl tlh tpi tpm ukr ukr vie vie vie wal wbm xho zom
difficulty (×100) using BPE-RNNLM with 0.4|V| merges difficulty (×100) using char-RNNLM
Difficulties on Bibles 10
“Difficulty” Models and languages What correlates with difficulty? And... is Translationese really easier?
11
50 100 150 200 −4 −2 2 4
bg cs da de el en et fi fr hu it lv lt nl pl pt ro sk sl es sv
MCC difficulty (×100, BPE-RNNLM) 50 100 150 200 −5 5 10
bg cs da de el en et fi fr hu it lv lt nl pl pt ro sk sl es sv
MCC difficulty (×100, char-RNNLM)
12
50 100 150 200 −4 −2 2 4
bg cs da de el en et fi fr hu it lv lt nl pl pt ro sk sl es sv
MCC difficulty (×100, BPE-RNNLM) 50 100 150 200 −5 5 10
bg cs da de el en et fi fr hu it lv lt nl pl pt ro sk sl es sv
MCC difficulty (×100, char-RNNLM)
...not particularly striking. Perhaps Finnish was an outlier in Cotterell et al. (2018)?
12
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)?
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences.
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)?
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences.
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences. Head-POS Entropy (Dehouck and Denis, 2018)?
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences. Head-POS Entropy (Dehouck and Denis, 2018)? ...neither mean and skew show correlation.
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences. Head-POS Entropy (Dehouck and Denis, 2018)? ...neither mean and skew show correlation. Average dependency length (computed using UDPipe (Straka et al., 2016))?
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences. Head-POS Entropy (Dehouck and Denis, 2018)? ...neither mean and skew show correlation. Average dependency length (computed using UDPipe (Straka et al., 2016))? ...correlation! But not significant after correcting for multiple hypotheses.
13
WALS: “Prefixing vs. Suffixing [...] Morphology” (for languages where present)? ...no visible differences. WALS: “Order of Subject, Object and Verb” (for languages where present)? ...no visible differences. Head-POS Entropy (Dehouck and Denis, 2018)? ...neither mean and skew show correlation. Average dependency length (computed using UDPipe (Straka et al., 2016))? ...correlation! But not significant after correcting for multiple hypotheses. This is disappointing.
13
Raw sequence length / # predictions → char-RNNLM difficulty Significant on:
i.e., for the char-RNNLM puˇ ccz is easier than Putschde!
14
Raw sequence length / # predictions → char-RNNLM difficulty Significant on:
i.e., for the char-RNNLM puˇ ccz is easier than Putschde! Raw vocabulary size → BPE-RNNLM difficulty Significant on:
p < .00000000001 i.e., the BPE-RNNLM still suffers if a language has high type-token-ratio!
14
Raw sequence length / # predictions → char-RNNLM difficulty Significant on:
i.e., for the char-RNNLM puˇ ccz is easier than Putschde! Raw vocabulary size → BPE-RNNLM difficulty Significant on:
p < .00000000001 i.e., the BPE-RNNLM still suffers if a language has high type-token-ratio! Wow! What is happening here? We have many conjectures...
14
“Difficulty” Models and languages What correlates with difficulty? And... is Translationese really easier?
15
Common assumption: Translationese is somehow simpler than “native” text.
16
Common assumption: Translationese is somehow simpler than “native” text. We have partial parallel data that we can use to evaluate our models: enoriginal entranslated deoriginal detranslated nloriginal nltranslated
··· Resumption... Wiederauf-... Hervatten... ··· The German... Der deutsche... De Duitse... ··· Thank you... Vielen Dank... Hartelijk... ··· ··· ··· ··· ··· ··· ···
16
Common assumption: Translationese is somehow simpler than “native” text. We have partial parallel data that we can use to evaluate our models: enoriginal entranslated deoriginal detranslated nloriginal nltranslated
··· Resumption... Wiederauf-... Hervatten... ··· The German... Der deutsche... De Duitse... ··· Thank you... Vielen Dank... Hartelijk... ··· ··· ··· ··· ··· ··· ···
...and indeed the original languages seem harder.
16
Common assumption: Translationese is somehow simpler than “native” text. We have partial parallel data that we can use to evaluate our models: enoriginal entranslated deoriginal detranslated nloriginal nltranslated
··· Resumption... Wiederauf-... Hervatten... ··· The German... Der deutsche... De Duitse... ··· Thank you... Vielen Dank... Hartelijk... ··· ··· ··· ··· ··· ··· ···
...and indeed the original languages seem harder. But we missed something!
16
en fr de es nl it pt sv el fi pl da ro hu sk cs sl lt bg et lv 5 10 15 20 languages, sorted by absolute # native sentences % native of sentences
17
We can rebalance a single language, leaving the others merged, i.e.: enoriginal entranslated de nl
··· Resumption... Wiederauf-... Hervatten... ··· The German... Der deutsche... De Duitse... ··· Thank you... Vielen Dank... Hartelijk... ··· ··· ··· ··· ··· ···
18
We can rebalance a single language, leaving the others merged, i.e.: enoriginal entranslated de nl
··· Resumption... Wiederauf-... Hervatten... ··· The German... Der deutsche... De Duitse... ··· Thank you... Vielen Dank... Hartelijk... ··· ··· ··· ··· ··· ···
And the result: the difficulties are now the same!
(more precisely, “native” is 0.004 ± 0.02 easier)
18
19
19
19
19
19
ACL 2019 Sabrina J. Mielke and Ryan Cotterell, Kyle Gorman, Brian Roark, Jason Eisner
Johns Hopkins University // City University of New York Graduate Center // Google
sjmielke@jhu.edu
Twitter: @sjmielke – paper and thread pinned! 20