SLIDE 1
Morphological Inflection
Zug
train (de)
Supervised Learning of Complete Morphological Paradigms Greg - - PowerPoint PPT Presentation
Supervised Learning of Complete Morphological Paradigms Greg - - PowerPoint PPT Presentation
Supervised Learning of Complete Morphological Paradigms Greg Durrett and John DeNero UC Berkeley / Google Morphological Inflection train (de) Zug Morphological Inflection N OM, S ING: Zug N OM, P LU: Zge train (de) Zug G EN, S ING:
SLIDE 2
SLIDE 3
Morphological Inflection
Zug
train (de)
(8 total) NOM, SING: Zug NOM, PLU: Züge GEN, SING: Zuges GEN, PLU: Züge
... ...
SLIDE 4
Morphological Inflection
Zug
train (de)
(8 total) NOM, SING: Zug NOM, PLU: Züge GEN, SING: Zuges GEN, PLU: Züge
... ...
u NOM, SING: u NOM, PLU: ü GEN, SING: u GEN, PLU: ü
... ...
SLIDE 5
Morphological Inflection
Zug
train (de)
(8 total) NOM, SING: Zug NOM, PLU: Züge GEN, SING: Zuges GEN, PLU: Züge
... ...
u NOM, SING: u NOM, PLU: ü GEN, SING: u GEN, PLU: ü
... ...
g NOM, SING: g NOM, PLU: ge GEN, SING: ges GEN, PLU: ge
... ...
SLIDE 6
express train (de)
u NOM, SING: u NOM, PLU: ü GEN, SING: u GEN, PLU: ü
... ...
g NOM, SING: g NOM, PLU: ge GEN, SING: ges GEN, PLU: ge
... ...
Schnellzug NOM, SING: Schnellzug NOM, PLU: Schnellzüge GEN, SING: Schnellzuges GEN, PLU: Schnellzüge
... ...
Morphological Inflection
SLIDE 7
valkuainen NOM, SING: valkuainen GEN, SING: valkuaisen PART, SING: valkuaista
... ... egg white (fi)
syyllinen
culprit (fi)
NOM, SING: syyllinen GEN, SING: syyllisen PART, SING: syyllistä
... ...
Morphological Inflection
SLIDE 8
valkuainen NOM, SING: valkuainen GEN, SING: valkuaisen PART, SING: valkuaista
... ... egg white (fi)
syyllinen
culprit (fi)
NOM, SING: syyllinen GEN, SING: syyllisen PART, SING: syyllistä
... ...
Morphological Inflection
SLIDE 9
valkuainen NOM, SING: valkuainen GEN, SING: valkuaisen PART, SING: valkuaista
... ... egg white (fi)
syyllinen
culprit (fi)
back vowels NOM, SING: syyllinen GEN, SING: syyllisen PART, SING: syyllistä
... ...
Morphological Inflection
SLIDE 10
valkuainen NOM, SING: valkuainen GEN, SING: valkuaisen PART, SING: valkuaista
... ... egg white (fi)
syyllinen
culprit (fi)
front vowels back vowels NOM, SING: syyllinen GEN, SING: syyllisen PART, SING: syyllistä
... ...
Morphological Inflection
SLIDE 11
valkuainen NOM, SING: valkuainen GEN, SING: valkuaisen PART, SING: valkuaista
... ... egg white (fi)
syyllinen
culprit (fi)
Suffix morphology can depend on context in a complex way front vowels back vowels NOM, SING: syyllinen GEN, SING: syyllisen PART, SING: syyllistä
... ...
Morphological Inflection
SLIDE 12
Base form (Word) Given
Morphological Inflection
SLIDE 13
Base form (Word) Attributes(1): Inflected form(1) Attributes(2): Inflected form(2)
... ...
Given Predict Inflection table (variants)
Morphological Inflection
SLIDE 14
Base form (Word) Attributes(1): Inflected form(1) Attributes(2): Inflected form(2)
... ...
Given Predict Inflection table (variants) Language/POS independence
Morphological Inflection
SLIDE 15
Base form (Word) Attributes(1): Inflected form(1) Attributes(2): Inflected form(2)
... ...
Given Predict Inflection table (variants) Handle multi-part paradigms Language/POS independence
Morphological Inflection
SLIDE 16
Base form (Word) Attributes(1): Inflected form(1) Attributes(2): Inflected form(2)
... ...
Given Predict Inflection table (variants) Handle multi-part paradigms Language/POS independence Rich way to predict inflections
Morphological Inflection
SLIDE 17
Outline
SLIDE 18
Outline
Rule extraction Kluft ... ... Zug ... ... (>1000)
SLIDE 19
Outline
Rule extraction g ... ... u ... ... Kluft ... ... Zug ... ... (>1000) (<50)
SLIDE 20
Outline
Rule extraction Paradigm prediction
,
) (
g ... ... u ... ... Kluft ... ... Zug ... ... Zug ... ... u ... ... (>1000) (<50)
SLIDE 21
Outline
Rule extraction Paradigm prediction
,
) (
g ... ... u ... ... Kluft ... ... Zug ... ... Zug ... ... u ... ... Base form Inflection table predictor (>1000) (<50)
SLIDE 22
Wiktionary Data
Thousands of full inflection tables for many languages and parts of speech
SLIDE 23
Wiktionary Data
Thousands of full inflection tables for many languages and parts of speech
http://en.wiktionary.org/wiki/Zug
SLIDE 24
Wiktionary Data
Thousands of full inflection tables for many languages and parts of speech 2763 more...
http://en.wiktionary.org/wiki/Zug
SLIDE 25
Previous Work
Dreyer and Eisner (2011) develops a semi- supervised Bayesian model
SLIDE 26
Previous Work
Dreyer and Eisner (2011) develops a semi- supervised Bayesian model Wicentowski (2004) predicts one variant at a time (the lemma)
SLIDE 27
Previous Work
Dreyer and Eisner (2011) develops a semi- supervised Bayesian model Wicentowski (2004) predicts one variant at a time (the lemma) u NOM, SING: u NOM, PLU: ü GEN, SING: u GEN, PLU: ü
... ...
SLIDE 28
Rule Extraction
SLIDE 29
Rule Extraction
dringen 1P
, PRES, SING: dringe
2P
, PRES, SING: dringst
3P
, PRES, SING: dringt
1P
, PAST, SING: drang
... ... to insist (de)
(27 total)
SLIDE 30
Rule Extraction
dr i ngen 1P
, PRES, SING: dringe
2P
, PRES, SING: dringst
3P
, PRES, SING: dringt
1P
, PAST, SING: drang
... ... to insist (de)
i (27 total) 1P
, PRES, SING: i
2P
, PRES, SING: i
3P
, PRES, SING: i
1P
, PAST, SING: a
... ...
SLIDE 31
Rule Extraction
dr i ng en 1P
, PRES, SING: dringe
2P
, PRES, SING: dringst
3P
, PRES, SING: dringt
1P
, PAST, SING: drang
... ... to insist (de)
i en (27 total) 1P
, PRES, SING: i
2P
, PRES, SING: i
3P
, PRES, SING: i
1P
, PAST, SING: a
... ...
1P
, PRES, SING: e
2P
, PRES, SING: st
3P
, PRES, SING: t
1P
, PAST, SING: -
... ...
SLIDE 32
INFINITIVE
d r i n g e n d r i n g e
1P
, PRES, SING
Alignment to Base Form
SLIDE 33
INFINITIVE
d r i n g e n d r i n g e
1P
, PRES, SING
Alignment to Base Form
D
SLIDE 34
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
SLIDE 35
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE
d r i n g e n d r i n g s t
2P
, PRES, SING
SLIDE 36
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE
d r i n g e n d r i n g s t
2P
, PRES, SING
S S
SLIDE 37
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
SLIDE 38
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE
d r i n g e n d r i n g t
3P
, PRES, SING
D S
SLIDE 39
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
SLIDE 40
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE
d r i n g e n d r a n g
1P
, PAST, SING
D D S
SLIDE 41
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Alignment to Base Form
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
SLIDE 42
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
Concatenation
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
SLIDE 43
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 44
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 45
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 46
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 47
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 48
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 49
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
SLIDE 50
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
d r i n g e d r i n g s t d r i n g t d r a n g
SLIDE 51
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
d r i n g e d r i n g s t d r i n g t d r a n g
SLIDE 52
INFINITIVE d r i n g e n
d r i n g e
1P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g s t
2P
, PRES, SING
INFINITIVE d r i n g e n
d r i n g t
3P
, PRES, SING
INFINITIVE d r i n g e n
d r a n g
1P
, PAST, SING
d r i n g e n
Concatenation
d r i n g e d r i n g s t d r i n g t d r a n g
SLIDE 53
(<50) (>1000)
Outline
Rule extraction Paradigm prediction
,
) (
g ... ... u ... ... Kluft ... ... Zug ... ... Zug ... ... u ... ... Base form Inflection table predictor
SLIDE 54
Paradigm Prediction
w i n d e n
to wind (de)
SLIDE 55
Paradigm Prediction
w i n d e n
i i i i a ... i1 to wind (de) en e st t
- ...
en1
SLIDE 56
Paradigm Prediction
w i n d e n
i i i i a ... i1 en e est et
- ...
en2 n
- st
t te ... n1 to wind (de) en e st t
- ...
en1
SLIDE 57
Paradigm Prediction
w i n d e n
i i i i a ... i1 en e est et
- ...
en2 n
- st
t te ... n1 i1 = = = en2 to wind (de) en e st t
- ...
en1
SLIDE 58
Paradigm Prediction
w i n d e n
i i i i a ... i1 en e est et
- ...
en2 n
- st
t te ... n1 i1 = = = = = = en2 to wind (de) en e st t
- ...
en1
SLIDE 59
Paradigm Prediction
w i n d e n
i i i i a ... i1 en e est et
- ...
en2 n
- st
t te ... n1 i1 n1 = = = = = = en2 to wind (de) en e st t
- ...
en1
SLIDE 60
Paradigm Prediction
w i n d e n
i i i i a ... i1 en e est et
- ...
en2 n
- st
t te ... n1 i1 n1 = = = = = = n1 en1 en2 to wind (de) en e st t
- ...
en1
SLIDE 61
Paradigm Prediction
w i n d e n
i1 n1 = = = = = = n1 en1 en2
SLIDE 62
Paradigm Prediction
w i n d e n
i1 n1 = = = = = = n1 en1 en2
Paths through this lattice are hypotheses
SLIDE 63
Paradigm Prediction
w i n d e n
i1 n1 = = = = = = n1 en1 en2
Model with semi-Markov CRF (Sarawagi and Cohen 2004) Paths through this lattice are hypotheses
SLIDE 64
w i n d e n
i1
Prediction Features
SLIDE 65
w i n d e n
i1
binden verbinden ...
Prediction Features
SLIDE 66
w i n d e n
i1
Prediction Features
binden verbinden ...
SLIDE 67
w i n d e n
i1
[i1] nd
Prediction Features
binden verbinden ...
SLIDE 68
w i n d e n
i1
Rule identity conjoined with 1- through 4-grams at offsets up to +/-5 [i1] nd
Prediction Features
binden verbinden ...
SLIDE 69
w i n d e n
i1
Rule identity conjoined with 1- through 4-grams at offsets up to +/-5 Coarse features shared between different rules [i1] nd
Prediction Features
binden verbinden ...
SLIDE 70
Learning
The gold inflection table of every training example can be produced using our rules
SLIDE 71
Learning
The gold inflection table of every training example can be produced using our rules Optimize conditional log-likelihood of the correct rule sequence
SLIDE 72
Base form Inflection table predictor u ... ... (<50) (>1000)
Outline
Rule extraction Paradigm prediction
,
) (
g ... ... u ... ... Kluft ... ... Zug ... ... Zug ... ...
SLIDE 73
Wiktionary Evaluation Setup
# lines/table Train size Test size German nouns 8 2364 200 German verbs 27 1627 200 Spanish verbs 57 3655 200 Finnish nouns/adjs 28 6000 200 Finnish verbs 53 6849 200
SLIDE 74
Wiktionary Evaluation Setup
# lines/table Train size Test size German nouns 8 2364 200 German verbs 27 1627 200 Spanish verbs 57 3655 200 Finnish nouns/adjs 28 6000 200 Finnish verbs 53 6849 200
Most common 200 inflection tables are not chosen for the test set; we can memorize these
SLIDE 75
25 50 75 100
74.39
Wiktionary Results
Suffixing baseline
Average inflected form accuracy
NAÏVE THIS WORK ORACLE FACTORED
SLIDE 76
25 50 75 100
94.93 74.39
Average inflected form accuracy
Wiktionary Results
NAÏVE THIS WORK ORACLE FACTORED
SLIDE 77
25 50 75 100
94.93 74.39 99.81 The best we can do with
- ur rules
Average inflected form accuracy
Wiktionary Results
NAÏVE THIS WORK ORACLE FACTORED
SLIDE 78
25 50 75 100
94.93 74.39
NAÏVE
99.81 94.41 Separate predictor for each inflected form
Average inflected form accuracy
Wiktionary Results
THIS WORK ORACLE FACTORED
SLIDE 79
25 50 75 100
78.6 86.1 99.4 40.0
Average whole table accuracy
Wiktionary Results
NAÏVE THIS WORK ORACLE FACTORED
SLIDE 80
Dreyer and Eisner (2011)
SLIDE 81
Dreyer and Eisner (2011)
Evaluate on German verbs in CELEX Results averaged over 10 random train/test splits Small train sets (50 or 100 observed tables) Large test sets (5415 verbs)
SLIDE 82
Dreyer and Eisner (2011)
Evaluate on German verbs in CELEX Results averaged over 10 random train/test splits Small train sets (50 or 100 observed tables) Large test sets (5415 verbs) Hierarchical Bayesian model of inflection Type-level transducers of variants can be trained in a supervised fashion Additionally incorporate unlabeled text with a token-level model
SLIDE 83
CELEX Results
85 90 95 100 50 100 200
Training set size (example tables) I n fl e c t e d f
- r
m a c c u r a c y
90.5 89.9 90.9 92.2 91.5 92.2
THIS WORK DE11+CORPUS DE11
SLIDE 84
CELEX Results
85 90 95 100 50 100 200
I n fl e c t e d f
- r
m a c c u r a c y
90.5 89.9 90.9 92.2 91.5 92.2
THIS WORK
93.2
Training set size (example tables) DE11+CORPUS DE11
SLIDE 85
Conclusion
Morphological inflection rules can be learned from supervised data, which is widely available
SLIDE 86
Conclusion
Morphological inflection rules can be learned from supervised data, which is widely available Structured prediction of entire tables at once is effective for inflecting unseen base forms
SLIDE 87
Conclusion
Morphological inflection rules can be learned from supervised data, which is widely available Structured prediction of entire tables at once is effective for inflecting unseen base forms Code and Wiktionary data is available at http://eecs.berkeley.edu/~gdurrett
SLIDE 88
Conclusion
Morphological inflection rules can be learned from supervised data, which is widely available
Thank you!
Structured prediction of entire tables at once is effective for inflecting unseen base forms Code and Wiktionary data is available at http://eecs.berkeley.edu/~gdurrett
SLIDE 89
Accuracy Breakdown
80 85 90 95 100 DE verbs DE nouns ES verbs FI verbs FI nouns/adjs
Inflected form accuracy
96.19 88.94 93.41 96.43 99.67