SLIDE 1
A Distributional and Orthographic Aggregation Model for English Derivational Morphology
Daniel Deutsch,* John Hewitt,* and Dan Roth
*equal contribution
2
Co-Authors
John Hewitt
Co-First Author
Dan Roth
Advisor
Aggregation Model for English Derivational Morphology Daniel - - PowerPoint PPT Presentation
Aggregation Model for English Derivational Morphology Daniel - - PowerPoint PPT Presentation
A Distributional and Orthographic Aggregation Model for English Derivational Morphology Daniel Deutsch,* John Hewitt,* and Dan Roth *equal contribution Co-Authors John Hewitt Dan Roth Co-First Author Advisor 2 Derivational Morphology
SLIDE 2
SLIDE 3
3
Derivational Morphology
employer employ employment intensely intense intensity
SLIDE 4
4
Derivational Morphology
employer employ employment intensely intense intensity
root word
transformation derived word
SLIDE 5
5
Derivational Morphology
employer employ employment intensely intense intensity
root word
transformation derived word
SLIDE 6
6
Motivation
- Machine translation
- Text simplification
- Language generation
SLIDE 7
7
Challenges
- Suffix ambiguity
- Orthographic irregularity
SLIDE 8
8
Suffix Ambiguity
“I have an observament!”
SLIDE 9
9
Suffix Ambiguity
ground grounding *groundation *groundment *groundal “I have an observament!”
Result
SLIDE 10
10
Suffix Ambiguity
valid validity *validness
Nominal
ground grounding *groundation *groundment *groundal “I have an observament!”
Result
SLIDE 11
11
Orthographic Irregularity
speak speech
Result
SLIDE 12
12
Orthographic Irregularity
speak creak speech
Result Result
SLIDE 13
13
Orthographic Irregularity
speak creak speech *creech
Result Result
SLIDE 14
*creech
14
Orthographic Irregularity
speak creak speech creaking
Result Result
SLIDE 15
*creech
15
Orthographic Irregularity
speak creak erupt speech creaking eruption
Result Result Result
SLIDE 16
*creech
16
Orthographic Irregularity
speak creak erupt bankrupt speech creaking eruption
Result Result Result Result
SLIDE 17
*creech
17
Orthographic Irregularity
speak creak erupt bankrupt speech creaking eruption *bankruption
Result Result Result Result
SLIDE 18
18
Orthographic Irregularity
speak speech creak *creech erupt eruption bankrupt *bankruption bankruptcy creaking
Result Result Result Result
SLIDE 19
19
Model Overview
wise
Adverb +
wisely aggregation distributional:
- rthographic irregularity
- rthographic:
suffix ambiguity
SLIDE 20
20
Model Overview
wise
Adverb +
SLIDE 21
21
Model Overview
distributional:
- rthographic irregularity
- rthographic:
suffix ambiguity
SLIDE 22
22
Model Overview
wisely aggregation
SLIDE 23
23
Model Overview
wise
Adverb +
wisely aggregation distributional:
- rthographic irregularity
- rthographic:
suffix ambiguity
SLIDE 24
24
Model Overview
- rthographic:
suffix ambiguity
SLIDE 25
25
Orthographic Model
- Seq2Seq baseline
- Dictionary-constrained decoding
- Reranking with frequency information
SLIDE 26
26
Seq2Seq Baseline
c
- m
p
- s
e #
Result
# c
- m
p
- s
i
SLIDE 27
27
Seq2Seq Baseline
c
- m
p
- s
e # #
Result
SLIDE 28
28
Seq2Seq Baseline
c
- m
p
- s
e # #
SLIDE 29
29
Seq2Seq Baseline
Result
SLIDE 30
30
Seq2Seq Baseline
c
- m
p
- s
i
SLIDE 31
- Seq2Seq models generate
many unattested words, but are reasonable guesses
31
Dictionary-Constrained Decoding
ground grounding *groundation *groundment *groundal
Result
Suffix Ambiguity
SLIDE 32
- Seq2Seq models generate
many unattested words, but are reasonable guesses
- Intuition: constrain model
to only generate known words
32
Dictionary-Constrained Decoding
ground grounding *groundation *groundment *groundal
Result
Suffix Ambiguity
SLIDE 33
33
Dictionary-Constrained Decoding
SLIDE 34
34
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … … … … … … #
SLIDE 35
35
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba #
SLIDE 36
36
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … … … … … … #
SLIDE 37
37
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba #
SLIDE 38
38
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … #
SLIDE 39
39
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … #
SLIDE 40
40
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … … … … … … #
SLIDE 41
41
Dictionary-Constrained Decoding
# a b ab aa bb aba abb baa bab ba … … … … … … #
SLIDE 42
42
Dictionary-Constrained Decoding
Search over trie induced from dictionary
# a b ab aba ba #
SLIDE 43
43
Reranking with Frequency Information
refute
Result
SLIDE 44
44
Reranking with Frequency Information
refution refutation refut refuty refutat
- 1.1
- 1.2
- 4.8
- 5.6
- 8.7
refute
Result
Model Output Model Score
SLIDE 45
45
Reranking with Frequency Information
refution refutation refut refuty refutat
- 1.1
- 1.2
- 4.8
- 5.6
- 8.7
refute
Result
Model Output Model Score
SLIDE 46
46
Reranking with Frequency Information
refution refutation refut refuty refutat
- 1.1
- 1.2
- 4.8
- 5.6
- 8.7
5.0 14.3 7.4 0.1 8.6
refute
Result
Model Output Model Score Log Corpus Freq
SLIDE 47
47
Reranking with Frequency Information
Model Output
refution refutation refut refuty refutat
Model Score
- 1.1
- 1.2
- 4.8
- 5.6
- 8.7
Log Corpus Freq
5.0 14.3 7.4 0.1 8.6
Reranker Output
refutation refution refut refuty refutat
Reranker Score
0.5
- 0.9
- 0.9
- 0.9
- 0.9
refute
Result
SLIDE 48
48
Model Overview
- rthographic:
suffix ambiguity
SLIDE 49
49
Model Overview
distributional:
- rthographic irregularity
SLIDE 50
- Orthographic information
can be unreliable
- Semantic transformation
remains the same
50
Distributional Model
*creech speak creak speech creaking Orthographic Irregularity
Result Result
SLIDE 51
51
Distributional Model
Intuition
SLIDE 52
52
Distributional Model
Intuition
SLIDE 53
53
Distributional Model
Intuition Learn non-linear function per transformation
SLIDE 54
54
Distributional Model
Intuition Learn non-linear function per transformation Independent of
- rthography
SLIDE 55
55
Distributional Model
non-linear function
SLIDE 56
56
Model Overview
distributional:
- rthographic irregularity
SLIDE 57
57
Model Overview
wisely aggregation
SLIDE 58
58
Aggregation Model
approval
Orthographic
approvation
- 0.2
Distributional
approval
- 0.1
non-linear function
SLIDE 59
59
Aggregation Model
Ortho Score Score Distributional
- 0.9
- 0.3
- 0.5
- 0.8
- 0.6
- 0.8
- 1.1
- 0.9
approvation bankruption expertly stroller approval bankruptcy expertly strolls
SLIDE 60
60
Aggregation Model
Ortho Score Score Distributional
- 0.9
- 0.3
- 0.8
- 0.6
- 0.8
- 0.9
approvation bankruption stroller approval bankruptcy strolls
SLIDE 61
61
Aggregation Model
Ortho Score Score Aggregation Selection
approvation bankruption expertly stroller
Distributional
approval bankruptcy expertly strolls
- 0.9
- 0.3
- 0.5
- 0.8
- 0.6
- 0.8
- 1.1
- 0.9
approval bankruption expertly stroller
SLIDE 62
Experiments
62
SLIDE 63
Dataset
Transformation Count Example
Adverb
1715
Result
1251
Agent
801
Nominal
354 recite recital
- verstate
- verstatement
simulate simulation wise wisely survive survivor yodel yodeler effective effectiveness pessimistic pessimism intense intensity
Cotterell et al. 2017
63
SLIDE 64
64
Experiment Details
- 30 random restarts
- Token information: Google Book NGrams
– 360k unigram types – Token counts aggregated
- Google News pre-trained word embeddings
- Evaluation: full-token match accuracy
SLIDE 65
65
Results Legend
Seq2Seq Distributional Aggregation Dictionary-Constrained Decoding Frequency-Based Reranking
SLIDE 66
66
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
SLIDE 67
Cotterell et al. 2017 Constrained
67
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained
SLIDE 68
68
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
SLIDE 69
69
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
Significant improvement when combining Dist and Seq
SLIDE 70
70
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
Frequency statistics are a valuable signal
SLIDE 71
71
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
Combined model still outperforms separate models
SLIDE 72
72
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained Cotterell et al. 2017
SLIDE 73
Cotterell et al. 2017
73
Results
40 45 50 55 60 65 70 75 80 85
Dist Seq Aggr Seq+Freq Aggr+Freq
T
- ken Accuracy
Unconstrained Constrained
22% and 37% relative error reductions over Seq
SLIDE 74
74
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Baseline Aggr Aggr+Freq+Dict Cotterell et al. 2017
SLIDE 75
Baseline Aggr Aggr+Freq+Dict 75
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Cotterell et al. 2017
SLIDE 76
Baseline Aggr Aggr+Freq+Dict 76
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Cotterell et al. 2017
SLIDE 77
Baseline Aggr 77
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Aggr+Freq+Dict Cotterell et al. 2017
SLIDE 78
78
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Baseline Aggr Aggr+Freq+Dict Cotterell et al. 2017
SLIDE 79
79
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Baseline Aggr Aggr+Freq+Dict Cotterell et al. 2017
SLIDE 80
80
Results by Transformation
10 20 30 40 50 60 70 80 90 100
Nominal Result Agent Adverb
T
- ken Accuracy
Baseline Aggr Aggr+Freq+Dict Cotterell et al. 2017
equivalent {equivalence, equivalency}
Nominal
SLIDE 81
81
What does each model do well?
SLIDE 82
82
What does each model do well?
Orthographic does best Distributional does best
SLIDE 83
83
What does each model do well?
Orthographic does best Distributional does best
SLIDE 84
84
What does each model do well?
Orthographic does best Distributional does best
SLIDE 85
85
What does each model do well?
*sponsorment sponsorship
Orthographic does best Distributional does best
SLIDE 86
86
What does each model do well?
Orthographic does best Distributional does best
SLIDE 87
87
What does each model do well?
Orthographic does best Distributional does best wise wisely
Adverb
quick quickly
Adverb
SLIDE 88
88
What does each model do well?
Orthographic does best Distributional does best wise wisely
Adverb
quick quickly
Adverb Result
renew renewal invest investment
Result
inspire inspiration
Result
SLIDE 89
89
Conclusion
- Aggregation model for English derivational
morphology
- Dictionary-constrained decoding
- Frequency-based reranking
- Distributional model per-transformation
- Best open- and closed-vocabulary models
demonstrate 22% and 37% reduction in error
– New state-of-the-art results
SLIDE 90
90
Code & Data
Code
https://github.com/danieldeutsch/derivational-morphology
Data
https://github.com/ryancotterell/derivational-paradigms Powered by
SLIDE 91
91
References
- Cotterell et al. 2017, Paradigm completion for
derivational morphology. In EMNLP
SLIDE 92