Supervised word sense disambiguation on polysemy with bidirectional - - PowerPoint PPT Presentation
Supervised word sense disambiguation on polysemy with bidirectional LSTM: A case study of BUN in Taiwan Hakka Huei-Ling Lai, Hsiao-Ling Hsu, Jyi-Shane Liu, Chia-Hung Lin and Yanhong Chen National Chengchi University, Taiwan Presented at The 21st
Supervised word sense disambiguation
A case study of BUN in Taiwan Hakka
Huei-Ling Lai, Hsiao-Ling Hsu, Jyi-Shane Liu, Chia-Hung Lin and Yanhong Chen National Chengchi University, Taiwan Presented at The 21st Chinese Lexical Semantics Workshop (CLSW2020), City University of Hong Kong, Hong Kong. May 28-30, 2020
TABLE OF CONTENTS
Introduction Related work
01 03 02 04 05 06
Methods
Overall Architecture
Input layers
Output layers
Experiments
Dataset and Evaluation Metrics
Results and Analysis
Conclusion and Future Work References
Polysemous BUN
Bidirectional LSTM for WSD task in Taiwan Hakka
INTRODUCTION
Polysemy is ubiquitous in languages may trigger some problems in some of the NLP tasks
performed by machines, for instance, part-of-speech (POS) tagging
Word sense disambiguation (WSD) Navigli (2009:3) defines WSD as ‘the ability to computationally
determine which sense of a word is activated by its use in a particular context ’.
Introduction
In the extant literature
WSD focuses on a few dominant languages, such as English and Chinese.
Findings based on a few dominant languages may lead to narrow applications.
Low-resource languages still receive little attention, for most WSD tasks are trained by supervised learning, which requires a large amount of labeled data : expensive and time-consuming
A language-specific WSD system is in need to implement in low-
resource languages, for instance, in Taiwan Hakka.
Introduction
Polysemous phenomena in Taiwan Hakka have been probed into by
quite a few studies.
A part-of-speech (POS) tagging system needs to be established for
classification and categorization of the corpus data.
Introduction
Our aim: to schematize a workable coding framework from integrating and
modifying the findings of previous studies on Taiwan Hakka polysemous phenomena
to develop a model for automatic polysemous word sense
disambiguation for Taiwan Hakka corpus
Introduction
Related work
Polysemous BUN in Taiwan Hakka
Related work
Table 1. The various usages of BUN.
Bidirectional LSTM for WSD task in Taiwan Hakka
and designed to simulate human understanding with statistical procedures to capture patterns of co-occurrences of words in context.
the task of word sense disambiguation based on the contextual hypotheses for words and senses (cf. Li and Jurafsky, 2015; Peters, et al., 2018).
employing a supervised and a semi-supervised LSTM neural network model.
Related work
Bidirectional LSTM for WSD task in Taiwan Hakka
○
We employ a bidirectional LSTM (Graves & Schmidhuber, 2005; Graves et al., 2013) and train the model on labeled data annotated by human to disambiguate and predict the sense of BUN.
○
In disambiguating the sense of polysemous BUN, the contextual and syntactic information of BUN are crucial and should be taken into account.
○
The basic idea of Bi-LSTM is to capture past and future information by presenting them to two hidden states and then the two hidden states are concatenated to form the final output.
Related work
Methods
Overall Architecture
Methods
Overall Architecture
Methods
Table 1. The number of tokens and types in each dataset Dataset Token
Type
Dataset 1 (manually annotated instances containing BUN ) Word embedding
64,278 5,695
Character embedding
89,126 2,164
POS
64,103
24 Dataset 2 (MOE Read Out Loud Tests ) Word embedding
68,012 7,322
Dataset 3 (Hans Christian Andersen's fairy tales (translation in Hakka)) Character embedding 835,534
3,910
Table 2. The number of tokens and types in each dataset
Experiments
Dataset and Evaluation Metrics
Experiments
Table 3. The occurrences of VA1, VA2, P1, and P2 in dataset 1. Dataset 1 Label Occurrence Training Set (Training + Dev) VA1 66 (4%) VA2 753 (46%) P1 238 (15%) P2 576 (35%) Subtotal 1,633 (100%) Test Set VA1 7 (4%) VA2 75 (46%) P1 24(15%) P2 57 (35%) Subtotal 163 (100%) Table 4. The number of samples in the training set, dev set and test set Dataset 1 Token Type Training (around 80%) Word embedding 14,410 2,463 Character embedding 14,410 1,226 POS 14,410 24 Dev (around 10%) Word embedding 1,610 672 Character embedding 1,610 513 POS 1,610 23 Test (around 10%; fixed) Word embedding 1,630 652 Character embedding 1,630 508 POS 1,630 22 Total (100%) Word embedding 17,650 2,708 Character embedding 17,650 1,283 POS 17,650 24
Results and Analysis
Experiments
Results and Analysis
Experiments
Results and Analysis
Experiments
Results and Analysis
Experiments
larger corpus data is needed sentence embeddings (cf. Wang and Chang, 2016)
Conclusion and Future Work
In this study, we propose a WSD model on the classification of polysemy in
Taiwan Hakka, a low-resource language in the world, especially in the field
The model proposed is a supervised bidirectional LSTM model trained and
tested on a small amount of labeled data.
Four kinds of input features are utilized POS only POS + word embeddings POS + character embeddings POS + word embeddings+ character embeddings the best
performance is achieved
Conclusion and Future Work
To enhance the robustness and stability of the model, we will design and
include other possible parameters to compare and contrast the performance of the experiments.
To test the model with different window spans (from L1R1 to L10R10)
and/or with whole sentences as inputs.
To improve the research design, we will try random selection without
considering the overall distribution of the four labels on the test set for other experiments in the future.
To test the model on other polysemous words in Taiwan Hakka.
Conclusion and Future Work
References
1.
Chiang, M. H. (2016). The Functions and Origin of Locative TU5 in Hailu Hakka, with a Note on the Origin of the Durative Marker TEN3. Bulletin of Chinese Linguistics, 9(1), 95- 120.
2.
Graves, A. and Schmidhuber, J. (2005). Framewise phoneme classification with bidirectional lstm networks. In Proceedings of the 2005 International Joint Conference on Neural Networks. Montreal, Canada.
3.
Graves, A., Mohamed, A., & Hinton, G. (2013). Speech recognition with deep recurrent neural
4.
Huang, H. C. (2014). Semantic Extensions and the Convergence of the Beneficiary Role: A Case Study of BUN and Lau in Hakka. Concentric: Studies in Linguistics, 40(1), 65-94.
5.
Iacobacci, I., Pilehvar, M. T., & Navigli, R. (2016). Embeddings for word sense disambiguation: An evaluation study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 897-907).
6.
Lai, H. L. (2001). On Hakka BUN: A case of polygrammaticalization. Language and Linguistics, 2(2), 137-153.
References
7.
Lai, H. L. (2015). Profiling Hakka BUN1 Causative Constructions. Language and Linguistics, 16(3), 369-395.
8.
Li, J., & Jurafsky, D. (2015). Do multi-sense embeddingss improve natural language understanding? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).
9.
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013a). Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR.
41(2), 1-69.
Deep contextualized word representations. In Proceeding of NAACL.
References
DI, DO, BUN, LAU, TUNG, ZIONG. Concentric: Studies in Linguistics, 38(2), 171-209.
sense disambiguation with neural models. In Proceeding of COLING, 1374–1385.
Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 2306-2315).
References