MID-TERM FOLLOW-UP Semantic mining: Unsupervised acquisition of - - PowerPoint PPT Presentation
PhD Mid-term Follow-up 16/10/2018 MID-TERM FOLLOW-UP Semantic mining: Unsupervised acquisition of multilingual semantic classes from texts Presenter: Zheng ZHANG Supervisors: Pierre ZWEIGENBAUM & Yue MA
Date
PhD Mid-term Follow-up 16/10/2018
Presenter: Zheng ZHANG Supervisors: Pierre ZWEIGENBAUM & Yue MA Evaluation committee: Vincent CLAVEAU & Alexandre ALLAUZEN
1
Word Co-occurrence Networks Using corpus2graph
2
3
couleurs pomme poire pêche prunelle
clustered by using distributional similarity measures
Introduction of Thesis Topic
4
couleurs pomme poire pêche prunelle
fruits
apple pear peach FR EN
Introduction of Thesis Topic
5
couleurs pomme poire pêche prunelle
fruits
apple pear peach FR EN
Introduction of Thesis Topic
5
couleurs pomme poire pêche prunelle
fruits
apple pear peach FR EN roux
Introduction of Thesis Topic
6
couleurs pomme poire pêche prunelle
fruits
apple pear peach FR EN roux
Introduction of Thesis Topic
6
pomme poire pêche prunelle
fruits
apple pear peach FR EN
Universal class
Introduction of Thesis Topic
7
word sentence document
alignment data level training stage
pre-preprocessing training post-embedding
mikolov2013exploiting artetxe2017learning gouws2015 bilbowa gouws2015simple Luong2015 bilingual levy2017strong
“neural” “count-based”
vulic2015bilingual
Multilingual word embeddings learning can be seen as an extension of (monolingual) word embeddings learning.
Introduction of Thesis Topic
8
9
every term in the Skip-gram objective.
noise distribution using logistic regression, where there are k negative examples for each data sample.
examples.
Results Achieved: GNEG*
*GNEG: Graph-Based Negative Sampling for word2vec, Zheng ZHANG, Pierre ZWEIGENBAUM, In Proceedings of ACL 2018, Melbourne, Australia 10
is only based on the word count.
word_id word_count word_id word_id lg(𝑄𝑜(𝑥))
word count Heat map of the negative examples distribution
Same !
Results Achieved: GNEG*
*GNEG: Graph-Based Negative Sampling for word2vec, Zheng ZHANG, Pierre ZWEIGENBAUM, In Proceedings of ACL 2018, Melbourne, Australia 11
unique terms of the document and whose edges represent co-occurrences between the terms within a fixed-size sliding window.
word co-occurrences for the context word selection, but not for the negative examples selection.
for Single-Document Keyword Extraction. https://safetyapp.shinyapps.io/GoWvis/
Results Achieved: GNEG*
*GNEG: Graph-Based Negative Sampling for word2vec, Zheng ZHANG, Pierre ZWEIGENBAUM, In Proceedings of ACL 2018, Melbourne, Australia 12
word_id word_id lg(𝑥𝑝𝑠𝑒 𝑑𝑝 − 𝑝𝑑𝑑𝑣𝑠𝑠𝑓𝑜𝑑𝑓)
the training words contexts distribution
Heat map of the word co-occurrence distribution
Results Achieved: GNEG*
*GNEG: Graph-Based Negative Sampling for word2vec, Zheng ZHANG, Pierre ZWEIGENBAUM, In Proceedings of ACL 2018, Melbourne, Australia 13
Word co-occurrence network (matrix) generation word2vec training
Results Achieved: GNEG*
*GNEG: Graph-Based Negative Sampling for word2vec, Zheng ZHANG, Pierre ZWEIGENBAUM, In Proceedings of ACL 2018, Melbourne, Australia
14
( “Don’t reinvent the wheel.” )
stemmer, sentence analyzer…)
corpus, tokens; by using multiprocessing)
vocabulary size, sentence analyzer…)
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US
15
corpus2graph processing
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US
16
The history of natural language processing generally started in the 1950s.
Word
& removing punctuation marks and(or) stop words
The histori of natur languag process gener start in the 0000s h n l p g s
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US
17
Word
& removing punctuation marks and(or) stop words
Sentence
analyzer
h n l p g s
dmax =distance=2 distance=1
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US
18
Word
& removing punctuation marks and(or) stop words
Sentence
analyzer
Word pair
h n l p g s
1 1 1 1 1 1 1 1 1 1 1
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US
19
Results Achieved: corpus2graph*
*Efficient Generation and Processing of Word Co-occurrence Networks Using corpus2graph, Zheng ZHANG, Ruiqing YIN, Pierre ZWEIGENBAUM, In Proceedings of NAACL 2018 Workshop on Graph-Based Algorithms for Natural Language Processing, New Orleans, US 20
21
word embeddings learning
Work in Progress
22
sampling is implicitly factorizing a word-context matrix.
Work in Progress
max(PMI( )-log k, 0) SVD
word co-occurrence matrix “enhanced” matrix
23
sampling is implicitly factorizing a word-context matrix.
Work in Progress
max(PMI( )-log k, 0) SVD
word co-occurrence matrix “enhanced” matrix
first order matrix Transition matrix of random walks
23
Work in Progress
first order matrix Transition matrix of random walks word co-occurrence matrix “enhanced” matrix
“+” option 1: matrix concatenation “+” option 2: mask based element wise merge
24
25
29
End of Oct. 2019 Defence Thesis submission Starting writing thesis Finishing future work (programming & experiment part) Finishing 3 works in progress
30