Representing symbolic linguistic structures for neural NLP: methods - - PowerPoint PPT Presentation

Representing symbolic linguistic structures for neural NLP: methods and applications

Alexander Panchenko Assistant Professor for NLP

Structure and goals of this talk

• Publishing in ACL and similar conferences, e.g. NAACL, EMNLP

, CoNLL:

• this is the top conference in NLP —> your work is visible
• Topics of this talk (all are based on forthcoming publications at ACL’19 and associated workshops):
• Encoding and using linguistic structures in neural NLP models
• Argument Mining

About myself: a decade of fun R&D in NLP

• 2019-now: Skoltech, Assistant Professor in

NLP , MOSCOW

• 2017-2019: University of Hamburg, Postdoc

in NLP , GERMANY

• 2015-2017: TU Darmstadt, Postdoc in NLP

, GERMANY

• 2013-2015: Startup in SNA, Research

Engineer in NLP , MOSCOW

• 2009-2013: Université catholique de Louvain,

PhD in Computational Linguistics, BELGIUM

• 2008: Xerox Research Centre Europe,

Research Intern, FRANCE

• 2002-2008: Bauman Moscow State Technical

University, Engineer in Information Systems, MOSCOW

• Publications in int’l conferences & journals:
• ACL
• EMNLP
• EACL
• ECIR
• NLE
• Best papers at Representation learning

workshop (ACL’2016) and SemEval’2019.

• Editor and co-chair:
• Cambridge Natural Language Engineering (NLE)
• Springer LNCS/CCIS: AIST conf.
• PC:
• ACL, NAACL, EMNLP

, CoNLL, LREC, RANLP , COLING, …

About myself: a decade of fun R&D in NLP

How to inform neural architectures for NLP with symbolic linguistic knowledge?

• Special issue of the Natural Language Engineering journal on informing neural architectures for

NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

How to inform neural architectures for NLP with symbolic linguistic knowledge?

• Special issue of the Natural Language Engineering journal on informing neural architectures for

NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

Some options:

• Graph embeddings
• Poincaré embeddings
• Regularisers that access the resource
• Structure of neural network is based on the

structure of the resource

• … other specialised embeddings?
• … invented by you?

Text: a sparse symbolic representation

Image source: https://www.tensorflow.org/tutorials/word2vec

Graph: a sparse symbolic representation

Embedding graph into a vector space

From a survey on graph embeddings [Hamilton et al., 2017]:

Learning with an autoencoder

From a survey on graph embeddings [Hamilton et al., 2017]:

A summary of well-known graph embedding algorithms

From a survey on graph embeddings [Hamilton et al., 2017]:

Graph Metric Embeddings

• A short paper at ACL 2019
• Paper: https://arxiv.org/abs/

1906.07040

• Code: http://github.com/uhh-

lt/path2vec

path2vec model

Computational gains compare to graph-based algorithms

Similarity computation: graph vs vectors

path2vec: evaluation results

• n three different graphs

Evaluation on different graphs on SimLex999 (left) and shortest path distance (middle, right).

path2vec evaluation inside a graph- based WSD algorithm (WordNet graph)

Graph embeddings for neural entity linking

• A short paper at ACL 2019

Student Research Workshop (main conference)

• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-sevgilietal- aclsrw-graphemb.pdf

• Code: https://github.com/

uhh-lt/kb2vec

What is Entity Linking?

Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Challenges of Entity Linking

Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Ambiguity ruin everything: Michael Jordan (NBA) vs Michael Jordan (LDA), etc.

Graph embeddings for neural entity linking

Graph embeddings for neural entity linking

Architecture of our feed-forward neural ED system: using Wikipedia hyperlink graph embeddings as an additional input representation of entity candidates

Graph embeddings for neural entity linking

end2end Entity Linking Model by Kolistas et al. (2018)

• The final score is used for

both the mention linking and entity disambiguation decisions.

• SOTA entity linking results.

Graph embeddings for neural entity linking

Poincaré embeddings for various NLP tasks

• ACL 2019 full paper
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-janaetal- aclmain-poincompo.pdf

• Code: https://github.com/

uhh-lt/poincare

Poincaré embeddings for various NLP tasks

Contributions:

• We devise a straightforward and

efficient approach for combining distributional and hypernymy information for the task of noun phrase compositionality

• prediction. As far as we are aware,

this is the first application of Poincaré embeddings to this task.

• We demonstrate consistent and

significant improvements on benchmark datasets in un- supervised and supervised settings.

Poincaré embeddings for various NLP tasks

• Poincaré ball:
• Distance on a ball between two points:
• Image source:

https://arxiv.org/pdf/1705.08039.pdf

Poincaré embeddings for various NLP tasks

Source of the image: https://arxiv.org/pdf/1902.00913.pdf

Training data:

• A set of relations (apple IsA fruit)
• Can be taken from WordNet
• … or extracted from text

Training objective:

Poincaré embeddings for noun compositionally

Evaluation results: comparison to the distributional models hot dog —> food BUT dog —> animal green apple —> fruit AND apple —> fruit

Noun compositionality for the Russian language

• The Balto-Slavic NLP

workshop at ACL 2019

• Paper: http://panchenko.me/

papers/bsnlp19.pdf

• Code: https://github.com/

slangtech/ru-comps

• A dataset for evaluation of

noun compositionally for Russian.

Noun Compositionality for Russian: Results

Poincaré embeddings for taxonomy induction

• A short paper at ACL 2019
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-alyetal- aclshort-hypertaxi.pdf

• Code: https://github.com/

uhh-lt/ Taxonomy_Refinement_Embe ddings

Abandoned children in a taxonomy problem

Attaching unconnected nodes in taxonomy provides large boosts in performance:

Poincaré embeddings for taxonomy induction

Outline of our taxonomy refinement method:

Poincaré embeddings for taxonomy induction

Comparative Argument Mining

• 6th Workshop on Argument Mining

at ACL 2019.

• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-panchenkoetal- argminingws-compsent.pdf

• Code: https://github.com/uhh-lt/

comparative

• Sentiment analysis ++
• … not only opinions but

also objective arguments.

• … from text only.
• Retrieve pros and cons

to make some informed decisions.

Comparative Argument Mining

Comparative Argument Mining

• Sentiment analysis ++
• … not only opinions but

also objective arguments.

• … from text only.
• Retrieve pros and cons

to make some informed decisions.

Categorizing Comparative Sentences

Contributions:

• We release a new dataset consisting of 7,199 sentences

containing item pairs (27% of the sentences are tagged as comparative and annotated with a preference);

• We present an experimental study of supervised classifiers and a

strong rule-based baseline from prior work.

Categorizing Comparative Sentences

Categorizing Comparative Sentences

Categorizing Comparative Sentences

Argument Mining Demo

• Demo paper at ACL 2019
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-chernodubetal- acl19demo-targer.pdf

• Code:
• A neural sequence tagger:

https://github.com/uhh-lt/targer

• A web application for AM: http://

github.com/achernodub/targer/

• Demo: http://

ltdemos.informatik.uni- hamburg.de/targer/

Argument Mining Demo

Tagger:

• BiLSTM-CNN-CRF
• A custom PyTorch implementation
• CoNLL as input: can be easily used for any sequence labelling task

Argument Mining Demo

Analyze Text: input field, drop-down model selection, colorised labels, and tagged result. http://ltdemos.informatik.uni-hamburg.de/targer/

Argument Mining Demo

Search Arguments: query box, field selectors, and result with link to the original document. http://ltdemos.informatik.uni-hamburg.de/targer/

Argument Mining Demo

How to publish NLP research in conferences like *ACL?

Select a relevant topic:

• “Hot topic” may boost interest to your work
• … but many people may be working on it at the same time (you need to be

fast).

• … especially if the idea is fairly straightforward extension of existing stuff.

Collaborate:

• Find strong collaborators which already published in the conferences you

are aiming at.

• Ideally your competences should complement one another.
• Splitting work into “experiments”, “writing”, “related work”, etc.

How to publish NLP research in conferences like *ACL?

Prepare it in advance:

• A deadline for a major conference happens once a year: be sure

*almost everything* is ready one week before the deadline.

• Avoid last minute submissions: a waste of your time and that of

reviewers. Read to understand rules of the community:

• The more recent papers from the conference you submit to you read

the better you understand how to write a paper for this conference.

• Some “must-have” rules may be not verbalised (and may be

different in different communities).

How to publish NLP research in conferences like *ACL?

ACL paper checklist:

• 1. Paper is written in a reasonably good English (use grammary.com, ask colleagues to read it)
• 2. Include explicitly the list of “contributions” in the intro.
• 3. Related work section is very important - it describes all most recent/strong alternatives to

your method. Potentially, you acknowledge work of people who will review your work.

• 4. Comparison to the SOTA - ideally you compare to all of these most recent/strong

alternatives of your method and show improvements.

• 5. Statistical significance: you not only report the numbers, but perform statistical testing.
• 6. Your paper is self-contained and is understandable without special knowledge.
• 7. In the introduction, your hypothesis/idea is clearly stated. Why should somebody read your

paper? What s/he will learn?

• 8. Paper is anonymised and the number of self-references is minimal.
• 9. Short paper: 1-2 experiments; Long paper: 2-3 experiments. Ideally you should have both

intrinsic and extrinsic experiments.

Interested in collaboration (BA/MA/PhD)

• n the following topics in the form of

co-supervision/joint project proposals on:

• Argument Mining
• Entity Linking
• Graph Embeddings
• Knowledge bases and lexical

resources for neural NLP

• Word sense induction and

disambiguation; lexical substitution

• Relation extraction
• Dialogue Systems

Image source: https://metro.co.uk/2018/02/08/freemasons-definitely-do- have-a-secret-handshake-but-they-wont-tell-us-what-it-is-7295849/

Drop me a line if interested: A.Panchenko@Skoltech.ru