Representing symbolic linguistic structures for neural NLP: methods - - PowerPoint PPT Presentation

Representing symbolic linguistic structures for neural NLP: methods and applications

Alexander Panchenko Assistant Professor for NLP

About myself: a decade of fun R&D in NLP

• 2002-2008: Bauman Moscow State

Technical University, Engineer in Information Systems, MOSCOW

• 2008: Xerox Research Centre Europe,

Research Intern, FRANCE

• 2009-2013: Université catholique de

Louvain, PhD in Computational Linguistics, BELGIUM

• 2013-2015: Startup in SNA, Research

Engineer in NLP , MOSCOW

• 2015-2017: TU Darmstadt, Postdoc in

NLP , GERMANY

• 2017-2019: University of Hamburg,

Postdoc in NLP , GERMANY

• Publications in int’l conferences & journals:
• ACL
• EMNLP
• EACL
• ECIR
• NLE
• Best papers at Representation learning

workshop (ACL’2016) and SemEval’2019.

• Editor and co-chair:
• Cambridge Natural Language Engineering (NLE)
• Springer LNCS/CCIS: AIST conf.
• PC:
• ACL, NAACL, EMNLP

, LREC, RANLP , COLING, …

About myself: a decade of fun R&D in NLP

About myself: my expertise and past/present research foci

• Lexical Semantics
• Semantic similarity
• Word sense

disambiguation

• Word/sense embedding
• Taxonomy induction,
• Frame induction, …
• Argument mining
• Graph clustering

Latest publications (2019-2018) are on argument mining and lexical semantics

• Lexical Semantics
• Semantic similarity
• Word sense

disambiguation

• Word/sense embedding
• Taxonomy induction,
• Frame induction, …
• Argument mining
• Graph clustering

How to inform neural architectures for NLP with symbolic linguistic knowledge?

• Special issue of the Natural Language Engineering journal on informing neural architectures for

NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

How to inform neural architectures for NLP with symbolic linguistic knowledge?

• Special issue of the Natural Language Engineering journal on informing neural architectures for

NLP with linguistic and background knowledge: https://sites.google.com/view/nlesi

Some options:

• Graph embeddings
• Poincaré embeddings
• Regularisers that access the resource
• Structure of neural network is based on the

structure of the resource

• … other specialised embeddings?
• … invented by you?

Text: a sparse symbolic representation

Image source: https://www.tensorflow.org/tutorials/word2vec

Graph: a sparse symbolic representation

Embedding graph into a vector space

From a survey on graph embeddings [Hamilton et al., 2017]:

Learning with an autoencoder

From a survey on graph embeddings [Hamilton et al., 2017]:

A summary of well-known graph embedding algorithms

From a survey on graph embeddings [Hamilton et al., 2017]:

Graph Metric Embeddings

• A short paper at ACL 2019
• Paper: https://arxiv.org/abs/

1906.07040

• Code: http://github.com/uhh-

lt/path2vec

path2vec model

Computational gains compare to graph-based algorithms

Similarity computation: graph vs vectors

path2vec: evaluation results

• n three different graphs

Evaluation on different graphs on SimLex999 (left) and shortest path distance (middle, right).

path2vec evaluation inside a graph- based WSD algorithm (WordNet graph)

Graph embeddings for neural entity linking

• A short paper at ACL 2019

Student Research Workshop (main conference)

• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-sevgilietal- aclsrw-graphemb.pdf

• Code: https://github.com/

uhh-lt/kb2vec

What is Entity Linking?

Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Challenges of Entity Linking

Source of image: https://medium.com/asgard-ai/how-to-enhance-automatic-text-analysis- with-entity-linking-29128a12b

Ambiguity ruin everything: Michael Jordan (NBA) vs Michael Jordan (LDA), etc.

Graph embeddings for neural entity linking

Graph embeddings for neural entity linking

Architecture of our feed-forward neural ED system: using Wikipedia hyperlink graph embeddings as an additional input representation of entity candidates

Graph embeddings for neural entity linking

Graph embeddings for neural entity linking

Poincaré embeddings for various NLP tasks

• ACL 2019 full paper
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-janaetal- aclmain-poincompo.pdf

• Code: https://github.com/

uhh-lt/poincare

Poincaré embeddings for various NLP tasks

Contributions:

• We devise a straightforward and

efficient approach for combining distributional and hypernymy information for the task of noun phrase compositionality

• prediction. As far as we are aware,

this is the first application of Poincaré embeddings to this task.

• We demonstrate consistent and

significant improvements on benchmark datasets in un- supervised and supervised settings.

Poincaré embeddings for various NLP tasks

• Poincaré ball:
• Distance on a ball between two points:
• Image source:

https://arxiv.org/pdf/1705.08039.pdf

Poincaré embeddings for various NLP tasks

Source of the image: https://arxiv.org/pdf/1902.00913.pdf

Training data:

• A set of relations (apple IsA fruit)
• Can be taken from WordNet
• … or extracted from text

Training objective:

Poincaré embeddings for noun compositionally

Evaluation results: comparison to the distributional models hot dog —> food BUT dog —> animal green apple —> fruit AND apple —> fruit

Noun compositionality for the Russian language

• A Balto-Slavic NLP workshop

at ACL 2019

• https://github.com/slangtech/

ru-comps

• … paper to paper soon online.
• A dataset for evaluation of

noun compositionally for Russian.

Poincaré embeddings for taxonomy induction

• A short paper at ACL 2019
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-alyetal- aclshort-hypertaxi.pdf

• Code: https://github.com/

uhh-lt/ Taxonomy_Refinement_Embe ddings

Poincaré embeddings for taxonomy induction

Outline of our taxonomy refinement method:

Poincaré embeddings for taxonomy induction

Comparative Argument Mining

• 6th Workshop on Argument Mining

at ACL 2019.

• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-panchenkoetal- argminingws-compsent.pdf

• Code: https://github.com/uhh-lt/

comparative

• Sentiment analysis ++
• … not only opinions but

also objective arguments.

• … from text only.
• Retrieve pros and cons

to make some informed decisions.

Comparative Argument Mining

Comparative Argument Mining

• Sentiment analysis ++
• … not only opinions but

also objective arguments.

• … from text only.
• Retrieve pros and cons

to make some informed decisions.

Categorizing Comparative Sentences

Contributions:

• We release CompSent-19, a new corpus consisting of 7,199

sentences containing item pairs (27% of the sentences are tagged as comparative and annotated with a preference);

• We present an experimental study of supervised classifiers and a

strong rule-based baseline from prior work.

Categorizing Comparative Sentences

Categorizing Comparative Sentences

Argument Mining Demo

• Demo paper at ACL 2019
• Paper: https://www.inf.uni-

hamburg.de/en/inst/ab/lt/ publications/2019-chernodubetal- acl19demo-targer.pdf

• Code:
• http://github.com/achernodub/

targer/

• https://github.com/uhh-lt/

targer

• Demo: http://

ltdemos.informatik.uni- hamburg.de/targer/

Argument Mining Demo

Argument Mining Demo

Analyze Text: input field, drop-down model selection, colorized labels, and tagged result.

Argument Mining Demo

Search Arguments: query box, field selectors, and result with link to the original document.

Argument Mining Demo

Interested in collaboration (BA/MA/PhD)

• n the following topics in the form of

co-supervision:

• Argument Mining
• Entity Linking
• Graph Embeddings
• Knowledge bases and lexical

resources for neural NLP

• Word sense induction and

disambiguation; lexical substitution

• Relation extraction

Image source: https://metro.co.uk/2018/02/08/freemasons-definitely-do- have-a-secret-handshake-but-they-wont-tell-us-what-it-is-7295849/