SLIDE 1
1
2
Zero-shot Sequence Labeling: Transferring Knowledge from Sentences - - PowerPoint PPT Presentation
Zero-shot Sequence Labeling: Transferring Knowledge from Sentences - - PowerPoint PPT Presentation
1 Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens Marek Rei Anders Sgaard 2 Sequence labeling Error detection: + + + - + + + + + - + I like to
SLIDE 2
SLIDE 3
3
Zero-shot sequence labeling
+ + + - + + + + + - + I like to playing the guitar and sing very louder . It was so long time to wait in the theatre . I look forward to receiving reply to my enquiry . This is a great opportunity to learn more about whales . Therefore, houses will be built on high supports .
SLIDE 4
4
Main idea
Neural sentence classification model
01 02 03
Make attention weights behave like sequence labeling
- utput
Based on self-attention
SLIDE 5
5
Model architecture
SLIDE 6
6
Based on softmax: Based on sigmoid + normalisation:
Soft attention weights
SLIDE 7
7
Optimising the attention
We can constrain the attention values based on the sentence-level label. 1. Only some, but not all, tokens in the sentence can have a positive label. 2. There are positive tokens in a sentence only if the overall sentence is positive.
SLIDE 8
8
Alternative methods
1. Labeling through backpropagation 2. Relative frequency 3. Supervised sequence labeling
Selvaraju et al (2016)
SLIDE 9
9
Evaluation: CoNLL 2010
Detection of uncertain language in scientific articles
SLIDE 10
10
Evaluation: FCE
Detecting grammatical errors in essays written by language learners.
SLIDE 11
11
Examples
SLIDE 12
12
Applications
Sequence labeling without data
01
Data exploration and feature analysis
02
Model visualisation and interpretation
03
SLIDE 13
13