Deep Learning for Broad Coverage Semantics: SRL, Coreference, and - - PowerPoint PPT Presentation
Deep Learning for Broad Coverage Semantics: SRL, Coreference, and Beyond Luke Zettlemoyer * Joint work with Luheng He , Kenton Lee , Matthew Peters* , Christopher Clark , Matthew Gardner*, Mohit Iyyer* , Mandar Joshi , Mike Lewis
Deep Learning for Broad Coverage Semantics: SRL, Coreference, and Beyond
Joint work with Luheng He†, Kenton Lee†, Matthew Peters*, Christopher Clark†, Matthew Gardner*, Mohit Iyyer*, Mandar Joshi†, Mike Lewis‡, Julian Michael†, Mark Neumann*
† Paul G. Allen School of Computer Science & Engineering, University of Washington,
‡ Facebook AI Research * Allen Institute for Artificial Intelligence
Luke Zettlemoyer†*
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3:
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3: Gather lots of training data!
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3:
Gather lots of training data!
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3:
Gather lots of training data! Apply Deep Learning!!
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3:
Gather lots of training data! Apply Deep Learning!!
Three Simple Steps that will Revolutionize Your ML Research
Step 1: Step 2: Step 3:
Gather lots of training data! Apply Deep Learning!! Observe Impressive Gains!!!
Coreference: clustering NPs
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100
killed in the crush as workers tried to flee the blaze in the four-story building.
ARG0 NASA PRED
ARG1 an X-ray flare 400 times brighter than usual TMP On January 5, 2015
Example Tasks:
Many applications:
Semantic Role Labeling: who did what, etc.
Question Answering Information Extraction Machine Translation
Does the Recipe Work for Broad Coverage Semantics?
Step 1: Gather lots of training data! Step 2: Apply Deep Learning!! Step 3: Observe Impressive Gains!!!
Does the Recipe Work for Broad Coverage Semantics?
Step 1: Gather lots of training data! Step 2: Apply Deep Learning!! Step 3: Observe Impressive Gains!!!
Challenge 1: Data is costly and limited (e.g. linguists required to label PennTreebank / OntoNotes)
Does the Recipe Work for Broad Coverage Semantics?
Step 1: Gather lots of training data! Step 2: Apply Deep Learning!! Step 3: Observe Impressive Gains!!!
Challenge 1: Data is costly and limited (e.g. linguists required to label PennTreebank / OntoNotes) Challenge 2: Pipeline of structured prediction problems with cascading errors (e.g. POS->Parsing->SRL->Coref)
Semantic Role Labeling: Coreference:
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100
killed in the crush as workers tried to flee the blaze in the four-story building.
ARG0 NASA PRED
ARG1 an X-ray flare 400 times brighter than usual TMP On January 5, 2015
New state-of-the-art results for two tasks:
Common themes:
no extra training data
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
predicate argument role label
who what when where why …
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
v subj
prep subj v prep adv
predicate argument role label
who what when where why …
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
v subj
prep subj v prep adv thing broken thing broken
predicate argument role label
who what when where why …
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
v subj
prep subj v prep adv thing broken thing broken breaker instrument pieces (final state) temporal
predicate argument role label
who what when where why …
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
v subj
prep Frame: break.01
role description
ARG0 breaker ARG1 thing broken ARG2 instrument ARG3 pieces ARG4 broken away from what? subj v prep adv thing broken thing broken breaker instrument pieces (final state) temporal
predicate argument role label
who what when where why …
My mug broke into pieces immediately. The robot broke my favorite mug with a wrench.
Semantic Role Labeling (SRL)
v subj
prep Frame: break.01
role description
ARG0 breaker ARG1 thing broken ARG2 instrument ARG3 pieces ARG4 broken away from what? subj v prep adv thing broken thing broken breaker instrument pieces (final state) temporal ARG0 ARG1 ARG2 ARG3 ARG1 ARGM-TMP
predicate argument role label
who what when where why …
80.3 (Toutanova et al, 2005) — 80.3 (FitzGerald et al, 2015)
Syntactic alternation Prepositional phrase attachment Long-range dependencies and common sense
SRL Systems
syntactic features candidate argument spans labeled arguments prediction labeling ILP/DP sentence, predicate argument id.
Pipeline Systems
Punyakanok et al., 2008 Täckström et al., 2015 FitzGerald et al., 2015
SRL Systems
syntactic features candidate argument spans labeled arguments prediction labeling ILP/DP sentence, predicate argument id.
Pipeline Systems
Punyakanok et al., 2008 Täckström et al., 2015 FitzGerald et al., 2015 sentence, predicate BIO sequence prediction
Deep BiLSTM + CRF layer
Viterbi context window features
End-to-end Systems
Collobert et al., 2011 Zhou and Xu, 2015 Wang et. al, 2015
SRL Systems
syntactic features candidate argument spans labeled arguments prediction labeling ILP/DP sentence, predicate argument id.
Pipeline Systems
Punyakanok et al., 2008 Täckström et al., 2015 FitzGerald et al., 2015 sentence, predicate BIO sequence prediction
Deep BiLSTM + CRF layer
Viterbi context window features
End-to-end Systems
Collobert et al., 2011 Zhou and Xu, 2015 Wang et. al, 2015 Deep BiLSTM
Hard constraints
BIO sequence prediction sentence, predicate
*This work
He et al., 2017
The cats love hats .
Input (sentence and predicate):
SRL as BIO Tagging Problem
The cats love hats .
Input (sentence and predicate): BIO output:
B-ARG0 I-ARG0 B-V I-ARG1 O
(Begin, Inside, Outside)
SRL as BIO Tagging Problem
The cats love hats .
Input (sentence and predicate): BIO output:
B-ARG0 I-ARG0 B-V I-ARG1 O
Final SRL output:
ARG0 V ARG1
(Begin, Inside, Outside)
SRL as BIO Tagging Problem
the cats love hats [ ] [ ] [V] [ ]
B-ARG0 0.4 I-ARG0 0.05 B-ARG1 0.5 I-ARG1 0.03 … … B-ARG0 0.1 I-ARG0 0.5 B-ARG1 0.1 I-ARG1 0.2 … … B-ARG0 0.001 I-ARG0 0.001 B-ARG1 0.001 … … B-V 0.95 B-ARG0 0.1 I-ARG0 0.1 B-ARG1 0.7 I-ARG1 0.2 … …
[He et al, 2017]
the cats love hats [ ] [ ] [V] [ ]
B-ARG0 0.4 I-ARG0 0.05 B-ARG1 0.5 I-ARG1 0.03 … … B-ARG0 0.1 I-ARG0 0.5 B-ARG1 0.1 I-ARG1 0.2 … … B-ARG0 0.001 I-ARG0 0.001 B-ARG1 0.001 … … B-V 0.95 B-ARG0 0.1 I-ARG0 0.1 B-ARG1 0.7 I-ARG1 0.2 … …
(1) Deep BiLSTM tagger
[He et al, 2017]
the cats love hats [ ] [ ] [V] [ ]
B-ARG0 0.4 I-ARG0 0.05 B-ARG1 0.5 I-ARG1 0.03 … … B-ARG0 0.1 I-ARG0 0.5 B-ARG1 0.1 I-ARG1 0.2 … … B-ARG0 0.001 I-ARG0 0.001 B-ARG1 0.001 … … B-V 0.95 B-ARG0 0.1 I-ARG0 0.1 B-ARG1 0.7 I-ARG1 0.2 … …
(1) Deep BiLSTM tagger (2) Highway connections
[He et al, 2017]
the cats love hats [ ] [ ] [V] [ ]
B-ARG0 0.4 I-ARG0 0.05 B-ARG1 0.5 I-ARG1 0.03 … … B-ARG0 0.1 I-ARG0 0.5 B-ARG1 0.1 I-ARG1 0.2 … … B-ARG0 0.001 I-ARG0 0.001 B-ARG1 0.001 … … B-V 0.95 B-ARG0 0.1 I-ARG0 0.1 B-ARG1 0.7 I-ARG1 0.2 … …
(1) Deep BiLSTM tagger (2) Highway connections (3) Variational dropout
[He et al, 2017]
the cats love hats [ ] [ ] [V] [ ]
B-ARG0 0.4 I-ARG0 0.05 B-ARG1 0.5 I-ARG1 0.03 … … B-ARG0 0.1 I-ARG0 0.5 B-ARG1 0.1 I-ARG1 0.2 … … B-ARG0 0.001 I-ARG0 0.001 B-ARG1 0.001 … … B-V 0.95 B-ARG0 0.1 I-ARG0 0.1 B-ARG1 0.7 I-ARG1 0.2 … …
(1) Deep BiLSTM tagger (2) Highway connections (4) Viterbi decoding with hard constraints (3) Variational dropout
[He et al, 2017]
Other Implementation Details …
matrices (Saxe et al., 2013)
(Hinton 2002)
F1 60 65 70 75 80 85 90 Ours* Ours Zhou FitzGerald* Täckström Toutanova* Punyakanok*
WSJ Test Brown (out-domain) Test
*:Ensemble models
2017 2015 2015 2015 2017 2008 2008 Datasets
CoNLL 2005 Results
CoNLL 2012 (OntoNotes) Results Ablations
F1 60 65 70 75 80 85 90 Ours* Ours Zhou FitzGerald* Täckström Toutanova* Punyakanok*
WSJ Test Brown (out-domain) Test
79.4 80.3 79.9 80.3 82.8 83.1 84.6 *:Ensemble models
2017 2015 2015 2015 2017 2008 2008 Datasets
CoNLL 2005 Results
CoNLL 2012 (OntoNotes) Results Ablations
F1 60 65 70 75 80 85 90 Ours* Ours Zhou FitzGerald* Täckström Toutanova* Punyakanok*
WSJ Test Brown (out-domain) Test
67.8 68.8 71.3 72.2 69.4 72.1 73.6 79.4 80.3 79.9 80.3 82.8 83.1 84.6 *:Ensemble models
2017 2015 2015 2015 2017 2008 2008 Datasets
CoNLL 2005 Results
CoNLL 2012 (OntoNotes) Results Ablations
F1 60 65 70 75 80 85 90 Ours* Ours Zhou FitzGerald* Täckström Toutanova* Punyakanok*
WSJ Test Brown (out-domain) Test
67.8 68.8 71.3 72.2 69.4 72.1 73.6 79.4 80.3 79.9 80.3 82.8 83.1 84.6 Pipeline models BiLSTM models *:Ensemble models
2017 2015 2015 2015 2017 2008 2008 Datasets
CoNLL 2005 Results
CoNLL 2012 (OntoNotes) Results Ablations
Ablations on Number of Layers (2,4,6 and 8)
13
F1 on CoNLL-05 Dev. 70 75 80 85 L2 L4 L6 L8
Greedy decoding Viterbi decoding
80.5 80.1 79.1 74.6
Ablations on Number of Layers (2,4,6 and 8)
13
F1 on CoNLL-05 Dev. 70 75 80 85 L2 L4 L6 L8
Greedy decoding Viterbi decoding
81.6 81.4 80.5 77.2 80.5 80.1 79.1 74.6
Ablations on Number of Layers (2,4,6 and 8)
13
F1 on CoNLL-05 Dev. 70 75 80 85 L2 L4 L6 L8
Greedy decoding Viterbi decoding
81.6 81.4 80.5 77.2 80.5 80.1 79.1 74.6 Performance increases as model goes deeper. Biggest jump from 2 to 4 layer.
Ablations on Number of Layers (2,4,6 and 8)
13
F1 on CoNLL-05 Dev. 70 75 80 85 L2 L4 L6 L8
Greedy decoding Viterbi decoding
81.6 81.4 80.5 77.2 80.5 80.1 79.1 74.6
Shallow models benefit more from constrained decoding.
Performance increases as model goes deeper. Biggest jump from 2 to 4 layer.
Semantic Role Labeling: Coreference:
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100
killed in the crush as workers tried to flee the blaze in the four-story building.
ARG0 NASA PRED
ARG1 an X-ray flare 400 times brighter than usual TMP On January 5, 2015
New state-of-the-art results for two tasks:
Common themes:
no extra training data
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Input document
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Input document
Cluster #1 A fire in a Bangladeshi garment factory the blaze in the four-story building
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building.
Cluster #1 A fire in a Bangladeshi garment factory the blaze in the four-story building Cluster #2 a Bangladeshi garment factory the four-story building
Input document
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building.
Cluster #1 A fire in a Bangladeshi garment factory the blaze in the four-story building Cluster #2 a Bangladeshi garment factory the four-story building Cluster #3 at least 37 people the deceased
Input document
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Input document
Mention detection
A fire in a Bangladeshi garment factory at least 37 people … the four-story building
Mention clustering
Cluster #1 A fire in a Bangladeshi garment factory the blaze in the four-story building Cluster #2 a Bangladeshi garment factory the four-story building Cluster #3 at least 37 people the deceased
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Candidate mentions A fire in a Bangladeshi garment factory garment factory at least 37 people dead and 100 hospitalized … Input document Hand-engineered rules Syntactic parser Mention #1 Mention #2 Coreferent? A fire in a Bangladeshi garment factory garment ✓/✗ garment factory ✓/✗ factory at least 37 people dead and 100 hospitalized ✓/✗ … … ✓/✗
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Candidate mentions A fire in a Bangladeshi garment factory garment factory at least 37 people dead and 100 hospitalized … Input document Hand-engineered rules Syntactic parser Mention #1 Mention #2 Coreferent? A fire in a Bangladeshi garment factory garment ✓/✗ garment factory ✓/✗ factory at least 37 people dead and 100 hospitalized ✓/✗ … … ✓/✗
Mention clustering: main source of improvement for many years!
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Candidate mentions A fire in a Bangladeshi garment factory garment factory at least 37 people dead and 100 hospitalized … Input document Hand-engineered rules Syntactic parser Mention #1 Mention #2 Coreferent? A fire in a Bangladeshi garment factory garment ✓/✗ garment factory ✓/✗ factory at least 37 people dead and 100 hospitalized ✓/✗ … … ✓/✗
Relies on parser for:
General Electric said the Postal Service contacted the company
Bidirectional LSTM Word & character embeddings
General Electric said the Postal Service contacted the company the Postal Service
+
Bidirectional LSTM Word & character embeddings Span representation
Bidirectional LSTM Word & character embeddings Span representation
General Electric said the Postal Service contacted the company the Postal Service
Boundary representations
General Electric said the Postal Service contacted the company the Postal Service
+
Bidirectional LSTM Word & character embeddings Head-finding attention Span representation
Attention mechanism to learn headedness
Bidirectional LSTM Word & character embeddings Head-finding attention Span representation
General Electric said the Postal Service contacted the company General Electric
+
Electric said the
+
the Postal Service
+
Service contacted the
+
the company
+
Compute all span representations
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Witnesses say the only exit door was on the ground floor, and that it was locked when the fire broke out.
Every span independently chooses an antecedent
Input document
y3 ∈ {✏, 1, 2}
✏
Span Antecedent 1 A 2 A fire 3 A fire in … … … M
y3 y2 y1 yM
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Witnesses say the only exit door was on the ground floor, and that it was locked when the fire broke out. Input document
Span Antecedent ( ) A A fire … … a Bangladeshi garment factory … … the four-story building a Bangladeshi garment factory … …
✏ ✏ ✏ ✏
yi
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Witnesses say the only exit door was on the ground floor, and that it was locked when the fire broke out. Input document
Span Antecedent ( ) A A fire … … a Bangladeshi garment factory … … the four-story building a Bangladeshi garment factory … …
✏ ✏ ✏ ✏
yi
Not a mention
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Witnesses say the only exit door was on the ground floor, and that it was locked when the fire broke out. Input document
Span Antecedent ( ) A A fire … … a Bangladeshi garment factory … … the four-story building a Bangladeshi garment factory … …
✏ ✏ ✏ ✏
yi
No link with previously occurring span
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. Witnesses say the only exit door was on the ground floor, and that it was locked when the fire broke out. Input document
Span Antecedent ( ) A A fire … … a Bangladeshi garment factory … … the four-story building a Bangladeshi garment factory … …
✏ ✏ ✏ ✏
yi
Predicted coreference link
P(y1, . . . , yM | D) =
M
Y
i=1
P(yi | D) =
M
Y
i=1
es(i,yi) P
y0∈Y(i) es(i,y0)
s(i, j) = ( sm(i) + sm(j) + sa(i, j) j 6= ✏ j = ✏
Factor coreference score to enable span pruning:
s(i, j)
P(y1, . . . , yM | D) =
M
Y
i=1
P(yi | D) =
M
Y
i=1
es(i,yi) P
y0∈Y(i) es(i,y0)
s(i, j) s(i, j) = ( sm(i) + sm(j) + sa(i, j) j 6= ✏ j = ✏
Factor coreference score to enable span pruning:
s(i, j)
Is this span a mention?
P(y1, . . . , yM | D) =
M
Y
i=1
P(yi | D) =
M
Y
i=1
es(i,yi) P
y0∈Y(i) es(i,y0)
s(i, j) s(i, j) = ( sm(i) + sm(j) + sa(i, j) j 6= ✏ j = ✏
Factor coreference score to enable span pruning:
s(i, j)
Is span j an antecedent of span i?
s(i, j)
Dummy antecedent has a fixed zero score
s(i, j) = ( sm(i) + sm(j) + sa(i, j) j 6= ✏ j = ✏
Factor coreference score to enable span pruning:
s(i, j) P(y1, . . . , yM | D) =
M
Y
i=1
P(yi | D) =
M
Y
i=1
es(i,yi) P
y0∈Y(i) es(i,y0)
Dataset: English OntoNotes (CoNLL-2012) Genres: Telephone conversations, newswire, newsgroups, broadcast conversation, broadcast news, weblogs Documents: 2802 training, 343 development, 348 test Aggressive pruning: Maximum span width, maximum sentence training, suppress spans with inconsistent bracketing, maximum number of antecedents Features: distance between spans, span width Metadata: speaker information, genre Longest document has 4009 words!
Test Avg. F1 (%) 50.0 54.0 58.0 62.0 66.0 70.0 Durrett & Klein (2013) Björkelund & Kuhn (2014) Martschat & Strube (2015)
62.5 61.6 60.3
Linear models
Test Avg. F1 (%) 50.0 54.0 58.0 62.0 66.0 70.0 Durrett & Klein (2013) Björkelund & Kuhn (2014) Martschat & Strube (2015) Wiseman et al. (2016) Clark & Manning (2016)
65.7 64.2 62.5 61.6 60.3
Linear models Neural models
Test Avg. F1 (%) 50.0 54.0 58.0 62.0 66.0 70.0 Durrett & Klein (2013) Björkelund & Kuhn (2014) Martschat & Strube (2015) Wiseman et al. (2016) Clark & Manning (2016)
65.7 64.2 62.5 61.6 60.3
Pipelined models
Test Avg. F1 (%) 50.0 54.0 58.0 62.0 66.0 70.0 Durrett & Klein (2013) Björkelund & Kuhn (2014) Martschat & Strube (2015) Wiseman et al. (2016) Clark & Manning (2016) Our model (single) Our model (ensemble)
68.8 67.2 65.7 64.2 62.5 61.6 60.3
Pipelined models End-to-end models
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. : Mention in a predicted cluster : Head-finding attention weight
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. : Mention in a predicted cluster : Head-finding attention weight
Attention-based head finder facilitates soft similarity cues
A fire in a Bangladeshi garment factory has left at least 37 people dead and 100 hospitalized. Most of the deceased were killed in the crush as workers tried to flee the blaze in the four-story building. : Mention in a predicted cluster : Head-finding attention weight
Good head-finding requires word-order information!
: Mention in a predicted cluster : Head-finding attention weight
The flight attendants have until 6:00 today to ratify labor concessions. The pilots' union and ground crew did so yesterday.
: Mention in a predicted cluster : Head-finding attention weight
The flight attendants have until 6:00 today to ratify labor concessions. The pilots' union and ground crew did so yesterday. Conflating relatedness with paraphrasing
Does the Recipe Work for Broad Coverage Semantics?
Step 1: Gather lots of training data! Step 2: Apply Deep Learning!! Step 3: Observe Impressive Gains!!!
Challenge 1: Data is costly and limited (e.g. linguists required to label PennTreebank / OntoNotes) Challenge 2: Pipeline of structured prediction problems with cascading errors (e.g. POS->Parsing->SRL->Coref)
Where Will the Data Come From???
Option 1: Semi-supervised learning
[Mikolov et al., 2013; Pennington et al., 2014]
Where Will the Data Come From???
Option 1: Semi-supervised learning
[Mikolov et al., 2013; Pennington et al., 2014]
Where Will the Data Come From???
Option 1: Semi-supervised learning
[Mikolov et al., 2013; Pennington et al., 2014]
Option 2: Supervised learning
Where Will the Data Come From???
Option 1: Semi-supervised learning
[Mikolov et al., 2013; Pennington et al., 2014]
Option 2: Supervised learning
supervision?
Where Will the Data Come From???
Learning Better Word Representations
Goal: Model contextualized syntax and semantics
R(plays, “The robot plays piano.”) R(plays, “The robot starred in many plays.”)
R(wi, w1 . . . wn) ∈ Rn
2 Layer Bidirectional LSTM Character convolutions
General Electric said the Postal Service contacted the company
Word Embeddings from a Language Model
Step 1: Train a large BiLM on unlabeled data
2 Layer Bidirectional LSTM Character convolutions
General Electric said the Postal Service contacted the company
Left and Right Per Word Softmaxs
contacted the Postal Electric
… … …
Word Embeddings from a Language Model
Step 1: Train a large BiLM on unlabeled data
2 Layer Bidirectional LSTM Character convolutions
General Electric said the Postal Service contacted the company
Word Embeddings from a Language Model
Step 2: Compute linear function of pre-trained model Step 1: Train a large BiLM on unlabeled data
Word Embeddings from a Language Model
2 Layer Bidirectional LSTM Character convolutions
General Electric said the Postal Service contacted the company
LM Embeddings
= α1 + α2 + α3
Step 2: Compute linear function of pre-trained model Step 1: Train a large BiLM on unlabeled data
Word Embeddings from a Language Model
2 Layer Bidirectional LSTM Character convolutions
General Electric said the Postal Service contacted the company
LM Embeddings
= α1 + α2 + α3
Step 2: Compute linear function of pre-trained model Step 1: Train a large BiLM on unlabeled data Step 3: Learn weights for each end task
Test Avg. F1 (%) 55.0 58.0 61.0 64.0 67.0 70.0
Feature Based Neural Neural+LM
70.4 67.2 62.5
SRL (+2.9 F1)
Coreference (+3.2 F1)
Test Avg. F1 (%) 75.0 77.0 79.0 81.0 83.0 85.0
Feature Based Neural Nueral+LM
84.6 81.7 79.9
40 50 60 70 80 90 100
SNLI SQuAD Coref SRL NER Sentiment (SST)
54.7 92.2 84.6 70.4 85.3 88.7 51.4 90.2 81.4 67.2 81.1 88 53.7 91.9 81.7 67.2 84.3 88.1
Previous SOTA Baseline Baseline+LM
Semantics:
[Miller et al.,1994]
classifier
65.0 66.2 67.4 68.6 69.8 71.0
Layer 1 Layer 2 Iacobacci (2016)
70.1 69.0 65.9
Semantics:
[Miller et al.,1994]
classifier
65.0 66.2 67.4 68.6 69.8 71.0
Layer 1 Layer 2 Iacobacci (2016)
70.1 69.0 65.9
Semantics:
[Miller et al.,1994]
classifier Syntax:
[Marcus et al., 1993]
N-th layer
Accuracy
95.0 95.6 96.2 96.8 97.4 98.0
Layer 1 Layer 2 Ling et al. (2015)
97.8 95.8 97.0
65.0 66.2 67.4 68.6 69.8 71.0
Layer 1 Layer 2 Iacobacci (2016)
70.1 69.0 65.9
Semantics:
[Miller et al.,1994]
classifier Syntax:
[Marcus et al., 1993]
N-th layer
Option 1: Semi-supervised learning
[Mikolov et al., 2013; Pennington et al., 2014]
Option 2: Supervised learning
supervision?
Where Will the Data Come From???
role inventory
predicate-argument relations
non-expert, part-time annotators
A First Data Step: QA-SRL
[He et al, 2015]
ARG1 ARG4 ARG3 ARG2
The rent rose 10% from $3000 to $3300
??? ??? ???
amount risen start point end point
inventory, requires syntactic parses
1) Identify the Frameset 2) Find arguments in the parse 3) Assign labels accordingly
The Proposition Bank: An Annotated Corpus of Semantic Roles, Palmer et al., 2005 http://verbs.colorado.edu/propbank/framesets-english/rise-v.html
Frameset: rise.01 , go up Arg1-: Logical subject, patient, thing rising Arg2-EXT: EXT, amount risen Arg3-DIR: start point Arg4-LOC: end point Argm-LOC: medium
Previous Method: Annotation with Frames
Our Annotation Scheme
They increased the rent this year .
Given sentence and a verb:
Our Annotation Scheme
Who increased something ? They increased the rent this year .
Given sentence and a verb: Step 1: Ask a question about the verb:
Our Annotation Scheme
Who increased something ? They They increased the rent this year .
Given sentence and a verb: Step 1: Ask a question about the verb: Step 2: Answer with words in the sentence:
Our Annotation Scheme
Who increased something ? They They increased the rent this year .
Given sentence and a verb: Step 1: Ask a question about the verb: Step 2: Answer with words in the sentence: Step 3: Repeat, write as many QA pairs as possible …
Our Annotation Scheme
Who increased something ? They What is increased ? the rent When is something increased ? this year They increased the rent this year .
Given sentence and a verb: Step 1: Ask a question about the verb: Step 2: Answer with words in the sentence: Step 3: Repeat, write as many QA pairs as possible …
Wh-Question Answer the rent What rose ? 10% $3000 $3300 How much did something rise ? What did something rise from ? What did something rise to ?
ARG1 ARG4 ARG3 ARG2
The rent rose 10% from $3000 to $3300
??? ??? ???
amount risen start point end point
Our Method: Q/A Pairs for Semantic Relations
Wh-words vs. PropBank Roles
Who What When Where Why How HowMuch ARG0 1575 414 3 5 17 28 2 ARG1 285 2481 4 25 20 23 95 ARG2 85 364 2 49 17 51 74 ARG3 11 62 7 8 4 16 31 ARG4 2 30 5 11 2 4 30 ARG5 1 2 AM-ADV 5 44 9 2 25 27 6 AM-CAU 3 1 23 1 AM-DIR 6 1 13 4 AM-EXT 4 5 5 AM-LOC 1 35 10 89 13 11 AM-MNR 5 47 2 8 4 108 14 AM-PNC 2 21 1 39 7 2 AM-PRD 1 1 1 AM-TMP 2 51 341 2 11 20 10
Advantages Limitations
Challenges
Does the Recipe Work for Broad Coverage Semantics?
Step 1: Gather lots of training data! Step 2: Apply Deep Learning!! Step 3: Observe Impressive Gains!!!
Challenge 1: Data is costly and limited (e.g. linguists required to label PennTreebank / OntoNotes) Challenge 2: Pipeline of structured prediction problems with cascading errors (e.g. POS->Parsing->SRL->Coref)
Models
coreference
Data
language model
Contributions
Future Directions
focus on out of domain performance
The End: Questions?
Recent Release