Statistical Models for Frame-Semantic Parsing Dipanjan Das * Google - - PowerPoint PPT Presentation
Statistical Models for Frame-Semantic Parsing Dipanjan Das * Google Frame Semantics in NLP: A Workshop in Honor of Chuck Fillmore June 27, 2014 * Thanks to Desai Chen, Kuzman Ganchev, Karl Moritz Hermann, Andr Martins, Nathan Schneider, Noah
Frame Semantics in NLP: A Workshop in Honor of Chuck Fillmore June 27, 2014
*Thanks to Desai Chen, Kuzman Ganchev, Karl Moritz Hermann,
André Martins, Nathan Schneider, Noah Smith and Jason Weston
2
Encodes an event or scenario
3
Participant or role for the frame
4
Participant or role for the frame
Experiencer
5
6
7
8
8
8
8
Bengal ’s massive stock of food was reduced to nothing
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Store or financial entity?
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Store of what? Of what size? Whose store?
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Store of what? Of what size? Whose store?
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
What was reduced? To what?
Bengal ’s massive stock of food wasreducedto nothing
N X A N ADP N V V ADP N
What was reduced? To what?
Bengal ’s massive stock of food wasreducedto nothing
N X A N ADP N V V ADP N
Bengal ’s massive stock of food was reduced to nothing
James Cameron directed Titanic
Artist Production
James Cameron directed Titanic
Artist Production
James Cameron is Titanic’s director
Production Artist
James Cameron directed Titanic
Artist Production
James Cameron is Titanic’s director
Production Artist
Grouping across syntactic categories
James Cameron filmed Titanic
Artist Production
James Cameron is Titanic’s director
Production Artist
Grouping across different lemmas
James Cameron filmed Titanic
ARG0
James Cameron is Titanic’s director
ARG2 ARG0 ARG1
from PropBank from NomBank
James Cameron filmed Titanic
ARG0
James Cameron is Titanic’s director
ARG2 ARG0 ARG1
from PropBank from NomBank
Hard to align different lemmas across resources
James Cameron filmed Titanic
ARG0
James Cameron is Titanic’s director
ARG2 ARG0 ARG1
from PropBank from NomBank
Very generic argument labels
James Cameron filmed Titanic
ARG0
James Cameron is Titanic’s director
ARG2 ARG0 ARG1
from PropBank from NomBank
Argument labels do not match up
James Cameron filmed Titanic in 1997
Artist Production
James Cameron filmed Titanic in 1997
Artist Production
λe.filmed.arg1(e, James Cameron)∧filmed.arg2(e, Titanic)∧filmed.in(e, 1997)
λe.filmed.arg1(e, /m/03 gd) ∧ filmed.arg2(e, /m/0dr 4) ∧ filmed.in(e, 1997)
James Cameron filmed Titanic in 1997
Artist Production
λe.filmed.arg1(e, James Cameron)∧filmed.arg2(e, Titanic)∧filmed.in(e, 1997)
λe.filmed.arg1(e, /m/03 gd) ∧ filmed.arg2(e, /m/0dr 4) ∧ filmed.in(e, 1997)
λe.filmed.arg1(e, James Cameron)∧filmed.arg2(e, Titanic)∧filmed.in(e, 1997)
λe.filmed.arg1(e, /m/03 gd) ∧ filmed.arg2(e, /m/0dr 4) ∧ filmed.in(e, 1997)
λe.filmed.arg1(e, James Cameron)∧filmed.arg2(e, Titanic)∧filmed.in(e, 1997)
λe.BEHIND THE SCENES.Artist(e, /m/03 gd)∧ ∧ BEHIND THE SCENES.Production(e, /m/0dr 4)∧ BEHIND THE SCENES.Time(e, 1997)
FrameNet
25
Slots for Spoken Dialogue Systems Using Frame- Semantic Parsing Chen, Wang and Rudnicky (ASRU 2013)
semantic structures
26
Xie et al. (ACL 2013)
news
frames and roles
27
Thomas Kleinbauer (PhD thesis, Saarland University)
summaries
28
29
PLACING
Agent Cause Goal Theme Area Time
PLACING
Agent Cause Goal Theme Area Time
PLACING
Agent Cause Goal Theme Area Time
PLACING
Agent Cause Goal Theme Area Time
PLACING
Agent Cause Goal Theme Area Time
PLACING
Agent Cause Goal Theme Area Time
archive.V, arrange.V, bag.V, bestow.V bin.V
PLACING
Agent Cause Goal Theme Area Time
DISPERSAL
Agent Cause Individuals Distance Time
TRANSITIVE_ACTION
Agent Cause Patient Event Place Time
INSTALLING
Agent Component Fixed_location Area Time
STORING
Agent Location Theme Area Time
STORE
Possessor Resource Supply Descriptor
PLACING
Agent Cause Goal Theme Area Time
DISPERSAL
Agent Cause Individuals Distance Time
TRANSITIVE_ACTION
Agent Cause Patient Event Place Time
INSTALLING
Agent Component Fixed_location Area Time
STORING
Agent Location Theme Area Time
STORE
Possessor Resource Supply Descriptor
PLACING
Agent Cause Goal Theme Area Time
DISPERSAL
Agent Cause Individuals Distance Time
TRANSITIVE_ACTION
Agent Cause Patient Event Place Time
INSTALLING
Agent Component Fixed_location Area Time
STORING
Agent Location Theme Area Time
STORE
Possessor Resource Supply Descriptor
Benchmark Dataset (SemEval 2007)
665 frames 720 role labels 8.4K unique predicate types
Training set:
2.2K sentences 11.2K predicate tokens
Test set: 120 sentences
Johansson and Nugues (2007)
Frame Identification Argument Filtering Argument Labeling
40
Johansson and Nugues (2007)
Frame Identification Argument Filtering Argument Labeling
41
Johansson and Nugues (2007)
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Johansson and Nugues (2007)
Find the best among all frames Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Johansson and Nugues (2007) Taken from an SVM classifier trained on ambiguous predicates
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Johansson and Nugues (2007)
To increase coverage, potential predicates were extracted from WordNet and automatically frame labels were selected for them.
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
50.0 61.3 72.5 83.8 95.0
LTH
57.3
F-Measure
Johansson and Nugues (2007)
Johansson and Nugues (2007)
Frame Identification Argument Filtering Argument Labeling
47
Bengal ’s massive stock of food was reduced to nothing
massive stock
Bengal ’s massive stock of food was reduced to nothing
massive stock Bengal ’s Bengal massive stock
food massive Bengal ’s massive to nothing to
’s
Bengal ’s massive stock of food was reduced to nothing
massive stock Bengal ’s Bengal massive stock
food massive Bengal ’s massive to nothing to
’s
Binary SVM Classification
Johansson and Nugues (2007)
Frame Identification Argument Filtering Argument Labeling
50
Bengal ’s massive stock of food was reduced to nothing
massive stock
Multiclass SVM Classification
Bengal ’s massive stock of food was reduced to nothing
massive stock Bengal ’s Bengal massive stock
food massive Bengal ’s massive to nothing to
’s
Multiclass SVM Classification
Bengal ’s massive stock of food was reduced to nothing
massive stock Bengal ’s Bengal massive stock
food massive Bengal ’s massive to nothing to
’s
Multiclass SVM Classification
35.0 43.8 52.5 61.3 70.0
LTH
44.4
F-Measure
Johansson and Nugues (2007)
Das, Chen, Martins, Schneider, Smith (2014)
Frame Identification Argument Identification
54
Das, Chen, Martins, Schneider, Smith (2014)
Frame Identification Argument Identification
55
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Logistic regression with a latent variable
Logistic regression with a latent variable
Logistic regression with a latent variable
Predicates evoking a frame in supervised data, e.g.
cargo.N, inventory.N, reserve.N, stockpile.N, store.N, supply.N
evoke STORE
stock.N stockpile.N
Bengal ’s massive stock
to nothing
N X A N ADP N V V ADP N
stock.N stockpile.N
Bengal ’s massive stock
to nothing
N X A N ADP N V V ADP N
LexSem = {synonym}
stock.N stockpile.N
Bengal ’s massive stock
to nothing
N X A N ADP N V V ADP N
LexSem = {synonym}
STORE stockpile.N
synonym LexSem
stock.N stockpile.N
Bengal ’s massive stock
to nothing
N X A N ADP N V V ADP N
LexSem = {synonym}
STORE stockpile.N
synonym LexSem
comes from WordNet!
New Data (FrameNet 1.5, 2010)
877 frames 1068 role labels 9.3K unique predicate types
Training set:
3.3K sentences 19.6K predicate tokens
Test set: 2420 sentences 4.5K predicate tokens Benchmark Dataset (SemEval 2007)
665 frames 720 role labels 8.4K unique predicate types
Training set:
2.2K sentences 11.2K predicate tokens
Test set: 120 sentences
New Data (FrameNet 1.5, 2010)
877 frames 1068 role labels 9.3K unique predicate types
Training set:
3.3K sentences 19.6K predicate tokens
Test set: 2420 sentences 4.5K predicate tokens Benchmark Dataset (SemEval 2007)
665 frames 720 role labels 8.4K unique predicate types
Training set:
2.2K sentences 11.2K predicate tokens
Test set: 120 sentences
50.0 61.3 72.5 83.8 95.0
LTH SEMAFOR
61 57.3
F-Measure
log-linear
50.0 61.3 72.5 83.8 95.0
LTH SEMAFOR
61 57.3
F-Measure
log-linear
auto predicates
50.0 61.3 72.5 83.8 95.0
LTH SEMAFOR
61 57.3
F-Measure
log-linear
auto predicates
50.0 61.3 72.5 83.8 95.0
SEMAFOR
83.0
Accuracy
log-linear
gold predicates
20.0 38.8 57.5 76.3 95.0
All Predicates
83.0
Accuracy
20.0 38.8 57.5 76.3 95.0
Unknown Predicates
23.1
Accuracy
Frame Identification
Knowledge of only 9,263 predicates in supervised data
Knowledge of only 9,263 predicates in supervised data However, English has lot more potential predicates (~65,000 in newswire English)
Knowledge of only 9,263 predicates in supervised data However, English has lot more potential predicates (~65,000 in newswire English)
Seed predicates
Seed predicates Unseen predicates
Seed predicates Unseen predicates
Seed predicates Unseen predicates
Seed predicates Unseen predicates
Continues till convergence...
15.0 28.8 42.5 56.3 70.0
Supervised Self-Training Graph-Based
42.7 18.9 23.1
Accuracy
Frame Identification
Das, Chen, Martins, Schneider, Smith (2014)
Frame Identification Argument Identification
81
Bengal ’s massive stock of food was reduced to nothing
Bengal ’s massive stock of food was reduced to nothing
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
PLACING
Agent Cause Goal Theme Area Time
archive.V, arrange.V, bag.V, bestow.V bin.V
PLACING
Agent Cause Goal Theme Area Time
archive.V, arrange.V, bag.V, bestow.V bin.V
If an agent places something, there cannot be a cause role in the sentence
PLACING
Agent Cause Goal Theme Area Time
archive.V, arrange.V, bag.V, bestow.V bin.V
The waiter placed food on the table. In Kabul, hauling water put food on the table.
Cause Agent
SIMILARITY
Dimension Differentiating_fact Entity_1 Entity_2 Degree
difference.N, resemble.V, unliike.A, vary.V
SIMILARITY
Dimension Differentiating_fact Entity_1 Entity_2 Degree
difference.N, resemble.V, unliike.A, vary.V
A mulberry resembles a loganberry.
SIMILARITY
Dimension Differentiating_fact Entity_1 Entity_2 Degree
difference.N, resemble.V, unliike.A, vary.V
A mulberry resembles.
!
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
stock
Bengal ’s Bengal massive stock
food massive Bengal ’s massive massive stock
Prevents overlap
more structural constraints
An integer linear program (ILP)
more structural constraints
An integer linear program (ILP)
Punyakanok, Roth and Yih (2008)
more structural constraints
An integer linear program (ILP) Often, very slow solutions
more structural constraints
An integer linear program (ILP) Fast ILP solvers proprietary
more structural constraints
An integer linear program (ILP) Dual Decomposition
35.0 43.8 52.5 61.3 70.0
LTH SEMAFOR
46.5 42.0
F-Measure
35.0 43.8 52.5 61.3 70.0
LTH SEMAFOR
46.5 42.0
F-Measure
auto predicates
35.0 43.8 52.5 61.3 70.0
LTH SEMAFOR
46.5 42.0
F-Measure
auto predicates
35.0 43.8 52.5 61.3 70.0
SEMAFOR
64.5
F-Measure
gold predicates
111
112
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
112
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
113
Discrete lexical features Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
114
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
Syntactic context words connected via a set of dependency paths
amod
prep pobj
nsubjpass auxpass ;
115
Bengal ’s massive stock of food was reduced to nothing
N X A N ADP N V V ADP N
amod
prep pobj
nsubjpass auxpass ;
amod
prep pobj
nsubjpass auxpass ;
Replace context words with off-the-shelf word embeddings
116
amod
prep pobj
nsubjpass auxpass ;
amod
prep pobj
nsubjpass auxpass ;
amod nsubj dobj >prep>pobj p <nsubjpass>auxpass
Input sparse embedding vector with context blocks
117
Frame instance space N × Rd
117
Frame instance space N × Rd Joint Space Rm
117
Frame instance space N × Rd Joint Space Rm Frame Instance Map M : Rd → Rm
117
Frame instance space N × Rd Joint Space Rm Frame Instance Map M : Rd → Rm Set of FrameNet labels
117
Frame instance space N × Rd Joint Space Rm Frame Instance Map M : Rd → Rm Set of FrameNet labels
Y ∈ RF ×m
117
Frame instance space N × Rd Joint Space Rm Frame Instance Map M : Rd → Rm Set of FrameNet labels
Y ∈ RF ×m
118
118
118
119
119
120
120
121
15.0 28.8 42.5 56.3 70.0
Supervised Self-Training Graph-Based Embeddings
46.15 42.7 18.9 23.1
Accuracy
Frame Identification
75.0 78.8 82.5 86.3 90.0
Supervised Self-Training Graph-Based Embeddings
86.49 83.6 82.3 83.0
Accuracy
Frame Identification
60.0 62.5 65.0 67.5 70.0
Supervised Graph-Based Embeddings
68.7 64.5 64.1
F-Score
Frame-Semantic Parsing
125
126
127
128
129
130