Generating FrameNets of various granularities: The FrameNet - - PowerPoint PPT Presentation

generating framenets of various granularities the
SMART_READER_LITE
LIVE PREVIEW

Generating FrameNets of various granularities: The FrameNet - - PowerPoint PPT Presentation

Generating FrameNets of various granularities: The FrameNet Transformer Josef Ruppenhofer, Jonas Sunde, & Manfred Pinkal Saarland University LREC, May 2010 Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various


slide-1
SLIDE 1

Generating FrameNets of various granularities: The FrameNet Transformer

Josef Ruppenhofer, Jonas Sunde, & Manfred Pinkal

Saarland University

LREC, May 2010

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 1 / 21

slide-2
SLIDE 2

Introduction

Predicate-argument structure has proven essential for many NLP applications

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 2 / 21

slide-3
SLIDE 3

Introduction

Predicate-argument structure has proven essential for many NLP applications Two prominent resources for modelling predicate-argument structure in English are PropBank (Palmer et al., 2005) and FrameNet (Baker et al., 1998)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 2 / 21

slide-4
SLIDE 4

Introduction

Predicate-argument structure has proven essential for many NLP applications Two prominent resources for modelling predicate-argument structure in English are PropBank (Palmer et al., 2005) and FrameNet (Baker et al., 1998) PropBank maps different syntactic realizations of one lemma to the same predicate-argument structure, using lemma-specific semantic roles

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 2 / 21

slide-5
SLIDE 5

Introduction

Predicate-argument structure has proven essential for many NLP applications Two prominent resources for modelling predicate-argument structure in English are PropBank (Palmer et al., 2005) and FrameNet (Baker et al., 1998) PropBank maps different syntactic realizations of one lemma to the same predicate-argument structure, using lemma-specific semantic roles FrameNet offers additional structure and detail, making it attractive for information-access tasks

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 2 / 21

slide-6
SLIDE 6

Pros and Cons of Using FrameNet

Pros Cons

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 3 / 21

slide-7
SLIDE 7

Pros and Cons of Using FrameNet

Pros Detail and richness

◮ Word senses grouped

into Frames

◮ Several types of frame

relations

◮ Parallel to frame

relations, FE relations

Cons

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 3 / 21

slide-8
SLIDE 8

Pros and Cons of Using FrameNet

Pros Detail and richness

◮ Word senses grouped

into Frames

◮ Several types of frame

relations

◮ Parallel to frame

relations, FE relations

Cons Many units are exemplified by relatively few annotated training instances (e.g. Kaisser & Webber 2007).

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 3 / 21

slide-9
SLIDE 9

Pros and Cons of Using FrameNet

Pros Detail and richness

◮ Word senses grouped

into Frames

◮ Several types of frame

relations

◮ Parallel to frame

relations, FE relations

Cons Many units are exemplified by relatively few annotated training instances (e.g. Kaisser & Webber 2007). Distinctions often too fine-grained (Burchardt et

  • al. 2009) to allow robust

shallow semantic parsing.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 3 / 21

slide-10
SLIDE 10

Coarsening FrameNet

We address the problems of data sparsity and too fine distinctions by coarsening FrameNet with the FN transformer tool

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 4 / 21

slide-11
SLIDE 11

Coarsening FrameNet

We address the problems of data sparsity and too fine distinctions by coarsening FrameNet with the FN transformer tool The tool efficiently generates coarser-grained variants of the FrameNet database.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 4 / 21

slide-12
SLIDE 12

Coarsening FrameNet

We address the problems of data sparsity and too fine distinctions by coarsening FrameNet with the FN transformer tool The tool efficiently generates coarser-grained variants of the FrameNet database.

◮ it reduces the number of word-senses (frames) per lemma ◮ it increases the number of annotated sentences per lexical unit and

frame.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 4 / 21

slide-13
SLIDE 13

Coarsening FrameNet

We address the problems of data sparsity and too fine distinctions by coarsening FrameNet with the FN transformer tool The tool efficiently generates coarser-grained variants of the FrameNet database.

◮ it reduces the number of word-senses (frames) per lemma ◮ it increases the number of annotated sentences per lexical unit and

frame.

we achieve this in two ways

◮ merging Frames ◮ merging LUs Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 4 / 21

slide-14
SLIDE 14

Coarsening FrameNet

We address the problems of data sparsity and too fine distinctions by coarsening FrameNet with the FN transformer tool The tool efficiently generates coarser-grained variants of the FrameNet database.

◮ it reduces the number of word-senses (frames) per lemma ◮ it increases the number of annotated sentences per lexical unit and

frame.

we achieve this in two ways

◮ merging Frames ◮ merging LUs Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 4 / 21

slide-15
SLIDE 15

Merging by frame

The idea is to merge frames in a principled way: by frame-relation

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 5 / 21

slide-16
SLIDE 16

Merging by frame

The idea is to merge frames in a principled way: by frame-relation Merging of senses would result as a side effect

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 5 / 21

slide-17
SLIDE 17

Merging by frame

The idea is to merge frames in a principled way: by frame-relation Merging of senses would result as a side effect Frame relations are redirected as needed

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 5 / 21

slide-18
SLIDE 18

Merging by frame

The idea is to merge frames in a principled way: by frame-relation Merging of senses would result as a side effect Frame relations are redirected as needed Parameters

◮ selection of frames that receive annotations ◮ selection of frames that disappear ◮ stop frames (e.g. Event, Entity,...) Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 5 / 21

slide-19
SLIDE 19

Choosing suitable relations

Good candidates

◮ Perspective on (Hiring → Employment start ← Get a job) ◮ Subframe ( Criminal process → Arrest, Arraignment, ...) ◮ Causative of (Killing → Death) ◮ Inchoative of (Death → Dead or alive)

Less reliable

◮ Using (Communication → Volubility) ◮ Inheritance (Transitive action → Cause to end) Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 6 / 21

slide-20
SLIDE 20

Crime scenario original

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 7 / 21

slide-21
SLIDE 21

Crime scenario after 1 iteration of frame-based merging

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 8 / 21

slide-22
SLIDE 22

Crime scenario after 2nd iteration of frame-based merging

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 9 / 21

slide-23
SLIDE 23

Lemma-based mode

Merges and migrates related (frame-specific) senses of a particular lemma Frame structure remains intact

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 10 / 21

slide-24
SLIDE 24

Lemma-based mode

Merges and migrates related (frame-specific) senses of a particular lemma Frame structure remains intact FN release 1.3 has 1316 lemmas that occur in more than one frame.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 10 / 21

slide-25
SLIDE 25

Lemma-based mode

Merges and migrates related (frame-specific) senses of a particular lemma Frame structure remains intact FN release 1.3 has 1316 lemmas that occur in more than one frame. Mostly they are involved in polysemy between 2 known senses but in some cases a lemma belongs to 9 different frames.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 10 / 21

slide-26
SLIDE 26

Lemma-based mode

Merges and migrates related (frame-specific) senses of a particular lemma Frame structure remains intact FN release 1.3 has 1316 lemmas that occur in more than one frame. Mostly they are involved in polysemy between 2 known senses but in some cases a lemma belongs to 9 different frames. These 1316 lemmas have a total of 2587 pairs of senses that could potentially be merged.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 10 / 21

slide-27
SLIDE 27

Lemma-based mode II

Two cases

◮ one LU’s frame is

ancestor of the other LU’s frame (530 potential pairs to merge)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 11 / 21

slide-28
SLIDE 28

Lemma-based mode II

Two cases

◮ one LU’s frame is

ancestor of the other LU’s frame (530 potential pairs to merge)

◮ neither LU’s frame is

an ancestor for the

  • ther: create a new LU

in a third frame, reflecting the broader semantic range covered by the combination.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 11 / 21

slide-29
SLIDE 29

Lemma-based mode II

Two cases

◮ one LU’s frame is

ancestor of the other LU’s frame (530 potential pairs to merge)

◮ neither LU’s frame is

an ancestor for the

  • ther: create a new LU

in a third frame, reflecting the broader semantic range covered by the combination.

user selects the types of relations to cross on the path from source to target LUs

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 11 / 21

slide-30
SLIDE 30

The FN Transformer

Java 1.6 no gui, reads user settings from xml file

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 12 / 21

slide-31
SLIDE 31

The FN Transformer

Java 1.6 no gui, reads user settings from xml file basic settings

◮ path to FrameNet data release ◮ path to an output directory ◮ logfile to be created Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 12 / 21

slide-32
SLIDE 32

The FN Transformer

Java 1.6 no gui, reads user settings from xml file basic settings

◮ path to FrameNet data release ◮ path to an output directory ◮ logfile to be created

  • utput is a format-compliant FrameNet release (xml files)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 12 / 21

slide-33
SLIDE 33

The FN Transformer

Java 1.6 no gui, reads user settings from xml file basic settings

◮ path to FrameNet data release ◮ path to an output directory ◮ logfile to be created

  • utput is a format-compliant FrameNet release (xml files)

for relation-based merger, output also includes .dot files that can be used for “visual diff” (via the open-source GraphViz software)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 12 / 21

slide-34
SLIDE 34

The FN Transformer

Java 1.6 no gui, reads user settings from xml file basic settings

◮ path to FrameNet data release ◮ path to an output directory ◮ logfile to be created

  • utput is a format-compliant FrameNet release (xml files)

for relation-based merger, output also includes .dot files that can be used for “visual diff” (via the open-source GraphViz software) in addition to the two automatic modes, there is a manual mode

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 12 / 21

slide-35
SLIDE 35

Evaluation

A baseline evaluation consists in confirming that we do obtain the expected improved accuracy of frame-semantic parsers trained on the modified data. In a further step, we perform a task-based evaluation to check whether we improve parsing accuracy at the cost of losing relevant information.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 13 / 21

slide-36
SLIDE 36

Parsing accuracy: setup

Compare the performance of the Shalmaneser semantic parser (Erk & Pad´

  • 2006) in two settings:

◮ Baseline: FrameNet release 1.3. ◮ Coarsened: modified FrameNets created by our transformer

Data: subset of lemmas that were affected by the transformation 10-fold cross-validation setting

◮ frame assignment ◮ argument recognition ◮ argument labeling Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 14 / 21

slide-37
SLIDE 37

Parsing accuracy

task cum. task cum. FN1.3 FN1.3R Frame assignment 0.94 0.94 0.94 0.94 Argument recognition 0.69 0.64 0.69 0.65 Argument labeling 0.71 0.46 0.75 0.49

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 15 / 21

slide-38
SLIDE 38

Parsing accuracy

task cum. task cum. FN1.3 FN1.3R Frame assignment 0.94 0.94 0.94 0.94 Argument recognition 0.69 0.64 0.69 0.65 Argument labeling 0.71 0.46 0.75 0.49 FN1.3 FN1.3LU Frame assignment 0.89 0.89 0.94 0.94 Argument recognition 0.69 0.62 0.66 0.62 Argument labeling 0.74 0.46 0.72 0.44

Table: Performance of Shalmaneser on FN release 1.3 and on transformations (10-fold cross-validation)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 15 / 21

slide-39
SLIDE 39

Preservation of relevant information - RTE

Ensure that better parser performance is not achieved at the cost of losing relevant information Evaluate our coarsened FrameNet versions in the context of the entailment recognition (RTE) task Entailment recognition is the task of determining whether a text T entails a hypothesis H in a common sense way.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 16 / 21

slide-40
SLIDE 40

Preservation of relevant information - RTE

Ensure that better parser performance is not achieved at the cost of losing relevant information Evaluate our coarsened FrameNet versions in the context of the entailment recognition (RTE) task Entailment recognition is the task of determining whether a text T entails a hypothesis H in a common sense way.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 16 / 21

slide-41
SLIDE 41

Preservation of relevant information - RTE

Ensure that better parser performance is not achieved at the cost of losing relevant information Evaluate our coarsened FrameNet versions in the context of the entailment recognition (RTE) task Entailment recognition is the task of determining whether a text T entails a hypothesis H in a common sense way. (3) T: An avalanche has struck a popular skiing resort in Austria, killing at least 11 people. H: Humans died in an avalanche.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 16 / 21

slide-42
SLIDE 42

Frame semantic information in the RTE task

Techniques for judging entailment include measuring lexical overlap, shallow syntactic parsing, and the use of WordNet relations

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 17 / 21

slide-43
SLIDE 43

Frame semantic information in the RTE task

Techniques for judging entailment include measuring lexical overlap, shallow syntactic parsing, and the use of WordNet relations Another kind of approach consists in using shallow semantic representations that abstract away from semantically irrelevant variations (5) T: An avalanche has struck a popular skiing resort in Austria, killing at least 11 people. H: Humans died in an avalanche.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 17 / 21

slide-44
SLIDE 44

Assessing the contribution of frame semantics to RTE

Burchardt et al 2009 performed an experiment on the gold standard data of the FATE corpus (Burchardt and Pennacchiotti 2008)

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 18 / 21

slide-45
SLIDE 45

Assessing the contribution of frame semantics to RTE

Burchardt et al 2009 performed an experiment on the gold standard data of the FATE corpus (Burchardt and Pennacchiotti 2008) FATE corpus: manual frame semantic annotations for the 800 entailment pairs of RTE-2 ; 4490 frame instances annotated.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 18 / 21

slide-46
SLIDE 46

Assessing the contribution of frame semantics to RTE

Burchardt et al 2009 performed an experiment on the gold standard data of the FATE corpus (Burchardt and Pennacchiotti 2008) FATE corpus: manual frame semantic annotations for the 800 entailment pairs of RTE-2 ; 4490 frame instances annotated. Key assumption: the more of the semantics of the hypothesis can be embedded into the text, the more likely it is that an entailment relation holds between text and hypothesis

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 18 / 21

slide-47
SLIDE 47

Assessing the contribution of frame semantics to RTE

Burchardt et al 2009 performed an experiment on the gold standard data of the FATE corpus (Burchardt and Pennacchiotti 2008) FATE corpus: manual frame semantic annotations for the 800 entailment pairs of RTE-2 ; 4490 frame instances annotated. Key assumption: the more of the semantics of the hypothesis can be embedded into the text, the more likely it is that an entailment relation holds between text and hypothesis Extracting frame-based statistical information from the positive and negative examples of the annotated corpus, respectively, and measuring the overlap of frame structures between text and hypothesis in an entailment pair. (9) T: An avalanche has struck a popular skiing resort in Austria, killing at least 11 people. H: Humans died in an avalanche.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 18 / 21

slide-48
SLIDE 48

Frame label overlap

Positive pairs Negative pairs Difference FN1.3 0.5711 0.4585 0.1126 FN1.3R 0.5913 0.4845 0.1068 FN1.3LU 0.5323 0.4348 0.0975

Table: Average frame label overlap on entailment pairs in three versions of the Fate corpus

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 19 / 21

slide-49
SLIDE 49

Conclusion

We presented a tool for semi-automatically deriving customized but format-compliant versions of the FrameNet database based on frame and frame element relations.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 20 / 21

slide-50
SLIDE 50

Conclusion

We presented a tool for semi-automatically deriving customized but format-compliant versions of the FrameNet database based on frame and frame element relations. In baseline evaluations, we found that coarsening FrameNet yields slightly better parsing accuracy and does not cause the loss of information for the RTE task

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 20 / 21

slide-51
SLIDE 51

Conclusion

We presented a tool for semi-automatically deriving customized but format-compliant versions of the FrameNet database based on frame and frame element relations. In baseline evaluations, we found that coarsening FrameNet yields slightly better parsing accuracy and does not cause the loss of information for the RTE task Allows users to produce FrameNet versions whose granularity is suitable for their particular applications.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 20 / 21

slide-52
SLIDE 52

Conclusion

We presented a tool for semi-automatically deriving customized but format-compliant versions of the FrameNet database based on frame and frame element relations. In baseline evaluations, we found that coarsening FrameNet yields slightly better parsing accuracy and does not cause the loss of information for the RTE task Allows users to produce FrameNet versions whose granularity is suitable for their particular applications. Additional experiments needed to assess whether the individual gains

  • f the two modes of transformation can be combined and what the

best settings are for each of them.

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 20 / 21

slide-53
SLIDE 53

Visual diff

Ruppenhofer, Sunde, Pinkal (Saarland U.) Generating FrameNets of various granularities LREC, May 2010 21 / 21