Extending Fine-Grained Semantic Relation Classification to - - PowerPoint PPT Presentation

▶

Nov 03, 2023 864 likes •1.27k views

Semantic Relations Annotation Experiments Conclusions and Future Work Extending Fine-Grained Semantic Relation Classification to Presupposition Relations between Verbs Galina Tremper and Anette Frank Department of Computational Linguistics

SLIDE 1

Semantic Relations Annotation Experiments Conclusions and Future Work

Extending Fine-Grained Semantic Relation Classification to Presupposition Relations between Verbs

Galina Tremper and Anette Frank

Department of Computational Linguistics Heidelberg University, Germany

Beyond semantics: Corpus-based investigations of pragmatic and discourse phenomena DGfS Workshop, G¨

ttingen

February 23-25, 2011

SLIDE 2

Semantic Relations Annotation Experiments Conclusions and Future Work

Motivation

Known Event On Sunday, Olaf Scholz won the state elections in Hamburg to gain absolute majority for the SPD.

SLIDE 3

Semantic Relations Annotation Experiments Conclusions and Future Work

Motivation

Known Event On Sunday, Olaf Scholz won the state elections in Hamburg to gain absolute majority for the SPD.

SLIDE 4

Semantic Relations Annotation Experiments Conclusions and Future Work

Content of Talk

1

Semantic Relations between Verbs

2

Challenges in the Annotation of Inference Relations

3

Automatic Classification Experiments

4

Conclusions and Ideas for Future Work

SLIDE 5

Semantic Relations Annotation Experiments Conclusions and Future Work

Semantic Relations Overview

Sem. Relation

Inference Pattern Example Substitution in Pattern Presupposition V1 presupposes V2, win - play winning presupposes playing not V1 presupposes V2 not winning presupposes playing Entailment V1 implies V2, kill - die killing implies dying not V1 doesn’t imply V2 not killing doesn’t imply dying Temporal V1 happens during V2 or snore - sleep snoring happens during sleeping Inclusion V1 is a special form of V2 mutter - talk muttering is a special form of talking Antonymy either V1 or V2, go - stay either going or staying V1 is the opposite of V2 going is the opposite of staying Other/unrelated none of the above jump - sing

Table: Semantic Relations and Inference Patterns for Guiding Annotation

SLIDE 6

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Notes

P(resupposition), E(ntailment), T(emporal Inclusion), A(ntonymy), S(ynonymy) V1 - trigger verb, V2 - inferred verb

SLIDE 7

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Presupposition verb pair (V1: win, V2: play)

win succeeds play

SLIDE 8

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Presupposition verb pair (V1: win, V2: play)

win succeeds play win → play

SLIDE 9

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Presupposition verb pair (V1: win, V2: play)

win succeeds play win → play not win → play - persistance under negation

SLIDE 10

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Presupposition verb pair (V1: win, V2: play)

win succeeds play win → play not win → play - persistance under negation not win → not play - cancellation

SLIDE 11

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 12

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 13

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 14

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 15

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 16

Semantic Relations Annotation Experiments Conclusions and Future Work

Properties of Semantic Relations

Behaviour under Negation V1 → V2 ¬V1 → V2 V1 → ¬V2 ¬V1 → ¬V2 V1 precedes V2 E E Temporal V1 succeeds V2 P P P Sequence E E No temporal E E sequence T T T A A S S

Table: Properties of Semantic Relations

Distinguishing Properties

Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

SLIDE 17

Semantic Relations Annotation Experiments Conclusions and Future Work

Annotation Tasks

Goal: Automatic classification of target relations in context Prerequisite: Annotations for training a classifier Building two Gold Standards:

Gold Standard 1: Type-based Annotation - verb pairs given as types, without context (e.g. win - play; learn - know)

A sample of 100 verb pairs We allow more than one label

Gold Standard 2: Token-based Annotation - verb pairs are presented in context (occurrence in one sentence)

The same 100 verb pairs Up to 10 randomly selected contexts for each verb pair Only one label per sentence is allowed

SLIDE 18

Semantic Relations Annotation Experiments Conclusions and Future Work

Gold Standards

1 Gold Standard 1 (GS1) - Type-based Annotation

Inter-annotator agreement 63% (Kappa value - 0.47)

2 Gold Standard 2 (GS2) - Token-based Annotation

Inter-annotator agreement 77.4% (Kappa value - 0.44)

3 Gold Standard 3 (GS3) - Type-based Annotation deduced

from GS2

Up to three most frequent annotations

SLIDE 19

Semantic Relations Annotation Experiments Conclusions and Future Work

Correlation between GS1 and GS2

62% - Overlap of labels (the same labels were assigned) 28% - Divergent contexts:

Labels assigned on the token level are not present on the type level (difficulty of considering all the verb meanings out of context) Labels assigned on the type level are not found on the token level (not enough contexts)

10% - Conflicting annotations (e.g. presupposition vs. entailment)

SLIDE 20

Semantic Relations Annotation Experiments Conclusions and Future Work

Automatic Classification

1 Experiment 1: Token-based classification

Applying 5 binary classifiers on each instance of the unlabeled verb pair in context Threshold for confidence is 0.75 Using a voting architecture to determine the most confident classification

2 Experiment 2: Type-based classification

The most confident classifications for instances are taken from Experiment 1 More than 10% of instances must be marked with the same label No more than three classes for one verb pair are accepted

Feature classes used:

Features detecting negation properties Features detecting temporal precedence properties

SLIDE 21

Semantic Relations Annotation Experiments Conclusions and Future Work

Examples of correct and wrong classification

Presupposition (classify - identify) Correct classification Presupposition: It was noted that of the thirteen issues identified in the report eight were classified as high priority. Wrong classification: System Label - Temp. Inclusion Temporal Inclusion: The meeting focussed on issues of identifying, classifying and marking up names in both corpora and analytical projects.

SLIDE 22

Semantic Relations Annotation Experiments Conclusions and Future Work

Examples of correct and wrong classification

Antonymy (disconnect - connect) Correct classification A click should be heard every time the antenna wire is connected

r disconnected.

Wrong classification None: This allows you to connect and disconnect easily , simply by clicking on the icon and selecting the relevant option.

SLIDE 23

Semantic Relations Annotation Experiments Conclusions and Future Work

Experiment 1 - Results

Semantic relation Precision Recall f-score Baseline Presupposition 23% 27% 25% 8% Entailment 18% 25% 21% 5%

Temp. Inclusion

10% 12% 11% 3% Antonymy 42% 68% 52% 5% Other/Unrelated 73% 59% 65% 79% Average 59% 54% 56% Table: Evaluation of the Results for Experiment 1: Token-based Classification

SLIDE 24

Semantic Relations Annotation Experiments Conclusions and Future Work

Experiment 2 - Results

Semantic Gold Standard 1 Gold Standard 3 relation Precision Recall f-score Baseline Precision Recall f-score Baseline Presupposition 43% 33% 37% 18% 50% 29% 37% 24% Entailment 36% 50% 42% 8% 36% 50% 42% 8%

Temp. Inclusion

50% 16% 24% 19% 33% 17% 22% 12% Antonymy 75% 75% 75% 12% 58% 70% 63% 10% Other/Unrelated 56% 74% 64% 43% 68% 85% 76% 46% Average 53% 53% 53% 59% 59% 59%

Table: Evaluation of the Results for Experiment 2: Type-based Classification

(against Gold Standards 1 and 3)

SLIDE 25

Semantic Relations Annotation Experiments Conclusions and Future Work

Annotation

Problems with current annotation tasks:

Type-based annotation task - no indication about the intended meaning of verbs Token-based annotation task - fails to select representative contexts that cover all readings, sometimes complex structure and interpretation difficulties, more time consuming

Proposed solutions:

Integration of prototypical arguments in type-based annotation task Question-based annotation scenario to elicit answers corresponding to inference patterns

SLIDE 26

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario

Figure: Decision Tree for distinguishing between semantic relations

SLIDE 27

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play John won the game. Did he play the game?

SLIDE 28

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play John won the game. Did he play the game? yes

SLIDE 29

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play John didn’t win the game. Did he play the game?

SLIDE 30

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play John didn’t win the game. Did he play the game? maybe

SLIDE 31

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play Did John win the game after playing it?

SLIDE 32

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play Did John win the game after playing it? yes

SLIDE 33

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario - Example

win - play Annotation: Presupposition

SLIDE 34

Semantic Relations Annotation Experiments Conclusions and Future Work

Question-based annotation scenario

Decision tree for deriving annotations

Exhaustive classification of all relations using maximally 3 questions.

SLIDE 35

Semantic Relations Annotation Experiments Conclusions and Future Work

Conclusion

Discriminative analysis and semantic classification of verb relations including presupposition Distinguishing semantic properties of 5 relations in terms of negation and temporal sequence Report on first annotation and classification experiments

Analysis of annotation problems Proposal for a question-based annotation guide accessible for non-linguists Using prototypical arguments for disambiguation Acquire larger data set using Amazon Mechanical Turk

Ultimate goal: annotating implicit semantic relations in context to enhance textual interpretation

SLIDE 36

Semantic Relations Annotation Experiments Conclusions and Future Work

Questions Thank your for your attention!

SLIDE 37

Semantic Relations Annotation Experiments Conclusions and Future Work

Questions Thank your for your attention!

Questions?

SLIDE 38

Semantic Relations Annotation Experiments Conclusions and Future Work

Examples of correct and wrong classification

Sem. Relation

Verb pair Correct classification Wrong classification (System label) Presupposition classify - It was noted that of the thirteen The meeting focussed on issues of identify issues identified in the report eight identifying, classifying and marking were classified as high priority. up names in both corpora and analytical projects. (Temp. Inclusion) Entailment click - send Clicking the Send feedback button You can send us your comments by will send any feedback you have simply clicking on this email. entered. (None)

Temp. Inclusion

reply - say Replying to the toast to the guests, 18 out of the 20 Rehabilitation Officers Dr Julia King said how privileged who replied said that there is somewhere the Faculty was to have two such they can take clients for equipment active alumni associations.

demonstrations. (None)

Antonymy disconnect - A click should be heard every time This allows you to connect and connect the antenna wire is connected or disconnect easily , simply by clicking disconnected.

n the icon and selecting the relevant
ption. (None)