Extending Fine-Grained Semantic Relation Classification to - PowerPoint PPT Presentation

Semantic Relations Annotation Experiments Conclusions and Future Work Extending Fine-Grained Semantic Relation Classification to Presupposition Relations between Verbs Galina Tremper and Anette Frank Department of Computational Linguistics Heidelberg University, Germany Beyond semantics: Corpus-based investigations of pragmatic and discourse phenomena DGfS Workshop, G¨ ottingen February 23-25, 2011

Semantic Relations Annotation Experiments Conclusions and Future Work Motivation Known Event On Sunday, Olaf Scholz won the state elections in Hamburg to gain absolute majority for the SPD.

Semantic Relations Annotation Experiments Conclusions and Future Work Motivation Known Event On Sunday, Olaf Scholz won the state elections in Hamburg to gain absolute majority for the SPD.

Semantic Relations Annotation Experiments Conclusions and Future Work Content of Talk Semantic Relations between Verbs 1 Challenges in the Annotation of Inference Relations 2 Automatic Classification Experiments 3 Conclusions and Ideas for Future Work 4

Semantic Relations Annotation Experiments Conclusions and Future Work Semantic Relations Overview Sem. Relation Inference Pattern Example Substitution in Pattern Presupposition V 1 presupposes V 2 , win - play winning presupposes playing not V 1 presupposes V 2 not winning presupposes playing Entailment V 1 implies V 2 , kill - die killing implies dying not V 1 doesn’t imply V 2 not killing doesn’t imply dying Temporal V 1 happens during V 2 or snore - sleep snoring happens during sleeping Inclusion V 1 is a special form of V 2 mutter - talk muttering is a special form of talking Antonymy either V 1 or V 2 , go - stay either going or staying V 1 is the opposite of V 2 going is the opposite of staying Other/unrelated none of the above jump - sing Table: Semantic Relations and Inference Patterns for Guiding Annotation

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Notes P(resupposition), E(ntailment), T(emporal Inclusion), A(ntonymy), S(ynonymy) V 1 - trigger verb, V 2 - inferred verb

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Presupposition verb pair (V 1 : win, V 2 : play ) win succeeds play

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Presupposition verb pair (V 1 : win, V 2 : play ) win succeeds play win → play

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Presupposition verb pair (V 1 : win, V 2 : play ) win succeeds play win → play not win → play - persistance under negation

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Presupposition verb pair (V 1 : win, V 2 : play ) win succeeds play win → play not win → play - persistance under negation not win → not play - cancellation

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Properties of Semantic Relations Behaviour under Negation V 1 → V 2 ¬ V 1 → V 2 V 1 → ¬ V 2 ¬ V 1 → ¬ V 2 V 1 precedes V 2 E E Temporal V 1 succeeds V 2 P P P Sequence E E No temporal E E sequence T T T A A S S Table: Properties of Semantic Relations Distinguishing Properties Presupposition/Temporal Inclusion vs. Entailment - persistance under negation Temporal Inclusion vs. Presupposition - temporal sequence Antonymy vs. Entailment/Presupposition/Temporal Inclusion - negation properties

Semantic Relations Annotation Experiments Conclusions and Future Work Annotation Tasks Goal: Automatic classification of target relations in context Prerequisite: Annotations for training a classifier Building two Gold Standards: Gold Standard 1: Type-based Annotation - verb pairs given as types, without context (e.g. win - play; learn - know ) A sample of 100 verb pairs We allow more than one label Gold Standard 2: Token-based Annotation - verb pairs are presented in context (occurrence in one sentence) The same 100 verb pairs Up to 10 randomly selected contexts for each verb pair Only one label per sentence is allowed

Semantic Relations Annotation Experiments Conclusions and Future Work Gold Standards 1 Gold Standard 1 (GS1) - Type-based Annotation Inter-annotator agreement 63% (Kappa value - 0.47) 2 Gold Standard 2 (GS2) - Token-based Annotation Inter-annotator agreement 77.4% (Kappa value - 0.44) 3 Gold Standard 3 (GS3) - Type-based Annotation deduced from GS2 Up to three most frequent annotations

Semantic Relations Annotation Experiments Conclusions and Future Work Correlation between GS1 and GS2 62% - Overlap of labels (the same labels were assigned) 28% - Divergent contexts: Labels assigned on the token level are not present on the type level (difficulty of considering all the verb meanings out of context) Labels assigned on the type level are not found on the token level (not enough contexts) 10% - Conflicting annotations (e.g. presupposition vs. entailment )