Semantically Constrained Multilayer Annotation The Case of - PowerPoint PPT Presentation

Semantically Constrained Multilayer Annotation The Case of Coreference Jakob Prange, Nathan Schneider, and Omri Abend

How to annotate coreference? DMR, August 1, 2019 Prange, Schneider, Abend 2

Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 3

Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 4

Did anyone else have these fears ? DMR, August 1, 2019 Prange, Schneider, Abend 5

Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 6

Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 7

Representing Coreference Is Not Trivial What to annotate How to annotate How to annotate as mentions? mention spans? coreference? Syntactic criteria Minimum spans Identity, Bridging, … (e.g., all nouns) (only head words) Semantic criteria Maximum spans Unordered clusters (e.g., all events) (plus args & mods) or ordered chains Tricky linguistic Singletons? Hybrid? phenomena (e.g., coordination) DMR, August 1, 2019 Prange, Schneider, Abend 8

Representing Coreference Is Not Trivial • Many of these decisions seem arbitrary • Many different approaches with different guidelines [Poesio et al., 2016] • Problematic for evaluation [Moosavi and Strube, 2016; Moosavi et al., 2019] • Seldom integrated with other layers of meaning DMR, August 1, 2019 Prange, Schneider, Abend 9

DMR, August 1, 2019 Prange, Schneider, Abend H10

Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend 10

Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend H11

Our Approach: Build upon a basic framework for semantic units that can be shared among many higher-level meaning representations. DMR, August 1, 2019 Prange, Schneider, Abend 11

Semantic Multilayering UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12

Semantic Multilayering Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12

Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H12

Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H13

Semantic Multilayering World Knowledge Coreference UCCA DMR, August 1, 2019 Prange, Schneider, Abend H13

Semantic Multilayering with UCCA [Abend & Rappoport, 2013] • Identify and relate ”scenes” (events) and participants → Basic semantic units • No assumptions about grammar or lexicon → Cross-linguistically applicable DMR, August 1, 2019 Prange, Schneider, Abend 14

UCoref: UCCA + Coreference Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 15

UCoref: UCCA + Coreference Scene-evoker Scene-evoker Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16

UCoref: UCCA + Coreference Scene-evoker Scene-evoker Participant Participant Participant Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16

UCoref: UCCA + Coreference Scene-evoker Scene-evoker Participant Participant Participant Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H16

UCoref: Streamlined Annotation • UCCA units can be filtered in preprocessing • All scene and participant units are automatically considered mentions DMR, August 1, 2019 Prange, Schneider, Abend 17

UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H18

UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H18

UCoref: Streamlined Annotation • UCCA units can be filtered in preprocessing • Human annotators identify remaining mentions and coreference clusters • Semantic heads serve as minimum span versions of complex mentions DMR, August 1, 2019 Prange, Schneider, Abend 19

UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20

UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20

UCoref: Streamlined Annotation Did anyone else have these fears ? How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend H20

Comprehensive Representation ✓ Flexible mention spans ✓ Event and entity coreference in one framework ✓ Anchored in predicate argument structure, which… … helps humans disambiguate … can be utilized in automatic resolution (as features, or in a joint or MTL setup) DMR, August 1, 2019 Prange, Schneider, Abend 21

Drawback: Need UCCA Annotations First • Efficiently annotatable by non-experts • Can use automatic parsers • It’s worth it! DMR, August 1, 2019 Prange, Schneider, Abend 22

Pilot Annotation • Small samples from 3 English coreference corpora • OntoNotes [Hovy et al., 2006] : Blog posts • GUM [Zeldes, 2017] : WikiHow articles • RED [O’Gorman et al., 2016] : Forum discussions • Similar genres, different guidelines Hypothesis: UCoref covers (most of) what existing schemes cover, and more. DMR, August 1, 2019 Prange, Schneider, Abend 23

Hypothesis: UCoref covers (most of) what existing schemes cover, and more. OntoNotes GUM RED UCoref Anchored in syntax syntax tokens semantics Mention criteria syntax syntax semantics semantics Mention spans max max min flexible ✗ ( ✓ ) ✓ ✓ Events ✗ ( ✓ ) ✓ ✓ Singletons Annotation guidelines point in this direction. DMR, August 1, 2019 Prange, Schneider, Abend 24

Hypothesis: UCoref covers (most of) what existing schemes cover, and more. UCoref UCoref mentions referents mentions referents < < OntoNotes 40 128 20 96 GUM 288 466 155 291 Numbers of mentions and referents confirm: UCoref covers more than other schemes. DMR, August 1, 2019 Prange, Schneider, Abend 25

Hypothesis: UCoref covers (most of) what existing schemes cover, and more. UCoref UCoref mentions referents mentions referents ≈ ≈ RED 120 117 82 78 RED is very similar to UCoref in terms of coverage. DMR, August 1, 2019 Prange, Schneider, Abend 26

High Recall Exact Match 100 75 50 % 25 0 OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27

Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match 100 100 75 75 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27

Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match Mention spans are 100 100 crucial for 75 75 evaluation. 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27

Iterative greedy 1-to-1 alignment High Recall based on Dice coefficient. Exact Match Parmal Match Mention spans are 100 100 crucial for Moosavi et al. [2019] 75 75 automatically extract evaluation. min spans. 50 50 % 25 25 0 0 OntoNotes GUM RED OntoNotes GUM RED Menmons Referents DMR, August 1, 2019 Prange, Schneider, Abend H27

Conclusions • Scattered coreference research needs to be unified • Related: Universal Coreference Initiative • Semantic representations should be semantically anchored and modular • UCoref is a first step in that direction • Main advantages: efficiency, consistency, and reusability DMR, August 1, 2019 Prange, Schneider, Abend 28

Ap Approa oaches es To Semantic Multilayer ering Highly modular Prague OntoNotes GUM [Böhmová et al., 2003] [Hovy et al., 2006] [Zeldes, 2017] Modular Decompositional Multi-sentence AMR Semantics [O’Gorman et al., 2018] [White et al., 2016] PropBank 1 Layer [Palmer et al., 2005] AMR FrameNet [Banarescu et al., 2013] RED [Baker et al., 1998] [O’Gorman et al., 2016] Sentence Semantics Syntax Token ANCHORING DMR, August 1, 2019 Prange, Schneider, Abend 29

Data & Code Paper Thank you! Questions? Jakob Prange, Nathan Schneider, and Omri Abend jakob@cs.georgetown.edu https://arxiv.org/abs/1906.00663 https://github.com/jakpra/UCoref

Appendix DMR, August 1, 2019 Prange, Schneider, Abend 31

Representing Coreference A: Did anyone else have these fears ? A: How did you get over them ? DMR, August 1, 2019 Prange, Schneider, Abend 32