Annotating Reduced Argument Scope Using QA-SRL Gabriel Stanovsky, - - PowerPoint PPT Presentation

▶
annotating reduced argument
SMART_READER_LITE
LIVE PREVIEW

Annotating Reduced Argument Scope Using QA-SRL Gabriel Stanovsky, - - PowerPoint PPT Presentation

Aug 21, 2022 175 likes •468 views

Annotating Reduced Argument Scope Using QA-SRL Gabriel Stanovsky, Ido Dagan and Meni Adler Contributions 1. Focus on minimal argument spans 2. Linguistic constructions characterizing minimality 3. Reliable crowdsourcing of minimal arguments

slide-1
SLIDE 1

Annotating Reduced Argument Scope Using QA-SRL

Gabriel Stanovsky, Ido Dagan and Meni Adler

slide-2
SLIDE 2

Contributions

  • 1. Focus on minimal argument spans
  • 2. Linguistic constructions characterizing minimality
  • 3. Reliable crowdsourcing of minimal arguments annotation
slide-3
SLIDE 3

Argument Span

slide-4
SLIDE 4

Argument Span

Obama, the 44th president, was born in Hawaii

  • Arguments are typically perceived as answering role questions
  • Who was born somewhere?
  • Where was someone born?
slide-5
SLIDE 5

Argument Span: “Inclusive” Approach

  • Arguments are full syntactic constituents
  • PropBank
  • FrameNet
  • AMR

Obama born president 44th the Hawaii

in

slide-6
SLIDE 6

Argument Span: “Inclusive” Approach

  • Arguments are full syntactic constituents
  • PropBank
  • FrameNet
  • AMR

Obama born Hawaii president 44th the

Who was born somewhere? Where was someone born?

in

slide-7
SLIDE 7

Can we go shorter?

Obama, the 44th president, was born in Hawaii

  • More concise, yet sufficient answer

Who was born somewhere?

slide-8
SLIDE 8

Motivation: Applications

  • Sentence Simplification

Barack Obama, the 44th president, thanked vice president Joe Biden and Hillary Clinton, the secretary of state

slide-9
SLIDE 9

Motivation: Applications

  • Sentence Simplification

Barack Obama, the 44th president, thanked vice president Joe Biden and Hillary Clinton, the secretary of state

slide-10
SLIDE 10

Motivation: Applications

  • Sentence Simplification

Barack Obama, the 44th president, thanked vice president Joe Biden and Hillary Clinton, the secretary of state

  • Knowledge Representation
  • Question Answering
slide-11
SLIDE 11

Motivation: Qualitative Evidence

  • Having shorter arguments improved performance in
  • Open IE (Corro et al., 2013)
  • TAC-KBP Slot Filling Task (Angeli et al. ,2015)
  • Text Comprehension (Stanovsky et al., 2015)
slide-12
SLIDE 12

What is a minimal span?

slide-13
SLIDE 13

Problem Formulation

  • Given:
  • 𝑞 - predicate in a sentence
  • Obama, the newly elected president, flew to Russia
  • 𝑏 = {𝑥1, … 𝑥𝑜} - non-reduced “PropBank” argument
  • Obama, the newly elected president
  • 𝑅(𝑞, 𝑏) - argument role question
  • Who flew somewhere?
slide-14
SLIDE 14

Problem Formulation

  • Find:

𝑁(𝑞, 𝑏)- a set of minimally scoped arguments, jointly answering 𝑹 Barack Obama, the 44th president, thanked vice president Joe Biden and Hillary Clinton, the secretary of state

  • 𝑅1: Who thanked someone?

𝑁(𝑅1): Barack Obama

  • 𝑅2: Who was thanked?

𝑁(𝑅2): Joe Biden; Hillary Clinton

slide-15
SLIDE 15

Background: QA-SRL Annotation

  • Recently, He et al. (2015) suggested pred-arg annotation by

explicitly asking and answering argument role questions

  • Published a large predicate-argument corpus annotated by QA pairs
  • Utilized in our annotation as follows…
slide-16
SLIDE 16

Expert Annotation Experiment

  • Using questions annotated in QA-SRL
  • Re-answer with minimal arguments
  • Annotated 260 arguments in 100 predicates
slide-17
SLIDE 17

Expert Annotation Experiment

  • Using questions annotated in QA-SRL
  • Re-answer with minimal arguments
  • Annotated 260 arguments in 100 predicates

Our criterion can be consistently annotated by experts

slide-18
SLIDE 18

Linguistic Characterization of Minimality

  • 1. Removal of tokens from 𝑏

=> Omission of non-restrictive modification

  • 2. Splitting 𝑏

=> Decoupling distributive coordinations

slide-19
SLIDE 19

Restrictive vs. Non-Restrictive

  • Restrictive
  • She wore the necklace that her mother gave her
  • Non – Restrictive
  • Obama , the newly elected president, flew to Russia
slide-20
SLIDE 20

Distributive vs. Non-Distributive

  • Distributive
  • Obama and Clinton were born in America
  • Non-Distributive
  • John and Mary met at the university
slide-21
SLIDE 21

Distributive vs. Non-Distributive

  • Distributive
  • Obama and Clinton were born in America
  • Non-Distributive
  • John and Mary met at the university

Obama was born in America Clinton was born in America John met at the university Mary met at the university

X V V X

slide-22
SLIDE 22

Impact on PropBank

The average reduced argument shrunk by 58% Arguments reduced 24% Non-Restrictive 19% Distributive 5%

Our annotation significantly reduces PropBank argument spans

slide-23
SLIDE 23

Non-expert Annotation

slide-24
SLIDE 24

Does QA-SRL Captures Minimality?

  • QA-SRL guidelines do not specifically aim to minimize arguments
slide-25
SLIDE 25

Does QA-SRL Captures Minimality?

  • QA-SRL guidelines do not specifically aim to minimize arguments

Non-experts intuitively minimize argument span

slide-26
SLIDE 26

Can We Do Better?

  • Ask turkers to re-answer the QA-SRL questions:
  • “Specify the shortest possible answer from which the entity is identifiable”
slide-27
SLIDE 27

Can We Do Better?

  • Ask turkers to re-answer the QA-SRL questions:
  • “Specify the shortest possible answer from which the entity is identifiable”

Explicit guidelines yield more consistent argument spans

slide-28
SLIDE 28

Conclusion

  • Minimal argument scope
  • Motivated by applications
  • Linguistic characterization of argument minimality
  • Removing non-restrictive modification (long paper in ACL)
  • Decoupling distributive coordinations
  • Consistent and intuitive non-expert annotation

Thanks for listening!