TAC KBP 2016 Linguistic Resources: Event Arguments (EA), Event - - PowerPoint PPT Presentation

TAC KBP 2016 Linguistic Resources: Event Arguments (EA), Event Nuggets (EN) and Belief/Sentiment (BeSt)

Joe Ellis (presenter), Jennifer Tracey (presenter), Jeremy Getman, Zhiyi Song, Ann Bies, Stephanie Strassel Linguistic Data Consortium University of Pennsylvania

Introduction and Overview

 Linguistic resources for TAC KBP

2016

 Eighth year LDC produced KBP

resources

 Twenty-nine new data sets

 Two primary goals

 Increase coordination across tracks  Increase multi-lingual evaluation tracks

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

 Yesterday

 Doc selection, ED&L, and Cold Start

 Today

 Event Arguments, Event Nuggets, and Belief/Sentiment (BeSt)

Entities, Relations, & Events (ERE)

 Entities, Relations, and Events (ERE)

 Ongoing annotation task developed by LDC for DARPA’s Deep

Exploration and Filtering of Text program (DEFT)

 Exhaustive labeling of entities, relations and events and their

attributes.

 ERE annotation performed as upstream task

 Provided inputs for multiple downstream tasks supporting

ED&L, EA, EN, and BeSt

 Primary means of meeting increased coordination of data goal

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

ERE Annotation

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

The Bo Xilai event was ignited by Lijun running into the US consulate in 2012 to bring Bogu Kailai’s killing to light. Will Bo Xilai end up in jail due to bribery and corruption; what will his wife end up with?

Entity

PER NAM Bo Xilai, Bo Xilai SPC IND PRO his PER NAM Lijun SPC IND PER NAM NOM Bogu Kailai his wife SPC IND GPE NAM US SPC IND LOC NOM US consulate SPC IND LOC NOM jail Non SPC

Filler (Rich ERE) Time 2012-XX-XX Crime bribery and corruption Relation R1 Physical.Located Lijun into US consulate R2 wife Personal.Family his his wife

Event Type.Subtype Realis Trigger Arguments H1 Movement. Transport Person Actual running Lijun US consulate 2012 H2 Conflict. Attack Actual killing Bogu Kailai H3 Justice. Jail Other jail Bo Xilai bribery and corruption H4 Life.Die Actual killing Bogu Kailai H5 Movement. Transport Person Other end up Bo Xilai jail

ERE Event Types

Conflict.Attack Manufacture.Artifact Justice.ArrestJail Conflict.Demonstrate Movement.TransportArtifact Life.Die Contact.Broadcast Movement.TransportPerson Life.Injure Contact.Contact Personnel.Elect Transaction.Transaction Contact.Correspondence Personnel.EndPosition Transaction.TransferMoney Contact.Meet Personnel.StartPosition Transaction.TransferOwnership

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

 ERE event type inventory reduced for 2016

 8 types and 18 subtypes in 2016 (listed below)  9 types and 38 subtypes in 2015 (listed in overview paper)

 Most of the dropped event types and subtypes are scarce in existing

data

ERE annotation counts

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

500 1000 1500 2000 2500 3000 3500 4000 NW DF NW DF NW DF Chinese Chinese English English Spanish Spanish Entities Fillers Relations Event Mentions Event Hoppers

ERE event mentions

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

100 200 300 400 500 600 700 CMN ENG SPA

Event Argument (EA) Overview

 Given new approach to evaluating EA in 2016,

data development procedure overhauled

 2014-2015

 Manual run  Argument-level assessment

 2016

 Gold Standard  Event-level cross-document task

• Queries, manual run, assessment

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

EA Gold Standard

ERE annotations on core source corpus Augmentation pass

 BBN script run over ERE data  Annotators review results  Inferred arguments  Locational containment

 Not annotated in ERE  Baghdad as Place of Conflict.Attack  Iraq added as 2nd Place

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

 Query selection

 51 simple, low-granularity queries  Event arguments in EAL Gold Standard  Annotators reviewed over 1300 potential queries

 Manual Run

 Exhaustive across full 30K English source corpus  Justification strings indicating presence of event hopper in doc

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

EA Cross-Document Queries & Manual Run

Personnel.End-Position Person – Thabo Mbeki

EA Cross-document Assessment and Results

 Does justification prove presence of query in document?

 Correct: Response contains query event  Event Type Match: Contains event of same type as query, but not

query event

 Wrong: Doesn’t contain query event or event of same type

 Low system recall on manually selected queries

 BBN produced 249 “derived” queries based on system responses

 No LDC manual run for these queries

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

EA Cross-Document Results

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

100 200 300 400 500 600 700 LDC Systems CORRECT ET_MATCH WRONG

Manual Queries

1000 2000 3000 4000 5000 6000 7000 Systems CORRECT ET_MATCH WRONG

Derived Queries

Event Nuggets and Linking (ENL)

 No separate ENL annotation task

 Data are entirely produced by running a script over ERE data to

extract and reformat a subset for use by ENL

TAC KBP 2016 Evaluation Workshop - NIST, November 14-15

500 1000 1500 2000 2500 3000 NW DF NW DF NW DF CMN CMN ENG ENG SPA SPA Event Nuggets Event Hoppers

Belief and Sentiment Annotation

 Only ERE entities are holders of belief and sentiment  Only ERE entities, relations and events are targets of belief

and sentiment

 For events only, belief marked for each argument as well as the

event itself

 Belief values: committed, non-committed, reported, n/a

 Polarity also marked

 Sentiment values (polarity): positive, negative  Sarcasm flag indicated when polarity annotated is opposite

• f literal meaning (based on context)

TAC KBP Workshop, November 14-15, 2016

Belief Annotation Example

TAC KBP Workshop, November 14-15, 2016

Ominous new action by UKRAINE’S SECURITY FORCES

• n Monday, including a raid on AN OPPOSITION

PARTY’S HEADQUARTERS, appeared to diminish

prospects for talks between THE GOVERNMENT and

PROTEST LEADERS, as WESTERN OFFICIALS grasped for

a way to defuse THE COUNTRY’s intensifying political crisis.

Relation: Ukraine’s security forces CB Event: raid CB Arguments also CB Event: talks NA Arguments also NA

BeSt Data Overview

 English evaluation data notably more dense in belief

annotations than training data

 Spanish evaluation data less dense than training data in

both belief and sentiment

TAC KBP Workshop, November 14-15, 2016

Language Belief annotations Sentiment annotations Training Evaluation Training Evaluation # Ann #/Doc # Ann #/Doc # Ann #/Doc # Ann #/Doc Chinese 13,192 66 12,163 76 27,982 140 18,982 118 English 18,915 77 21,188 128 38,664 157 25,358 154 Spanish 9,406 99 12,546 75 14,299 151 17,353 103

Comparison of Training and Eval Data - Belief

TAC KBP Workshop, November 14-15, 2016 2000 4000 6000 8000 10000 12000 14000 16000 Committed Non-Committed Reported N/A Committed Non-Committed Reported N/A Committed Non-Committed Reported N/A CMN CMN CMN CMN ENG ENG ENG ENG SPA SPA SPA SPA Training Evaluation

Comparison of Training and Eval Data - Sentiment

DEFT PI Meeting, May 28-29, 2015 – Boulder, CO 5000 10000 15000 20000 25000 30000 35000 Positive Negative None Positive Negative None Positive Negative None CMN CMN CMN ENG ENG ENG SPA SPA SPA Training Evaluation