Improving Information Extraction by Acquiring External Evidence - - PowerPoint PPT Presentation

▶
improving information extraction by acquiring external
SMART_READER_LITE
LIVE PREVIEW

Improving Information Extraction by Acquiring External Evidence - - PowerPoint PPT Presentation

Oct 29, 2023 197 likes •704 views

Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning Karthik Narasimhan, Adam Yala, Regina Barzilay CSAIL, MIT 1 Information Extraction: State of the Art Dependence on large training sets ACE: 300K

slide-1
SLIDE 1

Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning

Karthik Narasimhan, Adam Yala, Regina Barzilay CSAIL, MIT

1

slide-2
SLIDE 2

Information Extraction: State of the Art

  • Dependence on large training sets

ACE: 300K words Freebase: 24M rela6ons

2

Not available for many domains (ex. medicine, crime)

  • Even large corpora do not guarantee high performance

~ 75% F1 on relation extraction (ACE) ~ 58% F1 on event extraction (ACE)

slide-3
SLIDE 3

3

Task: Identify food carcinogens Coffee significantly reduced ER and cyclin D1 abundance in ER(+) cells … Coffee reduced the pAkt levels in both ER(+) and ER(-) cells.

A hard reading task for you

slide-4
SLIDE 4

3

Task: Identify food carcinogens Coffee significantly reduced ER and cyclin D1 abundance in ER(+) cells … Coffee reduced the pAkt levels in both ER(+) and ER(-) cells.

Is coffee a carcinogen?

A hard reading task for you

slide-5
SLIDE 5

A hard reading task for machines: IE

4

A 2 year old girl and four other people were wounded in a shoo6ng in West Englewood Thursday night, police said four

Extraction (NumWounded)

slide-6
SLIDE 6

5

The last shoo6ng leP five people wounded. five A 2 year old girl and four other people were wounded in a shoo6ng in West Englewood Thursday night, police said four

A hard reading task: IE (not always!)

Extraction (NumWounded)

slide-7
SLIDE 7

6

Incorporate External Evidence

Traditional formulation Our approach

extract + reason extra articles extract agg.

slide-8
SLIDE 8

Challenges

7

  • 1. Event Coreference
  • 2. Reconciling Predictions

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Several irrelevant articles! Inconsistent extractions

slide-9
SLIDE 9

Learning through Reinforcement

8

extract

Original

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Start with traditional extraction system

slide-10
SLIDE 10

Learning through Reinforcement

9

extract extract query

Original

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Perform a query and extract from a new article

slide-11
SLIDE 11

Learning through Reinforcement

10

extract

Original State

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

extract search Current New

slide-12
SLIDE 12

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State

New Curr

slide-13
SLIDE 13

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1

currentConf

State

New Curr

slide-14
SLIDE 14

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1

currentConf

0.4 0.6 0.3

newConf

State

New Curr

slide-15
SLIDE 15

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1

currentConf

0.4 0.6 0.3

newConf

1

matches

State

New Curr

slide-16
SLIDE 16

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1

currentConf

0.4 0.6 0.3

newConf

1

matches

0.65

docSim

State

New Curr

slide-17
SLIDE 17

RL: State

11

0.3 0.2 0.1 0.4 0.6 0.3

Conf

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1

currentConf

0.4 0.6 0.3

newConf

1

matches

0.65

docSim

1 ..

context

State

New Curr

slide-18
SLIDE 18

State 1

RL: Actions

  • 1. Reconcile (d) old values and new values.

Pick a single value, all values or no value from new set

12

New reconcile Curr

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

slide-19
SLIDE 19

RL: Actions

13

Final

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

New

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State 1

reconcile Curr

  • 2. Decide how to proceed:

Stop

slide-20
SLIDE 20

RL: Actions

14

select

q

extract search

State 2

  • 2. Decide how to proceed:

Select next query (q)

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

Shooter: Westerhuis NumKilled: 4 Location: Platte

New

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State 1

reconcile Curr

slide-21
SLIDE 21

Queries

Query templates are induced automa<cally

  • Title of original ar6cle
  • Content words having high mutual informa6on with gold

values

<title> <title> + ( suspect | shooter | said | men | arrested | …) <title> + ( injured | wounded | victims | shot | … )

15

slide-22
SLIDE 22
  • Change in accuracy
  • Small penalty for each transi6on

Rewards

R(s, a) = X

entityj

Acc(ej

cur) − Acc(ej prev)

16

= 1

Shooter: Scott Westerhuis NumKilled: 6 NumWounded: 0 Location: Platte

Current Values

Shooter: Scott Westerhuis NumKilled: 6 NumWounded: 1 Location: Platte

Previous Values

slide-23
SLIDE 23

Deep Q-Network

State space is continuous: requires function approximation Q(s, a) ≈ Q(s, a; θ)

17

Trained to maximize cumulative reward

(reconcile) (query)

slide-24
SLIDE 24

Acquiring External Evidence

18

  • 1. Select a query to search for articles on the same event
  • 2. Use base extractor to obtain values for entities of interest
  • 3. Reconcile old and new extractions

extract

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

slide-25
SLIDE 25

Related Work

19

  • Open Information Extraction (Etzioni et al.,

2011; Fader et al., 2011; Wu and Weld, 2010)

slide-26
SLIDE 26

Related Work

19

  • Open Information Extraction (Etzioni et al.,

2011; Fader et al., 2011; Wu and Weld, 2010)

  • Slot filling (Surdeanu et al., 2010; Ji and

Grishman, 2011)

slide-27
SLIDE 27

Related Work

19

  • Open Information Extraction (Etzioni et al.,

2011; Fader et al., 2011; Wu and Weld, 2010)

  • Slot filling (Surdeanu et al., 2010; Ji and

Grishman, 2011)

  • Searching for additional sources on the web

(Banko et al., 2002, West et al., 2014; Kanani and McCallum, 2012)

slide-28
SLIDE 28

Datasets

  • 1. Mass shootings in the United States

20

Train Test Dev Source 306 292 66 Downloaded 8k 7.9k 1.6k

Shooter Name Num Killed Num Wounded City

slide-29
SLIDE 29

Datasets

  • 2. Adulteration events from Foodshield EMA

21

Train Test Dev Source 292 148 42 Downloaded 7.6k 5.3k 1.5k

Food Adulterant Location

slide-30
SLIDE 30

Base Extraction Model

22

Indirect supervision: Project database values onto articles Maximum entropy model with contextual features

(Chieu and Ng, 2002; Bunescu et al., 2005)

slide-31
SLIDE 31

Baselines (1)

Simple Aggregation systems:

  • Confidence-based: Choose entity value with

highest confidence

23

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

0.3 0.2 0.1 0.4 0.6 0.3 0.7 0.2 0.1

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Original Extra Final

(Skounakis and Craven, 2003)

slide-32
SLIDE 32

Baselines (1)

Simple Aggregation systems:

  • Majority-based: Choose entity value extracted

the most from all articles on the event

24

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

Original Extra Final

(Skounakis and Craven, 2003)

slide-33
SLIDE 33

Baselines (2)

25

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Shooter: Westerhuis NumKilled: 0 Location: Platte

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Shooter: Scott NumKilled: 2 Location: S.D

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Shooter: Westerhuis NumKilled: 4 Location: Platte

Shooter: Scott Westerhuis NumKilled: 2 Location: S.D

Meta-classifier:

  • Same input space S and set of reconciliation

decisions as RL agent.

Original Extra Reconciled

slide-34
SLIDE 34

Baselines (2)

26

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Final

Confidence agg.

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Shooter: Westerhuis NumKilled: 0 Location: Platte

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D

Shooter: Scott NumKilled: 2 Location: S.D

Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

Shooter: Westerhuis NumKilled: 4 Location: Platte

Shooter: Scott Westerhuis NumKilled: 2 Location: S.D

Original Extra Reconciled

Meta-classifier:

  • Same input space S and set of reconciliation

decisions as RL agent.

slide-35
SLIDE 35

Accuracy (Shootings)

27

Accuracy

60 65 70 75 80

Maxent Confidence Agg. Meta-Classifier RL-Extract

77.6 70.7 70.3 69.7

NumKilled

slide-36
SLIDE 36

Accuracy (Shootings)

28

Accuracy

60 65 70 75 80

Maxent Confidence Agg. Meta-Classifier RL-Extract

77.6 70.7 70.3 69.7

NumKilled

slide-37
SLIDE 37

Accuracy (Adulterations)

29

Food

Accuracy

50 53.75 57.5 61.25 65

Maxent Majority Agg. Meta-Classifier RL-Extract

59.6 55.4 56.7 56.0

slide-38
SLIDE 38

Oracle

  • Given:
  • Same base extractor
  • Same set of queries
  • Agent performing perfect reconciliation and querying

decisions.

  • Upper-bound on performance of any system given

these extra articles on each event.

30

slide-39
SLIDE 39

Accuracy (Shootings)

31

Accuracy 50 60 70 80 90

Maxent RL-Extract Oracle

86.4 77.6 69.7

NumKilled

slide-40
SLIDE 40

RL-Extract

32

select

q

extract search

State 2

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

Shooter: Westerhuis NumKilled: 4 Location: Platte

New Old

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State 1

reconcile

Both reconciliation and querying

slide-41
SLIDE 41

RL-Basic

33

select

q

extract search

State 2

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

Shooter: Westerhuis NumKilled: 4 Location: Platte

New Old

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State 1

reconcile

Documents are presented in round robin order from different query lists

slide-42
SLIDE 42

RL-Query

34

select

q

extract search

State 2

Shooter: Scott Westerhuis NumKilled: 6 Location: S.D

Shooter: Westerhuis NumKilled: 4 Location: Platte

New Old

Shooter: Scott Westerhuis NumKilled: 4 Location: S.D Shooter: Scott Westerhuis NumKilled: 6 Location: Platte

State 1

reconcile

Reconciliation is confidence-based

slide-43
SLIDE 43

RL Models

Accuracy 50 57.5 65 72.5 80

RL-Basic RL-Query RL-Extract

77.6 66.6 71.2

Both reconciliation and querying are important and inter-linked

35

NumKilled

slide-44
SLIDE 44

Agent learns to balance all entity choices simultaneously

36

  • Avg. Reward

ShooterName NumKilled NumWounded City

Evolution of Test Accuracy

slide-45
SLIDE 45

Examples

37

Text Shooter Name Basic Extractor A source tells Channel 2 Ac6on News that Thomas Lee has been arrested in Mississippi ... Sgt . Stewart Smith, with the Troup County Sheriff’s office, said.

Stewart

RL-Extract Lee is accused of killing his wife, Chris6e; …

Lee

slide-46
SLIDE 46

Examples

38

Text NumKilled Basic Extractor Shoo6ng leaves 25 year old Pi_sfield man dead , 4 injured RL-Extract One man is dead aPer a shoo6ng Saturday night at the intersec6on of Dewey Avenue and Linden Street. 1

Our system finds alternative sources of information for reliable extraction

slide-47
SLIDE 47

Adulteration Detection

39

slide-48
SLIDE 48

Conclusion

  • Alternative paradigm to improve Information

Extraction, especially for low-resource domains.

  • Use of Reinforcement Learning to find and

incorporate external information.

40

Code and data available at: http://people.csail.mit.edu/karthikn/rl-ie/

slide-49
SLIDE 49

41