improving information extraction by acquiring external
play

Improving Information Extraction by Acquiring External Evidence - PowerPoint PPT Presentation

Oct 29, 2023 •197 likes •701 views

Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning Karthik Narasimhan, Adam Yala, Regina Barzilay CSAIL, MIT 1 Information Extraction: State of the Art Dependence on large training sets ACE: 300K

  1. Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning Karthik Narasimhan, Adam Yala, Regina Barzilay CSAIL, MIT 1

  2. Information Extraction: State of the Art • Dependence on large training sets ACE: 300K words Freebase: 24M rela6ons Not available for many domains (ex. medicine, crime) • Even large corpora do not guarantee high performance ~ 75% F1 on relation extraction (ACE) ~ 58% F1 on event extraction (ACE) 2

  3. A hard reading task for you Task : Identify food carcinogens Coffee significantly reduced ER and cyclin D1 abundance in ER(+) cells … Coffee reduced the pAkt levels in both ER(+) and ER(-) cells. 3

  4. A hard reading task for you Task : Identify food carcinogens Coffee significantly reduced ER and cyclin D1 abundance in ER(+) cells … Coffee reduced the pAkt levels in both ER(+) and ER(-) cells. Is coffee a carcinogen? 3

  5. A hard reading task for machines: IE Extraction (NumWounded) A 2 year old girl and four other people were wounded in a shoo6ng in West four Englewood Thursday night, police said 4

  6. A hard reading task: IE (not always!) Extraction (NumWounded) A 2 year old girl and four other people were wounded in a shoo6ng in West four Englewood Thursday night, police said The last shoo6ng leP five people five wounded. 5

  7. Incorporate External Evidence extract + reason Traditional formulation extract agg. Our approach extra articles 6

  8. Challenges 1. Event Coreference 2. Reconciling Predictions Shooter : Scott Westerhuis NumKilled : 4 Location : S.D Shooter : Scott Westerhuis NumKilled : 6 Location : Platte Several irrelevant articles! Inconsistent extractions 7

  9. Learning through Reinforcement Original extract Shooter : Scott Westerhuis NumKilled : 4 Location : S.D Start with traditional extraction system 8

  10. Learning through Reinforcement Original extract Shooter : Scott Westerhuis NumKilled : 4 Location : S.D extract Shooter : Scott Westerhuis query NumKilled : 6 Location : Platte Perform a query and extract from a new article 9

  11. Learning through Reinforcement Current Original extract Shooter : Scott Westerhuis NumKilled : 4 Location : S.D extract Shooter : Scott Westerhuis search NumKilled : 6 Location : Platte State New 10

  12. RL: State Conf 0.3 Shooter : Scott Westerhuis Curr 0.2 NumKilled : 4 Location : S.D 0.1 0.4 Shooter : Scott Westerhuis New NumKilled : 6 0.6 Location : Platte 0.3 State 11

  13. RL: State 0.3 0.2 currentConf 0.1 Conf 0.3 Shooter : Scott Westerhuis Curr 0.2 NumKilled : 4 Location : S.D 0.1 0.4 Shooter : Scott Westerhuis New NumKilled : 6 0.6 Location : Platte 0.3 State 11

  14. RL: State 0.3 0.2 currentConf 0.1 Conf 0.4 0.3 Shooter : Scott Westerhuis 0.6 newConf Curr 0.2 NumKilled : 4 0.3 Location : S.D 0.1 0.4 Shooter : Scott Westerhuis New NumKilled : 6 0.6 Location : Platte 0.3 State 11

  15. RL: State 0.3 0.2 currentConf 0.1 Conf 0.4 0.3 Shooter : Scott Westerhuis 0.6 newConf Curr 0.2 NumKilled : 4 0.3 Location : S.D 0.1 1 0 matches 0.4 0 Shooter : Scott Westerhuis New NumKilled : 6 0.6 Location : Platte 0.3 State 11

  16. RL: State 0.3 0.2 currentConf 0.1 Conf 0.4 0.3 Shooter : Scott Westerhuis 0.6 newConf Curr 0.2 NumKilled : 4 0.3 Location : S.D 0.1 1 0 matches 0.4 0 Shooter : Scott Westerhuis New NumKilled : 6 0.6 0.65 docSim Location : Platte 0.3 State 11

  17. RL: State 0.3 0.2 currentConf 0.1 Conf 0.4 0.3 Shooter : Scott Westerhuis 0.6 newConf Curr 0.2 NumKilled : 4 0.3 Location : S.D 0.1 1 0 matches 0.4 0 Shooter : Scott Westerhuis New NumKilled : 6 0.6 0.65 docSim Location : Platte 0.3 1 0 State .. context 0 0 11

  18. RL: Actions Shooter : Scott Westerhuis Shooter : Scott Westerhuis reconcile Curr NumKilled : 4 NumKilled : 6 Location : S.D Location : S.D Shooter : Scott Westerhuis New NumKilled : 6 Location : Platte State 1 1. Reconcile (d) old values and new values. Pick a single value, all values or no value from new set 12

  19. RL: Actions Shooter : Scott Westerhuis Shooter : Scott Westerhuis reconcile Curr NumKilled : 4 NumKilled : 6 Location : S.D Location : S.D Final Shooter : Scott Westerhuis New NumKilled : 6 Location : Platte State 1 2. Decide how to proceed: Stop 13

  20. RL: Actions Shooter : Scott Westerhuis Shooter : Scott Westerhuis reconcile Curr NumKilled : 4 NumKilled : 6 Location : S.D Location : S.D select Shooter : Westerhuis Shooter : Scott Westerhuis q New NumKilled : 6 NumKilled : 4 extract Location : Platte Location : Platte search State 2 State 1 2. Decide how to proceed: Select next query (q) 14

  21. Queries Query templates are induced automa<cally • Title of original ar6cle • Content words having high mutual informa6on with gold values <title> <title> + ( suspect | shooter | said | men | arrested | …) <title> + ( injured | wounded | victims | shot | … ) 15

  22. Rewards • Change in accuracy Current Values Previous Values Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 6 NumKilled : 6 NumWounded : 1 NumWounded : 0 Location : Platte Location : Platte X = 1 Acc( e j cur ) − Acc( e j R ( s, a ) = prev ) entity j • Small penalty for each transi6on 16

  23. Deep Q-Network State space is continuous: requires function approximation (reconcile) (query) Q ( s, a ) ≈ Q ( s, a ; θ ) Trained to maximize cumulative reward 17

  24. Acquiring External Evidence 1. Select a query to search for articles on the same event 2. Use base extractor to obtain values for entities of interest Shooter : Scott Westerhuis extract NumKilled : 6 Location : Platte 3. Reconcile old and new extractions Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 4 NumKilled : 6 Location : S.D Location : Platte 18

  25. Related Work • Open Information Extraction (Etzioni et al., 2011; Fader et al., 2011; Wu and Weld, 2010) 19

  26. Related Work • Open Information Extraction (Etzioni et al., 2011; Fader et al., 2011; Wu and Weld, 2010) • Slot filling (Surdeanu et al., 2010; Ji and Grishman, 2011) 19

  27. Related Work • Open Information Extraction (Etzioni et al., 2011; Fader et al., 2011; Wu and Weld, 2010) • Slot filling (Surdeanu et al., 2010; Ji and Grishman, 2011) • Searching for additional sources on the web (Banko et al., 2002, West et al., 2014; Kanani and McCallum, 2012) 19

  28. Datasets 1. Mass shootings in the United States Shooter Name Num Killed Num Wounded City Train Test Dev Source 306 292 66 Downloaded 8k 7.9k 1.6k 20

  29. Datasets 2. Adulteration events from Foodshield EMA Food Adulterant Location Train Test Dev Source 292 148 42 Downloaded 7.6k 5.3k 1.5k 21

  30. Base Extraction Model Maximum entropy model with contextual features (Chieu and Ng, 2002; Bunescu et al., 2005) Indirect supervision: Project database values onto articles 22

  31. Baselines (1) Simple Aggregation systems: • Confidence-based : Choose entity value with highest confidence 0.3 Shooter : Scott Westerhuis Original NumKilled : 4 0.2 Location : S.D 0.1 Shooter : Scott Westerhuis 0.4 Shooter : Scott Westerhuis NumKilled : 6 0.6 NumKilled : 6 Location : Platte Location : Platte 0.3 Extra Final 0.7 Shooter : Scott Westerhuis 0.2 NumKilled : 6 Location : S.D (Skounakis and Craven, 2003) 0.1 23

  32. Baselines (1) Simple Aggregation systems: • Majority-based : Choose entity value extracted the most from all articles on the event Shooter : Scott Westerhuis Original NumKilled : 4 Location : S.D Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 6 NumKilled : 6 Location : S.D Location : Platte Extra Final Shooter : Scott Westerhuis NumKilled : 6 Location : S.D (Skounakis and Craven, 2003) 24

  33. Baselines (2) Meta-classifier: • Same input space S and set of reconciliation decisions as RL agent. Original Extra Reconciled Shooter : Scott Westerhuis Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 4 NumKilled : 6 NumKilled : 6 Location : S.D Location : Platte Location : Platte Shooter : Westerhuis Shooter : Westerhuis Shooter : Scott Westerhuis NumKilled : 0 NumKilled : 4 NumKilled : 4 Location : S.D Location : Platte Location : Platte Shooter : Scott Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 2 NumKilled : 4 NumKilled : 2 Location : S.D Location : S.D Location : S.D 25

  34. Baselines (2) Meta-classifier: • Same input space S and set of reconciliation decisions as RL agent. Original Extra Reconciled Confidence agg. Shooter : Scott Westerhuis Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 4 NumKilled : 6 NumKilled : 6 Location : S.D Location : Platte Location : Platte Shooter : Westerhuis Shooter : Westerhuis Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 0 NumKilled : 4 NumKilled : 4 NumKilled : 6 Location : S.D Location : Platte Location : Platte Location : Platte Final Shooter : Scott Shooter : Scott Westerhuis Shooter : Scott Westerhuis NumKilled : 2 NumKilled : 4 NumKilled : 2 Location : S.D Location : S.D Location : S.D 26

  35. Accuracy (Shootings) 80 NumKilled 77.6 75 Accuracy 70.7 70 70.3 69.7 65 60 Maxent Confidence Agg. Meta-Classifier RL-Extract 27

  36. Accuracy (Shootings) 80 NumKilled 77.6 75 Accuracy 70.7 70 70.3 69.7 65 60 Maxent Confidence Agg. Meta-Classifier RL-Extract 28

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Document Modeling with Document Modeling with External Attention for Sentence External Attention

Document Modeling with Document Modeling with External Attention for Sentence External Attention

Document Modeling with Document Modeling with External Attention for Sentence External Attention for Sentence Extraction Extraction Shashi Narayan, Ronald Cardenas, Nikos Papasarantopoulos, Shay B. Cohen, Mirella Lapata, Jiangsheng Yu and Yi

977 views • 80 slides
Improving Hypernymy Extraction with Distributional Semantic Classes Introduction May 10, 2018

Improving Hypernymy Extraction with Distributional Semantic Classes Introduction May 10, 2018

Alexander Panchenko, Dmitry Ustalov, Stefano Faralli, Simone Paolo Ponzetto, and Chris Biemann Improving Hypernymy Extraction with Distributional Semantic Classes Introduction May 10, 2018 Improving Hypernymy Extraction with Distributional

642 views • 53 slides
GreenFIE: A Green Form-Based Information-Extraction System for Historical Documents

GreenFIE: A Green Form-Based Information-Extraction System for Historical Documents

GreenFIE: A Green Form-Based Information-Extraction System for Historical Documents (Self-improving Extraction Systems) Tae Woo Kim David W. Embley Stephen W. Liddle A Green Information Extraction System Green systems improve with

692 views • 23 slides
Declarative Information Extraction Declarative Information Extraction Using Datalog Datalog with

Declarative Information Extraction Declarative Information Extraction Using Datalog Datalog with

Declarative Information Extraction Declarative Information Extraction Using Datalog Datalog with Embedded with Embedded Using Extraction Predicates Extraction Predicates Warren Shen, AnHai Doan, Jeffrey Naughton University of Wisconsin,

566 views • 25 slides
Information Extraction Pedro Szekely Information Sciences Institute, USC Viterbi School of

Information Extraction Pedro Szekely Information Sciences Institute, USC Viterbi School of

Information Extraction Pedro Szekely Information Sciences Institute, USC Viterbi School of Engineering 1 Agenda Information extraction classification Text extraction techniques Storing extractions in knowledge graphs myDIG demo Summary

1.29k views • 64 slides
SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released

SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released

SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released third quarter GM profit-increase 10% results showing a 10% in profit over the same period last year. John Doe was convicted Tuesday on John Doe

779 views • 40 slides
SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released

SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released

SI485i : NLP Set 13 Information Extraction Information Extraction Yesterday GM released third quarter GM profit-increase 10% results showing a 10% in profit over the same period last year. John Doe was convicted Tuesday on John Doe

579 views • 41 slides
SI425 : NLP Set 13 Information Extraction Information Extraction Yesterday GM released third

SI425 : NLP Set 13 Information Extraction Information Extraction Yesterday GM released third

SI425 : NLP Set 13 Information Extraction Information Extraction Yesterday GM released third quarter GM profit-increase 10% results showing a 10% in profit over the same period last year. John Doe was convicted Tuesday on John Doe

609 views • 41 slides
Data Mining l The Extraction of useful information from data l The automated extraction of hidden

Data Mining l The Extraction of useful information from data l The automated extraction of hidden

Data Mining l The Extraction of useful information from data l The automated extraction of hidden predictive information from (large) databases l Business, Huge data bases, customer data, mine the data Also Medical, Genetic, Astronomy, etc. l

535 views • 32 slides
Querying Probabilistic Information Extraction Daisy Zhe Wang, Michael J. Franklin, Minos

Querying Probabilistic Information Extraction Daisy Zhe Wang, Michael J. Franklin, Minos

Querying Probabilistic Information Extraction Daisy Zhe Wang, Michael J. Franklin, Minos Garofalakis, Joseph M. Hellerstein VLDB, Singapore, 16 th September, 2010 Outline Information Extraction Systems Information Extraction (IE)

324 views • 29 slides
Chapter VI: Information Extraction Information Retrieval &amp; Data Mining Universitt des

Chapter VI: Information Extraction Information Retrieval &amp; Data Mining Universitt des

Chapter VI: Information Extraction Information Retrieval &amp; Data Mining Universitt des Saarlandes, Saarbrcken Winter Semester 2011/12 Chapter VI: Information Extraction VI.1 Motivation and Overview IE systems: Wolfram Alpha, Yago-Naga,

343 views • 23 slides
HANDLING UNCERTAINTY IN INFORMATION EXTRACTION Maurice van Keulen and Mena Badieh Habib URSW 23

HANDLING UNCERTAINTY IN INFORMATION EXTRACTION Maurice van Keulen and Mena Badieh Habib URSW 23

HANDLING UNCERTAINTY IN INFORMATION EXTRACTION Maurice van Keulen and Mena Badieh Habib URSW 23 Oct 2011 INFORMATION EXTRACTION Information Unstructured Web of Data extraction Text Inherently imperfect process Word Paris source:

254 views • 8 slides
Named Entity Recognition &amp; Sequence Labeling CSCI 699: ML for Knowledge Extraction &amp;

Named Entity Recognition &amp; Sequence Labeling CSCI 699: ML for Knowledge Extraction &amp;

Named Entity Recognition &amp; Sequence Labeling CSCI 699: ML for Knowledge Extraction &amp; Reasoning Instructor: Xiang Ren USC Computer Science Recap: Information Extraction Information extraction (IE) systems Find and understand

1.82k views • 142 slides
Improving Glucose for Incremental SAT Solving with Assumptions: Application to MUS Extraction

Improving Glucose for Incremental SAT Solving with Assumptions: Application to MUS Extraction

Introduction GLUCOSE and MUSER Conclusion Improving Glucose for Incremental SAT Solving with Assumptions: Application to MUS Extraction Gilles Audemard Jean-Marie Lagniez and Laurent Simon SAT 2013 Glucose and MUS SAT 2013 1 / 17

605 views • 37 slides
Selective Sampling for Information Extraction with a Committee of Classifiers Evaluating Machine

Selective Sampling for Information Extraction with a Committee of Classifiers Evaluating Machine

Selective Sampling for Information Extraction with a Committee of Classifiers Evaluating Machine Learning for Information Extraction, Track 2 Ben Hachey, Markus Becker, Claire Grover &amp; Ewan Klein University of Edinburgh Overview

504 views • 36 slides
Convex relaxations for weakly supervised information extraction Edouard Grave Columbia

Convex relaxations for weakly supervised information extraction Edouard Grave Columbia

Convex relaxations for weakly supervised information extraction Edouard Grave Columbia University edouard.grave@gmail.com Information Extraction Extract structured information from unstructured documents. Information Extraction Extract

1.03k views • 81 slides
Mol2Net-04 Chemometric Characterization of Chemlali Extra-Virgin Olive Oil Adulteration Mixed

Mol2Net-04 Chemometric Characterization of Chemlali Extra-Virgin Olive Oil Adulteration Mixed

Mol2Net-04 , 2018 , BIOCHEMPHYS-01 (pages 1- x, type of paper, doi: xxx-xxxx http://sciforum.net/conference/mol2net-4 SciForum Mol2Net-04 Chemometric Characterization of Chemlali Extra-Virgin Olive Oil Adulteration Mixed with Soybean Oil, Corn

628 views • 9 slides
You Cant Fool the Bladder Police Effective Use of Urine Drug Screening Wh Why Te Test?

You Cant Fool the Bladder Police Effective Use of Urine Drug Screening Wh Why Te Test?

You Cant Fool the Bladder Police Effective Use of Urine Drug Screening Wh Why Te Test? Accountability Create and maintain safe treatment environment Compliance with licensing or policy Collection Supervised or

489 views • 37 slides
Dietary Supplements Stakeholder Forum Michael McGuffin, Chair Summary Discussions Wednesday,

Dietary Supplements Stakeholder Forum Michael McGuffin, Chair Summary Discussions Wednesday,

Dietary Supplements Stakeholder Forum Michael McGuffin, Chair Summary Discussions Wednesday, June 1, 2016 What We Heard USP Updates and Discussions Adulterants Database Stakeholders consumers, marketers, regulators will work

299 views • 18 slides
5/31/2016 The Community Guide to Adult Oral Health Program Implementation Holly Greuling RDN

5/31/2016 The Community Guide to Adult Oral Health Program Implementation Holly Greuling RDN

5/31/2016 The Community Guide to Adult Oral Health Program Implementation Holly Greuling RDN ACL National Nutritionist June 2016 Background Oral health is an integral part of an older persons life, as it greatly affects ones overall

193 views • 7 slides
Chair: Vicknasingam B Kasinather, Universiti Sains Malaysia, Malaysia, vickna@usm.my

Chair: Vicknasingam B Kasinather, Universiti Sains Malaysia, Malaysia, vickna@usm.my

Identifying Knowledge Gaps as Kratom Emerges on the Global Stage Summary Slides National Institute on Drug Abuse (NIDA) International Symposium at College on Problems of Drug Dependence (CPDD) San Diego, June 9 2018 We lead Chair:

532 views • 19 slides
INLs Global Drug Demand Reduction Program Bruno Bui and Bill McGlynn Global Drug Demand

INLs Global Drug Demand Reduction Program Bruno Bui and Bill McGlynn Global Drug Demand

INLs Global Drug Demand Reduction Program Bruno Bui and Bill McGlynn Global Drug Demand Reduction Programs Bureau of International Narcotics and Law Enforcement Affairs U.S. Department of State Washington, D.C. Drug Demand Reduction in

318 views • 15 slides
Nothing to Disclose Craig Smollin MD Medical Director California Poison Control System, SF

Nothing to Disclose Craig Smollin MD Medical Director California Poison Control System, SF

3/8/18 Novel Toxicants in the Environment Cases from Poison Control Nothing to Disclose Craig Smollin MD Medical Director California Poison Control System, SF Division Associate Professor of Emergency Medicine University of California, San

485 views • 16 slides
Terry Clifford, MSN, RN, CPAN ASPAN 2010-2011 Immediate Past President Mercy Hospital, Portland,

Terry Clifford, MSN, RN, CPAN ASPAN 2010-2011 Immediate Past President Mercy Hospital, Portland,

Terry Clifford, MSN, RN, CPAN ASPAN 2010-2011 Immediate Past President Mercy Hospital, Portland, ME Discuss pharmacology of Ketamine. List indications for the use of Ketamine infusions in pain management services. Describe adverse

1.01k views • 61 slides

More recommend