dynamic programming for linear time incremental parsing
play

Dynamic Programming for Linear-Time Incremental Parsing Liang Huang - PowerPoint PPT Presentation

Dec 21, 2023 •316 likes •1.21k views

Dynamic Programming for Linear-Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California (Joint work with Kenji Sagae, USC/ICT) JHU CLSP Seminar September 14, 2010 Remembering Fred Jelinek

  1. Dynamic Programming for Linear-Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California (Joint work with Kenji Sagae, USC/ICT) JHU CLSP Seminar September 14, 2010

  2. Remembering Fred Jelinek (1932-2010) Prof. Jelinek hosted my visit and this talk on his last day. He was very supportive of this work, which is related to his work on structured language models, and I dedicate my work to his memory.

  3. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally DP for Incremental Parsing 3

  4. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; DP for Incremental Parsing 3

  5. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  6. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  7. Ambiguity and Incrementality • NLP is (almost) all about ambiguity resolution • human-beings resolve ambiguity incrementally One morning in Africa, I shot an elephant in my pajamas; how he got into my pajamas I’ll never know. DP for Incremental Parsing 3

  8. Ambiguities in Translation CS 562 - Intro 4

  9. Ambiguities in Translation CS 562 - Intro 4

  10. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  11. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  12. Ambiguities in Translation Google translate: carefully slide CS 562 - Intro 4

  13. If you are stolen... CS 562 - Intro 5

  14. If you are stolen... Google translate: Once the theft to the police CS 562 - Intro 5

  15. or even... CS 562 - Intro 6

  16. or even... clear evidence that NLP is used in real life! CS 562 - Intro 6

  17. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  18. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  19. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  20. Ambiguities in Parsing I feed cats nearby in the garden ... • let’s focus on dependency structures for simplicity • ambiguous attachments of nearby and in • ambiguity explodes exponentially with sentence length • must design efficient (polynomial) search algorithm • typically using dynamic programming (DP); e.g. CKY DP for Incremental Parsing 7

  21. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) DP for Incremental Parsing 8

  22. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) DP for Incremental Parsing 8

  23. But full DP is too slow... I feed cats nearby in the garden ... • full DP (like CKY) is too slow (cubic-time) • while human parsing is fast & incremental (linear-time) • how about incremental parsing then? • yes, but only with greedy search (accuracy suffers) • explores tiny fraction of trees (even w/ beam search) • can we combine the merits of both approaches? • a fast, incremental parser with dynamic programming? • explores exponentially many trees in linear-time? DP for Incremental Parsing 8

  24. Linear-Time Incremental DP incremental parsing greedy (e.g. shift-reduce) search ☹ (Nivre 04; Collins/Roark 04; ...) this work: full DP principled fast shift-reduce parsing (e.g. CKY) search with dynamic programming ☺ (Eisner 96; Collins 99; ...) ☹ ☺ fast slow (linear-time) (cubic-time) DP for Incremental Parsing 9

  25. Big Picture natural programming languages languages ☺ ☹ human psycholinguistics ☹ NLP ☺ computer compiler theory (LR, LALR, ...) DP for Incremental Parsing 10

  26. Big Picture natural programming languages languages ☺ ☹ human psycholinguistics ☹ NLP ☺ computer compiler theory (LR, LALR, ...) DP for Incremental Parsing 10

  27. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 parsing time (secs) 1.2 1 0.8 0.6 0.4 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing 11

  28. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y k e parsing time (secs) a l i e 1.2 n k r r a 1 e h B C MST 0.8 0.6 0.4 this work 0.2 0 0 10 20 30 40 50 60 70 sentence length DP for Incremental Parsing 11

  29. Preview of the Results • very fast linear-time dynamic programming parser • best reported dependency accuracy on PTB/CTB • explores exponentially many trees (and outputs forest) 1.4 y 10 10 DP: exponential k e parsing time (secs) a l i e 1.2 n k 10 8 r r a 1 e h B 10 6 C MST 0.8 0.6 10 4 0.4 10 2 this work 0.2 non-DP beam search 10 0 0 0 10 20 30 40 50 60 70 0 10 20 30 40 50 60 70 sentence length sentence length DP for Incremental Parsing 11

  30. Outline • Motivation • Incremental (Shift-Reduce) Parsing • Dynamic Programming for Incremental Parsing • Experiments DP for Incremental Parsing 12

  31. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - feed cats nearby ... cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 13

  32. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 14

  33. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... cats nearby in ... nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 15

  34. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I nearby in the ... nearby in the ... in the garden ... DP for Incremental Parsing 16

  35. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I nearby in the ... in the garden ... DP for Incremental Parsing 17

  36. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I in the garden ... DP for Incremental Parsing 18

  37. Shift-Reduce Parsing I feed cats nearby in the garden. action stack queue I feed cats ... 0 - I feed cats nearby ... 1 shift I feed 2 shift cats nearby in ... feed 3 l-reduce cats nearby in ... I 4 shift feed cats nearby in the ... I 5a r-reduce feed nearby in the ... cats I 5b shift in the garden ... feed cats nearby I DP for Incremental Parsing 19

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences

Dynamic Programming for Linear Time Incremental Parsing Liang Huang Information Sciences Institute University of Southern California Kenji Sagae Institute for Creative Technologies University of Southern California ACL 2010, Uppsala, Sweden,

1.1k views • 64 slides
Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1

Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1

Linear Time Constituency Parsing with RNNs and Dynamic Programming Juneki Hong 1 Liang Huang 1,2 1 Oregon State University 2 Baidu Research Silicon Valley AI Lab Span Parsing is SOTA in Constituency Parsing Cross+Huang 2016 introduced Span

691 views • 53 slides
TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras,

TAG, Dynamic Programming, and the Perceptron for Efficent, Feature-Rich Parsing Xavier Carreras, Michael Collins and Terry Koo MIT CSAIL Discriminative Models for Parsing Structured Prediction methods like CRF or Perceptron train linear models

403 views • 27 slides
Computational Geometry Linear programming Exercise session #8: Incremental construction &

Computational Geometry Linear programming Exercise session #8: Incremental construction &

Computational Geometry Linear programming Exercise session #8: Incremental construction & Problem : maximize c T x s.t. Ax b Arrangements c = ( c 1 ,, c d ) T Linear Programming x = ( x 1 ,, x d ) T

521 views • 4 slides
Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to

Lecture 11 Dynamic Programming 11.1 Overview Dynamic Programming is a powerful technique that allows one to solve many different types of problems in time O ( n 2 ) or O ( n 3 ) for which a naive approach would take exponential time. In this

259 views • 7 slides
Linear Programming Linear Programming In a linear programming problem, there is a set of

Linear Programming Linear Programming In a linear programming problem, there is a set of

Linear Programming Linear Programming In a linear programming problem, there is a set of variables, and we want to assign real values to CISC5835, Algorithms for Big Data them so as to satisfy a set of linear equations and/or linear

414 views • 6 slides
Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs

Robust Incremental Neural Semantic Graph Parsing Jan Buys and Phil Blunsom Dependency Parsing vs Semantic Parsing Dependency parsing models the syntactic structure between words in a sentence. Dependency Parsing vs Semantic Parsing

637 views • 31 slides
Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two

Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two

CS11-747 Neural Networks for NLP Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2020/ Two Types of Linguistic Structure Dependency: focus on relations between words ROOT I saw a girl with a

730 views • 38 slides
Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two

Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two

CS11-747 Neural Networks for NLP Parsing with Dynamic Programming Graham Neubig Site https://phontron.com/class/nn4nlp2017/ Two Types of Linguistic Structure Dependency: focus on relations between words ROOT I saw a girl with a

396 views • 37 slides
A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management

A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management

Online Linear Programming Main Results and Key Ideas Related and More Recent Work A Dynamic Near-Optimal Algorithm for Online Linear Programming Yinyu Ye Department of Management Science and Engineering and Institute of Computational and

662 views • 52 slides
Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State

Incremental Parsing in Bounded Memory William Schuler Department of Linguistics The Ohio State University September 16, 2010 William Schuler Incremental Parsing in Bounded Memory Motivation Goal: simple processing model, matches observations

1.27k views • 80 slides
Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City

Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City

Type-driven Incremental Semantic Parsing with Polymorphism Kai Zhao and Liang Huang City University of New York The Problem weakly supervised semantic fully supervised semantic parsing parsing aka. parsing from Q/A pairs

547 views • 44 slides
Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time

Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time

6. Space-time tradeoffs and dynamic programming D. Keil Analysis of Algorithms 1/11 Topic 6: Space/time tradeoffs; dynamic programming; y g g transform and conquer 1. Space/time tradeoffs 2. Dynamic programming 3 Example: sequence

498 views • 18 slides
Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 ,

Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 ,

Batch-Incremental vs. Instance-Incremental Learning in Dynamic and Evolving Data Jesse Read 1 , Albert Bifet 2 , Bernhard Pfahringer 2 , Geoff Holmes 2 1 Department of Signal Theory and Communications Universidad Carlos III Madrid, Spain 2

568 views • 19 slides
CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap

CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap

CKY Parsing Ling 571 Deep Processing Techniques for NLP January 12, 2011 Roadmap Motivation: Parsing (In) efficiency Dynamic Programming Cocke-Kasami-Younger Parsing Algorithm Chomsky Normal Form Conversion

689 views • 42 slides
Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May

Programming Languages:Parsing Programming Languages: Parsing Onur Tolga S ehito glu Computer Engineering,METU 27 May 2009 Programming Languages:Parsing Outline 1 Parsing Top-down Parsing Recursive Descent Parser LL Parsers

215 views • 8 slides
Distributed Task Scheduling for Physics Fusion Applications J. Herrera A. Cappa E. Huedo M.A.

Distributed Task Scheduling for Physics Fusion Applications J. Herrera A. Cappa E. Huedo M.A.

Enabling Grids for E-sciencE Distributed Task Scheduling for Physics Fusion Applications J. Herrera A. Cappa E. Huedo M.A. Tereshchenko R.S. Montero F. Castejn I.M. Llorente www.eu-egee.org EGEE-II INFSO-RI-031688 EGEE and gLite are

627 views • 15 slides
Stability of traveling waves with a point vortex Samuel Walsh (University of Missouri) joint work

Stability of traveling waves with a point vortex Samuel Walsh (University of Missouri) joint work

Stability of traveling waves with a point vortex Samuel Walsh (University of Missouri) joint work with Kristoffer Varholm (NTNU) and Erik Wahl en (Lund University) Water Waves Workshop ICERM, April 24, 2017 Supported in part by the NSF

1.37k views • 72 slides
Guarantees on the Probability of Good Selection David J. Eckman Shane G. Henderson Cornell

Guarantees on the Probability of Good Selection David J. Eckman Shane G. Henderson Cornell

Guarantees on the Probability of Good Selection David J. Eckman Shane G. Henderson Cornell University Cornell University Operations Research & Info Eng Operations Research & Info Eng r

715 views • 46 slides
Social Networks M A R I O L . S M A L L A N D M A L E A H F E K E T E H A R V A R D U N I V

Social Networks M A R I O L . S M A L L A N D M A L E A H F E K E T E H A R V A R D U N I V

Social Networks M A R I O L . S M A L L A N D M A L E A H F E K E T E H A R V A R D U N I V E R S I T Y The authors Mario L. Small Maleah Fekete https://mariosmall.com https://sociology.fas.harvard.edu/ people/maleah-fekete Background

472 views • 23 slides
America's Quest for Equality of Opportunity Lane Kenworthy November 18, 2014 Americans like

America's Quest for Equality of Opportunity Lane Kenworthy November 18, 2014 Americans like

America's Quest for Equality of Opportunity Lane Kenworthy November 18, 2014 Americans like equality of opportunity "Agree our society should do what is necessary to make sure everyone has an equal opportunity to succeed." Data

766 views • 47 slides
= Tensors More Examples Entity Keywords Tensor Matrix Relation Reviews Entity Users

= Tensors More Examples Entity Keywords Tensor Matrix Relation Reviews Entity Users

On Optimizing Distributed Tucker Decomposition for Sparse Tensors Venkatesan Chakaravarthy, Jee W. Choi , Douglas Joseph, Prakash Murali, Shivmaran S Pandian, Yogish Sabharwal, and Dheeraj Sreedhar IBM Research = Tensors More Examples

643 views • 23 slides
Technology Nodes Campbell Millar, Synopsys, Glasgow, UK ESSDERC/ ESSCIRC Workshop Process

Technology Nodes Campbell Millar, Synopsys, Glasgow, UK ESSDERC/ ESSCIRC Workshop Process

Simulation Tools for DTCO of Advanced Technology Nodes Campbell Millar, Synopsys, Glasgow, UK ESSDERC/ ESSCIRC Workshop Process Variations from Equipment Effects to Circuit and Design Impacts September 3, 2018, Dresden, Germany Slide 1

266 views • 12 slides
RXGK Benjamin Kaduk 20 June 2019 Benjamin Kaduk RXGK rxkad is bad roots in Kerberos 4,

RXGK Benjamin Kaduk 20 June 2019 Benjamin Kaduk RXGK rxkad is bad roots in Kerberos 4,

RXGK Benjamin Kaduk 20 June 2019 Benjamin Kaduk RXGK rxkad is bad roots in Kerberos 4, modified piecemeal as the ecosystem moved around us 56-bit keys, susceptible to brute force for O($100). confidentiailty protection for data transfer is

418 views • 12 slides

More recommend