top down tree long short term memory networks
play

Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang - PowerPoint PPT Presentation

Dec 03, 2022 •257 likes •746 views

Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang Lu, Mirella Lapata School of Informatics, University of Edinburgh 12th June, 2016 Zhang et al., 2016 Tree LSTM 12th June, 2016 1 / 18 Sequential Language Models n P (

  1. Top-down Tree Long Short-Term Memory Networks Xingxing Zhang , Liang Lu, Mirella Lapata School of Informatics, University of Edinburgh 12th June, 2016 Zhang et al., 2016 Tree LSTM 12th June, 2016 1 / 18

  2. Sequential Language Models n � P ( S = w 1 , w 2 , . . . , w n ) = P ( w i | w 1: i − 1 ) (1) i =1 State of the Art based on Long Short Term Memory Network Language Model (Hochreiter and Schmidhuber, 1997; Sundermeyer et al., 2012) Billion word benchmark results reported in Jozefowicz et al., (2016) Models PPL KN5 67.6 LSTM 30.6 LSTM+CNN INPUTS 30.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 2 / 18

  3. Will tree structures help LMs? Zhang et al., 2016 Tree LSTM 12th June, 2016 3 / 18

  4. Will tree structures help LMs? Probably yes LMs based on Constituency Parsing (Chelba and Jelinek, 2000; Roark, 2001; Charniak, 2001) LMs based on Dependency Parsing (Shen et al., 2008; Zhang, 2009; Sennrich, 2015) Zhang et al., 2016 Tree LSTM 12th June, 2016 3 / 18

  5. LSTMs + Dependency Trees = TreeLSTMs + Why? Sentence Length N v.s. Tree Height log ( N ) Zhang et al., 2016 Tree LSTM 12th June, 2016 4 / 18

  6. LSTMs + Dependency Trees = TreeLSTMs + Why? Sentence Length N v.s. Tree Height log ( N ) How? Top-down Generation Breadth-first search reminiscent of Eisner (1996) Zhang et al., 2016 Tree LSTM 12th June, 2016 4 / 18

  7. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  8. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  9. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  10. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  11. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  12. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  13. Generation Process (Unlabeled Trees) The luxury auto manufacturer last year sold 1,214 cars in the U.S. Zhang et al., 2016 Tree LSTM 12th June, 2016 5 / 18

  14. Tree LSTM n � P ( S ) = P ( w i | w 1: i − 1 ) (2) i =1 ⇓ � P ( S | T ) = P ( w |D ( w )) (3) w ∈ BFS( T ) \ root D ( w ) is the Dependency Path of w . D ( w ) is a generated sub-tree. Works on projective and unlabeled dependency trees. Zhang et al., 2016 Tree LSTM 12th June, 2016 6 / 18

  15. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  16. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  17. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  18. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  19. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  20. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  21. Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 7 / 18

  22. One Limitation of Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 8 / 18

  23. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  24. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  25. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  26. Left Dependent Tree LSTM Zhang et al., 2016 Tree LSTM 12th June, 2016 9 / 18

  27. Experiments Zhang et al., 2016 Tree LSTM 12th June, 2016 10 / 18

  28. MSR Sentence Completion Challenge Training set: 49 million words (around 2 million sentences) development set: 4000 sentences test set: 1040 completion questions. Zhang et al., 2016 Tree LSTM 12th June, 2016 11 / 18

  29. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  30. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  31. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  32. Zhang et al., 2016 Tree LSTM 12th June, 2016 12 / 18

  33. Dependency Parsing Reranking Rerank 2nd Order MSTParser (McDonald and Pereira, 2006) We train TreeLSTM and LdTreeLSTM as language models. We only use words as input features; POS tags, dependency labels or composition features are not used. Zhang et al., 2016 Tree LSTM 12th June, 2016 13 / 18

  34. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  35. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  36. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  37. Dependency Parsing Reranking NN: Chen & Manning, 2014; S-LSTM: Dyer et al., 2015 Zhang et al., 2016 Tree LSTM 12th June, 2016 14 / 18

  38. Tree Generation Four binary classifiers: Add Left? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  39. Tree Generation Four binary classifiers: Add Right? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  40. Tree Generation Four binary classifiers: Add Right? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  41. Tree Generation Four binary classifiers: Add Next Right? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  42. Tree Generation Four binary classifiers: Add Left? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  43. Tree Generation Four binary classifiers: Add Left? Yes! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  44. Tree Generation Four binary classifiers: Add Next Left? No! Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  45. Tree Generation Four binary classifiers: Add Left? Add Right? Add Next Left? Add Next Right? Features: hidden states and word embeddings Classifiers Accuracies Add-Left 94.3 Add-Right 92.6 Add-Nx-Left 93.4 Add-Nx-Right 96.0 Zhang et al., 2016 Tree LSTM 12th June, 2016 15 / 18

  46. Tree Generation Zhang et al., 2016 Tree LSTM 12th June, 2016 16 / 18

  47. Conclusions Syntax can help language modeling. Predicting tree structures with Neural Networks is possible. Next Steps: Sequence to Tree Models Tree to Tree Models code available: https://github.com/XingxingZhang/td-treelstm Thanks & Questions? Zhang et al., 2016 Tree LSTM 12th June, 2016 17 / 18

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks by Kai Sheng Tai, Richard Socher, Christopher D. Manning Daniel Perez tuvistavie CTO @ Claude Tech M2 @ The University of Tokyo October 2, 2017

546 views • 22 slides
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks Kai Sheng Tai , Richard Socher , and Christopher D. Manning Stanford University, MetaMind July 29, 2015 Distributed Word Representations

556 views • 36 slides
An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange

An Introduction to Neural Networks Long Short Term Memory (LSTM) and the Attention mechanism Ange Tato Universit du Qubec Montral Montreal, Canada Agenda v Recurrent Neural Network (RNN) v Long Short Term Memory (LSTM) v

506 views • 27 slides
Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal

Lecture 23: Recurrent Neural Networks, Long Short Term Memory Networks, Conntectionist Temporal Classifjcation, Decision Trees Instructor: Prof. Ganesh Ramakrishnan . . . . . . . . . . . . . . . . . . . . . . . . . . .

699 views • 31 slides
Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate

Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate

Memory Memory processing is the ability to: Acquire (Short term memory) Manipulate (Working memory) Retain (Long term memory) Memory Retrieve (Long term memory) processing A difficulty with any one or more of these skills

361 views • 6 slides
Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline

Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline

3/3/2020 Recurrent neural networks and Lo Long-short term memory (L (LSTM) Jeong Min Lee CS3750, University of Pittsburgh Outline RNN RNN Unfolding Computational Graph Backpropagation and weight update Explode / Vanishing

760 views • 35 slides
Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Sequential Data with Neural Networks Recurrent Neural Networks Sequential input / output Greg

Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Recurrent Neural Networks Long Short-Term Memory Temporal Convolutional Networks Examples Sequential Data with Neural Networks Recurrent Neural

303 views • 4 slides
Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified.

Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified.

Review Vanishing Gradient Example RNN Forget Gate LSTM Backprop Conclusion Long/Short-Term Memory Mark Hasegawa-Johnson All content CC-SA 4.0 unless otherwise specified. University of Illinois ECE 417: Multimedia Signal Processing, Fall

970 views • 55 slides
Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes

Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes

Chapter 3 - Cognition Types of human memory Short term memory and cognitive processes Long term memory Learning Unlearning Attention Cognitive load Working memory load Multitasking Measurement of

710 views • 32 slides
Boosting the deep multidimensional long short- term memory network for handwritten recognition

Boosting the deep multidimensional long short- term memory network for handwritten recognition

Boosting the deep multidimensional long short- term memory network for handwritten recognition systems Dayvid Castro 1 Byron L. D. Bezerra 1 Muser Valena 1 1 Polytechnic School of Pernambuco University of Pernambuco Recife, Brazil The 16th

693 views • 32 slides
Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials

Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials

Class 15 - Long Short-Term Memory (LSTM) Class 15 - Long Short-Term Memory (LSTM) Study materials Study materials http://colah.github.io/posts/2015-08-Understanding-LSTMs/ (http://colah.github.io/posts/2015-08-Understanding-LSTMs/)

641 views • 23 slides
the human sensory, short-term , long-term Information processed and applied

the human sensory, short-term , long-term Information processed and applied

the human Information i/ o chapter 1 visual, auditory, haptic, m ovem ent Information stored in memory the human sensory, short-term , long-term Information processed and applied reasoning, problem solving,

431 views • 7 slides
Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim

Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim

Large Vocabulary Continuous Speech Recognition with Long Short-Term Recurrent Networks Ha sim Sak, Andrew Senior, Oriol Vinyals, Georg Heigold, Erik McDermott, Rajat Monga, Mark Mao, Fran coise Beaufays andrewsenior/hasim@google.com See

578 views • 37 slides
Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel

Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel

Transition-Based Dependency Parsing with Stack Long Short-Term Memory Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, Noah A. Smith Association for Computational Linguistics (ACL), 2015 Presented By: Lavisha Aggarwal (lavisha2)

253 views • 23 slides
Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory

Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory

3/16/17 Long-Term Memory Introduction STM versus LTM Episodic Memory Semantic Memory Procedural Memory Encoding in Long-Term Memory Depth of Processing (or Levels of Processing) Self Reference Effect

863 views • 19 slides
Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S

Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S

Joint Learning of Speech-Driven Facial Motion with Bidirectional Long-Short Term Memory N AJMEH S ADOUGHI AND C ARLOS B USSO Multimodal Signal Processing (MSP) lab The University of Texas at Dallas Erik Jonsson School of Engineering and Computer

426 views • 26 slides
Natural Language Processing (CSE 517): Dependency Syntax and Parsing Noah A. Smith Swabha

Natural Language Processing (CSE 517): Dependency Syntax and Parsing Noah A. Smith Swabha

Natural Language Processing (CSE 517): Dependency Syntax and Parsing Noah A. Smith Swabha Swayamdipta 2018 c University of Washington { nasmith,swabha } @cs.washington.edu May 11, 2018 1 / 93 Recap: Phrase Structure S NP NP VP JJ

964 views • 93 slides
Natural Language Processing (CSE 517): Dependency Structure Noah Smith 2016 c University of

Natural Language Processing (CSE 517): Dependency Structure Noah Smith 2016 c University of

Natural Language Processing (CSE 517): Dependency Structure Noah Smith 2016 c University of Washington nasmith@cs.washington.edu February 24, 2016 1 / 45 Why might you want to use a generative classifier, such as Naive Bayes, as opposed

866 views • 45 slides
Motivation Moores Law continues More transistors & memory controllers on modern

Motivation Moores Law continues More transistors & memory controllers on modern

LMC: Automatic Resource-Aware Program-Optimized Memory Partitioning Hsin-Jung Yang , Kermin E. Fleming , Michael Adler , Felix Winterstein , and Joel Emer Massachusetts Institute of Technology, Intel Corporation,

1.12k views • 53 slides
ECE 697J Advanced Topics Advanced Topics ECE 697J in Computer Networks in Computer

ECE 697J Advanced Topics Advanced Topics ECE 697J in Computer Networks in Computer

ECE 697J Advanced Topics Advanced Topics ECE 697J in Computer Networks in Computer Networks Microengine Programming I 11/18/03 Tilman Wolf 1 Overview Overview Lab 2: IP forwarding on IXP1200 Any problems with Part I?

791 views • 24 slides
Draft version: Please do not circulate or quote without authors permission WINNING IN ASIA,

Draft version: Please do not circulate or quote without authors permission WINNING IN ASIA,

Draft version: Please do not circulate or quote without authors permission WINNING IN ASIA, AMERICAL STYLE: THE CASE OF THE AUTOMOBILE INDUSTRY Revised July 2002 Beverly Crawford, U.C. Berkeley (bev@socrates.berkeley.edu) Nick Biziouras,

748 views • 60 slides
You Are How You Drive: Peer and Temporal-Aware Representation Learning for Driving Behavior

You Are How You Drive: Peer and Temporal-Aware Representation Learning for Driving Behavior

You Are How You Drive: Peer and Temporal-Aware Representation Learning for Driving Behavior Analysis Pengyang Wang, Yanjie Fu, Jiawei Zhang, Pengfei Wang, Yu Zheng, Charu Aggarwal Outline 1 Background and Motivation Definition and

1.09k views • 38 slides
Drupal Core Auto-Update Drupal Core Auto-Update Architecture Architecture Peter Wolanin Peter

Drupal Core Auto-Update Drupal Core Auto-Update Architecture Architecture Peter Wolanin Peter

Drupal Core Auto-Update Drupal Core Auto-Update Architecture Architecture Peter Wolanin Peter Wolanin David Strauss David Strauss Peter Wolanin Peter Wolanin BioRAFT Engineering & Drupal BioRAFT Engineering & Drupal Security

407 views • 23 slides
Financial Econometrics Econ 40357 ARIMA (Auto Regressive Integrated Moving Average) Models Part

Financial Econometrics Econ 40357 ARIMA (Auto Regressive Integrated Moving Average) Models Part

Financial Econometrics Econ 40357 ARIMA (Auto Regressive Integrated Moving Average) Models Part 1. N.C. Mark University of Notre Dame and NBER August 26, 2020 1 / 20 Overview: Univariate, parametric time-series models Time series are

573 views • 20 slides

More recommend