more is less non parametric language models and efficiency
play

More is Less? Non-parametric Language Models and Efficiency Graham - PowerPoint PPT Presentation

Sep 10, 2022 •141 likes •833 views

Carnegie Mellon University More is Less? Non-parametric Language Models and Efficiency Graham Neubig Carnegie Mellon University Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick 1 Carnegie Mellon University Parametric Language Models

  1. Carnegie Mellon University More is Less? Non-parametric Language Models and Efficiency Graham Neubig Carnegie Mellon University Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick 1

  2. Carnegie Mellon University Parametric Language Models Parametric LM <s> 2

  3. Carnegie Mellon University Parametric Language Models I Parametric LM <s> 3

  4. Carnegie Mellon University Parametric Language Models I Parametric LM <s> I 4

  5. Carnegie Mellon University Parametric Language Models I ordered Parametric LM <s> I 5

  6. Carnegie Mellon University Parametric Language Models I ordered a Parametric LM <s> I ordered 6

  7. Carnegie Mellon University Parametric Language Models I ordered a pizza with sauce . </s> Parametric LM <s> I ordered a pizza with sauce . 7

  8. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Non-parametric datastore (typically training dataset) 8

  9. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Prototype sentence I ordered a burger with fries Non-parametric datastore (typically training dataset) 9

  10. Carnegie Mellon University Non-Parametric Language Models Non-Parametric LM Prototype sentence <s> I ordered a burger with fries Non-parametric datastore (typically training dataset) 10

  11. Carnegie Mellon University Non-Parametric Language Models I Non-Parametric LM Prototype sentence <s> I ordered a burger with fries Non-parametric datastore (typically training dataset) 11

  12. Carnegie Mellon University Non-Parametric Language Models I ordered a pizza with sauce . </s> Non-Parametric LM Prototype sentence <s> I ordered a pizza with sauce . I ordered a burger with fries Non-parametric datastore (typically training dataset) 12

  13. Carnegie Mellon University Non-Parametric Language Models I ordered a pizza with sauce . </s> Non-Parametric LM Prototype sentence <s> I ordered a pizza with sauce . I ordered a burger with fries Non-parametric datastore (typically training dataset) 13

  14. Carnegie Mellon University A Human Processing Analogy 14

  15. Carnegie Mellon University A Human Processing Analogy Our Language Ability/ Knowledge 14

  16. Carnegie Mellon University A Human Processing Analogy Our Language Ability/ Our Language Ability/ Knowledge + Knowledge Wikipedia/Google 14

  17. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP 15

  18. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. 15

  19. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. 15

  20. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. 15

  21. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. •Feed retrieved states into model: •Learning to Remember Rare Events. Kaiser et al. 2017. 15

  22. Carnegie Mellon University A (Very Incomplete) Survey of Neural Non-parametric Models for NLP •Count-based non-parametric data stores: • Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016. • Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016. • Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017. •Bias probabili6es based on count-based aggrega6on of retrieved sentences: •Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018. •Deep Weighted Averaging Classifiers. Card et al. 2019. •Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020. •A;end to retrieved sentences: • Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018. • Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018. • REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020. •Feed retrieved states into model: •Learning to Remember Rare Events. Kaiser et al. 2017. •Fine-tune models on retrieved sentences: •One Sentence One Model for Neural Machine Transla/on. Li et al. 2016. 15

  23. Carnegie Mellon University Non-Parametric Language Models: Pros/Cons Pros • Micgate pressure on the parametric models • Improved interpretability in the modeling process

  24. Carnegie Mellon University Non-Parametric Language Models: Pros/Cons Pros • Micgate pressure on the parametric models • Improved interpretability in the modeling process Cons • In almost all cases, large parametric datastore leads to significant issues with memory and speed efficiency at test cme

  25. Carnegie Mellon University A Concrete Example: Prototype-based Language Models (Guu et al. TACL 2018) 18

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural

The Significance of Errors to Parametric Models of Language Acquisition Paula Buttery Natural Language and Information Processing Group Computer Laboratory, Cambridge University paula.buttery@cl.cam.ac.uk Paula Buttery, 03/2004 Classification

395 views • 16 slides
Semi-parametric and response setup non-parametric approaches to Parametric models

Semi-parametric and response setup non-parametric approaches to Parametric models

Christian Ritz Concentration- Semi-parametric and response setup non-parametric approaches to Parametric models concentration-response modelling Non- parametric approach Semi- parametric Christian Ritz models Results University of

492 views • 23 slides
Parametric vs Nonparametric Models Parametric models assume some finite set of parameters .

Parametric vs Nonparametric Models Parametric models assume some finite set of parameters .

Parametric vs Nonparametric Models Parametric models assume some finite set of parameters . Given the parameters, future predictions, x , are independent of the observed data, D : P ( x | , D ) = P ( x | ) therefore capture

679 views • 5 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric:

Models for Probability Distributions and Density Functions 1 General Concepts Parametric: E.g., Gaussian, Gamma, Binomial Non-Parametric: E.g., kernel estimates Intermediate models: Mixture Models 2 Gaussian Mixture Model

382 views • 22 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use

h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use

Linear Signal Models Overview Introduction Introduction h ( n ) w ( n ) x ( n ) Linear nonparametric vs. parametric models Many researchers use signal models to analyze stationary Equivalent representations univariate time series

507 views • 5 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) &gt; p lm (small the is

1.06k views • 51 slides
Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo

Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo

Introduction &amp; Aims and Motivations Classification of Choices Testing Methodology Conclusions References Testing for Parametric Orderings Efficiency Sergio Ortobelli 1 , 3 Nikolas Topaloglou 4 Matteo Malavasi 2 , 1 2 Department of Actuarial

786 views • 23 slides
Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model)

Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model)

Chapter 3 Parametric Models and Methods Models: Weibull ( includes the exponential model) log-normal log-logistic gamma Remark: Except for the gamma distribution, these lifetime dis- tributions have the property that the

731 views • 34 slides
Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting

Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting

Stochastic Model Efficiency Applications: Cluster-Distance Sampling and Parametric Curve Fitting to Tackle Sampling Errors and Bias Yvonne C. Chueh, PhD, ASA Paul H. Johnson, Jr., PhD Yongxue Qi Joint work between the University of Illinois at

525 views • 37 slides
Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus

Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus

A Non-Parametric Model for the Discovery of Inflectional Paradigms from Plain Text using Graphical Models over Strings Markus Dreyer Center for Language and Speech Processing (CLSP) Human Language Technology Center of Excellence (HLTCOE) Johns

2.43k views • 200 slides
Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe

Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe

Tutorial on Parametric Timed Automata for RT Scheduling Examples of scheduling models Giuseppe Lipari and tienne Andr CRIStAL - Universit de Lille Outline Models Example 1: response time calculation Example 2: Sensitivity on

555 views • 27 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the

Non-Parametric Models Review of last class: Decision Tree Learning dealing with the overlearning problem: pruning ensemble learning boosting Agenda Nearest neighbor models Finding nearest neighbors with kd trees

493 views • 27 slides
Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems

Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems

Automatic Generation of Minimal and Reduced Models for Structured Parametric Dynamical Systems Igor Pontes Duff Joint work with Peter Benner (MPI, Magdeburg) Pawan Goyal (MPI, Magdeburg) ICERM Workshop - Mathematics of Reduced Order Models

1.46k views • 108 slides
Composition Closure of Linear Extended Top-down Tree Transducers Zoltn Flp and Andreas

Composition Closure of Linear Extended Top-down Tree Transducers Zoltn Flp and Andreas

Composition Closure of Linear Extended Top-down Tree Transducers Zoltn Flp and Andreas Maletti maletti@ims.uni-stuttgart.de Leipzig April 8, 2014 Zoltn Flp and Andreas Maletti Composition Closure of Linear XTOP The problem

1.28k views • 98 slides
Precise large deviation probabilities for a heavy-tailed random walk 1 Thomas Mikosch

Precise large deviation probabilities for a heavy-tailed random walk 1 Thomas Mikosch

Precise large deviation probabilities for a heavy-tailed random walk 1 Thomas Mikosch University of Copenhagen www.math.ku.dk/ mikosch 1 Conference in Honor of Sren Asmussen, Sandbjerg, August 1-5, 2011 1 2 3 4 Large deviations for a

472 views • 32 slides
Switchgrass Seed Growing Experiences at Nott Farms reap-canada.com reap-canada.com

Switchgrass Seed Growing Experiences at Nott Farms reap-canada.com reap-canada.com

Switchgrass Seed Growing Experiences at Nott Farms reap-canada.com reap-canada.com reap-canada.com WEED CONTROL Need Good Establishment Emphasis on pre-emergent herbicides to reduce crop damage and get better kill Post emerge

544 views • 9 slides
New Hardness Results for the Permanent Using Linear Optics Daniel Grier Luke Schae ff er MIT

New Hardness Results for the Permanent Using Linear Optics Daniel Grier Luke Schae ff er MIT

New Hardness Results for the Permanent Using Linear Optics Daniel Grier Luke Schae ff er MIT Permanent Review Permanent: Given matrix Example: &lt;latexit

488 views • 19 slides
Multiwavelength studies of HESS J1825-137 of the ISM next to HESS J1825137 and preliminary

Multiwavelength studies of HESS J1825-137 of the ISM next to HESS J1825137 and preliminary

Multiwavelength studies of HESS J1825-137 of the ISM next to HESS J1825137 and preliminary results from other TeV source Nanten meeting 2015 F. Voisin Overall: HESS J1825-137 and HESS J1826-130 TeV source powered by 140 -12.80 Nanten

455 views • 20 slides
1 Observation error Second finding: Randomness The true daily gains vary at random. = +

1 Observation error Second finding: Randomness The true daily gains vary at random. = +

Outline Filtering techniques applied to monitoring of daily gain in slaughter pigs: The Kalman filter Introduction - and other methods Basic monitoring Anders Ringgaard Kristensen Shewart control charts DLM and the Kalman

115 views • 7 slides
Form Etiquette The Format Use label to identify the label Need to use id in input (not

Form Etiquette The Format Use label to identify the label Need to use id in input (not

Form Etiquette The Format Use label to identify the label Need to use id in input (not name) Identifies the correct semantics Use div &lt;p&gt; may give the right styling But the wrong semantics If you need a tag to format,

150 views • 4 slides
Welcome to the Routing wg at RIPE 72 Joo Damas Rob Evans Agenda A. Welcome 00:05 Thanks

Welcome to the Routing wg at RIPE 72 Joo Damas Rob Evans Agenda A. Welcome 00:05 Thanks

Welcome to the Routing wg at RIPE 72 Joo Damas Rob Evans Agenda A. Welcome 00:05 Thanks to the scribe, jabber scribe and stenographer. Reminder of microphone etiquette. Minutes from RIPE 71. B. Appointment of new co-chair. 00:05 C.

58 views • 5 slides

More recommend