discriminative language models
play

Discriminative Language Models Prof. Sameer Singh CS 295: - PowerPoint PPT Presentation

Sep 19, 2022 •293 likes •649 views

Discriminative Language Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 26, 2017 Based on slides from Noah Smith, Richard Socher, and everyone else they copied from. Language Models Probability of a Sentence Is a

  1. Discriminative Language Models Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 26, 2017 Based on slides from Noah Smith, Richard Socher, and everyone else they copied from.

  2. Language Models Probability of a Sentence • Is a given sentence something you would expect to see? • Syntactically (grammar) and Semantically (meaning) Probability of the Next Word • Predict what comes next for a given sequence of words. • Think of it as V‐way classification CS 295: STATISTICAL NLP (WINTER 2017) 2

  3. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 3

  4. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 4

  5. Logistic Regression Model CS 295: STATISTICAL NLP (WINTER 2017) 5

  6. N‐Grams as Logistic Reg. CS 295: STATISTICAL NLP (WINTER 2017) 6

  7. Other features… CS 295: STATISTICAL NLP (WINTER 2017) 7

  8. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 8

  9. Logistic Reg. w/ Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 9

  10. Neural Networks CS 295: STATISTICAL NLP (WINTER 2017) 10

  11. Activation Functions sigmoid softmax tanh And many others… ReLUs, PReLUs, ELU, step, max, and so on.. CS 295: STATISTICAL NLP (WINTER 2017) 11

  12. Why do they work? https://colah.github.io CS 295: STATISTICAL NLP (WINTER 2017) 12

  13. Why do they work? z x2 y x1 CS 295: STATISTICAL NLP (WINTER 2017) 13

  14. Simulated Example https://github.com/clab/cnn/blob/master/examples/xor.cc CS 295: STATISTICAL NLP (WINTER 2017) 14

  15. Simple Feedforward NN LM Bigram Model CS 295: STATISTICAL NLP (WINTER 2017) 15

  16. Simple Feedforward NN LM N‐gram Model CS 295: STATISTICAL NLP (WINTER 2017) 16

  17. Deep Feedforward NN LM Bengio et al. 2003 CS 295: STATISTICAL NLP (WINTER 2017) 17

  18. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 18

  19. Sequence View of Simple NNs CS 295: STATISTICAL NLP (WINTER 2017) 19

  20. Recurrent Neural Networks CS 295: STATISTICAL NLP (WINTER 2017) 20

  21. Example: “I love food” love food <eos> love food I CS 295: STATISTICAL NLP (WINTER 2017) 21

  22. Power of RNNs: Characters! http://karpathy.github.io/2015/05/21/rnn‐effectiveness/ CS 295: STATISTICAL NLP (WINTER 2017) 22

  23. Char‐RNNs: Shakespeare! CS 295: STATISTICAL NLP (WINTER 2017) 23

  24. Char‐RNNs: Wikipedia! CS 295: STATISTICAL NLP (WINTER 2017) 24

  25. Char‐RNNs: Linux Code! CS 295: STATISTICAL NLP (WINTER 2017) 25

  26. Extension: Stacking CS 295: STATISTICAL NLP (WINTER 2017) 26

  27. Extension: Bidirectional RNNs CS 295: STATISTICAL NLP (WINTER 2017) 27

  28. Deep Bidirectional RNNs CS 295: STATISTICAL NLP (WINTER 2017) 28

  29. Extension: GRUs Gated Recurrent Units CS 295: STATISTICAL NLP (WINTER 2017) 29

  30. Extension: GRUs Gated Recurrent Units CS 295: STATISTICAL NLP (WINTER 2017) 30

  31. Estimating Parameters Beyond the scope of the course • Lots of tricks, heuristics, “domain knowledge” • Lot of engineering for efficiency, e.g. GPUs • New training algorithms being proposed every year • sometimes, architecture‐specific • Lots of available tools you can use! • Tensorflow, Torch, Keras, MxNET, etc. CS 295: STATISTICAL NLP (WINTER 2017) 31

  32. Outline Discriminative Language Models Feed‐forward Neural Networks Recurrent Neural Networks Upcoming.. CS 295: STATISTICAL NLP (WINTER 2017) 32

  33. Homework 1 so far… Public Private CS 295: STATISTICAL NLP (WINTER 2017) 33

  34. Ruslan Salakhutdinov Professor at Carnegie Mellon University Director of Artificial Intelligence, Apple Inc. Learning Deep Unsupervised and Multimodal Models Location : DBH 6011 Time : 11am ‐ 12pm Date: January 27, 2017 Meeting with PhD students, will post on Piazza CS 295: STATISTICAL NLP (WINTER 2017) 34

  35. Upcoming… • Homework 1 is due tonight: January 26, 2017 • Write‐up, data, and code for Homework 2 is up Homework • Homework 2 is due: February 9, 2017 • Proposal is due: February 7, 2017 (~2 weeks) Project • Only 2 pages CS 295: STATISTICAL NLP (WINTER 2017) 35

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology

Discriminative Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Discriminative Models 1(11) 1. Generative and Discriminative Models 2. Log-Linear Models 3. Local Discriminative Models

278 views • 11 slides
MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by

MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by

MaxEnt Models and Discriminative Estimation Gerald Penn CS224N/Ling284 [based on slides by Christopher Manning and Dan Klein] Introduction So far weve looked at generative models Language models, Naive Bayes, IBM MT In

593 views • 57 slides
Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni,

Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni,

Structured Discriminative Models for Speech Recognition Mark Gales - work with Anton Ragni, Austin Zhang, Rogier van Dalen September 2012 Cambridge University Engineering Department Symposium on Machine Learning in Speech and Language

697 views • 54 slides
Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using

Three models for discriminative machine Three models for discriminative machine translation using Global Lexical Selection translation using Global Lexical Selection and Sentence Reconstruction and Sentence Reconstruction Sriram Venkatapathy

645 views • 32 slides
Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu

Discriminative Metric Learning in Nearest Neighbor Models for Image Annotation Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek &amp; Cordelia Schmid LEAR Team, INRIA Rh one-Alpes Grenoble, France Discriminative Metric Learning in Nearest

555 views • 40 slides
Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
Models w/ Latent Random Variables Chunting Zhou, Junxian He Site

Models w/ Latent Random Variables Chunting Zhou, Junxian He Site

CS11-747 Neural Networks for NLP Models w/ Latent Random Variables Chunting Zhou, Junxian He Site https://phontron.com/class/nn4nlp2019/ Slides from Graham Neubig Discriminative vs. Generative Models Discriminative model: calculate the

932 views • 58 slides
Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language

Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language

Discriminative Bimodal Networks for Visual Localization and Detection with Natural Language Queries Yuting Zhang , Luyao Yuan, Yijie Guo, Zhiyuan He, I - An Huang, Honglak Lee University of Michigan, Ann Arbor Detection with natural language

336 views • 9 slides
Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin Layton &amp; Mark Gales 9 December 2005 Cambridge University Engineering Department NIPS 2005 Augmented Statistical Models: Exploiting Generative

726 views • 25 slides
Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19,

Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19,

Generative Models for Discriminative Problems Chris Dyer DeepMind ASRU 2017 December 19, 2017 Terminological clarification A discriminative problem : for some input x , find Y ( x ) the most likely y in a set A

1.71k views • 91 slides
Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken

Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken

Hallucinating system outputs for discriminative language modeling Brian Roark Center for Spoken Language Understanding, OHSU Joint work with D. Bikel, C. Callison-Burch, Y. Cao, A. C elebi, E. Dikici, N. Glenn, K. Hall, E. Hasler, D.

836 views • 32 slides
Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography

Discriminative Regularization for Latent Variable Models with Applications to Electrocardiography Poster #53 Andrew C. Miller with Ziad Obermeyer, John P . Cunningham, and Sendhil Mullainathan &lt;latexit

674 views • 9 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017)

Natural Language Processing Info 159/259 Lecture 3: Text classification 2 (Aug 31, 2017) David Bamman, UC Berkeley Generative vs. Discriminative models Generative models specify a joint distribution over the labels and the data. With

611 views • 38 slides
Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and

A New Framework For Hybrid Models An Application: Deep Hybrid Models Supervised and Semi-Supervised Experiments Deep Hybrid Models: Bridging Discriminative and Generative Approaches Volodymyr Kuleshov and Stefano Ermon Department of Computer

523 views • 29 slides
Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More

Generative vs. discriminative Generative Discriminative Belief network A is more More robust modular Dont need precise model Class-conditional densities are specification, so long as it is likely to be local, characteristic

1.09k views • 15 slides
Evan Smith, Ellie White, Richard Prestage, &amp; Martin Braun FOSDEM 2018 What is Open Source

Evan Smith, Ellie White, Richard Prestage, &amp; Martin Braun FOSDEM 2018 What is Open Source

observe the invisible universe ... Evan Smith, Ellie White, Richard Prestage, &amp; Martin Braun FOSDEM 2018 What is Open Source Radio Telescopes? An open, collaborative, and free collection of ideas and designs for radio telescope

338 views • 23 slides
Gross Negligence Manslaughter in Healthcare where are we now (how did we get here) and where

Gross Negligence Manslaughter in Healthcare where are we now (how did we get here) and where

30 July 2019 Gross Negligence Manslaughter in Healthcare where are we now (how did we get here) and where are we going? Leslie Hamilton LLM FRCS Cardiac Surgeon [retired (early), licence to practise past Council, Royal College of Surgeons

1.29k views • 51 slides
AIRS View of Monsoon Intraseasonal Waves Joshua Xiouhua Fu, Bin Wang, and Li Tao International

AIRS View of Monsoon Intraseasonal Waves Joshua Xiouhua Fu, Bin Wang, and Li Tao International

AIRS View of Monsoon Intraseasonal Waves Joshua Xiouhua Fu, Bin Wang, and Li Tao International Pacific Research Center (IPRC) SOEST, University of Hawaii (UH) at Manoa Honolulu, Hawaii 96822 Acknowledgements: AIRS data used in this study are

575 views • 32 slides
Computer Science I Course at D-BAUG at ETH Zurich Autumn 2019 1 Material The Team Lecturers

Computer Science I Course at D-BAUG at ETH Zurich Autumn 2019 1 Material The Team Lecturers

General Informations about the Course Organisation, Tools, Exercises, Exams Hermann Lehner, Felix Friedrich Computer Science I Course at D-BAUG at ETH Zurich Autumn 2019 1 Material The Team Lecturers Hermann Lehner 1 Felix Friedrich Chief

305 views • 4 slides
The nilpotency of elements of the stable homotopy groups of spheres Richard Wong eCHT Kan

The nilpotency of elements of the stable homotopy groups of spheres Richard Wong eCHT Kan

Introduction Nilpotence for elements of order p . Nilpotence for all elements. The nilpotency of elements of the stable homotopy groups of spheres Richard Wong eCHT Kan seminar Fall 2019 Richard Wong University of Texas at Austin The

601 views • 42 slides
Q3 2017 October 11 th , 2017, 11 AM Eastern About Advisen Leading the way to smarter and more

Q3 2017 October 11 th , 2017, 11 AM Eastern About Advisen Leading the way to smarter and more

Quarterly D&amp;O Claims Trends: Q3 2017 October 11 th , 2017, 11 AM Eastern About Advisen Leading the way to smarter and more efficient risk and insurance communities, Advisen delivers: the right information into the right hands at the right

510 views • 20 slides
Vectorising Bitmaps into Semi-Transparent Gradient Layers 1 Christian Richardt 1,2 2 Jorge

Vectorising Bitmaps into Semi-Transparent Gradient Layers 1 Christian Richardt 1,2 2 Jorge

Vectorising Bitmaps into Semi-Transparent Gradient Layers 1 Christian Richardt 1,2 2 Jorge Lopez-Moreno 1,3 Adrien Bousseau 1 3 Maneesh Agrawala 4 4 George Drettakis 1 1 photos drawings vector art 2 Vector art representations 3 Vector

878 views • 60 slides
Introduction to Descriptive Set Theory (MATH40350) Dr Richard Smith

Introduction to Descriptive Set Theory (MATH40350) Dr Richard Smith

Introduction to Descriptive Set Theory (MATH40350) Dr Richard Smith (http://maths.ucd.ie/~rsmith) Dr Richard Smith (maths.ucd.ie/~rsmith) Intro to Desc Set Theory (MATH40350) Semester 2 2011 2012 1 / 31 Metric spaces Polish spaces

983 views • 61 slides

More recommend