language models
play

Language Models Machine Translation Lecture 3 Instructor: Chris - PowerPoint PPT Presentation

Nov 13, 2023 •372 likes •941 views

Language Models Machine Translation Lecture 3 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn No MT yet Today we will talk about models of p (sentence) The rest of this semester will deal

  1. Language Models Machine Translation Lecture 3 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn

  2. No MT yet • Today we will talk about models of p (sentence) • The rest of this semester will deal with 
 p (translated sentence | input sentence) • Why do it this way? • Conditioning on more stuff makes modeling more complicated. That is: p (sentence) is easier than p (translated sentence | input sentence). • Language models are arguably the most important models in statistical MT

  3. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  4. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  5. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  6. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  7. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  8. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  9. My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

  16. Language Models Matter • Language models play the role of ... • a judge of grammaticality • a judge of semantic plausibility • an enforcer of stylistic consistency • a repository of knowledge (?)

  17. What is the probability of a sentence? • Requirements • Assign a probability to every sentence (i.e., string of words) • Questions • How many sentences are there in English? • Too many :)

  18. What is the probability of a sentence? • Requirements • Assign a probability to every sentence (i.e., string of words) • Questions X p LM ( e ) = 1 • How many sentences are there in e ∈ Σ ∗ English? p LM ( e ) ≥ 0 ∀ e ∈ Σ ∗ • Too many :)

  19. Why do we want to estimate the probability of a sentence? • Goal: Assign a higher probability to good sentences in English p LM (the house is small) > p LM (small the is house) translations of German Haus: home, house … p LM (I am going home) > p LM (I am going house)

  20. n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × = p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × Vector-valued random variable p ( e 4 | e 1 , e 2 , e 3 ) × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 )

  21. n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × ≈ p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × p ( e 4 | e 1 , e 2 , e 3 ) × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 )

  22. Chain rule The chain rule is derived from a repeated application 
 of the definition of conditional probability: p ( a, b, c, d ) = p ( a | b, c, d ) p ( b, c, d ) = p ( a | b, c, d ) p ( b | c, d ) p ( c, d ) = p ( a | b, c, d ) p ( b | c, d ) p ( c | d ) p ( d )

  23. Conditional Independence p ( a, b, c ) = p ( a | b, c ) p ( b, c ) = p ( a | b, c ) p ( b | c ) p ( c ) “If I know B, then C doesn’t tell me about A” p ( a | b, c ) = p ( a | b ) p ( a, b, c ) = p ( a | b, c ) p ( b, c ) = p ( a | b, c ) p ( b | c ) p ( c ) = p ( a | b ) p ( b | c ) p ( c )

  24. Is the Markov assumption valid for Language? • the old man are/is • the pictures are/is • The old man in the pictures is my dad.

  25. n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × p ( e 1 ) × ≈ ≈ p ( e 2 | e 1 ) × p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × p ( e 3 | e 1 , e 2 ) × p ( e 4 | e 1 , e 2 , e 3 ) × p ( e 4 | e 1 , e 2 , e 3 ) × · · · × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 ) p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 ) Which do you think is better? Why?

  26. n -gram LMs p LM ( e ) = p ( e 1 , e 2 , e 3 , . . . , e ` ) p ( e 1 ) × ≈ p ( e 2 | e 1 ) × p ( e 3 | e 1 , e 2 ) × p ( e 4 | e 1 , e 2 , e 3 ) × · · · × p ( e ` | e 1 , e 2 , . . . , e ` − 2 , e ` − 1 ) ` Y = p ( e 1 | START) × p ( e i | e i − 1 ) × p (STOP | e ` ) i =2

  27. START my friends call me Alex STOP p ( my | START ) × p ( friends | my ) × p ( call | friends ) × p ( me | call ) × p ( Alex | me ) × p ( STOP | Alex ) START my friends dub me Alex STOP p ( my | START ) × p ( friends | my ) × p ( dub | friends ) × p ( me | dub ) × p ( Alex | me ) × p ( STOP | Alex ) These sentences have many terms in common.

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution

Models of Language Evolution models thereof its evolution language Models of Language Evolution ? What is language? never start with a dictionary definition!! the language of Google search query completion A language is a dialect with

1.13k views • 30 slides
4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language

4 Language Models 2: Log-linear Language Models This chapter will discuss another set of language models: log-linear language models [9, 4], which take a very di ff erent approach than the count-based n -grams described above. 11 4.1 Model

501 views • 8 slides
Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models

Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house is small) > p lm (small the is

1.06k views • 51 slides
Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan

CSEP 517 Natural Language Processing Language Models Luke Zettlemoyer Slides adapted from Dan Klein, Michael Collins, Yejin Choi, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation: perplexity

530 views • 50 slides
Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for

Statistical language models Statistical language models in IR Discussion Introduction to Information Retrieval http://informationretrieval.org IIR 12: Language Models for IR Hinrich Sch utze Institute for Natural Language Processing,

1.69k views • 141 slides
3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine

3 Language Models 1: n -gram Language Models While the final goal of a statistical machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

106 views • 8 slides
3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine

3 Language Models 1: n -gram Language Models While the final goal of a probabilistic machine translation system is to create a model of the target sentence E given the source sentence F , P ( E | F ), in this chapter we will take a step back, and

458 views • 8 slides
Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some

Maxent Models (III), & Neural Language Models CMSC 473/673 UMBC September 25 th , 2017 Some slides adapted from 3SLP Recap from last time Maximum Entropy Models a more general language model argmax ) ()

1.46k views • 66 slides
CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted

CSE 447/547 Natural Language Processing Winter 2020 Language Models Yejin Choi Slides adapted from Dan Klein, Michael Collins, Luke Zettlemoyer, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation:

1.14k views • 65 slides
CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from

CSE 490 Natural Language Processing Spring 2016 Language Models Yejin Choi Slides adapted from Dan Klein, Michael Collins, Luke Zettlemoyer, Dan Jurafsky Overview The language modeling problem N-gram language models Evaluation:

991 views • 68 slides
Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic

Language Models Language Models Dan Klein, John DeNero UC Berkeley Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

757 views • 22 slides
Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic

Language Models Dan Klein, John DeNero UC Berkeley Language Models Language Models Acoustic Confusions the station signs are in deep in english -14732 the stations signs are in deep in english -14735 the station signs are in deep into

1.44k views • 86 slides
Introduction So far: Point-wise classification (geometric models) Whats next: Structured

Introduction So far: Point-wise classification (geometric models) Whats next: Structured

University of Oslo : Department of Informatics INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Probabilities and Language Models Stephan Oepen & Milen Kouylekov Language Technology Group (LTG) October 15,

556 views • 27 slides
Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and

Computational Models of Language Learning Jelle Zuidema Institute for Logic, Language and Computation, U. of Amsterdam MSc Brain & Cognitive Science, Artificial Intelligence, Logic MoL Guest Lecture Computational Models of Language

945 views • 58 slides
III.4 Statistical Language Models 1. Basics of Statistical Language Models 2. Query-Likelihood

III.4 Statistical Language Models 1. Basics of Statistical Language Models 2. Query-Likelihood

III.4 Statistical Language Models 1. Basics of Statistical Language Models 2. Query-Likelihood Approaches 3. Smoothing Methods 4. Divergence Approaches 5. Extensions Based on MRS Chapter 12 and

459 views • 45 slides
http://demo.clab.cs.cmu.edu/algo4nlp19/ https://piazza.com/class/jy617kmo6ub134

http://demo.clab.cs.cmu.edu/algo4nlp19/ https://piazza.com/class/jy617kmo6ub134

http://demo.clab.cs.cmu.edu/algo4nlp19/ https://piazza.com/class/jy617kmo6ub134 Chan Young Park Jong Hyuk Park Slide credit: Noah Smith Slide credit: Noah Smith Slide credit: Noah Smith Slide credit: Noah

940 views • 80 slides
Degrees, Power Laws and Popularity Gonzalo Mateos Dept. of ECE and Goergen Institute for Data

Degrees, Power Laws and Popularity Gonzalo Mateos Dept. of ECE and Goergen Institute for Data

Degrees, Power Laws and Popularity Gonzalo Mateos Dept. of ECE and Goergen Institute for Data Science University of Rochester gmateosb@ece.rochester.edu http://www.ece.rochester.edu/~gmateosb/ February 13, 2020 Network Science Analytics

498 views • 45 slides
Bounds on the epsilon expansion Matthijs Hogervorst Ecole polytechnique f ed erale de

Bounds on the epsilon expansion Matthijs Hogervorst Ecole polytechnique f ed erale de

Bounds on the epsilon expansion Matthijs Hogervorst Ecole polytechnique f ed erale de Lausanne, Fields and Strings Laboratory September 18, 2019 Joint ICTP/SISSA string seminar based on 1909. with Chiara Toldo ( Ecole

600 views • 26 slides
Estimating MultiWay Fixed Effect Models with reghdfe Sergio Correia, Duke University 2016

Estimating MultiWay Fixed Effect Models with reghdfe Sergio Correia, Duke University 2016

Estimating MultiWay Fixed Effect Models with reghdfe Sergio Correia, Duke University 2016 Stata Conference, Chicago Illinois Introduction reghdfe implements the estimator from: Correia, S. (2016). Linear Models with High-Dimensional

648 views • 33 slides
Lesson 6: Case study: Polio Aaron A. King, Edward L. Ionides, and Kidus Asfaw 1 / 68 Outline

Lesson 6: Case study: Polio Aaron A. King, Edward L. Ionides, and Kidus Asfaw 1 / 68 Outline

Lesson 6: Case study: Polio Aaron A. King, Edward L. Ionides, and Kidus Asfaw 1 / 68 Outline Covariates 1 A POMP model for polio 2 A pomp representation of the POMP model 3 Logistics for the computations 4 Persistence of polio 5

960 views • 68 slides
Jonathan Siegel Stanford EE Computer Systems Colloquium Jan 22, 2013 Embracing Failure

Jonathan Siegel Stanford EE Computer Systems Colloquium Jan 22, 2013 Embracing Failure

Jonathan Siegel Stanford EE Computer Systems Colloquium Jan 22, 2013 Embracing Failure Failure Failing s a P x u a F Flunking Floundering Failure Fiasco Falling flat n g l d i F o Flopping g n i l z z i F F

1.2k views • 94 slides
Lecture 8 Health Worker Support Winter 2015 Richard Anderson 2/25/2015 University of

Lecture 8 Health Worker Support Winter 2015 Richard Anderson 2/25/2015 University of

Computing and Global Health Lecture 8 Health Worker Support Winter 2015 Richard Anderson 2/25/2015 University of Washington, Winter 2015 1 Todays topics Health workers Worker tracking and supervision Worker support tools

832 views • 38 slides
MOTORE: a Mobile Haptic Interface for Neuro-Rehabilitation Carlo A. Avizzano * , Massimo

MOTORE: a Mobile Haptic Interface for Neuro-Rehabilitation Carlo A. Avizzano * , Massimo

MOTORE: a Mobile Haptic Interface for Neuro-Rehabilitation Carlo A. Avizzano * , Massimo Satler * , Giovanni Cappiello # , Andrea Scoglio # , Emanuele Ruffaldi * and Massimo Bergamasco * * P ERC R O L AB ., S CUOLA S

423 views • 26 slides

More recommend