Language Modeling Prof. Sameer Singh CS 295: STATISTICAL NLP - PowerPoint PPT Presentation

Language Modeling Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 January 24, 2017 Based on slides from Dan Jurafsky, Noah Smith, and everyone else they copied from.

Outline Wrapup Word Embeddings Introduction to Language Models N-Gram Based Language Models Smoothing Language Models CS 295: STATISTICAL NLP (WINTER 2017) 2

Outline Wrapup Word Embeddings Introduction to Language Models N-Gram Based Language Models Smoothing Language Models CS 295: STATISTICAL NLP (WINTER 2017) 3

Predict surrounding words A bottle of tezguino is on the table. u v CS 295: STATISTICAL NLP (WINTER 2017) 4

Negative Sampling CS 295: STATISTICAL NLP (WINTER 2017) 5

Neural View of Embeddings CS 295: STATISTICAL NLP (WINTER 2017) 6

Word embeddings Variations • Skip-gram: predict context from word CBOW: predict word from context bag of words • • Dependencies: a better description of context Uses • Similarity: Grammar: • • Analogies Gender: • Facts: • CS 295: STATISTICAL NLP (WINTER 2017) 7

Outline Wrapup Word Embeddings Introduction to Language Models N-Gram Based Language Models Smoothing Language Models CS 295: STATISTICAL NLP (WINTER 2017) 8

Language Models Probability of a Sentence Is a given sentence something you would expect to see? • Syntactically (grammar) and Semantically (meaning) • Probability of the Next Word Predict what comes next for a given sequence of words. • Think of it as V-way classification • CS 295: STATISTICAL NLP (WINTER 2017) 9

Task: Speech Recognition “eyes awe of an” OR “I saw a van” CS 295: STATISTICAL NLP (WINTER 2017) 10

Task: Machine Translation CS 295: STATISTICAL NLP (WINTER 2017) 11

Task: Handwriting Recognition http://www.cedar.buffalo.edu/handwriting/HRoverview.html CS 295: STATISTICAL NLP (WINTER 2017) 12

Task: Image Captioning CS 295: STATISTICAL NLP (WINTER 2017) 13

Task: Spelling Correction The office is about fifteen minuets from my house P(about fifteen minutes from) >> P(about fifteen minuets from) CS 295: STATISTICAL NLP (WINTER 2017) 14

Other Applications Summarization Question Answering Dialog Systems CS 295: STATISTICAL NLP (WINTER 2017) 15

Evaluating Language Models Best choice: Extrinsic 2nd choice: Intrinsic CS 295: STATISTICAL NLP (WINTER 2017) 16

Perplexity CS 295: STATISTICAL NLP (WINTER 2017) 17

Generating Text from an LM CS 295: STATISTICAL NLP (WINTER 2017) 18

Outline Wrapup Word Embeddings Introduction to Language Models N-Gram Based Language Models Smoothing Language Models CS 295: STATISTICAL NLP (WINTER 2017) 19

Direct Language Modeling P(“I do not like green eggs and ham”) P(w | “I do not like green eggs and ”) CS 295: STATISTICAL NLP (WINTER 2017) 20

Applying the Chain Rule CS 295: STATISTICAL NLP (WINTER 2017) 21

Markov Assumption 22 CS 295: STATISTICAL NLP (WINTER 2017)

Unigram Language Model CS 295: STATISTICAL NLP (WINTER 2017) 23

Bigram Language Model CS 295: STATISTICAL NLP (WINTER 2017) 24

Berkeley Restaurant Project CS 295: STATISTICAL NLP (WINTER 2017) 25

Berkeley Restaurant Project CS 295: STATISTICAL NLP (WINTER 2017) 26

N-Gram Language Models “The computer which I had just put into the “The computer which I had just put into the dining room on the fifth floor crashed.” dining room on the fifth floor had lunch.” CS 295: STATISTICAL NLP (WINTER 2017) 27

Shakespeare CS 295: STATISTICAL NLP (WINTER 2017) 28

Wall Street Journal CS 295: STATISTICAL NLP (WINTER 2017) 29

Implementation Tips Use Logs Prevent underflow • Sums, instead of products • Filter out n-grams • Rare n-grams are noisy/have low prob Use unigrams to filter bigrams… • CS 295: STATISTICAL NLP (WINTER 2017) 30

Outline Wrapup Word Embeddings Introduction to Language Models N-Gram Based Language Models Smoothing Language Models CS 295: STATISTICAL NLP (WINTER 2017) 31

Zero Probability Problem Training set: Test set • … denied the allegations … denied the offer … denied the reports … denied the loan … denied the claims … denied the request P(“offer” | denied the) = 0 Rare words/combinations Mispellings New words Because corpus is finite.. “minuets” Truthiness • • • #letalonethehashtags • bigly • CS 295: STATISTICAL NLP (WINTER 2017) 32

Laplace Smoothing CS 295: STATISTICAL NLP (WINTER 2017) 33

Intuition Behind Smoothing When we have sparse statistics: P(w | denied the) 3 allegations allegations 2 reports reports 1 claims outcome … 1 request request attack claims man 7 total Steal probability mass to generalize better P(w | denied the) 2.5 allegations allegations allegations 1.5 reports 0.5 claims outcome … reports 0.5 request attack man claims request 2 other 7 total CS 295: STATISTICAL NLP (WINTER 2017) 34

Berkeley Restaurant Project CS 295: STATISTICAL NLP (WINTER 2017) 35

Berkeley Restaurant Project CS 295: STATISTICAL NLP (WINTER 2017) 36

Backoff and Interpolation Backoff Use trigram, unless rare • Then use bigram, unless rare • Then use unigram.. • Interpolation Combine all three! • Linear function with parameters • Learn on held out data • CS 295: STATISTICAL NLP (WINTER 2017) 37

Upcoming… Homework 1 is due: January 26, 2017 • Homework Write-up, data, and code for Homework 2 is up • Homework 2 is due: February 9, 2017 • • Proposal is due: February 7, 2017 (~2 weeks) Make things more concrete: approach, metrics, baselines • Project • Mention progress, and address my concerns, if any Only 2 pages • CS 295: STATISTICAL NLP (WINTER 2017) 38