Lecture 4: Language Model Evaluation and Advanced methods Kai-Wei - - PowerPoint PPT Presentation

Lecture 4: Language Model Evaluation and Advanced methods

Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/NLP16

1 6501 Natural Language Processing

This lecture

v Kneser-Ney smoothing v Discriminative Language Models v Neural Language Models v Evaluation: Cross-entropy and perplexity

2 6501 Natural Language Processing

Recap: Smoothing

v Add-one smoothing v Add-𝜇 smoothing

vparameters tuned by the cross-validation

v Witten-Bell Smoothing

vT: # word types N: # tokens vT/(N+T): total prob. mass for unseen words vN/(N+T): total prob. mass for observed tokens

v Good-Turing

vReallocate the probability mass of n-grams that

• ccur r+1 times to n-grams that occur r times.

3 6501 Natural Language Processing

Recap: Back-off and interpolation

v Idea: even if we’ve never seen “red glasses”, we know it is more likely to occur than “red abacus” v Interpolation:

paverage(z | xy) = µ3 p(z | xy) + µ2 p(z | y) + µ1 p(z) where µ3 + µ2 + µ1 = 1 and all are ≥ 0

4 6501 Natural Language Processing

Absolute Discounting

v Save ourselves some time and just subtract 0.75 (or some d)!

v But should we really just use the regular unigram P(w)?

5

) ( ) ( ) ( ) , ( ) | (

1 1 1 1 scounting AbsoluteDi

w P w w c d w w c w w P

i i i i i i − − − −

+ − = λ

discounted bigram unigram

Interpolation weight 6501 Natural Language Processing

Kneser-Ney Smoothing

v Better estimate for probabilities of lower-order unigrams!

vShannon game: I can’t see without my reading___________? v“Francisco” is more common than “glasses” v… but “Francisco” always follows “San”

6

Francisco glasses

6501 Natural Language Processing

Kneser-Ney Smoothing

v Instead of P(w): “How likely is w” v Pcontinuation(w): “How likely is w to appear as a novel continuation?

vFor each word, count the number of bigram types it completes vEvery bigram type was a novel continuation the first time it was seen

7

P

CONTINUATION(w)∝ {wi−1 :c(wi−1,w) > 0}

6501 Natural Language Processing

Kneser-Ney Smoothing

v How many times does w appear as a novel continuation: v Normalized by the total number of word bigram types

8

P

CONTINUATION(w) =

{wi−1 :c(wi−1,w) > 0} {(wj−1,wj):c(wj−1,wj) > 0}

P

CONTINUATION(w)∝ {wi−1 :c(wi−1,w) > 0}

{(wj−1,wj):c(wj−1,wj) > 0}

6501 Natural Language Processing

Kneser-Ney Smoothing

v Alternative metaphor: The number of # of word types seen to precede w v normalized by the # of words preceding all words: v A frequent word (Francisco) occurring in only one context (San) will have a low continuation probability

9

P

CONTINUATION(w) =

{wi−1 :c(wi−1,w) > 0} {w'i−1 :c(w'i−1,w') > 0}

w'

∑

|{wi−1 :c(wi−1,w) > 0}|

6501 Natural Language Processing

Kneser-Ney Smoothing

10

P

KN(wi | wi−1) = max(c(wi−1,wi)− d,0)

c(wi−1) + λ(wi−1)P

CONTINUATION(wi)

λ(wi−1) = d c(wi−1) {w :c(wi−1,w) > 0}

λ is a normalizing constant; the probability mass we’ve discounted

the normalized discount The number of word types that can follow wi-1 = # of word types we discounted = # of times we applied normalized discount

6501 Natural Language Processing

Kneser-Ney Smoothing: Recursive formulation

11

P

KN (wi | wi−n+1 i−1 ) = max(cKN (wi−n+1 i

)− d,0) cKN (wi−n+1

i−1 )

+ λ(wi−n+1

i−1 )P KN (wi | wi−n+2 i−1

) cKN(•) = count(•) for the highest order continuationcount(•) for lower order ! " # $ #

Continuation count = Number of unique single word contexts for Ÿ

6501 Natural Language Processing

Practical issue: Huge web-scale n-grams v How to deal with, e.g., Google N-gram corpus v Pruning

vOnly store N-grams with count > threshold.

v Remove singletons of higher-order n-grams

12 6501 Natural Language Processing

Huge web-scale n-grams

v Efficiency

vEfficient data structures

v e.g. tries

vStore words as indexes, not strings vQuantize probabilities (4-8 bits instead of 8-byte float)

13

https://en.wikipedia.org/wiki/Trie

6501 Natural Language Processing

600.465 - Intro to NLP - J. Eisner 14

Smoothing

This dark art is why NLP is taught in the engineering school.

14

There are more principled smoothing methods, too. We’ll look next at log-linear models, which are a good and popular general technique.

6501 Natural Language Processing

Conditional Modeling

v Generative language model (tri-gram model):

v Then, we compute the conditional probabilities by maximum likelihood estimation

v Can we model 𝑄 𝑥$ 𝑥%, 𝑥' directly? v Given a context x, which outcomes y are likely in that context?

P (NextWord=y | PrecedingWords=x)

15 600.465 - Intro to NLP - J. Eisner 15

𝑄(𝑥), … 𝑥+)= P 𝑥) 𝑄 𝑥0 𝑥) …𝑄 𝑥+ 𝑥+10,𝑥+1)

6501 Natural Language Processing

Modeling conditional probabilities

v Let’s assume

𝑄(𝑧|𝑦) = exp(score x,y )/∑ exp (𝑡𝑑𝑝𝑠𝑓 𝑦,𝑧D )

ED

Y: NextWord, x: PrecedingWords

v𝑄(𝑧|𝑦) is high ⇔ score(x,y) is high vThis is called soft-max vRequire that P(y | x) ≥ 0, and ∑ 𝑄(𝑧|𝑦)

E

= 1; not true of score(x,y)

16 6501 Natural Language Processing

Linear Scoring

v Score(x,y): How well does y go with x? v Simplest option: a linear function of (x,y).

But (x,y) isn’t a number ⇒ describe it by some numbers (i.e. numeric features)

v Then just use a linear function of those numbers.

17

Ranges over all features Whether (x,y) has feature k(0 or 1) Or how many times it fires (≥ 0) Or how strongly it fires (real #) Weight of the kth feature. To be learned …

6501 Natural Language Processing

What features should we use?

v Model p wI w$1), w$10):

𝑔

'(“𝑥$1), 𝑥$10”, “𝑥$”) for Score(“𝑥$1), 𝑥$10”, “𝑥$”) can be

v # “𝑥$1)” appears in the training corpus. v 1, if “𝑥$” is an unseen word; 0, otherwise. v 1, if “𝑥$1), 𝑥$10” = “a red”; 0, otherwise. v 1, if “𝑥$10” belongs to the “color” category; 0 otherwise.

18 6501 Natural Language Processing

What features should we use?

v Model p ”𝑕𝑚𝑏𝑡𝑡𝑓𝑡” ”𝑏 𝑠𝑓𝑒”):

𝑔

'(“𝑠𝑓𝑒”, “𝑏”, “𝑕𝑚𝑏𝑡𝑡𝑓𝑡”) for Score(“𝑠𝑓𝑒”, “𝑏”, “𝑕𝑚𝑏𝑡𝑡𝑓𝑡”)

v # “𝑠𝑓𝑒” appears in the training corpus. v 1, if “𝑏” is an unseen word; 0, otherwise. v 1, if “a 𝑠𝑓𝑒” = “a red”; 0, otherwise. v 1, if “𝑠𝑓𝑒” belongs to the “color” category; 0 otherwise.

19 6501 Natural Language Processing

Log-Linear Conditional Probability

20

600.465 - Intro to NLP - J. Eisner 20

where we choose Z(x) to ensure that unnormalized prob (at least it’s positive!) thus,

Partition function

6501 Natural Language Processing

v n training examples v feature functions f1, f2, … v Want to maximize p(training data|θ) v Easier to maximize the log of that: v Alas, some weights θi may be optimal at -∞ or +∞. When would this happen? What’s going “wrong”?

Training θ

21

This version is “discriminative training”: to learn to predict y from x, maximize p(y|x).

Whereas in “generative models”, we learn to model x, too, by maximizing p(x,y).

6501 Natural Language Processing

Generalization via Regularization

v n training examples v feature functions f1, f2, … v Want to maximize p(training data|θ) ⋅ pprior(θ) v Easier to maximize the log of that v Encourages weights close to 0.

v “L2 regularization”: Corresponds to a Gaussian prior

22

𝑞 𝜄 ∝ 𝑓 X Y/ZY

6501 Natural Language Processing

Gradient-based training

v Gradually adjust θ in a direction that improves

23

Gradient ascent to gradually increase f(θ): while (∇f(θ) ≠ 0) // not at a local max or min θ = θ + 𝜃∇f(θ) // for some small 𝜃 > 0 Remember: ∇f(θ) = (∂f(θ)/∂θ1, ∂f(θ)/∂θ2, …) update means: θk += ∂f(θ) / ∂θk

6501 Natural Language Processing

Gradient-based training

v Gradually adjust θ in a direction that improves v Gradient w.r.t 𝜄

24 6501 Natural Language Processing

More complex assumption?

v 𝑄(𝑧|𝑦) = exp(score x,y )/ ∑ exp(𝑡𝑑𝑝𝑠𝑓 𝑦,𝑧′ )

𝑧′

Y: NextWord, x: PrecedingWords v Assume we saw:

What is P(shoes; blue)?

v Can we learn categories of words(representation) automatically? v Can we build a high order n-gram model without blowing up the model size?

25

red glasses; yellow glasses; green glasses; blue glasses red shoes; yellow shoes; green shoes;

6501 Natural Language Processing

Neural language model

v Model 𝑄(𝑧|𝑦) with a neural network

26

Example 1: One hot vector: each component of the vector represents one word [0, 0, 1, 0, 0] Example 2: word embeddings

6501 Natural Language Processing

Neural language model

v Model 𝑄(𝑧|𝑦) with a neural network

27

Learned matrices to project the input vectors Obtain (y|x) by performing softmax Concatenate projected vectors Non-linear function e.g., ℎ = tanh (𝑋b 𝑑 ⃑ + 𝑐)

6501 Natural Language Processing

Why?

v Potentially generalize to unseen contexts

vExample: P(“red” | “the”, “shoes”, “are”) vThis does not occurs in training corpus but [“the”, ”glasses”, ”are”, “red”] does. vIf the word representations of “red” and “blue” are similar, then the model can generalize.

v Why are “red” and “blue” similar?

vBecause NN saw “red skirt”, “blue skirt”, “red pen”, ”blue pen”, etc.

28 6501 Natural Language Processing

Training neural language models

v Can use gradient ascent as well v Using the chain rule to derive the gradient a.k.a. back propagation v More complex NN architectures can be used – e.g., LSTM, char-based models

29 6501 Natural Language Processing

Language model evaluation

v How to compare models?

v we need an unseen text set, why?

v Information theory: study resolution of uncertainty.

vPerplexity: measure how well a probability distribution predicts a sample

30 6501 Natural Language Processing

Cross-Entropy

v A common measure of model quality

v Task-independent v Continuous – slight improvements show up here even if they don’t change # of right answers on task

v Just measure probability of (enough) test data

v Higher prob means model better predicts the future v There’s a limit to how well you can predict random stuff v Limit depends on “how random” the dataset is (easier to predict weather than headlines, especially in Arizona)

31 6501 Natural Language Processing

32

Cross-Entropy (“xent”)

v Want prob of test data to be high:

p(h | <S>, <S>) * p(o | <S>, h) * p(r | h, o) * p(s | o, r) … 1/8 * 1/8 * 1/8 * 1/16 …

v high prob → low xent by 3 cosmetic improvements:

v Take logarithm (base 2) to prevent underflow:

log (1/8 * 1/8 * 1/8 * 1/16 …) = log 1/8 + log 1/8 + log 1/8 + log 1/16 … = (-3) + (-3) + (-3) + (-4) + …

v Negate to get a positive value in bits 3+3+3+4+… v Divide by length of text à 3.25 bits per letter (or per word)

6501 Natural Language Processing

Average? Geometric average

• f 1/23,1/23, 1/23, 1/24

= 1/23.25 ≈ 1/9.5

33

Cross-Entropy (“xent”)

v Want prob of test data to be high:

p(h | <S>, <S>) * p(o | <S>, h) * p(r | h, o) * p(s | o, r) … 1/8 * 1/8 * 1/8 * 1/16 …

v Cross-entropy à 3.25 bits per letter (or per word)

v Want this to be small (equivalent to wanting good compression!) v Lower limit is called entropy – obtained in principle as cross-entropy of the true model measured on an infinite amount of data

v perplexity = 2xent (meaning ≈9.5 choices)

Average? Geometric average

• f 1/23,1/23, 1/23, 1/24

= 1/23.25 ≈ 1/9.5

6501 Natural Language Processing

More math: Entropy H(X)

v The entropy H(𝑞) of a discrete random variable 𝑌 is the expected negative log probability:

H p = − ∑ 𝑞 𝑦 log0 𝑄(𝑦)

k

v Entropy is measure of uncertainty

34 6501 Natural Language Processing

Entropy of coin tossing

v Toss a coin P(H)=𝑞, P(T)=1 − p v H(p)= −𝑞 log0 𝑞 − 1 − 𝑞 log0 1 − 𝑞

vp=0.5: H(p)= 1 vP=1: H(p) = 0

6501 Natural Language Processing 35

Entropy of coin tossing

v Toss a coin P(H)=𝑞, P(T)=1 − p v H(p)= −𝑞 log0 𝑞 − 1 − 𝑞 log0 1 − 𝑞

vp=0.5: H(p)= 1 vP=1: H(p) = 0

6501 Natural Language Processing 36

How many bits to encode messages

v Consider three letters (A, B, C, D): v If p=(½, ½, 0, 0), how many bits per letter in average to encode a message ~ p?

vEncode A as 0, B as 1;AAABBBAA ⇒ 00011100

v If p=(¼ , ¼ , ¼ , ¼ )

vA: 00, B: 01, C:10, D:11; ABDA ⇒ 00011100

v How about p=(½, ¼, ¼, 0)

vA: 0, B:10, C:11; AAACBA ⇒ 00011100

6501 Natural Language Processing 37

More math: Cross Entropy

v Cross-entropy:

• vAvg. # bits to encode events ~ p(x) using a coding

scheme m(x) vH p, 𝑛 = − ∑ 𝑞 𝑦 log0 𝑛(𝑦)

k

vNot symmetric: H p, 𝑛 ≠ H 𝑛,𝑞 vLower bounded by H(p)

v Let p=(½, ¼, ¼, 0)

vWe encode A:00, B:01, C:10, D:11 (i,e., m= (¼, ¼, ¼, ¼)) vAAACBA?

38 6501 Natural Language Processing

000000100100

Perplexity and geometric mean v

6501 Natural Language Processing 39

it assi Perplexity(w1... wN) = 2H(w1. . . wN) = 2− 1

N log2 m(w1. . . wN)

= m(w1... wN)− 1

N

=

N

s 1 m(w1... wN)

An experiment

v Train: 38M WSJ text, |V|= 20k v Test: 1.5M WSJ text v Word level LSTM ~85 v Char level ~79

6501 Natural Language Processing 40

Results:

Unigram Bigram Trigram Perplexity 962 170 109