Neural Language Models CMSC 723 / LING 723 / INST 725 M ARINE C - - PowerPoint PPT Presentation

Neural Language Models

CMSC 723 / LING 723 / INST 725 MARINE CARPUAT

marine@cs.umd.edu

With slides from Graham Neubig and Philipp Koehn

Roadmap

• Modeling Sequences

– First example: language model – What are n-gram models? – How to estimate them? – How to evaluate them? – Neural language models

Probabilistic Language Modeling

• Goal: compute the probability of a sentence or

sequence of words P(W) = P(w1,w2,w3,w4,w5…wn)

• Related task: probability of an upcoming word

P(w5|w1,w2,w3,w4)

• A model that computes either of these:

P(W) or P(wn|w1,w2…wn-1)

is called a language model.

Evaluation: How good is our model?

• Does our language model prefer good sentences to bad
• nes?

– Assign higher probability to “real” or “frequently observed” sentences

• Than “ungrammatical” or “rarely observed” sentences?
• Extrinsic vs intrinsic evaluation

Intrinsic evaluation: intuition

• The Shannon Game:

– How well can we predict the next word? – Unigrams are terrible at this game. (Why?)

• A better model of a text assigns a higher

probability to the word that actually occurs

I always order pizza with cheese and ____ The 33rd President of the US was ____ I saw a ____

mushrooms 0.1 pepperoni 0.1 anchovies 0.01 …. fried rice 0.0001 …. and 1e-100

Intrinsic evaluation metric: perplexity

Perplexity is the inverse probability of the test set, normalized by the number of words: Chain rule: For bigrams: Minimizing perplexity is the same as maximizing probability

The best language model is one that best predicts an unseen test set

• Gives the highest P(sentence)

PP(W) = P(w1w2...wN )

• 1

N

= 1 P(w1w2...wN )

N

Perplexity as branching factor

• Let’s suppose a sentence consisting of N random

digits

• What is the perplexity of this sentence according to a

model that assign P=1/10 to each digit?

Lower perplexity = better model

• Training 38 million words, test 1.5 million words, WSJ

N-gram Order Unigram Bigram Trigram Perplexity 962 170 109

Pros and cons of n-gram models

• N-gram models

– Really easy to build, can train on billions and billions of words – Smoothing helps generalize to new data – Only work well for word prediction if the test corpus looks like the training corpus – Only capture short distance context “Smarter” LMs can address some of these issues, but they are order of magnitudes slower…

Roadmap

• Modeling Sequences

– First example: language model – What are n-gram models? – How to estimate them? – How to evaluate them? – Neural language models

NE NEURAL AL NE NETWO WORKS KS

Aside

Recall the person/not-person classification problem

Given an introductory sentence in Wikipedia predict whether the article is about a person

Formalizing binary prediction

The Perceptron:

a “machine” to calculate a weighted sum

sign

𝑗=1 𝐽

𝑥𝑗 ⋅ ϕ𝑗 𝑦

φ“A” = 1 φ“site” = 1 φ“,” = 2 φ“located” = 1 φ“in” = 1 φ“Maizuru”= 1 φ“Kyoto” = 1 φ“priest” = 0 φ“black” = 0

• 3

2

• 1

The Perceptron: Geometric interpretation

O X O X O X

The Perceptron: Geometric interpretation

O X O X O X

Limitation of perceptron

• can only find linear separations between

positive and negative examples

X O O X

Neural Networks

• Connect together multiple perceptrons

φ“A” = 1 φ“site” = 1 φ“,” = 2 φ“located” = 1 φ“in” = 1 φ“Maizuru”= 1 φ“Kyoto” = 1 φ“priest” = 0 φ“black” = 0

• 1
• Motivation: Can represent non-linear functions!

Neural Networks: key terms

φ“A” = 1 φ“site” = 1 φ“,” = 2 φ“located” = 1 φ“in” = 1 φ“Maizuru”= 1 φ“Kyoto” = 1 φ“priest” = 0 φ“black” = 0

• 1
• Input (aka features)
• Output
• Nodes
• Layers
• Activation function

(non-linear)

• Multi-layer

perceptron

Example

• Create two classifiers

X O O X

φ0(x2) = {1, 1} φ0(x1) = {-1, 1} φ0(x4) = {1, -1} φ0(x3) = {-1, -1}

sign sign

φ0[0] φ0[1] 1 1 1

• 1
• 1
• 1
• 1

φ0[0] φ0[1]

φ1[0] φ0[0] φ0[1] 1

w0,0 b0,0

φ1[1]

w0,1 b0,1

Example

• These classifiers map to a new space

X O O X

φ0(x2) = {1, 1} φ0(x1) = {-1, 1} φ0(x4) = {1, -1} φ0(x3) = {-1, -1}

1 1

• 1
• 1
• 1
• 1

φ1 φ2

φ1[1] φ1[0]

φ1[0] φ1[1]

φ1(x1) = {-1, -1}

X O

φ1(x2) = {1, -1}

O

φ1(x3) = {-1, 1} φ1(x4) = {-1, -1}

Example

• In new space, the examples are linearly separable!

X O O X

φ0(x2) = {1, 1} φ0(x1) = {-1, 1} φ0(x4) = {1, -1} φ0(x3) = {-1, -1}

1 1

• 1
• 1
• 1
• 1

φ0[0] φ0[1]

φ1[1] φ1[0] φ1[0] φ1[1] φ1(x1) = {-1, -1} X O φ1(x2) = {1, -1} O φ1(x3) = {-1, 1} φ1(x4) = {-1, -1}

1 1 1

φ2[0] = y

Example wrap-up: Forward propagation

• The final net

tanh tanh

φ0[0] φ0[1] 1 φ0[0] φ0[1] 1 1 1

• 1
• 1
• 1
• 1

1 1 1 1

tanh

φ1[0] φ1[1] φ2[0]

24

Softmax Function for multiclass classification

• Sigmoid function for multiple classes
• Can be expressed using matrix/vector ops

𝑄 𝑧 ∣ 𝑦 = 𝑓𝐱⋅ϕ 𝑦,𝑧

𝑧 𝑓𝐱⋅ϕ 𝑦, 𝑧

Current class Sum of other classes

𝐬 = exp 𝐗 ⋅ ϕ 𝑦 𝐪 = 𝐬

𝑠∈𝐬

𝑠

Stochastic Gradient Descent

Online training algorithm for probabilistic models w = 0 for I iterations for each labeled pair x, y in the data w += α * dP(y|x)/dw In other words

• For every training example, calculate the gradient

(the direction that will increase the probability of y)

• Move in that direction, multiplied by learning rate α

• 10

10 0.1 0.2 0.3 0.4 w*phi(x) dp(y|x)/dw*phi(x)

Gradient of the Sigmoid Function

Take the derivative of the probability

𝑒 𝑒𝑥 𝑄 𝑧 = 1 ∣ 𝑦 = 𝑒 𝑒𝑥 𝑓𝐱⋅ϕ 𝑦 1 + 𝑓𝐱⋅ϕ 𝑦 = ϕ 𝑦 𝑓𝐱⋅ϕ 𝑦 1 + 𝑓𝐱⋅ϕ 𝑦

2

𝑒 𝑒𝑥 𝑄 𝑧 = −1 ∣ 𝑦 = 𝑒 𝑒𝑥 1 − 𝑓𝐱⋅ϕ 𝑦 1 + 𝑓𝐱⋅ϕ 𝑦 = −ϕ 𝑦 𝑓𝐱⋅ϕ 𝑦 1 + 𝑓𝐱⋅ϕ 𝑦

2

Learning: We Don't Know the Derivative for Hidden Units!

For NNs, only know correct tag for last layer

y=1

ϕ 𝑦 𝑒𝑄 𝑧 = 1 ∣ 𝐲 𝑒𝐱𝟓 = 𝐢 𝑦 𝑓𝐱𝟓⋅𝐢 𝑦 1 + 𝑓𝐱𝟓⋅𝐢 𝑦

2

𝐢 𝑦 𝑒𝑄 𝑧 = 1 ∣ 𝐲 𝑒𝐱𝟐 = ? 𝑒𝑄 𝑧 = 1 ∣ 𝐲 𝑒𝐱𝟑 = ? 𝑒𝑄 𝑧 = 1 ∣ 𝐲 𝑒𝐱𝟒 = ?

w1 w2 w3 w4

Answer: Back-Propagation

Calculate derivative with chain rule

𝑒𝑄 𝑧 = 1 ∣ 𝑦 𝑒𝐱𝟐 = 𝑒𝑄 𝑧 = 1 ∣ 𝑦 𝑒𝐱𝟓𝐢 𝐲 𝑒𝐱𝟓𝐢 𝐲 𝑒ℎ1 𝐲 𝑒ℎ1 𝐲 𝑒𝐱𝟐 𝑓𝐱𝟓⋅𝐢 𝑦 1 + 𝑓𝐱𝟓⋅𝐢 𝑦

2

𝑥1,4

Error of next unit (δ4) Weight Gradient of this unit

𝑒𝑄 𝑧 = 1 ∣ 𝐲 𝐱𝐣 = 𝑒ℎ𝑗 𝐲 𝑒𝐱𝐣

𝑘

δ𝑘 𝑥𝑗,𝑘

In General Calculate i based

• n next units j:

Backpropagation = Gradient descent + Chain rule

Feed Forward Neural Nets

All connections point forward

y

ϕ 𝑦

It is a directed acyclic graph (DAG)

Neural Networks

• Non-linear classification
• Prediction: forward propagation

– Vector/matrix operations + non-linearities

• Training: backpropagation + stochastic gradient

descent

For more details, see Cho chap 3 or CIML Chap 7

NE NEURAL AL NE NETWO WORKS KS

Aside

Back to language modeling…

Representing words

• “one hot vector”

dog = [ 0, 0, 0, 0, 1, 0, 0, 0 …] cat = [ 0, 0, 0, 0, 0, 0, 1, 0 …] eat = [ 0, 1, 0, 0, 0, 0, 1, 0 …]

• That’s a large vector! practical solutions:

– limit to most frequent words (e.g., top 20000) – cluster words into classes

• WordNet classes, frequency binning, etc.

Feed-Forward Neural Language Model

Map each word into a lower-dimensional real-valued space using shared weight matrix C Embedding layer Bengio et al. 2003

Word Embeddings

• Neural language models produce word embeddings as

a by product

• Words that occurs in similar contexts tend to have

similar embeddings

• Embeddings are useful features in many NLP tasks

[Turian et al. 2009]

Word embeddings illustrated

Recurrent Neural Networks

Recurrent Neural Nets (RNN)

Part of the node outputs return as input

y

ϕ𝐮 𝑦 𝐢𝐮−𝟐

Why? It is possible to “memorize”

Training: backpropagation through time

After processing a few training examples, Update through the unfolded recurrent neural network

Recurrent neural language models

• Hidden layer plays double duty

– Memory of the network – Continuous space representation to predict output words

• Other more elaborate architectures

– Long Short Term Memory – Gated Recurrent Units

Neural Language Models in practice

• Much more expensive to train than n-grams!
• But yielded dramatic improvement in hard extrinsic tasks

– speech recognition (Mikolov et al. 2011) – and more recently machine translation (Devlin et al. 2014)

• Key practical issue:

– softmax requires normalizing over sum of scores for all possible words – What to do?

• Ignore – a score is a score (Auli and Gao, 2014)
• Integrate normalization into objective function (Devlin et al. 2014)

What we know about modeling sequences so far…

– First example: language model – What are n-gram models? – How to estimate them? – How to evaluate them? – Neural language models