Neural Autoregressive Distribution Estimation Instructor: John - - PowerPoint PPT Presentation

Neural Autoregressive Distribution Estimation

Discussion Board: Available on Ed! Zoom Link: Available on Canvas Instructor Contact: thickstn@cs.washington.edu Course Webpage: https://courses.cs.washington.edu/courses/cse599i/20au/ CSE 599I: Generative Models University of Washington Lecture 4 Instructor: John Thickstun

Autoregressive Modeling

CSE 599I: Generative Models University of Washington Lecture 4

• Factor the joint distribution into conditionals:
• Learn the conditional distributions
• Linear autoregressive model:

p(x) =

T

Y

t=1

p(xt|x<t).

ˆ p(xt|x<t).

xt ∼ N(f(x<t), σ2), f(x<t) = ρ(xt−1 − µ) + µ.

Binary Autoregressive Models

CSE 599I: Generative Models University of Washington Lecture 4

• Sigmoid function
• A log-linear autoregressive model.

sigmoid(u) = 1/(1 + e−u)

Masked Autoregressive Models

CSE 599I: Generative Models University of Washington Lecture 4

• Let such that
• Then can be equivalently defined by

m ∈ {0, 1}T ×T

f(x<t) ⌘ hP<t, x<ti = hP, mt xi.

mt,s = 1s<t.

f(x<t)

Discrete Autoregressive Models

CSE 599I: Generative Models University of Washington Lecture 4

• Softmax function defined by
• What is the input encoding of the discrete inputs ?

softmax(u) = eu Pd

k=1 euk .

softmax : Rd → Rd

wt ∈ V

xt ∈ Rd

CSE 599I: Generative Models University of Washington Lecture 4

Input Encoding

• Discrete sequences .
• Tokens for some finite vocabulary , where
• Need to encode tokens as vectors in a Euclidean space.
• Two popular options:

w ∈ VT

wt ∈ V

V

|V| = d.

• 1. One-hot encoding: where
• 2. Word Embeddings (word2vec, Glove, etc.):

x

w

x ∈ Rd

xk = 1k=w.

xk ∈ Rk (k < d).

Training a FVSBN

CSE 599I: Generative Models University of Washington Lecture 4

• Maximize the likelihood! Cross-entropy minimization:

1 n

n

X

i=1

− log p(wi) = 1 n

n

X

i=1 T

X

t=1

− log p(wi

t|wi <t).

Neural Parameterization (NADE)

CSE 599I: Generative Models University of Washington Lecture 4

• Fully-Visible Softmax Belief Network:
• Neural Autoregressive Distribution Estimation (NADE):

ft(x<t) = X

s<t

Psxs, P ∈ RT ×d×d. ht = sigmoid ct + X

s<t

Vsxs ! , V ∈ RT ×k×d, c ∈ RT ×k, ft(x<t) = bt + X

s≤t

Wshs, W ∈ RT ×d×k, b ∈ RT ×d.

Training a NADE

CSE 599I: Generative Models University of Washington Lecture 4

• Optimize with SGD (choose a random ):
• Maximize the likelihood:
• Consider adding momentum (e.g. NAG) and adaptivity (Adam).

1 n

n

X

i=1

− log pθ(wi) = 1 n

n

X

i=1 T

X

t=1

− log pθ(wi

t|wi <t).

j, t

θ(i) = θ(i−1) + ηrθ log pθ(wj

t|wj <t).

Back-Propagation Through Time

CSE 599I: Generative Models University of Washington Lecture 4

• NADE models see a limited window of context/history (order ).
• Mini-batch SGD over time steps (one call to the model):
• Also mini-batch over k data points (k calls to the model):

p

θ(i) = θ(i−1) + ηrθ log pθ(wj

t, . . . , wj t+p)

= θ(i−1) + ηrθ

p

X

s=1

log pθ(wj

t+s|wj t, . . . , wj t+s−1).

θ(i) = θ(i−1) + ηrθ

k

X

j=1 p

X

s=1

log pθ(wj

t+s|wj t, . . . , wj t+s−1).

5-Minute Break

CSE 599I: Generative Models University of Washington Lecture 4

BPTT in NLP: The Dirty Details

CSE 599I: Generative Models University of Washington Lecture 4

• Suppose we have an NLP corpus of documents (e.g. Wikitext-2).
• Concatenate all the documents: one long sequence of length .
• Chunk the sequence up into segments of length .
• Treat the corpus as data points, each of length .

p

T

n = T/p

p

Weight-Sharing

CSE 599I: Generative Models University of Washington Lecture 4

• Neural Autoregressive Distribution Estimator (order ):
• Set Vs = (Ws)T .

p

ht = sigmoid ct + X

s<t

Vsxs ! , V ∈ Rp×k×d, c ∈ Rp×k, ft(x<t) = bt + X

s≤t

Wshs, W ∈ Rp×d×k, b ∈ Rp×d.

Deep NADE

CSE 599I: Generative Models University of Washington Lecture 4

• Multiple hidden layers (compare to a deep fully-connected network)

h0,t = ReLU c0,t + X

s<t

(Ws)T xs ! , W ∈ Rp×d×k, c0 ∈ Rp×k, h`,t = ReLU @c`,t + X

s≤t

V`,sh`−1,s 1 A , V ∈ R(L−1)×p×k×k, c ∈ R(L−1)×p×k, f✓,t(x<t) = bt + X

s≤t

WshL,s, b ∈ Rp×d.

A Recurrent NADE

CSE 599I: Generative Models University of Washington Lecture 4

• Parameter counts in NADE scale with the autoregressive order .
• We can share weights using a recursively defined model (RNN):

p

• No dependency on ! Can set for further sharing.

p

Wx = WT

• <latexit sha1_base64="qxKpF/uzvxWesWzCaXsW9E7S8+c=">ADmnicdVLbhMxEHWzXMpyS+ERkFZEkcpLlY2QKqQqjYPQEKTcpGyKvdza16svKdkqC8RNfwyt8DX+DN0lLm5KRLB+fOTOesSctGNWm0fizVQlu3Lx1e/tOePfe/QcPqzuPelpOFYEukUyqQYo1MCqga6hMCgUYJ4y6KenR6W/fwZKUyk6Zl7AiOJoDkl2HhqXH2WGJiZNLd9N5Fb6JLR/mlM67WGnuNhUXQbwCNbSy9ninAkmyZSDMIRhrYdxozAji5WhIELk6mGApNTPIGhwJz0CO76MNFdc9kUS6VX8JEC/ZyhMVc6zlPvZJjc6LXfSX5X1/JGCmZXivA5K9HlopiakCQ5f35lEVGRuVbRlVQAybe4CJor6FiJxghYnxLxqGSQt8jwra2Ouylv8DTr1nYBMq8oyezdywObK7zr5wth5GUS1O9pOslGn6DZLvyX6tWQ9SQK+Esk5Ftl5qLOtsT3+4C4yXRX1sHI2KbtS3JYH96+aj579VIDCRiqbMDmhxtnltkGj6YRLmjl7DjbnymCZrNw3ZO54Xjms63ABl0KSsgpY9TZwu4Qeun1dkjbGAilR9d5vxgxutjeB30mnux59f1g7er0Z0Gz1Bz9EuitErdIDeojbqIoJ+oJ/oF/odPA0Og3fB8VJa2VrFPEZXLOj8BU5XNKk=</latexit><latexit sha1_base64="qxKpF/uzvxWesWzCaXsW9E7S8+c=">ADmnicdVLbhMxEHWzXMpyS+ERkFZEkcpLlY2QKqQqjYPQEKTcpGyKvdza16svKdkqC8RNfwyt8DX+DN0lLm5KRLB+fOTOesSctGNWm0fizVQlu3Lx1e/tOePfe/QcPqzuPelpOFYEukUyqQYo1MCqga6hMCgUYJ4y6KenR6W/fwZKUyk6Zl7AiOJoDkl2HhqXH2WGJiZNLd9N5Fb6JLR/mlM67WGnuNhUXQbwCNbSy9ninAkmyZSDMIRhrYdxozAji5WhIELk6mGApNTPIGhwJz0CO76MNFdc9kUS6VX8JEC/ZyhMVc6zlPvZJjc6LXfSX5X1/JGCmZXivA5K9HlopiakCQ5f35lEVGRuVbRlVQAybe4CJor6FiJxghYnxLxqGSQt8jwra2Ouylv8DTr1nYBMq8oyezdywObK7zr5wth5GUS1O9pOslGn6DZLvyX6tWQ9SQK+Esk5Ftl5qLOtsT3+4C4yXRX1sHI2KbtS3JYH96+aj579VIDCRiqbMDmhxtnltkGj6YRLmjl7DjbnymCZrNw3ZO54Xjms63ABl0KSsgpY9TZwu4Qeun1dkjbGAilR9d5vxgxutjeB30mnux59f1g7er0Z0Gz1Bz9EuitErdIDeojbqIoJ+oJ/oF/odPA0Og3fB8VJa2VrFPEZXLOj8BU5XNKk=</latexit><latexit sha1_base64="qxKpF/uzvxWesWzCaXsW9E7S8+c=">ADmnicdVLbhMxEHWzXMpyS+ERkFZEkcpLlY2QKqQqjYPQEKTcpGyKvdza16svKdkqC8RNfwyt8DX+DN0lLm5KRLB+fOTOesSctGNWm0fizVQlu3Lx1e/tOePfe/QcPqzuPelpOFYEukUyqQYo1MCqga6hMCgUYJ4y6KenR6W/fwZKUyk6Zl7AiOJoDkl2HhqXH2WGJiZNLd9N5Fb6JLR/mlM67WGnuNhUXQbwCNbSy9ninAkmyZSDMIRhrYdxozAji5WhIELk6mGApNTPIGhwJz0CO76MNFdc9kUS6VX8JEC/ZyhMVc6zlPvZJjc6LXfSX5X1/JGCmZXivA5K9HlopiakCQ5f35lEVGRuVbRlVQAybe4CJor6FiJxghYnxLxqGSQt8jwra2Ouylv8DTr1nYBMq8oyezdywObK7zr5wth5GUS1O9pOslGn6DZLvyX6tWQ9SQK+Esk5Ftl5qLOtsT3+4C4yXRX1sHI2KbtS3JYH96+aj579VIDCRiqbMDmhxtnltkGj6YRLmjl7DjbnymCZrNw3ZO54Xjms63ABl0KSsgpY9TZwu4Qeun1dkjbGAilR9d5vxgxutjeB30mnux59f1g7er0Z0Gz1Bz9EuitErdIDeojbqIoJ+oJ/oF/odPA0Og3fB8VJa2VrFPEZXLOj8BU5XNKk=</latexit><latexit sha1_base64="qxKpF/uzvxWesWzCaXsW9E7S8+c=">ADmnicdVLbhMxEHWzXMpyS+ERkFZEkcpLlY2QKqQqjYPQEKTcpGyKvdza16svKdkqC8RNfwyt8DX+DN0lLm5KRLB+fOTOesSctGNWm0fizVQlu3Lx1e/tOePfe/QcPqzuPelpOFYEukUyqQYo1MCqga6hMCgUYJ4y6KenR6W/fwZKUyk6Zl7AiOJoDkl2HhqXH2WGJiZNLd9N5Fb6JLR/mlM67WGnuNhUXQbwCNbSy9ninAkmyZSDMIRhrYdxozAji5WhIELk6mGApNTPIGhwJz0CO76MNFdc9kUS6VX8JEC/ZyhMVc6zlPvZJjc6LXfSX5X1/JGCmZXivA5K9HlopiakCQ5f35lEVGRuVbRlVQAybe4CJor6FiJxghYnxLxqGSQt8jwra2Ouylv8DTr1nYBMq8oyezdywObK7zr5wth5GUS1O9pOslGn6DZLvyX6tWQ9SQK+Esk5Ftl5qLOtsT3+4C4yXRX1sHI2KbtS3JYH96+aj579VIDCRiqbMDmhxtnltkGj6YRLmjl7DjbnymCZrNw3ZO54Xjms63ABl0KSsgpY9TZwu4Qeun1dkjbGAilR9d5vxgxutjeB30mnux59f1g7er0Z0Gz1Bz9EuitErdIDeojbqIoJ+oJ/oF/odPA0Og3fB8VJa2VrFPEZXLOj8BU5XNKk=</latexit>

h0 = Winit, Winit ∈ Rk, ht = sigmoid(c + Whht−1 + Wxxt), Wh ∈ Rk×k, Wx ∈ Rk×d, c ∈ Rk, ft(x<t) = b + Woht−1, Wo ∈ Rd×k, b ∈ Rd.

Hyper-Parameter Selection

CSE 599I: Generative Models University of Washington Lecture 4

• In general: make the model very wide and very deep. Use large .
• Don’t regularize with model size! Regularize by (in order of effectiveness):
• Use validation data, train multiple models (pay the Amazon tax).
• If your model is constrained by GPU memory, you are on the right track.
• 1. Getting more training data.
• 2. Dataset augmentation (e.g. input/embedding dropout)
• 3. Early stopping/L2 regularization/dropout

p

Exposure Bias

CSE 599I: Generative Models University of Washington Lecture 4

• Factor the joint distribution into conditionals:
• Learn the conditional distributions
• Iteratively sample to construct
• The data we condition on to predict has the wrong distribution!

p(x) =

T

Y

t=1

p(xt|x<t).

ˆ p(xt|x<t).

ˆ xt ∼ ˆ p(·|ˆ x<t)

Logistics

CSE 599I: Generative Models University of Washington Lecture 4

• Homework 1 is out on the website.
• If you downloaded it early, get a fresh copy: I made some corrections.
• You could start looking at the code for Problem 8 (except transformers.py)