More is Less? Non-parametric Language Models and Efficiency Graham - - PowerPoint PPT Presentation

Carnegie Mellon University

More is Less? Non-parametric Language Models and Efficiency

Graham Neubig

1

Carnegie Mellon University Based on Work by Junxian He w/ Taylor Berg-Kirkpatrick

Carnegie Mellon University

Parametric Language Models

2

<s> Parametric LM

Carnegie Mellon University

Parametric Language Models

3

<s> Parametric LM I

Carnegie Mellon University

Parametric Language Models

4

<s> I Parametric LM I

Carnegie Mellon University

Parametric Language Models

5

<s> I Parametric LM I ordered

Carnegie Mellon University

Parametric Language Models

6

<s> I ordered Parametric LM I ordered a

Carnegie Mellon University

Parametric Language Models

7

<s> I ordered a pizza with sauce . Parametric LM I ordered a pizza with sauce . </s>

Carnegie Mellon University

Non-Parametric Language Models

8

Non-Parametric LM Non-parametric datastore (typically training dataset)

Carnegie Mellon University

Non-Parametric Language Models

9

Non-Parametric LM I ordered a burger with fries Prototype sentence Non-parametric datastore (typically training dataset)

Carnegie Mellon University

Non-Parametric Language Models

10

<s> Non-Parametric LM I ordered a burger with fries Prototype sentence Non-parametric datastore (typically training dataset)

Carnegie Mellon University

Non-Parametric Language Models

11

<s> Non-Parametric LM I I ordered a burger with fries Prototype sentence Non-parametric datastore (typically training dataset)

Carnegie Mellon University

Non-Parametric Language Models

12

<s> I ordered a pizza with sauce . Non-Parametric LM I ordered a pizza with sauce . </s> I ordered a burger with fries Prototype sentence Non-parametric datastore (typically training dataset)

Carnegie Mellon University

Non-Parametric Language Models

13

<s> I ordered a pizza with sauce . Non-Parametric LM I ordered a pizza with sauce . </s> I ordered a burger with fries Prototype sentence Non-parametric datastore (typically training dataset)

Carnegie Mellon University

A Human Processing Analogy

14

Carnegie Mellon University

A Human Processing Analogy

14

Our Language Ability/ Knowledge

Carnegie Mellon University

A Human Processing Analogy

14

Our Language Ability/ Knowledge Our Language Ability/ Knowledge + Wikipedia/Google

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

• Count-based non-parametric data stores:
• Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
• Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
• Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

• Count-based non-parametric data stores:
• Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
• Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
• Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
• Bias probabili6es based on count-based aggrega6on of retrieved sentences:
• Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
• Deep Weighted Averaging Classifiers. Card et al. 2019.
• Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

• Count-based non-parametric data stores:
• Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
• Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
• Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
• Bias probabili6es based on count-based aggrega6on of retrieved sentences:
• Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
• Deep Weighted Averaging Classifiers. Card et al. 2019.
• Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
• A;end to retrieved sentences:
• Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
• Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
• REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

• Count-based non-parametric data stores:
• Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
• Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
• Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
• Bias probabili6es based on count-based aggrega6on of retrieved sentences:
• Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
• Deep Weighted Averaging Classifiers. Card et al. 2019.
• Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
• A;end to retrieved sentences:
• Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
• Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
• REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.
• Feed retrieved states into model:
• Learning to Remember Rare Events. Kaiser et al. 2017.

Carnegie Mellon University

A (Very Incomplete) Survey of Neural Non-parametric Models for NLP

15

• Count-based non-parametric data stores:
• Neural Machine Transla/on with External Phrase Memory. Tang et al. 2016.
• Incorpora/ng Discrete Transla/on Lexicons into Neural Machine Transla/on. Arthur et al. 2016.
• Generalizing and Hybridizing Count-based and Neural Language Models. Neubig and Dyer 2017.
• Bias probabili6es based on count-based aggrega6on of retrieved sentences:
• Guiding Neural Machine Transla/on with Retrieved Transla/on Pieces. Zhang et al. 2018.
• Deep Weighted Averaging Classifiers. Card et al. 2019.
• Nearest Neighbor Machine Transla/on. Khandelwal et al. 2020.
• A;end to retrieved sentences:
• Search Engine Guided Non-Parametric Neural Machine Transla/on. Gu et al. 2018.
• Genera/ng Sentences by Edi/ng Prototypes. Guu et al. 2018.
• REALM: Retrieval-Augmented Language Model Pre-Training. Guu et al. 2020.
• Feed retrieved states into model:
• Learning to Remember Rare Events. Kaiser et al. 2017.
• Fine-tune models on retrieved sentences:
• One Sentence One Model for Neural Machine Transla/on. Li et al. 2016.

Carnegie Mellon University

Non-Parametric Language Models: Pros/Cons

Pros

• Micgate pressure on the parametric models
• Improved interpretability in the modeling process

Carnegie Mellon University

Non-Parametric Language Models: Pros/Cons

Pros

• Micgate pressure on the parametric models
• Improved interpretability in the modeling process

Cons

• In almost all cases, large parametric datastore leads to significant

issues with memory and speed efficiency at test cme

Carnegie Mellon University

A Concrete Example: Prototype-based Language Models (Guu et al. TACL 2018)

18

Carnegie Mellon University

Previous Approach

Prototype distribution

0 1 2 3 N

p(t)

Uniform

Guu et al. 2018

Carnegie Mellon University

Previous Approach

I ordered a burger with fries

Prototype distribution

Prototype sentence

0 1 2 3 N

N

tn

p(t)

Uniform

Guu et al. 2018

Carnegie Mellon University

Previous Approach

I ordered a burger with fries

Prototype distribution

Prototype sentence Edit vector

0 1 2 3 N

N

tn zn

p(t)

Uniform

p(z)

Guu et al. 2018

Carnegie Mellon University

Previous Approach

I ordered a burger with fries I ordered a pizza with sauce

Prototype distribution

Prototype sentence Edit vector

0 1 2 3 N

N

Observed sentence

xn tn zn

p(t)

Uniform

p(z)

Guu et al. 2018

Carnegie Mellon University

Proposed Model: Sparse Prototypes for Text Generacon (He et al. NeurIPS 2020)

23

Carnegie Mellon University

Prototypes can be Sparse

I ordered a burger with fries I ordered a pizza with sauce

Guu et al. 2018

I ordered a turkey with sauce I ordered a sandwich with fries and coke I ordered a lobster roll with oysters and spicy sauce

Prototype Sentence Data Examples

Carnegie Mellon University

Sparse Neural Editor

I ordered a burger with fries I ordered a pizza with sauce

Prototype sentence Edit vector

N

Observed sentence

xn tn zn

p(z)

α

Dirichlet hyperparam

θ

Carnegie Mellon University

Sparse Neural Editor

I ordered a burger with fries I ordered a pizza with sauce

Sparse Prototype distribution

Prototype sentence Edit vector

N

Observed sentence

xn tn zn

p(z)

α

Dirichlet hyperparam

θ

0 1 2 3 N

Carnegie Mellon University

Learning and Inference

27

= Eq(t,z,θ|x) log p(x|t, z, θ) − DKL(q(t, z, θ)||p(t, z, θ))

log p(x) ≥ LELBO

Approximate posterior

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

28

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

28

q(θ) = Dir(θ; λ)

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

BERT Data example x

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

BERT Data example x BERT Prototype xk

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

29

BERT Data example x BERT Prototype xk

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Only the linear transformacon is learned while BERT is

• fixed. We use REINFORCE algorithm to propagate gradients
• ver discrete prototype variables

Carnegie Mellon University

The Approximate Posterior

30

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

The Approximate Posterior

30

I

• rdered

sweet potato fries

I

• rdered

a burger

∅

= = —

↔

↔

Edit Operation Data Embed input to inference network

xn

Prototype tn

λ

θ xn tn zn

N

q(zn|tn, xn) q(tn|xn)

Carnegie Mellon University

Experiments

31

Carnegie Mellon University

Experiments

• Datasets
• MSCOCO image capcon (40K training)
• Yelp restaurant reviews (17M training)

Carnegie Mellon University

Perplexity Results

prototypes: 17M 2K Yelp Large prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

prototypes: 17M 2K Yelp Large prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

34.2

prototypes: 17M 2K Yelp Large prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

26.9 34.2

prototypes: 17M 2K Yelp Large prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20

prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

18.6 20

prototypes: 779 Yelp Large prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

18.6 20

prototypes: 779 Yelp Large

PPL 20 40 60 80 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

18.6 20

prototypes: 779 Yelp Large

PPL 20 40 60 80 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

74.7

prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Perplexity Results

PPL 10 20 30 40 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

20.2 26.9 34.2

prototypes: 17M 2K Yelp Large

PPL 6.25 12.5 18.75 25 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

18.6 20

prototypes: 779 Yelp Large

PPL 20 40 60 80 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

61.9 74.7

prototypes: 1.5k Yelp Medium

Carnegie Mellon University

Processing Speed Results

prototypes: 17M 2K Yelp Large

Carnegie Mellon University

Processing Speed Results

Sent/Sec 0.01 1 1000 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

prototypes: 17M 2K Yelp Large

Carnegie Mellon University

Processing Speed Results

Sent/Sec 0.01 1 1000 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

272

prototypes: 17M 2K Yelp Large

Carnegie Mellon University

Processing Speed Results

Sent/Sec 0.01 1 1000 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

0.1 272

prototypes: 17M 2K Yelp Large

Carnegie Mellon University

Processing Speed Results

Sent/Sec 0.01 1 1000 NLM Neural Editor (Guu et al. 2018) Sparse Neural Editor

108 0.1 272

prototypes: 17M 2K Yelp Large

Carnegie Mellon University

How Does Sparsity Affect Prototypes?

Dense prototypes see more semancc overlap, sparse prototypes are more syntaccc.

Table 2: Qualitacve examples of prototypes when using denser and sparser prototype supports.

Carnegie Mellon University

How Does Sparsity Affect Prototypes?

Model Overall NOUN DET AUX PRON ADJ VERB 31K prototypes 91.2K 14.4K 9.6K 9.3K 9.0K 7.2K 6.4K 1.5K prototypes 74.7K 9.9K 8.5K 8.2K 7.3K 5.6K 4.4K Relative Change

• 18.1%
• 31.3%
• 11.5%
• 11.8%
• 18.9%
• 22.2%
• 31.3K

Table 1: Number of matching tokens between examples and their prototypes. Results are reported in cluster of POS tags.

Dense prototypes see more overlapping content words.

Carnegie Mellon University

Example Generacon Results

37

Carnegie Mellon University

Conclusion

38

Carnegie Mellon University

Summary

• Prototype support can be sparse to improve efficiency — superior over

previous neural editor model with only 1‰ of datastore and 1‰ of test cme

• Prototypes tend to drop semancc but keep syntaccc informacon when they

grow sparser

Carnegie Mellon University

Future Direccons?

40

• MANY other promising non-parametric models! How do they play with

sparsity?

• What is the interaccon between big non-parametric models and small

parametric models? Do we need 100 billion parameters if we have Google search?

hops://github.com/jxhe/sparse-text-prototype

Code available, try it out!