Low Rank Ensembles Eric Xing Ankur Parikh Avneesh Saluja Chris - - PowerPoint PPT Presentation

Language Modeling with Power Low Rank Ensembles

1

Ankur Parikh Avneesh Saluja Chris Dyer Eric Xing

Overview

2

Overview

• Model: Framework for language modeling using

ensembles of low rank matrices and tensors

• Relations: Includes existing n-gram smoothing

techniques as special cases

2

Overview

• Model: Framework for language modeling using

ensembles of low rank matrices and tensors

• Relations: Includes existing n-gram smoothing

techniques as special cases

• Performance: Consistently outperforms state-of-the-art

Kneser Ney baselines for same context length

• Speed: Easily scalable since no partition function

required

2

Introduction Background Rank Power Ensembles Experiments

Outline

• Introduction
• Background on n-gram smoothing
• Our Approach
• Rank
• Power
• Constructing the Ensemble
• Experiments

3

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences

4

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences

4

Linear algebra is awesome

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences

4

𝑄 𝑥1, . . , 𝑥4 = 0.3648

Linear algebra is awesome

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences

4

𝑄 𝑥1, . . , 𝑥4 = 0.3648

Linear algebra is awesome Linear algebra is boring

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences

4

𝑄 𝑥1, . . , 𝑥4 = 0.3648

Linear algebra is awesome Linear algebra is boring

𝑄 𝑥1, . . , 𝑥4 = 0.1922

Introduction Background Rank Power Ensembles Experiments

Language Modeling

• Evaluate probabilities of sentences
• Very useful in downstream applications such as machine

translation and speech recognition.

4

𝑄 𝑥1, . . , 𝑥4 = 0.3648

Linear algebra is awesome Linear algebra is boring

𝑄 𝑥1, . . , 𝑥4 = 0.1922

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

𝑥𝑗

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗 𝑥𝑗 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 count 𝑥𝑗, 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 count 𝑥𝑗, 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 𝑥𝑗−2 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 count 𝑥𝑗, 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

• Predominant approach to language modeling

N-grams

5

count 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 𝑥𝑗−2 𝑥𝑗 𝑥𝑗 𝑥𝑗−1 count 𝑥𝑗, 𝑥𝑗−1 count 𝑥𝑗, 𝑥𝑗−1, 𝑥𝑗−2

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

N-gram Smoothing

• Alleviate data sparsity problem

6

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

• “Fine-to-coarse”, captures various levels of dependence
• Very fast
• O(N) test complexity
• Low context sizes sufficient

Advantages of f N-gram Models

7

𝑄 𝑥𝑗 𝑥𝑗−1, 𝑥𝑗−2) 𝑄 𝑥𝑗 𝑥𝑗−1) 𝑄(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) 𝑄 𝑥𝑗 𝑥𝑗−1)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) (house, decrepit) 𝑄 𝑥𝑗 𝑥𝑗−1)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) (house, decrepit) 𝑄 𝑥𝑗 𝑥𝑗−1) (house)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) (house, decrepit) (house, old) (house, shabby) 𝑄 𝑥𝑗 𝑥𝑗−1) (house)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) (house, decrepit) (house, old) (house, shabby) 𝑄 𝑥𝑗 𝑥𝑗−1) (house)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗) (house, decrepit) (house, {synonym of old} ) (house, old) (house, shabby) 𝑄 𝑥𝑗 𝑥𝑗−1) (house)

Introduction Background Rank Power Ensembles Experiments

Cla lassic Dis isadvantage of f N-gram Models

• No notion of similarity between words

8

𝑄(𝑥𝑗)

?

(house, decrepit) (house, {synonym of old} ) (house, old) (house, shabby) 𝑄 𝑥𝑗 𝑥𝑗−1) (house)

Introduction Background Rank Power Ensembles Experiments

Motivation For Low Rank Methods

• Project words to lower-dimensional space

9

Introduction Background Rank Power Ensembles Experiments

Motivation For Low Rank Methods

• Project words to lower-dimensional space

9

≈

Introduction Background Rank Power Ensembles Experiments

Motivation For Low Rank Methods

• Project words to lower-dimensional space
• Words with similar contexts will have similar projections

9

≈

Introduction Background Rank Power Ensembles Experiments

Motivation For Low Rank Methods

• Project words to lower-dimensional space
• Words with similar contexts will have similar projections

9

≈

house cabin flat

Introduction Background Rank Power Ensembles Experiments

Motivation For Low Rank Methods

• Project words to lower-dimensional space
• Words with similar contexts will have similar projections

9

≈

house cabin flat

• ld

shabby decrepit

Introduction Background Rank Power Ensembles Experiments

Low Rank Approaches

10

Introduction Background Rank Power Ensembles Experiments

Low Rank Approaches

• Low rank approximation successful in many ML

applications

• Collaborate filtering (Netflix)
• Matrix completion

10

Introduction Background Rank Power Ensembles Experiments

Low Rank Approaches

• Low rank approximation successful in many ML

applications

• Collaborate filtering (Netflix)
• Matrix completion
• These solutions have been attempted in language

modeling

• Saul and Pereira 1997
• Hutchinson et al. 2011

10

Introduction Background Rank Power Ensembles Experiments

Low Rank Approaches

• Low rank approximation successful in many ML

applications

• Collaborate filtering (Netflix)
• Matrix completion
• These solutions have been attempted in language

modeling

• Saul and Pereira 1997
• Hutchinson et al. 2011
• Unfortunately, not generally competitive with Kneser

Ney

10

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

11

If rank is too small……

≈

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

11

If rank is too small……

≈

(break, spring)

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

11

If rank is too small……

≈

(break, spring)

Probability gets diluted since “break” has many synonyms

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

12

≈

If rank is too large….

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

12

≈

If rank is too large….

(domicile, dilapidated)

Introduction Background Rank Power Ensembles Experiments

Proble lem: Low Rank Methods Operate at Fix ixed Granularity

12

≈

If rank is too large….

Probabilities of rare words a problem, since representation is too fine grained

(domicile, dilapidated)

Introduction Background Rank Power Ensembles Experiments

Our Approach

13

Introduction Background Rank Power Ensembles Experiments

Our Approach

• Construct ensembles of low rank matrices/tensors to

model language at multiple granularities

13

Introduction Background Rank Power Ensembles Experiments

Our Approach

• Construct ensembles of low rank matrices/tensors to

model language at multiple granularities

• Includes existing n-gram techniques as special cases
• Absolute discounting
• Jelinek Mercer (deleted-interpolation)
• Kneser Ney

13

Introduction Background Rank Power Ensembles Experiments

Our Approach

• Construct ensembles of low rank matrices/tensors to

model language at multiple granularities

• Includes existing n-gram techniques as special cases
• Absolute discounting
• Jelinek Mercer (deleted-interpolation)
• Kneser Ney
• Preserves advantages of standard n-gram approaches
• Effective for short context lengths
• Fast evaluation at test time

13

Introduction Background Rank Power Ensembles Experiments

Outline

• Introduction
• Background on Kneser Ney smoothing
• Our Approach
• Rank
• Power
• Constructing the Ensemble
• Experiments

14

Introduction Background Rank Power Ensembles Experiments

• Lower order distribution

should be altered

Kneser Ney - Intuition

56

Introduction Background Rank Power Ensembles Experiments

• Lower order distribution

should be altered

• Consider two words, York and door
• York only follows very few words i.e. New York
• Door can follow many words i.e. “the door”, “red door”, “my door” etc.

𝑄 𝑥𝑗 = door backed − off on 𝑥𝑗−1) > 𝑄(𝑥𝑗 = York | backed − off on 𝑥𝑗−1)

Kneser Ney - Intuition

57

Introduction Background Rank Power Ensembles Experiments

• Lower order distribution

should be altered

• Consider two words, York and door
• York only follows very few words i.e. New York
• Door can follow many words i.e. “the door”, “red door”, “my door” etc.

𝑄 𝑥𝑗 = door backed − off on 𝑥𝑗−1) > 𝑄(𝑥𝑗 = York | backed − off on 𝑥𝑗−1)

Kneser Ney - Intuition

58

Introduction Background Rank Power Ensembles Experiments

Kneser Ney Unigram Distribution

16

𝑂− 𝑥𝑗 = | 𝑥 ∶ 𝑑 𝑥𝑗, 𝑥 > 0 |

Diversity of 𝒙𝒋

′𝒕 history

Introduction Background Rank Power Ensembles Experiments

Kneser Ney Unigram Distribution

16

𝑄𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗) = 𝑂− 𝑥𝑗 𝑥 𝑂− 𝑥

𝑂− 𝑥𝑗 = | 𝑥 ∶ 𝑑 𝑥𝑗, 𝑥 > 0 |

Diversity of 𝒙𝒋

′𝒕 history

Introduction Background Rank Power Ensembles Experiments

Discounting

17

Introduction Background Rank Power Ensembles Experiments

Discounting

17

𝑄𝑒 𝑥𝑗 𝑥𝑗−1) = max(𝑑 𝑥𝑗, 𝑥𝑗−1 − 𝑒, 0) 𝑥 𝑑 𝑥, 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

Discounting

17

𝑄𝑒 𝑥𝑗 𝑥𝑗−1) = max(𝑑 𝑥𝑗, 𝑥𝑗−1 − 𝑒, 0) 𝑥 𝑑 𝑥, 𝑥𝑗−1

𝑄

𝑙𝑜𝑓𝑧 𝑥𝑗 𝑥𝑗−1) =

𝑄

𝑒 𝑥𝑗 𝑥𝑗−1) + 𝛿 𝑥𝑗−1

𝑄

𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

Discounting

17

𝑄𝑒 𝑥𝑗 𝑥𝑗−1) = max(𝑑 𝑥𝑗, 𝑥𝑗−1 − 𝑒, 0) 𝑥 𝑑 𝑥, 𝑥𝑗−1 Where 𝜹 𝒙𝒋−𝟐 is the leftover probability

𝑄

𝑙𝑜𝑓𝑧 𝑥𝑗 𝑥𝑗−1) =

𝑄

𝑒 𝑥𝑗 𝑥𝑗−1) + 𝛿 𝑥𝑗−1

𝑄

𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗)

Introduction Background Rank Power Ensembles Experiments

Lower Order Marginal Aligns!

18

𝑄 𝑥𝑗 =

𝑥𝑗−1

𝑄𝑙𝑜𝑓𝑧 𝑥𝑗 𝑥𝑗−1) 𝑄 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

19

Kneser Ney Power Low Rank Ensembles

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

19

Kneser Ney

• Ensemble composed of

unsmoothed n-grams Power Low Rank Ensembles

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

19

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories Power Low Rank Ensembles

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

19

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

19

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

? ? ?

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

20

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

? ? ?

Introduction Background Rank Power Ensembles Experiments

In In General, Bigram is Full Rank

21

Introduction Background Rank Power Ensembles Experiments

In Independence = Rank 1

• If 𝑥𝑗 and 𝑥𝑗−1 are independent

73

𝑄(𝑥𝑗, 𝑥𝑗−1) = 𝑄 𝑥𝑗 𝑄 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

In Independence = Rank 1

• If 𝑥𝑗 and 𝑥𝑗−1 are independent

74

𝑄(𝑥𝑗, 𝑥𝑗−1) = 𝑄 𝑥𝑗 𝑄 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

In Independence = Rank 1

• If 𝑥𝑗 and 𝑥𝑗−1 are independent

75

𝑄(𝑥𝑗, 𝑥𝑗−1) = 𝑄 𝑥𝑗 𝑄 𝑥𝑗−1

=

𝑄(ℎ𝑝𝑣𝑡𝑓) 𝑄(𝑝𝑚𝑒) 𝑄(ℎ𝑝𝑣𝑡𝑓, 𝑝𝑚𝑒)

Introduction Background Rank Power Ensembles Experiments

In Independence = Rank 1

• If 𝑥𝑗 and 𝑥𝑗−1 are independent
• But what if 𝑥𝑗 and 𝑥𝑗−1 are not independent? What

does the best rank 1 approximation give?

76

𝑄(𝑥𝑗, 𝑥𝑗−1) = 𝑄 𝑥𝑗 𝑄 𝑥𝑗−1

=

𝑄(ℎ𝑝𝑣𝑡𝑓) 𝑄(𝑝𝑚𝑒) 𝑄(ℎ𝑝𝑣𝑡𝑓, 𝑝𝑚𝑒)

Introduction Background Rank Power Ensembles Experiments

Rank

• Let 𝑪 be the matrix such that

𝑪 𝑥𝑗, 𝑥𝑗−1 = 𝑑 𝑥𝑗, 𝑥𝑗−1

• Let
• Then

𝑵1 𝑥𝑗, 𝑥𝑗−1 ∝ 𝑄 𝑥𝑗 𝑄 𝑥𝑗−1

77

𝑵1 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪 − 𝑵 𝐿𝑀

=

Generalized KL [Lee and Seung 2001]

Introduction Background Rank Power Ensembles Experiments

Rank

• MLE unigram is normalized rank 1 approx. of MLE

bigram under KL:

24

𝑄 𝑥𝑗 = 𝑵1 𝑥𝑗, 𝑥𝑗−1 𝑥𝑗 𝑵1(𝑥𝑗, 𝑥𝑗−1)

Introduction Background Rank Power Ensembles Experiments

Rank

• MLE unigram is normalized rank 1 approx. of MLE

bigram under KL:

• Vary rank to obtain quantities between bigram and

unigram

24

𝑄 𝑥𝑗 = 𝑵1 𝑥𝑗, 𝑥𝑗−1 𝑥𝑗 𝑵1(𝑥𝑗, 𝑥𝑗−1)

Introduction Background Rank Power Ensembles Experiments

Rank

• MLE unigram is normalized rank 1 approx. of MLE

bigram under KL:

• Vary rank to obtain quantities between bigram and

unigram

24

𝑄 𝑥𝑗 = 𝑵1 𝑥𝑗, 𝑥𝑗−1 𝑥𝑗 𝑵1(𝑥𝑗, 𝑥𝑗−1)

full rank rank 1

Introduction Background Rank Power Ensembles Experiments

Rank

• MLE unigram is normalized rank 1 approx. of MLE

bigram under KL:

• Vary rank to obtain quantities between bigram and

unigram

24

𝑄 𝑥𝑗 = 𝑵1 𝑥𝑗, 𝑥𝑗−1 𝑥𝑗 𝑵1(𝑥𝑗, 𝑥𝑗−1)

full rank low rank rank 1

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

25

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

? ?

• Ensemble composed of

unsmoothed n-grams plus other low rank matrices/tensors

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

26

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

? ?

• Ensemble composed of

unsmoothed n-grams plus other low rank matrices/tensors

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

row sum

𝟓 𝟔 𝟑

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

row sum

𝟓 𝟔 𝟑

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔

row sum

𝟓 𝟔 𝟑

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔

row sum row sum

𝟓 𝟔 𝟑 𝟒. 𝟓 𝟑. 𝟑 𝟐. 𝟓

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔

row sum row sum

𝟓 𝟔 𝟑 𝟒. 𝟓 𝟑. 𝟑 𝟐. 𝟓

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏 𝟐 𝟐 𝟐 𝟏 𝟐 𝟐 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔 𝑪𝟏

row sum row sum

𝟓 𝟔 𝟑 𝟒. 𝟓 𝟑. 𝟑 𝟐. 𝟓

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏 𝟐 𝟐 𝟐 𝟏 𝟐 𝟐 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔 𝑪𝟏

row sum row sum row sum

𝟓 𝟔 𝟑 𝟒. 𝟓 𝟑. 𝟑 𝟐. 𝟓 𝟒 𝟐 𝟐

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

27

𝟐 𝟑 𝟐 𝟏 𝟑 𝟔 𝟏 𝟏 𝟏 𝟐 𝟐 𝟐 𝟏 𝟐 𝟐 𝟏 𝟏 𝟏

𝑪

𝟐 𝟐. 𝟓 𝟐 𝟏 𝟐. 𝟓 𝟑. 𝟑 𝟏 𝟏 𝟏

𝑪𝟏.𝟔 𝑪𝟏

emphasis on diversity

row sum row sum row sum

𝟓 𝟔 𝟑 𝟒. 𝟓 𝟑. 𝟑 𝟐. 𝟓 𝟒 𝟐 𝟐

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

𝑵1

0 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪𝟏 − 𝑵 𝐿𝑀

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

𝑵1

0 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪𝟏 − 𝑵 𝐿𝑀

𝑄𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗) = 𝑵1

0 𝑥𝑗, 𝑥𝑗−1

𝑥 𝑵1

0 𝑥, 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

𝑵1

0 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪𝟏 − 𝑵 𝐿𝑀

𝑄𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗) = 𝑵1

0 𝑥𝑗, 𝑥𝑗−1

𝑥 𝑵1

0 𝑥, 𝑥𝑗−1

power = 1 full rank

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

𝑵1

0 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪𝟏 − 𝑵 𝐿𝑀

𝑄𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗) = 𝑵1

0 𝑥𝑗, 𝑥𝑗−1

𝑥 𝑵1

0 𝑥, 𝑥𝑗−1

power = 1 full rank power = 0 full rank power

Introduction Background Rank Power Ensembles Experiments

Consider Elementwise Power

28

𝑵1

0 = 𝑛𝑗𝑜𝑵:𝑵≥0,𝑠𝑏𝑜𝑙 𝑵 =1 𝑪𝟏 − 𝑵 𝐿𝑀

𝑄𝑙𝑜−𝑣𝑜𝑗(𝑥𝑗) = 𝑵1

0 𝑥𝑗, 𝑥𝑗−1

𝑥 𝑵1

0 𝑥, 𝑥𝑗−1

power = 1 full rank power = 0 full rank power = 0 rank = 1 power low rank

Introduction Background Rank Power Ensembles Experiments

Vary rying Rank and Power

• Construct matrices of varying rank and power

29

power = 1 full rank power = 0 rank = 1

Introduction Background Rank Power Ensembles Experiments

Vary rying Rank and Power

• Construct matrices of varying rank and power

29

power = 1 full rank power = 0.5 low rank power = 0 rank = 1

Introduction Background Rank Power Ensembles Experiments

Vary rying Rank and Power

• Generalizes to higher orders

30

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

31

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

?

• Ensemble composed of

unsmoothed n-grams plus other low rank matrices/tensors

• Alter lower order distributions

by elementwise power

Introduction Background Rank Power Ensembles Experiments

Generalizing KN to PLRE

32

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

?

• Ensemble composed of

unsmoothed n-grams plus other low rank matrices/tensors

• Alter lower order distributions

by elementwise power

Introduction Background Rank Power Ensembles Experiments

Key Requirements

• Marginal constraint must hold:
• Evaluation of conditional probabilities must be fast

33

𝑄 𝑥𝑗 =

𝑥𝑗−1

𝑄

𝑡𝑛 𝑥𝑗 𝑥𝑗−1)

𝑄 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 1: Compute discounts on powered counts such that marginal

constraint holds. Each count gets a different discount

34

1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 1: Compute discounts on powered counts such that marginal

constraint holds. Each count gets a different discount

34

discount discount 1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 1: Compute discounts on powered counts such that marginal

constraint holds. Each count gets a different discount

34

discount discount 1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 2: Take low rank approximation of discounted quantities

such that marginal constraint still holds

35

discount discount 1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 2: Take low rank approximation of discounted quantities

such that marginal constraint still holds

35

discount discount low rank low rank 1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 2: Take low rank approximation of discounted quantities

such that marginal constraint still holds

35

discount discount low rank low rank 1 0.5

Introduction Background Rank Power Ensembles Experiments

Our Approach: Two Step Procedure

• Step 2: Take low rank approximation of discounted quantities

such that marginal constraint still holds

35

discount discount low rank low rank

power low rank ensemble

1 0.5

Introduction Background Rank Power Ensembles Experiments

Why It It Works

• Low rank approximations with respect to KL preserve

row/column sums

36

Introduction Background Rank Power Ensembles Experiments

Why It It Works

• Low rank approximations with respect to KL preserve

row/column sums

36

low rank

Introduction Background Rank Power Ensembles Experiments

Why It It Works

• Low rank approximations with respect to KL preserve

row/column sums

36

low rank

Introduction Background Rank Power Ensembles Experiments

Why It It Works

• Low rank approximations with respect to KL preserve

row/column sums

36

low rank

Introduction Background Rank Power Ensembles Experiments

Why It It Works

• Low rank approximations with respect to KL preserve

row/column sums

• Therefore, discounting / leftover weight are preserved

under the low rank approximation

36

low rank

Introduction Background Rank Power Ensembles Experiments

Normalizer can be Precomputed

• Low rank approximations with respect to KL preserve

row/column sums

37

low rank

Introduction Background Rank Power Ensembles Experiments

Normalizer can be Precomputed

• Low rank approximations with respect to KL preserve

row/column sums

• Compute normalizers on sparse counts

37

low rank

Introduction Background Rank Power Ensembles Experiments

Normalizer can be Precomputed

• Low rank approximations with respect to KL preserve

row/column sums

• Compute normalizers on sparse counts
• No partition functions!

37

low rank

Introduction Background Rank Power Ensembles Experiments

Marginal Constraint Holds

38

𝑄 𝑥𝑗 =

𝑥𝑗−1

𝑄𝑞𝑚𝑠𝑓 𝑥𝑗 𝑥𝑗−1) 𝑄 𝑥𝑗−1

Introduction Background Rank Power Ensembles Experiments

• Generalized discounting

scheme: First compute discounts on powered counts, then take low rank approximation

Generalizing KN to PLRE

39

Kneser Ney

• Ensemble composed of

unsmoothed n-grams

• Alter lower order distributions by

using count of unique histories

• Use absolute discounting to

interpolate different n-grams and preserve lower order marginal constraint Power Low Rank Ensembles

• Ensemble composed of

unsmoothed n-grams plus other low rank matrices/tensors

• Alter lower order distributions

by elementwise power

Introduction Background Rank Power Ensembles Experiments

Training Procedure

40

Introduction Background Rank Power Ensembles Experiments

Training Procedure

40

count from corpus

Introduction Background Rank Power Ensembles Experiments

Training Procedure

40

count from corpus count from corpus

Introduction Background Rank Power Ensembles Experiments

Training Procedure

40

count from corpus count from corpus Use alternating minimization (EM) to compute low rank approximation with respect to KL [Lee and Seung 2001]

Introduction Background Rank Power Ensembles Experiments

Training Procedure

• Because of ensemble representation, required rank is
• nly about 100, even for billion word datasets

40

count from corpus count from corpus Use alternating minimization (EM) to compute low rank approximation with respect to KL [Lee and Seung 2001]

Introduction Background Rank Power Ensembles Experiments

Test Time

KN Test Complexity: 𝑃(𝑜) PLRE Test Complexity: 𝑃 𝑜𝐿

41

𝑜 = 𝑝𝑠𝑒𝑓𝑠, 𝐿 = 𝑠𝑏𝑜𝑙

Introduction Background Rank Power Ensembles Experiments

Test Time

KN Test Complexity: 𝑃(𝑜) PLRE Test Complexity: 𝑃 𝑜𝐿

41

=

𝑜 = 𝑝𝑠𝑒𝑓𝑠, 𝐿 = 𝑠𝑏𝑜𝑙

Introduction Background Rank Power Ensembles Experiments

Test Time

KN Test Complexity: 𝑃(𝑜) PLRE Test Complexity: 𝑃 𝑜𝐿

41

=

𝑳 𝑳

𝑜 = 𝑝𝑠𝑒𝑓𝑠, 𝐿 = 𝑠𝑏𝑜𝑙

Introduction Background Rank Power Ensembles Experiments

Outline

• Introduction
• Background on n-gram smoothing
• Our Approach
• Rank
• Power
• Constructing the Ensemble
• Experiments

42

Introduction Background Rank Power Ensembles Experiments

Experiments

• Evaluate on English and Russian
• Baselines
• modKN – Modified Kneser Ney (back-off)
• modint-KN- Modified Interpolated Kneser Ney
• Other comparisons: Class-based models, Neural Networks,

Hierarchical Pitman Yor

43

Introduction Background Rank Power Ensembles Experiments

Small Datasets - Perplexity

• English-Small [Bengio et al. 2003]
• 20K vocabulary
• 14 million tokens
• Russian-Small
• 77K vocabulary
• 3.5 million tokens

44

Introduction Background Rank Power Ensembles Experiments

Small Datasets - Perplexity

• English-Small [Bengio et al. 2003]
• 20K vocabulary
• 14 million tokens
• Russian-Small
• 77K vocabulary
• 3.5 million tokens

44

class KN mod-KN modint-KN PLRE English-Small 119.7 104.55 100.07 95.15 Russian-Small 284.09 283.7 260.19 238.96

Introduction Background Rank Power Ensembles Experiments

Small English Comparisons

45

Introduction Background Rank Power Ensembles Experiments

Small English Comparisons

46

Model Context Size Perplexity mod-KN(4) 3 128 modint-KN(4) 3 116.6 infinity-gram HPYP [Wood et al. 2009] infinity 111.8

Introduction Background Rank Power Ensembles Experiments

Small English Comparisons

47

Model Context Size Perplexity mod-KN(4) 3 128 modint-KN(4) 3 116.6 infinity-gram HPYP [Wood et al. 2009] infinity 111.8 PLRE(4) 3 108.7

Introduction Background Rank Power Ensembles Experiments

Small English Comparisons

48

Model Context Size Perplexity mod-KN(4) 3 128 modint-KN(4) 3 116.6 infinity-gram HPYP [Wood et al. 2009] infinity 111.8 PLRE(4) 3 108.7 LBL [Mnih and Hinton 2007] 5 117 LBL [Mnih and Hinton 2007] 10 107.8 RNN-ME [Mikolov et al. 2012] infinity 82.1

Introduction Background Rank Power Ensembles Experiments

Large Datasets - Perplexity

• English-Large
• 836,000 types
• 837 million tokens
• Russian-Large
• 1.3 million types
• 521 million tokens
• On 8 cores, PLRE (with optimal parameter settings)

completes training on English-Large in 3.2 hrs and Russian-Large in 7.7 hours

49

Introduction Background Rank Power Ensembles Experiments

Large Datasets - Perplexity

• English-Large
• 836,000 types
• 837 million tokens
• Russian-Large
• 1.3 million types
• 521 million tokens
• On 8 cores, PLRE (with optimal parameter settings)

completes training on English-Large in 3.2 hrs and Russian-Large in 7.7 hours

49

modint-KN PLRE English-Large 77.90 +/- 0.20 75.66 +/- 0.19 Russian-Large 289.6 +/-6.82 264.59 +/- 5.84

Introduction Background Rank Power Ensembles Experiments

Machine Translation Task

• English to Russian translation task

(Language model is used as a feature in the translation system)

• Unlike other recent works, we use

PLRE instead of modint-KN (not both)

• To deal with the non-determinism,

the model is only trained once, using modint-KN. The same feature weights are then used for both PLRE and modint-KN

50

Introduction Background Rank Power Ensembles Experiments

Machine Translation Task

• English to Russian translation task

(Language model is used as a feature in the translation system)

• Unlike other recent works, we use

PLRE instead of modint-KN (not both)

• To deal with the non-determinism,

the model is only trained once, using modint-KN. The same feature weights are then used for both PLRE and modint-KN

50

Method BLEU modint-KN 17.63 +/- 0.11 PLRE 17.79 +/- 0.07 Smallest Diff PLRE+0.05 Largest Diff PLRE+0.29

Conclusion

• We presented a novel technique for language modeling

called power low rank ensembles

• Consistently outperforms state-of-the-art Kneser Ney

baselines

• Effective for small context sizes
• No partition function required
• Part of broader theme of exploiting connection between

linear algebra and probability to develop new solutions for NLP

51

Thanks!

52

Code/data available at http://www.cs.cmu.edu/~apparikh/plre