[PPT] - Numeracy for Language Models: Evaluating and Improving their Ability PowerPoint Presentation

SLIDE 1

Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers

Georgios Spithourakis, Steffen Petersen, Sebastian Riedel

Machine Reading Group

SLIDE 2

words

Numeracy

π

ℤ

i 4 2018

ne

two three four dog 0.001 fox cat mat sat jumped brown

numbers numerals

the sleeping 2

ℝ ℂ ℕ

3.14 1.73 2

ℚ

2 2 2/3 5/8 1

1

7 3.14…

0. ത

9 1+2i −1 2000

SLIDE 3

Literate Language Models

‘I eat an apple’ ‘An apple eats me’ ‘I eats an apple’

Plausible (semantically, grammatically, etc.)

‘A apple eats I’

𝑄𝑀𝑁 𝑢𝑓𝑦𝑢

SLIDE 4

Numerate Language Models

‘John is 0 m tall’ ‘John is 1.7 m tall’ ‘John is 2 m tall’

𝑄𝑀𝑁 𝑢𝑓𝑦𝑢

‘John is 999 m tall’

1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2

𝑄 ℎ𝑓𝑗𝑕ℎ𝑢

SLIDE 5

Numeracy Matters

0.5 ‘Unemployment of the US is 5 %’ 50 500 ‘Our model is 10 times better than the baseline’ 100 1000 0.0 23.2 ‘Patient’s temperature is 36.6 degrees’ 41.9 98.6

SLIDE 6

Q1: Are existing LMs numerate? Q2: How to improve the numeracy of LMs?

SLIDE 7

Q1: Are existing LMs numerate? Q2: How to improve the numeracy of LMs?

SLIDE 8

A Neural Language Model

RNN 𝑞(𝑥𝑢|ℎ𝑢) ℎ𝑢 ℎ𝑢−1 𝑓𝑢 … Input 𝑥𝑢−1 … Output

SLIDE 9

A Neural Language Model

RNN 𝑞(𝑥𝑢|ℎ𝑢) ℎ𝑢 ℎ𝑢−1 𝑓𝑢 … Input 𝑥𝑢−1 … Output = softmax

𝑊

(𝑥𝑢)

the cat mat sat UNK V 2 1 1.7 2018 UNKNUM 2.1 … 0.731 9,846,321 2018.3 petrichor unothrorgaphy Spithourakis … ht

SLIDE 10

Evaluation: Adjusted Perplexity

Perplexity 2.1

… 0.731 9,846,321 2018.3

John is 2.1 m tall 𝑞 2.1 = 𝑞 BUT

+ 𝑞 0.731 + 𝑞 9,846,321 + ⋮

UNKNUM

SLIDE 11

Evaluation: Adjusted Perplexity

𝑞 2.1 = 𝑞 𝑥 ∈ Perplexity Adjusted Perplexity from test data

UNKNUM [Ueberla, 1994]

2.1

… 0.731 9,846,321 2018.3

John is 2.1 m tall 𝑞 2.1 = 𝑞 BUT

+ 𝑞 0.731 + 𝑞 9,846,321 + ⋮

UNKNUM UNKNUM [Ahn et al., 2016]

a.k.a. Unknown-Penalised Perplexity

SLIDE 12

Datasets

Clinical Dataset 16,015 clinical patient reports Source: London Chest Hospital Scientific Dataset 20,962 paragraphs from scientific papers Source: ARXIV

96% 4% words numerals

SLIDE 13

Results: Adjusted Perplexity

8.91 5.99 5 10

all tokens words numerals

80.62 51.83 50 100

all tokens words numerals

3,505,856.25

Scientific

58,443.72

(Lower is better) Clinical

SLIDE 14

Results: Adjusted Perplexity

8.91 5.99 5 10

all tokens words numerals

80.62 51.83 50 100

all tokens words numerals

3,505,856.25

Scientific

58,443.72

(Lower is better) Clinical

SLIDE 15

Results: Adjusted Perplexity

PMF

8.91 5.99 5 10

all tokens words numerals

80.62 51.83 50 100

all tokens words numerals

3,505,856.25

Scientific

58,443.72

softmax

Assumptions Reality (?)

PDF (Lower is better) UNKNUM large small large small Clinical

SLIDE 16

Q1: Are existing LMs numerate? Q2: How to improve the numeracy of LMs?

SLIDE 17

Strategy: Softmax & Hierarchical Softmax

softmax

V the cat mat sat UNK word numeral ht 𝑥𝑝𝑠𝑒𝑡

softmax softmax

the cat mat sat UNK 2 1 1.7 2018 UNKNUM ht 𝑢𝑧𝑞𝑓s

𝒒(𝒐𝒗𝒏𝒇𝒔𝒃𝒎|𝒊𝒖)

SLIDE 18

Strategy: Softmax & Hierarchical Softmax

softmax

V the cat mat sat UNK 2 1 1.7 2018 UNKNUM word numeral ht 𝑥𝑝𝑠𝑒𝑡 𝑜𝑣𝑛𝑓𝑠𝑏𝑚𝑡

softmax softmax softmax

the cat mat sat UNK 2 1 1.7 2018 UNKNUM ht 𝑢𝑧𝑞𝑓s

SLIDE 19

Strategy: Softmax & Hierarchical Softmax

softmax

V the cat mat sat UNK word numeral ht 𝑥𝑝𝑠𝑒𝑡

softmax softmax 𝒒(𝒐𝒗𝒏𝒇𝒔𝒃𝒎|𝒊𝒖)

h-softmax
digit-by-digit
from PDF
etc.

the cat mat sat UNK 2 1 1.7 2018 UNKNUM ht 𝑢𝑧𝑞𝑓s

SLIDE 20

Strategy: Digit-by-Digit Composition

SOS 2 . 1 2 . 1 EOS ℎ𝑢 𝑞 2.1 = 𝑞 2 𝑞 . |2 𝑞 1 2. )𝑞 𝐹𝑃𝑇 2.1)

d-RNN

SLIDE 21

Strategy: Digit-by-Digit Composition

SOS 2 . 1 2 . 1 EOS ℎ𝑢 𝑞 2.1 = 𝑞 2 𝑞 . |2 𝑞 1 2. )𝑞 𝐹𝑃𝑇 2.1)

d-RNN UNKNUM 1.98

1.99

1.97 1.96 1.95 1.94

2.00

2.01 2.02

SLIDE 22

Strategy: from continuous PDF

0.5 1 1.5 2 2.5 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3 3.1 3.2 3.3 3.4 3.5

PDF

𝑞 𝐨𝐯𝐧𝐟𝐬𝐛𝐦 = 2.1 = 𝑞𝑸𝑵𝑮 2.05 < 𝐨𝐯𝐧𝐜𝐟𝐬 < 2.15 |precision = 1 × 𝑞 precision = 1

SLIDE 23

Strategy: from continuous PDF

0.5 1 1.5 2 2.5 1 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 3 3.1 3.2 3.3 3.4 3.5

PDF

𝑞 𝐨𝐯𝐧𝐟𝐬𝐛𝐦 = 2.1 = 𝑞𝑸𝑵𝑮 2.05 < 𝐨𝐯𝐧𝐜𝐟𝐬 < 2.15 |precision = 1 × 𝑞 precision = 1

ht MoG softmax

𝑑𝑝𝑛𝑞𝑝𝑜𝑓𝑜𝑢

𝑞 precision = 𝑞𝑆𝑂𝑂 𝑒𝑒𝑒𝑒

𝑞𝑠𝑓𝑑𝑗𝑡𝑗𝑝𝑜

𝐹𝑃𝑇 Frozen 𝜈s and 𝜏s

SLIDE 24

Overview of Strategies

<SOS> 2 . 1 2 . 1 <EOS> MoG d-RNN h-softmax

PDF

2 1.7 2018 UNKNUM 𝑜𝑣𝑛𝑓𝑠𝑏𝑚𝑡

softmax

SLIDE 25

Overview of Strategies

<SOS> 2 . 1 2 . 1 <EOS> MoG d-RNN h-softmax ht softmax

𝑡𝑢𝑠𝑏𝑢𝑓𝑕𝑗𝑓𝑡

combination

PDF

2 1.7 2018 UNKNUM 𝑜𝑣𝑛𝑓𝑠𝑏𝑚𝑡

softmax

SLIDE 26

Results: Language Modelling (1)

5.99 4.96 4.95 4.99 4.96 5 10