Aoccdrnig to a rscheearch at Cmabrigde Uinerv4sy, it deosnt m8aer in - - PowerPoint PPT Presentation

Robsut Wrod Reocginiton

via Semi-Character RNN

Keisuke Sakaguchi Kevin Duh Ma< Post Ben Van Durme

Aoccdrnig to a rscheearch at Cmabrigde Uinerv4sy, it deosn’t m8aer in waht

• redr the l8eers in a word are, the olny

iprmoetnt 4hng is taht the frist and lsat l8eer be at the rghit pclae. The rset can be a toatl mses and you can sitll raed it wouthit porbelm. Tihs is bcuseae the huamn mnid deos not raed ervey lteter by istlef, but the wrod as a wlohe.

1

Masked Priming

Forward Mask (500 milliseconds)

GARDEN gadren ########

Prime (60 milliseconds) Target

2

Forster, K. I.; Davis, C.; Schoknecht, C.; and Carter, R. 1987. Masked priming with graphemically related forms: Repe44on or par4al ac4va4on? The Quarterly Journal of Experimental Psychology 39(2):211–251.

Eye movement tracking

3

Condi@on Example

#fixa@on Regression (%) Avg. Fixa@on (ms) Normal The boy could not solve the problem so he asked for help. 10.4 15.0 236 Internal The boy cuold not slove the probelm so he aksed for help. 11.4 17.6 244 End The boy coudl not solev the problme so he askde for help. 12.6 17.5 246 Begin The boy oculd not oslve the rpoblem so he saked for help. 13.0 21.5 259

Rayner, K.; White, S. J.; Johnson, R. L.; and Liversedge, S. P. 2006. Raeding wrods with jubmled le8res: There is a cost. Psychological Science 17(3):192–193.

Semi-Character Recurrent Net (scRNN)

b i e

LSTM

b i e

LSTM

b i e

LSTM

b i e

LSTM

• Aoccdrnig

Softmax Softmax Softmax Softmax

• to

a rscheearch

• According

to a research

4

Input representa4on:

The word “University” is represented as

5

bn = {U = 1} en = {y = 1} in = {e = 1, i = 2, n = 1, s = 1, r = 1, t = 1, v = 1}

xn =   bn in en  

Alterna4ves

• Character vs. Word in output layer
• Different input representa4on, e.g. CharCNN

6

Kim, Y.; Jernite, Y.; Sontag, D.; and Rush, A. M. 2016. Character-aware neural language

• models. AAAI

Non-Neural Alterna4ves

• Search all permuta4ons

– Should work for most words – Except for anagrams: being/begin, quiet/quite, creditors/directors, views/wives, center/recent, licensed/declines

• Edit distance

– Spelling checkers

7

Experiment setup: Spelling correc4on

• Add noise to Penn Treebank:

– Jumble: Cambridge à Cmbarigde – Delete: Cambridge à Camridge – Insert: Cambridge à Cambpridge – One type of noise to every word, except short words and numbers

• Training: Noisy text à Normal text
• Evalua4on: % words corrected

8

Accuracy Results

Jumble Delete Insert CharCNN (Kim et. al.) 16 19 35 Enchant 57 35 89 Commercial A 54 60 93 Commercial B 54 71 73 scRNN 99 85 97

9

Accuracy Results

Jumble Delete Insert CharCNN (Kim et. al.) 16 19 35 Enchant 57 35 89 Commercial A 54 60 93 Commercial B 54 71 73 scRNN 99 85 97

10

Accuracy Results

Jumble Delete Insert CharCNN (Kim et. al.) 16 19 35 Enchant 57 35 89 Commercial A 54 60 93 Commercial B 54 71 73 scRNN 99 85 97

11

Effect of context

12

BPTT

Effect of model size

13

Summary

b i e

LSTM

b i e

LSTM

b i e

LSTM

b i e

LSTM

• Aoccdrnig

Softmax Softmax Softmax Softmax

• to

a rscheearch

• According

to a research

14

A simple model for correc4ng jumbled text

Discussion

• We can achieve 99% accuracy for this

matched condi4on, but…

• One model for all noisy condi4ons?

– Cooooool à cool – Speling mistake -> Spelling mistake – Noise beyond word level

15

More info

• Mul4lingual analysis of jumbled text:

h8p://www.mrc-cbu.cam.ac.uk/personal/ ma8.davis/Cmabrigde/

• Paper (AAAI2017):

h8p://www.cs.jhu.edu/~kevinduh/a/ sakaguchi17robsut.pdf

16