a neural model of adaptation in reading

play

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal - PowerPoint PPT Presentation

Jan 19, 2024 •515 likes •1.06k views

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018 Department of Cognitive Science, Johns Hopkins University van Schijndel and Linzen November 4, 2018 1 / 24 Thought experiment cassowary

A Neural Model of Adaptation in Reading Marten van Schijndel and Tal Linzen November 4, 2018 Department of Cognitive Science, Johns Hopkins University van Schijndel and Linzen November 4, 2018 1 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
Thought experiment …cassowary … …cassowary? … …cassowary … …cassowary! … You are now less surprised when this person says ‘Cassowary’ van Schijndel and Linzen November 4, 2018 2 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning Humans adapt to syntactic structures van Schijndel and Linzen November 4, 2018 3 / 24
A psycholinguistic example of adaptation (Fine & Jaeger, 2016) The soldiers warned about the dangers conducted the raid. Unreduced: The soldiers (who were) warned about the dangers … By end of experiment, subjects expected RRC more than at beginning van Schijndel and Linzen November 4, 2018 3 / 24 • Humans adapt to syntactic structures
Adaptation is studied in NLP Learn new words from context But can we model human adaptation? van Schijndel and Linzen November 4, 2018 4 / 24 • Domain adaptation (Kuhn & de Mori, 1990; McClosky, 2010) News Model → Biomedical Text • Handling unknown words (Grave et al., 2015) • Style adaptation (Jaech & Ostendorf, 2017) Lawyer A → Lawyer B
Adaptation is studied in NLP Learn new words from context But can we model human adaptation? van Schijndel and Linzen November 4, 2018 4 / 24 • Domain adaptation (Kuhn & de Mori, 1990; McClosky, 2010) News Model → Biomedical Text • Handling unknown words (Grave et al., 2015) • Style adaptation (Jaech & Ostendorf, 2017) Lawyer A → Lawyer B
Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24
Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24
Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24
Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24
Our proposed model LSTM language model (Gives prob of next word in sequence) Base Model: Trained on Wikipedia (90M words) (Gulordava et al., 2018) Adaptation algorithm: 1 Test on a sentence 2 Update weights based on that sentence 3 Repeat on remaining sentences van Schijndel and Linzen November 4, 2018 5 / 24
Experiment 1 (standard): Does adaptation improve model accuracy? van Schijndel and Linzen November 4, 2018 6 / 24
Accuracy Evaluation Data Test data: Natural Stories Corpus (Futrell et al., 2017) van Schijndel and Linzen November 4, 2018 7 / 24 • 10 texts (485 sentences) • 7 Fairy Tales • 3 Documentaries
Accuracy Results van Schijndel and Linzen November 4, 2018 8 / 24 Full Corpus Separate Story Types 180 Wikipedia Wikipedia+Adaptation 160 140 120 Perplexity 100 80 60 40 20 0 Natural Stories Fairy Tales Documentaries
Experiment 2: Evaluate model against human adaptation van Schijndel and Linzen November 4, 2018 9 / 24
Evaluation Measure: Surprisal Reading times can be predicted with surprisal (Smith and Levy, 2013) van Schijndel and Linzen November 4, 2018 10 / 24 Surprisal ( w i ) = − log P ( w i | w 1 .. i − 1 )
Evaluation Data: Reading Times non-adaptive surprisal van Schijndel and Linzen November 4, 2018 11 / 24 • Timeline of adaptation is similar to human adaptation • Adaptive surprisal predicts reading times better than
Evaluation: Reading Times The soldiers (who were) warned about the dangers conducted the raid. Figure 1: Human reading times Figure 2: Adaptive model surprisal van Schijndel and Linzen November 4, 2018 12 / 24

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

More recommend