SLIDE 1
Kadlec, R., Bajgar, O., Hrinčár, P., Kleindienst, J. IBM Watson, Prague lab based on https://openreview.net/pdf?id=rJM69B5xx
Finding a Jack-of-All-Trades: An Examination of Transfer Learning in - - PowerPoint PPT Presentation
Finding a Jack-of-All-Trades: An Examination of Transfer Learning in - - PowerPoint PPT Presentation
Finding a Jack-of-All-Trades: An Examination of Transfer Learning in Text Comprehension Kadlec, R., Bajgar, O., Hrinr , P., Kleindienst, J. IBM Watson, Prague lab based on https://openreview.net/pdf?id=rJM69B5xx Generalization is the key
SLIDE 2
SLIDE 3
Cloze style questions
Children’s Book Test (Hill et al 2015)
Hill, F., Bordes, A., Chopra, S., & Weston, J. (2015). The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations
~ 200k examples (CN+NE)
SLIDE 4
Starting point
Bajgar, O., Kadlec, R., & Kleindienst, J. (2016). Embracing data abundance: BookTest Dataset for Reading Comprehension. http://arxiv.org/abs/1610.00956
BookTest (Bajgar et al, 2016) 14M examples Train Children’s Book Test
(Hill et al, 2015)
CBT dev/test 2k examples Test ASReader
(Kadlec et al, 2016)
ML Model
SLIDE 5
Trained on more data (BookTest) than the previous models!
BookTest
5
SLIDE 6
Transfer learning?
AS Reader Children’s Book Test (Hill et al, 2015) BookTest (Bajgar et al, 2016) 14M examples Train Test bAbI
(Weston et al, 2015)
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merrienboer, B., Joulin, A., & Mikolov,
- T. (2015). Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks.
SLIDE 7
Simple testing tasks: bAbI tasks
SLIDE 8
Simple testing tasks: bAbI tasks
SLIDE 9
Can it it generalize what it it le learned? Not really .. ...
9
BAD!
SLIDE 10
Finetuning - bAbI
AS Reader BookTest Train Test bAbI
bAbI 10
bAbI 100
bAbI 1k
SLIDE 11
2nd Experiment: It It does better wit ith target-adjustment!
11
SLIDE 12
2nd Experiment: It It does better wit ith target-adjustment!
12
SLIDE 13
13
SLIDE 14
b
11 bAbI tasks mean
SLIDE 15
Finetuning - SQuAD
ML BookTest (Bajgar et al, 2016) 14M examples Train Test SQuAD dev SQuAD SQuAD
(Rajpurkar et al 2016)
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text A subset with single word answers
SLIDE 16
16
subset
SOTA is around 75% -> we are missing something, however pre-training still helps.
SLIDE 17
3rd Experiment: Where is is the useful knowledge? ? Part rtial pretraining
17
Input Model parameters Output
SLIDE 18
18
SLIDE 19
19
SLIDE 20
Conclusions:
20
- Pre-school helps
- But it‘s not enough!
- More work to be done!
SLIDE 21
Questions?
21