spell once summon anywhere a two level open vocabulary
play

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language - PowerPoint PPT Presentation

Jan 12, 2023 •460 likes •1.53k views

Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model AAAI 2019 Technical Track Sabrina J. Mielke and Jason Eisner sjmielke@jhu.edu , jason@cs.jhu.edu Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

  1. Language modeling: a generative story of text p ( the cat chased the ) = p ( the ) · p ( cat | the ) · p ( chased | the cat ) · p ( the | the cat chased ) Lexicon / vocabulary Text generation with an RNN type spelling embedding σ ( w ) e ( w ) RNN RNN RNN RNN w cell cell cell cell � [ 0.2, ··· ,0.0 ] the 1 � [ 0.4, ··· ,0.5 ] the cat c a g e d the cat 2 UNK � [ − 0.1, ··· ,0.2 ] chased 3 � [ 0.3, ··· ,0.1 ] UNK 4 ...but what is the word? Pure character-level model as the solution? Ugh, spelling the again ... RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e t h e c a t c h a s e d 2

  2. Language modeling: a generative story of text p ( the cat chased the ) = p ( the ) · p ( cat | the ) · p ( chased | the cat ) · p ( the | the cat chased ) Lexicon / vocabulary Text generation with an RNN type spelling embedding σ ( w ) e ( w ) RNN RNN RNN RNN w cell cell cell cell � [ 0.2, ··· ,0.0 ] the 1 � [ 0.4, ··· ,0.5 ] the cat c a g e d the cat 2 UNK � [ − 0.1, ··· ,0.2 ] chased 3 � [ 0.3, ··· ,0.1 ] UNK 4 ...but what is the word? Pure character-level model as the solution? Ugh, spelling the again ... RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell ...can’t we memorize it? t h e t h e c a t c h a s e d 2

  3. Our model: Spell once, summon anywhere 3

  4. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 � ) e ( 2 � ) e ( 3 3

  5. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN cell t � ) e ( 2 � ) e ( 3 3

  6. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN cell cell t h � ) e ( 2 � ) e ( 3 3

  7. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) e ( 2 � ) e ( 3 3

  8. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 � ) e ( 3 3

  9. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 3

  10. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  11. Our model: Spell once, summon anywhere Known words only have to be spelled out once, � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  12. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  13. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 � ) e ( 3 RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  14. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: � h 1 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  15. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 embeddings � = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  16. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the � ) e ( 3 σ ( w 1 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  17. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN � � h 1 h 2 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  18. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 embeddings � � = 1 = 2 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  19. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat � ) e ( 3 σ ( w 1 ) σ ( w 2 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  20. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN � � � h 1 h 2 h 3 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  21. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 embeddings � � � = 1 = 2 = 3 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  22. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 w 4 embeddings � � � � = 1 = 2 = 3 = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  23. Our model: Spell once, summon anywhere Known words only have to be spelled out once, and can then be summoned anywhere: RNN RNN RNN � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e look up � ) σ ( 1 w 1 w 2 w 3 w 4 embeddings � � � � = 1 = 2 = 3 = 1 � ) e ( 2 RNN RNN RNN look up cell cell cell c a t spellings � ) σ ( 2 the cat chased the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 3 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 3

  24. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 4 ? � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  25. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 4 ? � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  26. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  27. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  28. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  29. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  30. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  31. Our model: Spell once, summon anywhere – the open-vocabulary case Known words only have to be spelled out once, and can then be summoned anywhere. Unknown words are spelled out “on-demand” using the same character-level model. e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN RNN cell cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 4

  32. Samples from the model Sampled text from our model: Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . 5

  33. Samples from the model Sampled text from our model: Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  34. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  35. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  36. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC of Fullett . novel word with contextually appropriate spelling 5

  37. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . novel word with contextually appropriate spelling 5

  38. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word with contextually appropriate spelling 5

  39. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually appropriate spelling 5

  40. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling 5

  41. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling valuables migrations ❀ 5

  42. Samples from the model novel spelling sampled known spelling Sampled text from our model: ❀ from its embedding grounded stipped ❀ Following the death of Edward Mc- Cartney in 1060 , the new defini- differ coronate ❀ tion was transferred to the WDIC Clive Dickey ❀ of Fullett . Southport Strigger ❀ novel word Carl Wuly ❀ with contextually Chants Tranquels ❀ appropriate spelling valuables migrations ❀ So why is this a good way of modeling language? 5

  43. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 6

  44. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 “ Meaningless ” character composition should be separate from “ meaningful ” word composition! 6

  45. Linguistic notions: duality of patterning The meaningful elements in any language—"words" in everyday parlance [ ... ] — [ ... ] are represented by [ a ] small stock of distinguishable sounds which are in characters themselves wholly meaningless. – Hockett, 1960 “ Meaningless ” character composition should be separate from “ meaningful ” word composition! We should need a word’s spelling only to define it – not to later use it. 6

  46. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? 7

  47. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! 7

  48. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! 7

  49. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: 7

  50. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! 7

  51. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! ...and they’re slow as hell... 7

  52. Duality of patterning �→ conditional independence! So? Why does this linguistics blurb matter? • Irregular words have uncommon spellings children ...yet we use them like regular words! • Function words have uncommon spellings the , of ...yet we use them all the time without feeling weird! We should need a word’s spelling only to define it – not to later use it. Recall: i.e. character-level models do it wrong! ...and they’re slow as hell... usage ⊥ spelling | embedding 7

  53. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? 8

  54. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. – de Saussure, 1916, translated 8

  55. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated 8

  56. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. 8

  57. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! 8

  58. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! 8

  59. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but use spellings as prior / regularization! 8

  60. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but Outliers ( children , the , ...) use spellings as prior / regularization! may have idiosyncratic embeddings! 8

  61. The arbitrariness of the sign �→ allowing for idiosyncracy How should a word’s embedding and its spelling be connected? The connection between the signifier and the signified is arbitrary. spelling meaning – de Saussure, 1916, translated Meaning is not fully predictable from spellings. Example: neither silly nor folly is an adverb, even though they both end in - ly ! “construction” models like e ( caged ) : = CNN ( c a g e d ) ignore this! ⇒ Allow any pairing a priori, but Outliers ( children , the , ...) use spellings as prior / regularization! may have idiosyncratic embeddings! regularize embeddings, don’t construct them 8

  62. Recap: how does our model implement these ideas? Embeddings and spellings are connected on the type level, ensuring conditional independence of usage and spelling while assigning positive probability to any pairing! e ( UNK ) � � � � h 1 h 2 h 3 h 4 � ) e ( 1 RNN RNN RNN cell cell cell t h e � ) σ ( 1 w 1 w 2 w 3 w 4 = UNK � � � = 1 = 2 = 1 � ) e ( 2 RNN RNN RNN cell cell cell c a t RNN RNN RNN RNN RNN cell cell cell cell cell � ) σ ( 2 the cat c a g e d the � ) e ( 3 s ∼ p spell ( · | � σ ( w 1 ) σ ( w 2 ) σ ( w 4 ) h 3 ) RNN RNN RNN RNN RNN RNN cell cell cell cell cell cell c h a s e d � ) σ ( 3 9

  63. How do we evaluate open-vocabulary language models? 1. Report likelihood p ( held-out text ) as perplexity? ( ↓ lower is better) 10

  64. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) 10

  65. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) 2. no UNKing allowed! 10

  66. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. 10

  67. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! 10

  68. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. → we must predict every character of the text, regardless of vocabulary size ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! 10

  69. How do we evaluate open-vocabulary language models? bits per character 1. Report likelihood p ( held-out text ) as perplexity ( ↓ lower is better) no UNKing allowed! ∗ 2. → we must predict every character of the text, regardless of vocabulary size ______________ ∗ Yes, we call some words “UNK” temporarily , but we still generate them fully ! ⇒ A tunable “vocabulary size” hyperparameter decides what is temporary-UNK. 10

  70. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia 11

  71. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d 11

  72. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) 11

  73. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed 11

  74. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed 11

  75. Results test WikiText-2 (Merity et al., 2017) ← 1.8 1.4 � 2.5 million tokenized words from the English Wikipedia RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN � 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache � 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN � BPE: cell cell cell cell cell 1.468 the ca@ @ t cha@ @ sed � our full model: Spell Once, Summon Anywhere 1.455 11

  76. Results on dev data test WikiText-2 (Merity et al., 2017) rare novel frequent 2.5 million tokenized words from the English Wikipedia all words words words RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN RNN 3.89 2.08 1.38 1.775 cell cell cell cell cell cell cell cell cell cell cell cell cell cell cell t h e c a t c h a s e d HCLM + cache – – – 1.500 previous SOTA (Kawakami et al., 2017) RNN RNN RNN RNN RNN BPE: cell cell cell cell cell 4.01 1.70 1.08 1.468 the ca@ @ t cha@ @ sed our full model: Spell Once, Summon Anywhere 4.00 1.64 1.10 1.455 11

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The

Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The

4/24/13 Summon Implementation at IMSA: Illinois Mathematics and Science Academy Paula Garrett The Summon Decision IMSA environment High school; college-level research 70 databases (35 vendors) Federated search vs.

240 views • 4 slides
Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of

Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of

Nuspell: version 3 of the new spell checker FOSS spell checker implemented in C++17 with aid of Mozilla Sander van Geloven FOSDEM, Brussels February 1, 2020 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

611 views • 39 slides
Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla.

Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla.

Nuspell: the new spell checker FOSS spell checker implemented in C++14 with aid of Mozilla. Sander van Geloven FOSDEM, Brussels February 2, 2019 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

327 views • 21 slides
Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal

Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal

Open Vocabulary Learning on Source Code with a Graph-Structured Cache Milan Cvitkovic Badal Singh Anima Anandkumar Caltech, Amazon Web Services Amazon Web Services Caltech ICML, 2019-6-12 Open Vocabulary Learning Goal: Models that can

137 views • 10 slides
Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse

Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse

Spell Checking: Edit Distance VSM, session 8 CS6200: Information Retrieval Slides by: Jesse Anderton Spell Checking poiner sisters 10-15% of all queries contain spelling brimingham news errors, so spell checking can help a catamarn sailing

258 views • 7 slides
LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID

LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID

INCENTIVE INCOME o o o o o o o Darwin Platform Group of Companies 21 level5% LEVEL OPEN CONDITION:- 1 ID DIRECT 10000 = 1 LEVEL 1 ID DIRECT 25000 = 3 LEVEL 1 ID DIRECT 50000 = 5 LEVEL 1 ID DIRECT 100000 = 7 LEVEL o

730 views • 42 slides
Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018

Character-level Language Models With Word-level Learning Arvid Frydenlund March 16, 2018 Character-level Language models Want language models with an open vocabulary Character-level models give this for free Treat the probability of

1.07k views • 5 slides
+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin

+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin

+ A Quantitative Survey on the Use of the Cube Vocabulary in the Linked Open Data Cloud Karin Becker Instituto de Informtica - Federal University of Rio Grande do Sul, Brazil Shiva Jahangiri , Craig A. Knoblock Information Sciences

472 views • 32 slides
Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th

Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th

Teaching Vocabulary Pre-Teaching Vocabulary + Pre-Teaching Vocabulary: An Example for 2 nd -5 th Grades Teacher says the word and asks students to repeat the 1. word three times. Teacher states the word in context from the text. 2. Teacher

524 views • 31 slides
Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de

Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de

Generating Fine-Grained Open Vocabulary Entity Type Descriptions Rajarshi Bhowmik and Gerard de Melo Introduction Knowledge Graph Vast repository of structured facts Why short textual description? Can succinctly characterize

457 views • 20 slides
VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

ATI TEAS ENGLISH AND LANGUAGE USAGE REVIEW VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of the TEAS deal with figuring out the meaning of the word based on synonyms, context clues, tone, and your

411 views • 6 slides
VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of

ATI TEAS ENGLISH AND LANGUAGE USAGE REVIEW VOCABULARY ATI TEAS ENGLISH AND LANGUAGE USAGE VOCABULARY Vocabulary questions on this part of the TEAS deal with figuring out the meaning of the word based on synonyms, context clues, tone, and your

336 views • 4 slides
Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for

Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for

Building Science Vocabulary: Seeds of Science Roots of Reading Goal Review our model for vocabulary development Share some vocabulary instructional routines Approach to ELL vocabulary development Vocabulary Research Low

303 views • 28 slides
CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE

CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE

CLOSE THE VOCABULARY GAP! PROVEN STRATEGIES ECE ADMINISTRATORS CAN USE TO HELP TEACHERS ENHANCE VOCABULARY DEVELOPMENT IN THEIR CLASSROOMS Julie M. Wood, Ed.D. www.juliemwood.com Is Developing Young Childrens Vocabulary Knowledge Really

156 views • 15 slides
7 6 4-5 2-3 0-1 4. Fluency Speech sustained throughout Speech sustained most of Some

7 6 4-5 2-3 0-1 4. Fluency Speech sustained throughout Speech sustained most of Some

300 Listening/Speaking Oral Presentation Evaluation Rubric Maximum points = 49 7 6 4-5 2-3 0-1 1.Vocabulary Vocabulary is consistently Uses general vocabulary Some hesitation and Limited and/or inaccurate Consistently inadequate

161 views • 3 slides
Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ

Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ

Duration Models Introduction to Single Spell Models James J. Heckman University of Chicago Econ 312, Spring 2019 Heckman Duration Models The hazard function gives the probability that a spell, denoted by the nonnegative random variable T

643 views • 21 slides
Chatham House Rule Score Score

Chatham House Rule Score Score

Chatham House Rule Score Score Castor is a suitable crop for Mediterranean farming systems Castor-based polymer with an ancient story

712 views • 53 slides
A Group Membership Service for Large-Scale Grids* Fernando Castor Filho 1,4 , Raphael Y. Camargo 2

A Group Membership Service for Large-Scale Grids* Fernando Castor Filho 1,4 , Raphael Y. Camargo 2

A Group Membership Service for Large-Scale Grids* Fernando Castor Filho 1,4 , Raphael Y. Camargo 2 , Fabio Kon 3 , and Augusta Marques 4 1 Informatics Center, Federal University of Pernambuco 2 School of Arts, Sciences, and Humanities, University

490 views • 21 slides
OLEOCHEMICAL INDUSTRY IN MALAYSIA - RESEARCH, DEVELOPMENT & COMMERCIALIZATION Dr. Yeong

OLEOCHEMICAL INDUSTRY IN MALAYSIA - RESEARCH, DEVELOPMENT & COMMERCIALIZATION Dr. Yeong

OLEOCHEMICAL INDUSTRY IN MALAYSIA - RESEARCH, DEVELOPMENT & COMMERCIALIZATION Dr. Yeong Shoot Kian, Malaysian Palm Oil Board Introduction of MPOB and its Function MPOB is the premier government agency entrusted to serve the countrys

598 views • 39 slides
Major Issues in the Minor Prophets 2) The goodness and severity of God Arise, go unto

Major Issues in the Minor Prophets 2) The goodness and severity of God Arise, go unto

Major Issues in the Minor Prophets 2) The goodness and severity of God Arise, go unto Nineveh, that great city [the second time] 1 Why did the Ninevites repent? 1) Jonahs message Yet forty days, and Nineveh shall be

61 views • 5 slides
Binomials, Derangements, and Catalan Numbers Dr. Mattox Beckman University of Illinois at

Binomials, Derangements, and Catalan Numbers Dr. Mattox Beckman University of Illinois at

Introduction Fibonacci Numbers Binomial Coeffjcients Derangements Catalan Numbers Binomials, Derangements, and Catalan Numbers Dr. Mattox Beckman University of Illinois at Urbana-Champaign Department of Computer Science Introduction

175 views • 6 slides
Segal-type models of higher categories Simona Paoli Department of Mathematics University of

Segal-type models of higher categories Simona Paoli Department of Mathematics University of

Segal-type models of higher categories Simona Paoli Department of Mathematics University of Leicester Higher Structures Lisbon 2017 Simona Paoli (University of Leicester) July 2017 1 / 34 Two motivating examples Two prototype examples in

480 views • 34 slides
Harbin Institute of Technology Microsoft Research Asia Microsoft Advanced Technology Center

Harbin Institute of Technology Microsoft Research Asia Microsoft Advanced Technology Center

ACM MM 2010 Dong Liu , Xian-Sheng Hua, Meng Wang and Hong-Jiang Zhang Harbin Institute of Technology Microsoft Research Asia Microsoft Advanced Technology Center medici chapel, Firenze, Italy... Loggia dei lanzi, sword, honeymoon, ...

313 views • 29 slides
DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi ,

DL2: Training and Querying Neural Networks with Logic Ma Marc Fischer er , Mislav Balunovi , Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, Martin Vechev github.com/eth-sri/dl2 differencing neural networks finding inputs that deactivates [Pei

992 views • 13 slides

More recommend