Part-of-Speech Tagging for Historical English Yi Yang and Jacob - PowerPoint PPT Presentation

Part-of-Speech Tagging for Historical English Yi Yang and Jacob Eisenstein Georgia Tech

‣ Digital humaniEes research ‣ How does the portrayal of men and women differ in Shakespeare’s plays? ‣ What’s the language use paMerns in North American slave narraEves? [Muralidharan and Hearst, 2011&2012]

‣ Digital humaniEes research ‣ How does the portrayal of men and women differ in Shakespeare’s plays? ‣ What’s the language use paMerns in North American slave narraEves? ‣ NLP can help! [Muralidharan and Hearst, 2011&2012]

‣ Digital humaniEes research ‣ How does the portrayal of men and women differ in Shakespeare’s plays? ‣ What’s the language use paMerns in North American slave narraEves? ‣ NLP can help! ‣ Only if NLP works for historical texts … [Muralidharan and Hearst, 2011&2012]

Early Modern English Hee said nobody had said anything agt mee . [Henry Oxinden, 1660]

Early Modern English He He against me Hee said nobody had said anything agt mee . ‣ Spelling variaEon [Henry Oxinden, 1660]

Stanford POS Tagger Stanford: Hee said nobody had said anything agt mee . ‣ Spelling variaEon

Stanford POS Tagger Gold: X X X Stanford: Hee said nobody had said anything agt mee . ‣ Spelling variaEon

Transfer Loss for POS Tagging 25 20 Error rate 15 10 Modern English 5 3.0 0 [Rayson et al., 2007]

Transfer Loss for POS Tagging 25 Early Modern English 20 18.0 Error rate 15 10 Modern English 5 3.0 0 [Rayson et al., 2007]

Approaches ‣ Spelling normalizaEon } Rayson et al. (2007) ‣ Map from historical spellings to Scheible et al. (2011) contemporary forms. Bollmann (2011)

Approaches ‣ Spelling normalizaEon } Rayson et al. (2007) ‣ Map from historical spellings to Scheible et al. (2011) contemporary forms. Bollmann (2011) ‣ Domain adaptaEon (this work) ‣ Build robust NLP systems with } Yang & Eisenstein (2014) representaEon learning. Yang & Eisenstein (2015)

Spelling NormalizaEon Original: Hee said nobody had said anything agt mee . Normalized: Hee said nobody had said anything aged me . [VARD; Baron and Rayson, 2008]

Spelling NormalizaEon Original: Hee said nobody had said anything agt mee . Normalized: Hee said nobody had said anything aged me . X ‣ Correct normalizaEon [VARD; Baron and Rayson, 2008]

Spelling NormalizaEon against Original: Hee said nobody had said anything agt mee . Normalized: Hee said nobody had said anything aged me . X X ‣ Correct normalizaEon ‣ Incorrect normalizaEon [VARD; Baron and Rayson, 2008]

Spelling NormalizaEon He against Original: Hee said nobody had said anything agt mee . Normalized: Hee said nobody had said anything aged me . X X X ‣ Correct normalizaEon ‣ Incorrect normalizaEon ‣ False negaEve [VARD; Baron and Rayson, 2008]

Spelling NormalizaEon Gold: Stanford: Normalized: Hee said nobody had said anything aged me . X X X [VARD; Baron and Rayson, 2008]

Spelling NormalizaEon Gold: X X Stanford: Normalized: Hee said nobody had said anything aged me . X X X [VARD; Baron and Rayson, 2008]

RepresentaEon Learning Hee said nobody had said anything agt mee .

RepresentaEon Learning Hee said nobody had said anything agt mee .

RepresentaEon Learning Hee said nobody had said anything agt mee .

RepresentaEon Learning Hee said nobody had said anything agt mee . OOV Context IV Context said said } } He was was I Hee came came We told told … … …

Model

Feature Embeddings Hee said nobody had said anything agt mee . [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings Hee said nobody had said anything agt mee . [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings > v t � � p ( f t | f 2 ) ∝ exp u 2 Input Output embeddings embeddings CurrWord = hee v 1 } 1 NextWord = said u 2 2 Prefix1 = h features v 3 3 Suffix1 = e v 4 4 … [FEMA; Yang and Eisenstein, 2015]

Feature Embeddings > v t � � p ( f t | f 2 ) ∝ exp u 2 T X ` = log p ( f t | f 2 ) Input Output t 6 =2 embeddings embeddings CurrWord = hee v 1 } 1 NextWord = said u 2 2 Prefix1 = h features v 3 3 Suffix1 = e v 4 4 … [FEMA; Yang and Eisenstein, 2015]

Word Embeddings hee } ‣ Word embeddings 1 said 2 nobody words 3 had 4 … CurrWord = hee ‣ Feature embeddings } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [word2vec; Mikolov et al., 2013]

Word Embeddings hee } ‣ Word embeddings 1 said 2 ‣ Generic representaEons nobody words 3 had 4 … CurrWord = hee ‣ Feature embeddings } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [word2vec; Mikolov et al., 2013]

Word Embeddings hee } ‣ Word embeddings 1 said 2 ‣ Generic representaEons nobody words 3 had 4 … CurrWord = hee ‣ Feature embeddings } 1 NextWord = said ‣ Task-specific representaEons 2 Prefix1 = h features 3 Suffix1 = e 4 … [word2vec; Mikolov et al., 2013]

Word Embeddings hee } ‣ Word embeddings 1 said 2 ‣ Generic representaEons nobody words 3 ‣ Word co-occurrences had 4 … CurrWord = hee ‣ Feature embeddings } 1 NextWord = said ‣ Task-specific representaEons 2 Prefix1 = h features 3 Suffix1 = e 4 … [word2vec; Mikolov et al., 2013]

Word Embeddings hee } ‣ Word embeddings 1 said 2 ‣ Generic representaEons nobody words 3 ‣ Word co-occurrences had 4 … CurrWord = hee ‣ Feature embeddings } 1 NextWord = said ‣ Task-specific representaEons 2 Prefix1 = h features 3 ‣ Feature co-occurrences Suffix1 = e 4 … [word2vec; Mikolov et al., 2013]

Learning from MulEple Domains ‣ Previous work on unsupervised domain adaptaEon involves in two domains. [FEMA; Yang and Eisenstein, 2015]

Learning from MulEple Domains ‣ Previous work on unsupervised domain adaptaEon involves in two domains. ‣ Unsupervised mulE-domain adaptaEon [FEMA; Yang and Eisenstein, 2015]

Learning from MulEple Domains ‣ Previous work on unsupervised domain adaptaEon involves in two domains. ‣ Unsupervised mulE-domain adaptaEon [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Hee said nobody had said anything agt mee . [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Domain AMributes: Genre Epoch Hee said nobody had said anything agt mee . [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Domain AMributes: Genre Epoch leMers 1600+ Hee said nobody had said anything agt mee . [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Domain AMributes: Genre Epoch leMers 1600+ Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said 2 Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Domain AMributes: Genre Epoch leMers 1600+ Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said = + + 2 (shared) (leMers) (1600+) Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Domain AMributes: Genre Epoch leMers 1600+ Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said = + + 2 (shared) (leMers) (1600+) Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said = + + 2 (shared) (leMers) (1600+) Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings = + + u 2 = h (shared) + h (letters) + h (1600+) 2 2 2 Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said = + + 2 (shared) (leMers) (1600+) Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

MulEple Feature Embeddings > v t � � p ( f t | f 2 ) ∝ exp u 2 u 2 = h (shared) + h (letters) + h (1600+) 2 2 2 Hee said nobody had said anything agt mee . CurrWord = hee } 1 NextWord = said = + + 2 (shared) (leMers) (1600+) Prefix1 = h features 3 Suffix1 = e 4 … [FEMA; Yang and Eisenstein, 2015]

Experiments