Finding Canonical Forms for Historical German Text Bryan Jurish - - PowerPoint PPT Presentation

▶
finding canonical forms for historical german text
SMART_READER_LITE
LIVE PREVIEW

Finding Canonical Forms for Historical German Text Bryan Jurish - - PowerPoint PPT Presentation

Sep 12, 2023 211 likes •422 views

Finding Canonical Forms for Historical German Text Bryan Jurish jurish@bbaw.de Berlin-Brandenburgische Akademie der Wissenschaften J agerstrasse 22/23 10117 Berlin Germany September 30, 2008 KONVENS 2008 / Jurish / Finding canonical

slide-1
SLIDE 1

Finding Canonical Forms for Historical German Text

Bryan Jurish

jurish@bbaw.de

Berlin-Brandenburgische Akademie der Wissenschaften J¨ agerstrasse 22/23 · 10117 Berlin · Germany

September 30, 2008

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 1/20

slide-2
SLIDE 2

Overview

The Big Picture The Situation: unconventional text corpora The Problem: conventional tools low coverage The Proposal: conflation & canonical form(s) Conflation Methods Phonetic Identity Lemma Instantiation Heuristics Concluding Remarks Next Steps Summary

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 2/20

slide-3
SLIDE 3

The Big Picture

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 3/20

slide-4
SLIDE 4

The Situation: Corpora

“Unconventional” Text Corpora Historical text Spoken language transcriptions OCR output Non-standard dialects Lexical “Conventions” Extinct or dialect-specific lexemes Require manual attention Orthographic Conventions Extinct or dialect-specific lexical variants Can be handled automatically (to some extent)

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 4/20

slide-5
SLIDE 5

The Situation: Text Technologies

Conventional Text Technologies Document indexers Part-of-speech taggers Word stemmers Morphological analyzers Common Characteristics Fixed lexicon accessed via orthographic form Extant lexemes only Desideratum Apply existing tools to “unconventional” corpora

. . . but . . .

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 5/20

slide-6
SLIDE 6

The Problem

Conventional Tools + Unconventional Corpus = Soup Corpus variants missing from application lexicon Low coverage, poor recall, degraded accuracy, . . . Examples

Source: Deutsches Wörterbuch (DWB): Bartz et al., 2004

ir keinr nam war, wa ieder lag am rangen da sah ich sitzen siben frawen radweisz umb einen külen brunnen. vil manige sêle er zuhte dem tiuvel ûZ sînem rachen. genuoge wurden verbrant, versteinet und mit swerte erslagen

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 6/20

slide-7
SLIDE 7

The Proposal

Conflation & Canonical Form(s) Collect variant forms into equivalence classes Represent classes by (extant) canonical elements Analysis by Disjunction Analyze “extinct” form w by disjunction over extant members of its equivalence class [w]:

analyses(w) :=

  • v∈[w]

analyses(v)

Expect improved recall, some loss of precision

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 7/20

slide-8
SLIDE 8

. . . A Case in Point

Base Corpus Verse quotations from DWB

(Bartz et al., 2004)

6,581,501 tokens of 322,271 graphemic types Indexed with TAXI corpus indexing system Preprocessing & Filtering UTF-8 → ISO-8859-1 (e.g. œ → oe, e

  • → ö, ô → o, . . . )

removed non-alphabetic & foreign material 5,491,982 tokens of 318,383 graphemic types Conventional Analysis TAGH morphology FST

(Geyken & Hanneforth, 2006)

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 8/20

slide-9
SLIDE 9

Conflation Methods

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 9/20

slide-10
SLIDE 10

Phonetic Conflation: Sketch

Idea Map each word w to a unique phonetic form pho(w) Conflate words with identical phonetic forms

[w]pho := {v : pho(v) = pho(w)}

Phonetization: Letter-to-Sound (LTS) Conversion Well-known in text-to-speech (TTS) research ims german LTS rule-set

(Möhler et al., 2001)

for festival TTS system

(Black & Taylor, 1997)

slightly modified for historical input converted to finite-state transducer (FST)

  • ver 5.5 times faster than festival

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 10/20

slide-11
SLIDE 11

Phonetic Conflation: Coverage

Types Tokens Total

318,383 5,491,982

+TAGH 42.4 % 83.7 % +TAGH / pho 54.6 % 91.5 % Error Reduction 21.1 % 48.2 %

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 11/20

slide-12
SLIDE 12

Phonetic Conflation: Problems

Insufficient (too permissive) Phonetic Identity ⇒ Lexical Equivalence Precision Errors (conflated but not equivalent)

(hˆ an–Hahn), (niht–Niet), (vil–fiel), (usz–Uhus), . . .

Not too dangerous (yet) Unnecessary (too strict) Phonetic Identity ⇐ Lexical Equivalence Recall Errors (equivalent but not conflated)

(guot–gut), (pflag–pflegte), (tiuvel–Teufel), (umb–um), . . .

This is the more severe of the two problems!

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 12/20

slide-13
SLIDE 13

Lemma Instantiation: Sketch

Idea Exploit dictionary-corpus structure Assume each quote contains an instance of the associated dictionary lemma String Edit Distance

(Levenshtein, 1966; Baroni et al., 2002)

Relax strict identity criterion Pointwise Mutual Information

(McGill, 1955; Church & Hanks, 1990)

Filter out “random” phonetic similarities Restrict Comparisons Compare only lemma-instance pairs Over 10 thousand times faster (vs. all word pairs)

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 13/20

slide-14
SLIDE 14

Lemma Instantiation: Coverage

Types Tokens +TAGH 42.4 % 83.7 % +TAGH / pho 54.6 % 91.5 % +TAGH / li 66.7 % 94.4 % Error Reduction

  • vs. TAGH / pho

26.7 % 33.8 %

  • vs. TAGH

42.2 % 65.8 %

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 14/20

slide-15
SLIDE 15

Examples: Phonetic Conflation

da sah ich sitzen

sieben

siben frawen radweisz

  • radweise

umb einen külen

k¨ uhlen

brunnen.

viel, *fiel

  • vil

manige

Seele

  • sêle er zuhte

dem tiuvel ûZ

  • aus

sînem

seinem

rachen.

ihr

  • ir

keinr

nahm

  • nam

*war, wahr

  • war

wa ieder

*Ider

lag am rangen. genuoge wurden

verbrannt

  • verbrant,

versteinet und mit swerte erslagen

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 15/20

slide-16
SLIDE 16

Examples: Lemma Instantiation

da sah ich sitzen

sieben

siben frawen radweisz

  • radweise

umb einen külen

k¨ uhlen

brunnen.

viel, *fiel

  • vil

manige

Seele

  • sêle er

zuckte

zuhte dem tiuvel

Teufel

ûZ

  • aus

sînem

seinem

rachen.

ihr

  • ir

keinr

nahm

  • nam

*war, wahr

  • war

wa ieder

*Ider, jeder

lag am rangen. genuoge wurden

verbrannt

  • verbrant,

versteinet und mit swerte

?Schwert

erslagen

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 16/20

slide-17
SLIDE 17

Concluding Remarks

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 17/20

slide-18
SLIDE 18

Next Steps

Test Corpus Manually constructed gold standard

circa 11,000 tokens; 4,000 types

Quantitative analysis: precision & recall Status: 99% done (pending expert review) Robust Rewrite Cascades Weighted finite-state transducer cascades Generalized edit distance “Lazy” best-path lookup Status: Beta (gfsmxl, TAXI/DTA)

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 18/20

slide-19
SLIDE 19

Summary

Problem Historical text corpora and conventional tools don’t play together nicely Proposal Conflate lexical variants into equivalence classes . . . by phonetic identity . . . and/or by lemma-instantiation heuristics Results 94.4% tokens covered

65.8% fewer errors

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 19/20

slide-20
SLIDE 20

The End Thank you for listening!

KONVENS 2008 / Jurish / Finding canonical forms . . . – p. 20/20