[PPT] - Morphological Analysis Morphological Analysis and Generation for PowerPoint Presentation

SLIDE 1

Morphological Analysis Morphological Analysis and Generation for Pali and Generation for Pali

David Alfter Jürgen Knauth 18 September 2015

SLIDE 2

@daalft @daalft

SLIDE 3

Pali Pali

SLIDE 4

Pali Pali

(Dead) Indo-aryan language Fusional language Rich morphology Sandhi

SLIDE 5

Source: https://commons.wikimedia.org/wiki/File:BoreanLanguageTree.png

SLIDE 6

Fusional language Fusional language

Morphological information added by affigation No 1:1 correspondence

SLIDE 7

DEVO DEVO

Base: DEV- god/deity Ending: -O noun singular masculine nominative

SLIDE 8

Compounding Compounding

naccagītavāditavisūkadassanamālāgandhavilepanadhār aṇamaṇḍanavibhūsanaṭṭhānā

SLIDE 9

Compounding Compounding

naccagītavāditavisūka- dassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsana- ṭṭhānā dancing singing music show-watching garland perfume cosmetics wearing decoration decoration

SLIDE 10

Compounding Compounding

naccagītavāditavisūka- dassanamālāgandhavilepanadhāraṇamaṇḍanavibhūsana- ṭṭhānā dancing, singing, music, going to see entertainments, wearing garlands, using perfumes, and beautifying the body with cosmetics

SLIDE 11

7th precept 7th precept

naccagītavāditavisūkadassanamālāgandhavilepanadhāraṇamaṇḍana vibhūsanaṭṭhānā veramaṇi sikkhāpadaṃ samādiyāmi I adopt the precept of refraining from ...

SLIDE 12

Sandhi Sandhi

SLIDE 13

External sandhi External sandhi

evaṃ ca (and thus) → evañca

SLIDE 14

Internal sandhi Internal sandhi

paca + ti → pacati (he cooks) paca + mi → pacāmi (I cook) canda (moon) + udayo (rising) → candodayo (rising of the moon)

SLIDE 15

Internal sandhi Internal sandhi

paca + ti → pacati (he cooks) paca + mi → pacāmi (I cook) canda (moon) + udayo (rising) → candodayo (rising of the moon)

SLIDE 16

The Problem The Problem

SLIDE 17

Low-resource language Low-resource language

SLIDE 18

Why don't we adapt Why don't we adapt resources from resources from Sanskrit? Sanskrit?

SLIDE 19

Top Resources Top Resources

Dictionaries Dictionaries Morphological analyzers Morphological analyzers

SLIDE 20

Credit: http://iflizwerequeen.com

SLIDE 21

Lingua Franca Lingua Franca

SLIDE 22

Lingua Franca Lingua Franca

Written in different scripts

SLIDE 23

Lingua Franca Lingua Franca

Written in different scripts Introduces variation!

SLIDE 24

Scripts Scripts

Sinhalese Devanagari Burmese Transliterations ...

SLIDE 25

Literature Literature

SLIDE 26

Literature Literature

Scarce and not exhaustive

SLIDE 27

No annotated corpus No annotated corpus

SLIDE 28

Generation Generation

SLIDE 29

Generation Generation

and Overgeneration

SLIDE 30

Irregular Irregular

Dictionary lookup Rule based generation: Lemma => Stem Stem + Ending => Form

Regular Regular

Dictionary lookup

SLIDE 31

Word class specific lemma ending Word class specific lemma ending Lemma - Ending Lemma - Ending → Stem Stem Stem + Ending Stem + Ending → Surface Form Surface Form

SLIDE 32

Stem + Ending Stem + Ending → Form Form Ending Ending Ending Ending Ending Ending Ending Ending Ending Ending Ending Ending

SLIDE 33

Compiled Morphological Information

SLIDE 34

SLIDE 35

SLIDE 36

SLIDE 37

SLIDE 38

Lemma => Stem Lemma => Stem Stem + Ending => Stem + Ending => Form Form

deva => dev- dev- + -o => devo

SLIDE 39

Lemma => Stem Stem + Ending => Form

deva => dev- deva => dev- dev- + -o => devo dev- + -o => devo

SLIDE 40

SLIDE 41

I make I cook

karo + mi = karomi karo + mi = karomi paca + mi = pacāmi paca + mi = pacāmi

SLIDE 42

bhavaṃ (sir) bhavaṃ (sir)

stem: bhav- ending: -anto form: bhavanto bhanto

SLIDE 43

Lemma Lemma

Derive stem Select paradigm(s) based on word class Combine stem and endings Return generated forms and associated information

SLIDE 44

Verbs Verbs

Of Roots and Bases Of Roots and Bases

SLIDE 45

Abstract Root Abstract Root

√kar

(to make)

SLIDE 46

Base Base

→ karo √kar → paca √pac

(to make) (to cook) (to fight)

→ yujjha √yudh

SLIDE 47

Seven declension classes Seven declension classes

SLIDE 48

1+ bases 1+ bases

√cur

core-, coraya- (to steal)

SLIDE 49

1+ bases 1+ bases

√rudh

rundha-, rundhi-, rundhī-, rundhe-, rundho- (to obstruct)

SLIDE 50

Verb forms based on Verb forms based on Root or Base? Root or Base?

SLIDE 51

SLIDE 52

Irregular forms Irregular forms

Dictionary lookup Dictionary lookup

Full/Partial Irregularity

SLIDE 53

Output Output

JSON/XML JSON/XML

SLIDE 54

Key:Value pairs Receiver can decide what information to use

SLIDE 55

{" lemma":"eka","forms ":{"numeral":[{ "gender ":"masculine", "number ":" singular", "word ":" eko", "case":" nominative"}, {"gender ":"masculine", "number ":" singular","word ":"ekassa", "case":" genitive"},...

SLIDE 56

Analysis Analysis

SLIDE 57

Lookup Lookup

Dictionary/Table lookup

Heuristic approach Heuristic approach

Identify paradigmatic ending → Morphological Analysis → Separation Stem-Ending

SLIDE 58

buddhe

SLIDE 59

buddhe

SLIDE 60

Word Class Guesser Word Class Guesser

SLIDE 61

Heuristic Approach Heuristic Approach

Lemma Lemma Free Form Free Form

Identify possible endings Weigh by length Weigh by frequency Prune results Identify possible endings

SLIDE 62

if (ends(lemma, "a", "ā", "i", "ī", "u", "ū", "ant", "vā", "mā", "at")) { guesses.add("adjective"); } if (ends(lemma, "a", "i", "aṃ", "ma", "ya")) { guesses.add("numeral"); } if (ends(lemma, "uṃ")) { guesses.add("indeclinable"); }

Word Class Guesser: Lemma Word Class Guesser: Lemma

Code Excerpt

SLIDE 63

Results Results

Accuracy Nouns-Adjectives 99.96% Pronouns 88.57% Numerals 76.62% Verbs 63.37%

SLIDE 64

Sandhi Sandhi

SLIDE 65

Compound Sandhi Compound Sandhi

SLIDE 66

Intuition Intuition

Identify possible sandhi loci Split into n words such that

∀n : w ∈ D

n

SLIDE 67

Requires extensive Dictionary Requires extensive Dictionary More than one analysis possible More than one analysis possible Not a compound Not a compound

Problems Problems

SLIDE 68

External Sandhi External Sandhi

SLIDE 69

Sandhi-inducing words Sandhi-inducing words

ca (and) hi (because) pi (also)

Corpus-based resolution Corpus-based resolution

SLIDE 70

Hand-written rules Hand-written rules

Regular Expressions

SLIDE 71

Replacement rules \bpañca\b X ñca\b ṃ ca X pañca ñhi\b ṃ hi ñpi\b ṃ pi

SLIDE 72

Replacement rules \bpañca\b X ñca\b ṃ ca X pañca ñhi\b ṃ hi ñpi\b ṃ pi

SLIDE 73

Internal Sandhi Internal Sandhi

SLIDE 74

Internal Sandhi Internal Sandhi

SLIDE 75

Conclusion Conclusion

SLIDE 76

Paradigms for Paradigms for Generation and Generation and Analysis Analysis

SLIDE 77

Dictionary Integration Dictionary Integration for additional for additional information information

SLIDE 78

Rule-based and Rule-based and heuristic backup heuristic backup

SLIDE 79

RegEx-based External RegEx-based External Sandhi Resolution Sandhi Resolution

SLIDE 80

Lookup Lookup

SLIDE 81

Server Architecture Server Architecture

SLIDE 82

Well documented REST API Well documented REST API

Easy integration Easy integration

SLIDE 83

Data Processing Data Processing

SLIDE 84

Extract structured data Extract structured data from unstructured data from unstructured data

SLIDE 85

[n. ag. fr. abhijjhita in med. function] one who covets M <smallcaps>i.</smallcaps> 287 (T. abhijjhātar, v. l. °itar) = A <smallcaps>v.</smallcaps> 265 (T. °itar, v. l. °ātar).

SLIDE 86

[n. ag. fr. abhijjhita in med. function] one who covets M <smallcaps>i.</smallcaps> 287 (T. abhijjhātar, v. l. °itar) = A <smallcaps>v.</smallcaps> 265 (T. °itar, v. l. °ātar).

SLIDE 87

Pacati，[Ved．pacati，Idg．*peqǔō，Av．pac-； Obulg．peka to fry，roast，Lith，kepū bake，Gr．pέssw cook，pέpwn ripe] to cook，boil，roast Vin．IV，264； fig．torment in purgatory （trs．and intrs．）：Niraye pacitvā after roasting in N．S．II， 225，PvA．10，14．-- ppr．pacanto tormenting，Gen．pacato （+Caus．pācayato） D．I，52 （expld at DA．I，159，where read pacato for paccato，by pare daṇḍena pīḷentassa）．-- pp． pakka （q．v．）．‹-› Caus．pacāpeti & pāceti （q．v．）．-- Pass．paccati to be roasted or tormented （q．v．）．（Page 382）

SLIDE 88

Manual annotation Manual annotation

SLIDE 89

Open Problems Open Problems

SLIDE 90

Verbs Verbs

SLIDE 91

Use verb form table Use verb form table

Attested forms only

SLIDE 92

Internal Sandhi Internal Sandhi

SLIDE 93

Illustrating Calculation Illustrating Calculation

Splitting Internal Sandhi

SLIDE 94

"When two vowels meet, one may be elided." When two vowels meet: elide first vowel elide second vowel no elision

SLIDE 95

8 vowels n-vowel-word

N = (1 + (2 ∗ 8))n n = 2 → N = 289 n = 1 → N = 17 n = 3 → N = 4913

SLIDE 96

SLIDE 97

"A final dental is assimilated to "A final dental is assimilated to the following consonant" the following consonant"

SLIDE 98

"A final dental is assimilated to "A final dental is assimilated to the following consonant" the following consonant"

(DENTAL) (CONSONANT) : duplicate($2)

SLIDE 99

kk: t k kk: th k kk: d k kk: dh k kk: n k kk: l k kk: s k ... 224 possibilities

SLIDE 100

151 rules Sandhi merge rules

SLIDE 101

151 rules Sandhi merge rules Sandhi split rules 1103 rules

SLIDE 102

SLIDE 103

Overall architecture Overall architecture

SLIDE 104

Morphological analyzer and generator Dictionary

SLIDE 105

Morphological analyzer and generator Dictionary Server

SLIDE 106

Morphological analyzer and generator Dictionary Server Dictionary GUI Data processor and scripting engine Corpus management and processing tool

SLIDE 107

Thank you for your attention! Thank you for your attention!

SLIDE 108

Thank you for your attention! Thank you for your attention!

Questions?