[PPT] - Applications in finite state automata The lexc language Kurt Eberle PowerPoint Presentation

SLIDE 1

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Applications in finite state automata

The lexc language Kurt Eberle

kurt.eberle@uni-tuebingen.de

(includes material from Karttunen, Beesley, Butt and others)

November 29, 2016

1 / 56

SLIDE 2

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

2 / 56

SLIDE 3

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Topics of this session

Understand lexc

◮ lexc files ◮ Continuation classes ◮ Defining lexical transducers ◮ Include definitions in lexc ◮ Useful strategies ◮ Integration and testing 3 / 56

SLIDE 4

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

4 / 56

SLIDE 5

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc files

◮ for definition of one or several lexica ◮ structure

◮ Multichar symbols declaration ◮ Declarations (definitions) ◮ Named lexica

with one (start) lexicon named root

5 / 56

SLIDE 6

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

structure

6 / 56

SLIDE 7

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

The start lexicon

7 / 56

SLIDE 8

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

The start lexicon: notational variants

8 / 56

SLIDE 9

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

end of lexc files

9 / 56

SLIDE 10

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

special symbols

10 / 56

SLIDE 11

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc files

compile file

◮ read lexc < ex1-lex.txt

→ network: 1 stack element

◮ save stack ex1-lex.fst

11 / 56

SLIDE 12

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

compiled lexicon

12 / 56

SLIDE 13

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

lexc file

a kind of abbreviation

13 / 56

SLIDE 14

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

14 / 56

SLIDE 15

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

stems and continuations

◮ analyze lexical entries into ◮ stem (root) ◮ ending(s) (continuations)

15 / 56

SLIDE 16

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

example

16 / 56

SLIDE 17

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Syntax

17 / 56

SLIDE 18

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

continuation classes and morphotactics

◮ continuation classes inherited from Koskenniemi’s two-level

morphology

◮ more difficult to model:

◮ separated dependencies ◮ interdigitation ◮ infixation ◮ reduplication 18 / 56

SLIDE 19

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Multiple classes of words

◮ style: stem + affix

19 / 56

SLIDE 20

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Multiple classes of words

◮ PoS classified stems

20 / 56

SLIDE 21

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Optionality

◮ style: aspect morphem as an option

21 / 56

SLIDE 22

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Optionality

◮ variant: aspect morphem as an option

22 / 56

SLIDE 23

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Continuation classes

Loops

◮ several instances of a class in a string ◮ (N.B. ’no procedural reading’)

23 / 56

SLIDE 24

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

24 / 56

SLIDE 25

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Defining lexical transducers

Transitions in lexc

25 / 56

SLIDE 26

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Lexical transducers

regular expressions

26 / 56

SLIDE 27

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Lexical transducers

Multichars

◮ Introductory section!

27 / 56

SLIDE 28

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

28 / 56

SLIDE 29

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Include definitions in lexc

Definitions section

◮ Defined vars

29 / 56

SLIDE 30

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

30 / 56

SLIDE 31

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

◮ visualize the morphotactic templates before coding (informal

sketch)

◮ a lexicon should subsume a generally coherent collection of

morphemes

◮ a lexicon is a potential target for continuations from other

morphemes

◮ prefer most general solutions if possible (large classes) ◮ avoid multiple copies of a morpheme if possible ◮ separated dependencies are difficult to model with

continuation classes: use filters instead, if possible

◮ read error messages carefully. Be aware of loops and filters

increasing complexity heavily

31 / 56

SLIDE 32

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Visualization

32 / 56

SLIDE 33

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Organization into coherent lexicons

33 / 56

SLIDE 34

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Using sublexicons

34 / 56

SLIDE 35

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Classes and subclasses . . .

35 / 56

SLIDE 36

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Classes and subclasses: add irregularities 1

36 / 56

SLIDE 37

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Classes and subclasses: add irregularities 2 (better solution)

37 / 56

SLIDE 38

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Tag-names

◮ prime directive: appropriate and known tags (if possible) ◮ secondary directive: same name for same phenomenon ◮ tertiary directive: take up naming used in grammars of related

languages if possible

38 / 56

SLIDE 39

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Separated dependencies

◮ example: (pseudo) Arabic

39 / 56

SLIDE 40

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Useful strategies

Separated dependencies

40 / 56

SLIDE 41

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Separated dependencies

Bad strategy

41 / 56

SLIDE 42

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Separated dependencies

Overgeneration . . .

42 / 56

SLIDE 43

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Separated dependencies

Overgeneration examples

43 / 56

SLIDE 44

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Separated dependencies

Filter to avoid overgeneration

44 / 56

SLIDE 45

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Outline

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

45 / 56

SLIDE 46

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interation and testing

◮ get a feeling how components of a larger system interact ◮ thinking about languages ◮ thinking about transducers ◮ thinking about rules ◮ thinking about composition

46 / 56

SLIDE 47

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

languages

◮ avoid overgeneration ◮ avoid undergeneration ◮ model information via different modules

47 / 56

SLIDE 48

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

languages

48 / 56

SLIDE 49

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

transducers: Xerox ’world’

◮ upper language: description level ◮ lower language: string level (inflected form) ◮ analyze string (look up) ◮ generate string (look down)

49 / 56

SLIDE 50

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

Analyze string!

50 / 56

SLIDE 51

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

Generate string!

51 / 56

SLIDE 52

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

Be aware of setting

52 / 56

SLIDE 53

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

Be aware of setting

53 / 56

SLIDE 54

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interaction and testing

rules

◮ ’principle based’ ◮ easy to extend ◮ irregular (beau/belle): treat in lexc per transducer statement ◮ regular: use (postprocessing) rules

54 / 56

SLIDE 55

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interation and testing

composition

55 / 56

SLIDE 56

lexc files Continuation classes Defining lexical transducers Include definitions in lexc Useful strategies Interation and testing

Interation and testing

composition

56 / 56