Learnability-based Syntactic Annotation Design Roy Schwartz, Omri - - PowerPoint PPT Presentation

▶
learnability based syntactic annotation design
SMART_READER_LITE
LIVE PREVIEW

Learnability-based Syntactic Annotation Design Roy Schwartz, Omri - - PowerPoint PPT Presentation

Aug 18, 2022 124 likes •729 views

Learnability-based Syntactic Annotation Design Roy Schwartz, Omri Abend and Ari Rappoport The Hebrew University In proceedings of COLING 2012 Overview In many cases, there is more than one plausible way to annotate syntactic structures

slide-1
SLIDE 1

Learnability-based Syntactic Annotation Design

Roy Schwartz, Omri Abend and Ari Rappoport

The Hebrew University

In proceedings of COLING 2012

slide-2
SLIDE 2

Overview

  • In many cases, there is more than one plausible way to

annotate syntactic structures

– A single annotation must be selected

  • We propose learnability as a selection criterion

– A principled learnability-based methodology – Use parsers for annotation design

  • Selecting the more learnable annotation may result in up to

35.3% error reduction in parsing performance

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 2

slide-3
SLIDE 3

Different Syntactic Formalisms

Example

  • I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 3

slide-4
SLIDE 4

Different Syntactic Formalisms

Example

  • I want to eat the apple

I want to eat the apple PRP VBP TO VB DT NN

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 3

slide-5
SLIDE 5

Different Syntactic Formalisms

Example

  • I want to eat the apple

I want to eat the apple PRP VBP TO VB DT NN (S (NP I) (VP (VP want ) (VP to (VP eat (NP the apple) ) ) ) ) I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 3

slide-6
SLIDE 6

Different Syntactic Formalisms

Example

  • I want to eat the apple

I want to eat the apple PRP VBP TO VB DT NN (S (NP I) (VP (VP want ) (VP to (VP eat (NP the apple) ) ) ) ) I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 3

slide-7
SLIDE 7

Disagreement within the same Formalism

I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-8
SLIDE 8

Disagreement within the same Formalism

I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-9
SLIDE 9

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-10
SLIDE 10

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-11
SLIDE 11

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-12
SLIDE 12

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 4

slide-13
SLIDE 13

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Varying Syntactic Structures (VSS)

4

slide-14
SLIDE 14

Disagreement within the same Formalism

I want to eat the apple I want to eat the apple I want to eat the apple I want to eat the apple

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Varying Syntactic Structures (VSS) Annotation scheme

4

slide-15
SLIDE 15

Varying Syntactic Structures (VSS)

  • VSSs are very frequent

– More than 40% of the tokens in PTB participate in at least one VSS*

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

* Schwartz et al., ACL 2011

5

slide-16
SLIDE 16

Varying Syntactic Structures (VSS)

  • VSSs are very frequent

– More than 40% of the tokens in PTB participate in at least one VSS*

  • Evaluation Problems

– Different parsers train and evaluate against different annotation schemes

  • Selecting one alternative over the other in a VSS can affect the

performance of a specific parser**

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

* Schwartz et al., ACL 2011 ** Nilsson et al., ACL 2006

5

slide-17
SLIDE 17

Performance Differences

  • Learning the correct annotation for a VSS is easy

– Usually the direction of a single edge

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

theDT appleNN

6

slide-18
SLIDE 18

Performance Differences

  • Learning the correct annotation for a VSS is easy

– Usually the direction of a single edge

  • An annotation scheme is learned as a whole

– Not each VSS alone

  • There are second order effects

– The way in which the VSS attaches to the rest of the sentence – These can lead to performance differences

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

theDT appleNN

6

slide-19
SLIDE 19

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-20
SLIDE 20

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-21
SLIDE 21

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-22
SLIDE 22

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-23
SLIDE 23

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-24
SLIDE 24

MST parser

(McDonald et al. 2005)

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

7

slide-25
SLIDE 25

MST parser

(McDonald et al. 2005)

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

Predictable?

7

slide-26
SLIDE 26

MST parser

(McDonald et al. 2005)

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

Parser specific? Predictable?

7

slide-27
SLIDE 27

MST parser

(McDonald et al. 2005)

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Example

Performance Differences

Parser specific? Predictable? Magnitude?

7

slide-28
SLIDE 28

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Varying Syntactic Structures (VSS)

8

slide-29
SLIDE 29

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Varying Syntactic Structures (VSS)

Selecting one Annotation is Required

8

slide-30
SLIDE 30

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Varying Syntactic Structures (VSS)

Selecting one Annotation is Required Selection Can Affect Parsing Performance

8

slide-31
SLIDE 31

Our Solution: Learnability

  • A straightforward selection criterion

– Namely, how easy it is to learn a given annotation scheme using statistical tools

  • Learnability is justified practically

– Training on more learnable schemes results in more accurate parsers

  • This criterion is only applied on linguistically plausible

annotations

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 9

slide-32
SLIDE 32

Learnability

  • Learnability is widely used theoretically

– Learnability using distributional methods has been used as an important consideration in designing the phrase structure formalism* – It is also used recurrently in cognitive science**

* Chomsky 2006 ** Chater and Vitányi 2003, Perfors et al. 2011

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 10

slide-33
SLIDE 33

Learnability-based Methodology 1

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 11

slide-34
SLIDE 34

Learnability-based Methodology 1

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Standard Parsing Evaluation

parser1 parser2 parser3 result1 result2 result3 annotation scheme corpus

11

slide-35
SLIDE 35

Learnability-based Methodology 1

parser*

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Standard Parsing Evaluation

parser1 parser2 parser3 result1 result2 result3 annotation scheme corpus

11

slide-36
SLIDE 36

Learnability-based Methodology 1

parser*

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Standard Parsing Evaluation Our Approach

parser1 parser2 parser3 result1 result2 result3 annotation scheme corpus parser result1 result2 result3 corpus scheme1 scheme2 scheme3

11

slide-37
SLIDE 37

Learnability-based Methodology 1

parser*

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Standard Parsing Evaluation Our Approach

scheme*

parser1 parser2 parser3 result1 result2 result3 annotation scheme corpus parser result1 result2 result3 corpus scheme1 scheme2 scheme3

11

slide-38
SLIDE 38

Learnability-based Methodology 2

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3

12

slide-39
SLIDE 39

Learnability-based Methodology 2

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

scheme1

*

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3

12

slide-40
SLIDE 40

Learnability-based Methodology 2

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

scheme1

*

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3 parser2 result1 result2 result3 corpus scheme1 scheme2 scheme3

12

slide-41
SLIDE 41

Learnability-based Methodology 2

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

scheme1

*

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3 parser2 result1 result2 result3 corpus scheme1 scheme2 scheme3

scheme2

*

12

slide-42
SLIDE 42

Learnability-based Methodology 2

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

scheme1

*

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3 parser2 result1 result2 result3 corpus scheme1 scheme2 scheme3

scheme2

*

parser3 result1 result2 result3 corpus scheme1 scheme2 scheme3

scheme3

*

12

slide-43
SLIDE 43

Learnability-based Methodology 2

scheme*

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

scheme1

*

parser1 result1 result2 result3 corpus scheme1 scheme2 scheme3 parser2 result1 result2 result3 corpus scheme1 scheme2 scheme3

scheme2

*

parser3 result1 result2 result3 corpus scheme1 scheme2 scheme3

scheme3

*

12

slide-44
SLIDE 44

Experimental Setup 1

  • 6 VSSs, 26 = 64 annotation schemes

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

aboutIN everyoneNN JohnNNP andCC MaryNNP toTO eatVB JohnNNP DoeNNP canMD comeVB theDT appleNN

(a) Prepositional Phrases (b) Noun Phrases (c) Coordinations (f) Infinitive Verbs (e) Noun Sequences (d) Verb Groups

13

slide-45
SLIDE 45

Experimental Setup 2

  • 5 parsers of different types

– Graph based parsers

  • MST parser (McDonald et al. 2005)
  • DMV parser (Klein and Manning 2004)

– Transition based parsers

  • Su parser (Nivre 2009)
  • Clear parser (Choi and Nicolov 2009)

– Other

  • NonDir Parser (Goldberg and Elhadad 2010)

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 14

slide-46
SLIDE 46

Results

  • In 3/6 structures, a unanimously more learnable annotation

was found:

– Prepositions (and not NPs) as heads of PPs – Nouns (and not their determiners) as heads of NPs – Conjuncts as heads of coordination structures

aboutIN everyoneNN JohnNNP andCC MaryNNP

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

theDT appleNN

15

slide-47
SLIDE 47

Results

  • In 3/6 structures, a unanimously more learnable annotation

was found:

– Prepositions (and not NPs) as heads of PPs – Nouns (and not their determiners) as heads of NPs – Conjuncts as heads of coordination structures

aboutIN everyoneNN JohnNNP andCC MaryNNP

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

theDT appleNN

15

slide-48
SLIDE 48

Magnitude

  • Gains are substantial

– Up to 19.8% error reduction for a single structure

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 16

slide-49
SLIDE 49

Magnitude

  • Gains are substantial

– Up to 19.8% error reduction for a single structure

  • Gains are additive

– Selecting the more learnable annotation in all 3 VSSs results in an even more learnable scheme – Up to 35.3% error reduction by selecting the most vs. least learnable annotation

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 16

slide-50
SLIDE 50

And the Winner is…

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

Available @ http://www.cs.huji.ac.il/~roys02/

17

slide-51
SLIDE 51

Additional Experiments

  • Two learnability measures
  • High agreement between different parsers
  • Predictability – a simple information-theoretic measure that

yields similar results to learnability

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 18

slide-52
SLIDE 52

Additional Experiments

  • Two learnability measures
  • High agreement between different parsers
  • Predictability – a simple information-theoretic measure that

yields similar results to learnability

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

slide-53
SLIDE 53

Some more Ideas

  • Apply our methodology to different tasks

– POS tagging , Phrase Structure parsing, etc.

  • Apply our methodology to different languages

– Ballesteros and Nivre, CL 2013

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 19

slide-54
SLIDE 54

What does it all Mean?

  • Powerful results

– Some annotations are clearly more learnable than others

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 20

slide-55
SLIDE 55

What does it all Mean?

  • Powerful results

– Some annotations are clearly more learnable than others

  • Linguistic implications?

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 20

slide-56
SLIDE 56

What does it all Mean?

  • Powerful results

– Some annotations are clearly more learnable than others

  • Linguistic implications?
  • Cognitive implications?

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 20

slide-57
SLIDE 57

Summary

  • Varying Syntactic Structures (VSS)

– Sometimes you have to choose

  • Learnability as a selection criterion

– A principled learnability-based methodology – Use parsers as research tools

  • Selecting one alternative has a substantial and predictable

effect on parsing performance

– Parser independent – up to 35.3% error reduction

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 21

slide-58
SLIDE 58

Summary

  • Varying Syntactic Structures (VSS)

– Sometimes you have to choose

  • Learnability as a selection criterion

– A principled learnability-based methodology – Use parsers as research tools

  • Selecting one alternative has a substantial and predictable

effect on parsing performance

– Parser independent – up to 35.3% error reduction

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 21

slide-59
SLIDE 59
  • Richard Johansson
  • Roi Reichart
  • Shai Shalev-Shwartz
  • Valentin I. Spitkovsky

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012 22

slide-60
SLIDE 60

Learnability-based Syntactic Annotation Design @ Schwartz, Abend and Rappoport, COLING 2012

roys02@cs.huji.ac.il http://www.cs.huji.ac.il/~roys02/

23