Parsing to Stanford Dependencies: Trade-offs between speed and - - PowerPoint PPT Presentation

▶
parsing to stanford dependencies trade offs between speed
SMART_READER_LITE
LIVE PREVIEW

Parsing to Stanford Dependencies: Trade-offs between speed and - - PowerPoint PPT Presentation

Feb 23, 2023 210 likes •889 views

Parsing to Stanford Dependencies: Trade-offs between speed and accuracy Daniel Cer, Marie-Catherine de Marneffe Daniel Jurafsky, Christopher D. Manning Stanford Dependencies Overview About the Representation Widely used

slide-1
SLIDE 1

Parsing to Stanford Dependencies: Trade-offs between speed and accuracy

Daniel Cer, Marie-Catherine de Marneffe Daniel Jurafsky, Christopher D. Manning

slide-2
SLIDE 2

Stanford Dependencies

Overview

About the Representation Widely used Semantically-oriented Slow to extract Extraction Bottleneck Stanford lexicalized phrase structure parser Are There Faster and Better Approaches? Dependency parsing algorithms Alternate phrase structure parsers

slide-3
SLIDE 3

Stanford Dependencies

Organization

Brief Review of Stanford Dependencies Properties Extraction pipeline Experiments Comparing Parsing Approaches Search Space Pruning with Charniak-Johnson Dependency Phrase structure MaltParser Berkeley MSTParser Bikel Charniak

slide-4
SLIDE 4

Stanford Dependencies

What We’ll Show

Performing dependency parsing using a phrase structure parser followed by rule based extraction is more accurate and, in some cases, faster then using statistical dependency parsing algorithms.

slide-5
SLIDE 5

Stanford Dependencies

What We’ll Show

Performing dependency parsing using a phrase structure parser followed by rule based extraction is more accurate and, in some cases, faster then using statistical dependency parsing algorithms. For English using the Stanford Dependency formalism

slide-6
SLIDE 6

Stanford Dependencies

What We’ll Show

Performing dependency parsing using a phrase structure parser followed by rule based extraction is more accurate and, in some cases, faster then using statistical dependency parsing algorithms. For English using the Stanford Dependency formalism However, we suspect the results maybe more general

slide-7
SLIDE 7

Stanford Dependencies

Semantically Oriented

Capture relationships between content words Syntactic Dependencies Stanford Dependencies

slide-8
SLIDE 8

Stanford Dependencies

Basic Dependencies

Start out by extracting syntactic heads Results in a projective dependency tree

slide-9
SLIDE 9

Stanford Dependencies

Basic Dependencies

Start out by extracting syntactic heads Results in a projective dependency tree

slide-10
SLIDE 10

Stanford Dependencies

Collapsed Dependencies

Bills on ports and immigration

slide-11
SLIDE 11

Stanford Dependencies

Collapsed Dependencies

Bills on ports and immigration

slide-12
SLIDE 12

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Sentence Standard Pipeline

slide-13
SLIDE 13

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Sentence Constituent Parse Tree Standard Pipeline

slide-14
SLIDE 14

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Sentence Constituent Parse Tree Projective Basic Dependencies Standard Pipeline

slide-15
SLIDE 15

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Sentence Constituent Parse Tree Projective Basic Dependencies Collapsed Dependencies Standard Pipeline

slide-16
SLIDE 16

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Sentence Constituent Parse Tree Projective Basic Dependencies Collapsed Dependencies Standard Pipeline

slide-17
SLIDE 17

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Dependency parser Sentence Sentence Constituent Parse Tree Projective Basic Dependencies Collapsed Dependencies Standard Pipeline Direct Pipeline

slide-18
SLIDE 18

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Dependency parser Sentence Sentence Constituent Parse Tree Projective Basic Dependencies Projective Basic Dependencies Collapsed Dependencies Standard Pipeline Direct Pipeline

slide-19
SLIDE 19

Stanford Dependencies

Obtaining the Dependencies

Phrase Structure Parser Dependency parser Sentence Sentence Constituent Parse Tree Projective Basic Dependencies Projective Basic Dependencies Collapsed Dependencies Collapsed Dependencies Standard Pipeline Direct Pipeline

slide-20
SLIDE 20

RESULTS BY PIPELINE TYPE

Experimental

slide-21
SLIDE 21

Results by Pipeline Type

Method

Train Penn Treebank Sections 2 through 21 Test Penn Treebank Section 22 Dependency parsers RelEx CMU Link grammar parser in Stanford compatibility mode Malt Parser MSTParser Algorithm Nivre Eager, Nivre, Covington Eisner Classifier LibLinear, LibSVM Factored MIRA

slide-22
SLIDE 22

Results by Pipeline Type

Method

Train Penn Treebank Sections 2 through 21 Test Penn Treebank Section 22 Phrase Structure Parsers Charniak Charniak Johnson Reranking Bikel Berkeley Stanford

slide-23
SLIDE 23

Results by Pipeline Type

Phrase Structure Parser Accuracy

Best: Charniak Johnson Reranking Parser

82 83 84 85 86 87 88 89 Charniak Johnson Berkeley Bikel Stanford

Labeled Attachment F1

slide-24
SLIDE 24

Results by Pipeline Type

Phrase Structure Parser Speed

0.5 1 1.5 2 2.5 3 Berkeley Stanford Charniak Johnson Bikel

Sentences/Second Parse times similar except for Bikel

slide-25
SLIDE 25

Results by Pipeline Type

Dependency Parser Accuracy

Labeled Attachment

74 75 76 77 78 79 80 81 Nivre Eager LibSVM MSTParser Eisner Nivre Eager LibLinear

Best: Nivre Eager LibSVM F1

slide-26
SLIDE 26

Results by Pipeline Type

Dependency Parser Speed

20 40 60 80 100 120 Nivre Eager LibLinear Nivre Eager LibSVM MSTParser Eisner

Sentences/Second Fastest: Nivre Eager LibLinear

slide-27
SLIDE 27

SPEED AND ACCURACY TRADE-OFFS

Comparison of

slide-28
SLIDE 28

Speed and Accuracy Trade-Offs

Worst vs. Best Accuracy Worst Phrase Structure Parser vs. Best Dependency Parser

slide-29
SLIDE 29

Speed and Accuracy Trade-Offs

Worst vs. Best Accuracy

76 78 80 82 84 86 Bikel Stanford Nivre Eager LibSVM MSTParser Eisner

Labeled Attachment Phrase Structure Dependency Worst phrase structure parser better than Best dependency parser F1

slide-30
SLIDE 30

Speed and Accuracy Trade-Offs

Worst vs. Best Accuracy

76 78 80 82 84 86 Bikel Stanford Nivre Eager LibSVM MSTParser Eisner

Labeled Attachment Phrase Structure Dependency Worst phrase structure parser better than Best dependency parser F1

+3

slide-31
SLIDE 31

Speed and Accuracy Trade-Offs

Best vs. Best Accuracy Best Phrase Structure Parser vs. Best Dependency Parser

slide-32
SLIDE 32

Speed and Accuracy Trade-Offs

Best vs. Best Accuracy

76 78 80 82 84 86 88 90 Charniak Johnson Berkeley Nivre Eager LibSVM MSTParser Eisner

Labeled Attachment Phrase Structure Dependency Eight point difference between Best phrase structure and Best dependency parser F1

+8

slide-33
SLIDE 33

Speed and Accuracy Trade-Offs

Worst vs. Best Speed Worst Dependency Parser vs. Best Phrase Structure Parser

slide-34
SLIDE 34

Speed and Accuracy Trade-Offs

Worst vs. Best Speed

Sentences/Second Dependency Phrase Structure Worst dependency parser better than Best phrase structure parser

1 2 3 4 5 6 7 8 9 Nivre Eager LibSVM MSTParser Eisner Berkeley Stanford

slide-35
SLIDE 35

Speed and Accuracy Trade-Offs

Worst vs. Best Speed

Sentences/Second Dependency Phrase Structure Worst dependency parser better than Best phrase structure parser

1 2 3 4 5 6 7 8 9 Nivre Eager LibSVM MSTParser Eisner Berkeley Stanford

+2

slide-36
SLIDE 36

Speed and Accuracy Trade-Offs

Best vs. Best Speed Best Dependency Parser vs. Best Phrase Structure Parser

slide-37
SLIDE 37

20 40 60 80 100 120 Nivre Eager LibLinear Nivre Eager LibSVM Berkeley Stanford

Speed and Accuracy Trade-Offs

Best vs. Best Speed

Sentences/Second Dependency Phrase Structure 103 sentences/second difference between Best dependency and Best phrase structure parser

slide-38
SLIDE 38

20 40 60 80 100 120 Nivre Eager LibLinear Nivre Eager LibSVM Berkeley Stanford

Speed and Accuracy Trade-Offs

Best vs. Best Speed

Sentences/Second Dependency Phrase Structure 103 sentences/second difference between Best dependency and Best phrase structure parser

+103

slide-39
SLIDE 39

Speed and Accuracy Trade-Offs

Out-of-the-Box Summary

Accuracy Use Phrase Structure Parsers Best choice: Charniak Johnson Reranking Speed Use Dependency Parsers Best Choice: Nivre Eager* with LibLinear

* Actually, any parser in the MaltParser package will do.

slide-40
SLIDE 40

CHARNIAK JOHNSON SEARCH SPACE PRUNING

Making Use Of

slide-41
SLIDE 41

Charniak Johnson Search Space Pruning

Example Search

Best First Search

slide-42
SLIDE 42

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-43
SLIDE 43

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-44
SLIDE 44

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-45
SLIDE 45

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-46
SLIDE 46

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-47
SLIDE 47

Best First Search Charniak Johnson Search Space Pruning

Example Search

slide-48
SLIDE 48

First Complete Parse! Charniak Johnson Search Space Pruning

Example Search

slide-49
SLIDE 49

After the First Complete Parse Count edges expanded so far Then Expand Edge count x Pruning constant more edges

Pruning constant = T parameter /10

Charniak Johnson Search Space Pruning

Example Search

slide-50
SLIDE 50

Expand edge count x Constant more edges Charniak Johnson Search Space Pruning

Example Search

slide-51
SLIDE 51

Expand edge count x Constant more edges Charniak Johnson Search Space Pruning

Example Search

slide-52
SLIDE 52

Expand edge count x Constant more edges Charniak Johnson Search Space Pruning

Example Search

slide-53
SLIDE 53

Expand edge count x Constant more edges Charniak Johnson Search Space Pruning

Example Search

slide-54
SLIDE 54

Charniak Johnson Search Space Pruning

Pruning Effects on Accuracy

Minimal loss of accuracy for moderate pruning Labeled Attachment

70 75 80 85 90 T210 T50 T10 Default T210

F1

slide-55
SLIDE 55

Charniak Johnson Search Space Pruning

Pruning Effects on Accuracy

Minimal loss of accuracy for moderate pruning Labeled Attachment

70 75 80 85 90 T210 T50 T10 Default T210

F1

  • 2
slide-56
SLIDE 56

2 4 6 8 10 12 14 T10 T50 T210

Charniak Johnson Search Space Pruning

Pruning Effects on Speed

3x speed gains with moderate pruning

Default T210

Sentences/Second

slide-57
SLIDE 57

2 4 6 8 10 12 14 T10 T50 T210

Charniak Johnson Search Space Pruning

Pruning Effects on Speed

3x speed gains with moderate pruning

Default T210

Sentences/Second

+5

slide-58
SLIDE 58

Speed and Accuracy Trade-Offs

Best vs. Best Accuracy Best Phrase Structure Parser vs. Best Dependency Parser

with Pruning

slide-59
SLIDE 59

Pruning Speed and Accuracy Trade-Offs

Best vs. Best Accuracy

76 78 80 82 84 86 88 90 Charniak Johnson Berkeley Nivre Eager LibSVM MSTParser Eisner

Labeled Attachment Phrase Structure Dependency Remember this? Let’s insert Charniak Johnson T50 F1

+8

slide-60
SLIDE 60

72 74 76 78 80 82 84 86 88 90

Charniak Johnson Default Berkeley Charniak Johnson T50 Nivre Eager LibSVM MSTParser Eisner

Pruning Speed and Accuracy Trade-Offs

Best vs. Best Accuracy

Labeled Attachment Phrase Structure Dependency About as good as Berkeley

Charniak Johnson T50

F1

  • 1
slide-61
SLIDE 61

72 74 76 78 80 82 84 86 88 90

Charniak Johnson Default Berkeley Charniak Johnson T50 Nivre Eager LibSVM MSTParser Eisner

Pruning Speed and Accuracy Trade-Offs

Best vs. Best Accuracy

Labeled Attachment Phrase Structure Dependency Much better than best dependency parser

Charniak Johnson T50

F1

+6

slide-62
SLIDE 62

Speed and Accuracy Trade-Offs

Best vs. Best Speed Best Dependency Parser vs. Best Phrase Structure Parser

with Pruning

slide-63
SLIDE 63

20 40 60 80 100 120 Nivre Eager LibLinear Nivre Eager LibSVM Berkeley Stanford

Speed and Accuracy Trade-Offs

Best vs. Best Speed

Sentences/Second Dependency Phrase Structure Remember this? Let’s insert Charniak Johnson T50

+103

slide-64
SLIDE 64

20 40 60 80 100 120 Nivre Eager LibLinear Nivre Eager LibSVM Berkeley Stanford

Speed and Accuracy Trade-Offs

Best vs. Best Speed

Sentences/Second Dependency Phrase Structure Let’s insert Charniak Johnson T50 …Excluding the 100+ sentences/second parser

+103

slide-65
SLIDE 65

1 2 3 4 5 6 7 8 9 Nivre Eager LibSVM Charniak Johnson T50 Berkeley Stanford

Speed and Accuracy Trade-Offs

Best vs. Best Speed

Sentences/Second Dependency Phrase Structure Nearly as fast as Nivre Eager using LibSVM

Charniak Johnson T50

∆ < 1

slide-66
SLIDE 66

Speed and Accuracy Trade-Offs

Pruning Summary Charniak Johnson T50

Accuracy similar to best phrase structure parsers Speed similar to most dependency parsers

slide-67
SLIDE 67

Parsing to Stanford Dependencies

Conclusion

For Accuracy Charniak Johnson 89.1 Labeled Attachment F1 For Speed Nivre Eager with LibLinear +100 Sentences/second For A Good Balance Charniak Johnson T50 87.6 Labeled Attachment F1 Speed similar to most dependency parsers Thanks to Joakim Nivre Mihai Surdeanu