A comparison of selectional preference models for automatic verb - - PowerPoint PPT Presentation

Introduction Models Results References

A comparison of selectional preference models for automatic verb classification

Will Roberts and Markus Egg

Institut für Anglistik und Amerikanistik Humboldt Universität zu Berlin

Sunday, 26 October, 2014

Will Roberts Selprefs for verb classification 1 / 20

Introduction Models Results References

Outline

1

Introduction

2

Models

3

Results

Will Roberts Selprefs for verb classification 2 / 20

Introduction Models Results References

Selectional preferences

Predicates can select for their arguments: ? My aunt is a bachelor. (McCawley, 1968) We model verbs empirically: I eat meat bread fruit . . . newspaper Evaluate on an automatic verb classification task Baseline model clusters verbs based on subcategorisation

Will Roberts Selprefs for verb classification 3 / 20

Introduction Models Results References

Selectional preferences

Example Wir benutzen Ihre Umfragedaten nicht für eigene Zwecke. We use your survey data not for own purposes. We will not use your survey responses for private purposes. We will want to record that this instance of use has: Subject wir, we (pronoun, ignored) Direct object Umfragedatum, survey datum PP (für, for) Zweck, purpose We also include indirect objects (datives) A selectional preference model will map noun forms onto concept labels

Will Roberts Selprefs for verb classification 4 / 20

Introduction Models Results References

Hypothesis

verb clustering score

• nly subcat:
• ne concept

containing all nouns lexical preferences:

• ne concept

per noun effective SP model

• ptimal

concept granularity ineffective SP model

Will Roberts Selprefs for verb classification 5 / 20

Introduction Models Results References

Subcategorisation

Example Wir benutzen Ihre Umfragedaten nicht für eigene Zwecke. We use your survey data not for own purposes. We will not use your survey responses for private purposes. The combination of syntactic argument types is assigned a subcategorisation frame (SCF) code: benutzen ⇒ nap:für.Acc A verb’s distribution over SCF codes is its subcategorisation preference

Will Roberts Selprefs for verb classification 6 / 20

Introduction Models Results References

Pipeline

SdeWaC corpus mate-tools dependency parser SCF tagger test set hierarchical clustering (Ward’s) verb clusters gold standard selectional preferences model

Test set has 3 million verb instances Gold standard: 168 verbs in 43 classes

Will Roberts Selprefs for verb classification 7 / 20

Introduction Models Results References

Verb clustering

verb: p = 1 p = 0 6 12 3 7 2 12 11 scf1 scf2 scf3 scf4 . . . scf671 scf672 scf673 corpus counts discrete probability distribution = subcat prefs

Verb dissimilarity is computed with the Jensen-Shannon divergence

Will Roberts Selprefs for verb classification 8 / 20

Introduction Models Results References

Lexical preferences (LP)

Example Wir benutzen Ihre Umfragedaten nicht für eigene Zwecke. We use your survey data not for own purposes. We will not use your survey responses for private purposes. benutzen ⇒ nap:für.Acc*dobj-Umfragedatum*prep-Zweck To control data sparsity, we employ a parameter N: number of nouns included in the lexical preferences model

Nouns with rank > N are ignored (as if unseen)

Will Roberts Selprefs for verb classification 9 / 20

Introduction Models Results References

Sun/Korhonen

noun: p = 1 p = 0 3 8 4 10 4 7 11 verb1, subj verb1, obj verb2, subj verb2, prep . . . verbN, subj verbN, obj verbN, dative corpus counts discrete probability distribution

Partition N nouns into M classes (equivalence relation)

Will Roberts Selprefs for verb classification 10 / 20

Introduction Models Results References

Word space model (WSM)

Built on lemmatised SdeWaC Features are the 50,000 most common words (minus stop words) Sentences as windows Feature weighting: t-test scheme Context selection zeroes out infrequent features in the model Use cosine similarity and spectral clustering to partition N nouns into M classes

Will Roberts Selprefs for verb classification 11 / 20

Introduction Models Results References

GermaNet

Granularity is controlled using depth, d Nouns can belong to more than one concept: soft clustering

GNROOT_n_1 Stelle_n_1 Jahr_n_1 Menge_n_2 Entitaet_n_2 kognitives_Objekt_n_1 Zeitabschnitt_n_1 zyklische_Zeiteinheit_n_1 Jahr_n_2 0.5 0.375 0.125 target set, depth ≤ 1 Will Roberts Selprefs for verb classification 12 / 20

Introduction Models Results References

Latent Dirichlet Allocation (LDA)

Built with the same data used by the Sun/Korhonen model Each verb, grammatical relation pair has a distribution Φ over concepts Each concept z has a distribution Θ

• ver the N nouns

Number of concepts M is 50 or 100 α Φ z W β Θ M n G

Will Roberts Selprefs for verb classification 13 / 20

Introduction Models Results References

Results

SP model Parameters Granularity F-score SUN 10K nouns 1,000 noun classes 39.76 LDA (hard) 10K nouns 50 topics 39.09 LP 5K nouns 38.02 WSM 10K nouns 500 noun classes 36.92 LDA (soft) 10K nouns 50 topics 35.91 GermaNet depth = 5 8,196 synsets 34.41 Baseline 33.47

Will Roberts Selprefs for verb classification 14 / 20

Introduction Models Results References

Sparsity effects in LP

100 101 102 103 104 105 N 31 32 33 34 35 36 37 38 PairF 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Coverage

Will Roberts Selprefs for verb classification 15 / 20

Introduction Models Results References

Qualitative differences in noun partitions

SUN WSM F-score 39.76 F-score 36.92 syntagmatic information paradigmatic information synonym/co-hyponym structure thematic structure class size variance 37 class size variance 2800 semantically consistent large classes inconsistent

Will Roberts Selprefs for verb classification 16 / 20

Introduction Models Results References

Test set size

105 106 107 108 Number of verb instances 15 20 25 30 35 40 45 PairF Baseline lp wsm sun lda-hard

Will Roberts Selprefs for verb classification 17 / 20

Introduction Models Results References

Conclusions

1 Selectional preferences help automatic verb classification 2 Optimal concept granularity is relatively fine

Lexical preferences works very well if it is properly tuned Classification of proper names is useful: given names, corporations, medications, etc.

3 Syntagmatic information works better than paradigmatic Will Roberts Selprefs for verb classification 18 / 20

Introduction Models Results References

Summary

Selectional preference models have been compared before

Almost always under a plausibility or pseudoword paradigm!

We are interested in semantic verb clustering We evaluate several selectional preference models, comparing them using a manually constructed semantic verb classification We show that modelling selectional preferences is beneficial for verb clustering, no matter which selectional preference model we choose Other findings:

Capturing syntagmatic relations seems to work better than paradigmatic A simple lexical preferences model performs very well; data sparsity does not seem to be more of a problem for this model than for others

Will Roberts Selprefs for verb classification 19 / 20

Introduction Models Results References

References

James D. McCawley. The role of semantics in a grammar. In Emmon Bach and Robert Harms, editors, Universals in Linguistic Theory, pages 124–169. Holt, Rinehart and Winston, 1968.

Will Roberts Selprefs for verb classification 20 / 20