Leaving no token behind: comprehensive (and delicious) annotation - - PowerPoint PPT Presentation

SLIDE 1

1

Leaving no token behind: comprehensive (and delicious) annotation of MWEs and supersenses

nert

Nathan Schneider Georgetown University

LAW-MWE-CxG • 25 August 2018 • Santa Fe, NM

SLIDE 2

• Goal: corpora in a language annotated with

some form of lexical semantics 
 (for NLP , corpus linguistics)

• How to achieve this with good coverage,

quality, scalability?

2

SLIDE 3

Traditional Strategy

Start with a general lexicon like WordNet, 
 apply it to corpus.

LIMITATIONS: coverage (esp. for MWEs), 


granularity, language-specificity, cost

3

SLIDE 4

Lexicon-Free Lexical Semantics

Annotate general categories/criteria at the token level, and identify types as you go. Some options:

• 1. Focus on a syntactic domain of interest, and annotate

at the token level with general criteria [most work on

MWEs, e.g. PARSEME Shared Tasks 1.0, 1.1 focusing on VMWE subclasses; Savary et al. 2017, Ramisch et al. 2018]. 


Next session!

• 2. Focus on a semantic domain of interest, and annotate

at the token level while populating a lexicon or constructicon of types [Dunietz et al. 2015, 2017]. 
 Lori’s keynote tomorrow!

• 3. Comprehensively annotate at the token level

with coarse-grained categories. This talk!

4

SLIDE 5

Claim

Comprehensive annotation of MWEs and supersenses (semantic classes), without starting from a lexicon, is

• structurally simple (labeled segments)
• intuitive to annotate (reasonable agreement)
• robust to cover long tail of types &

constructions

• robust to gappy expressions
• scalable/cost-effective
• conducive to NLP

5

SLIDE 6

Roadmap

The story of the STREUSLE corpus:

• MWEs

✦ comprehensive annotation ✦ data exploration: notable MWEs/constructions

• Supersenses

✦ nouns & verbs ✦ prepositions

6

SLIDE 7

7

!

Supersense Tagged Repository of English with a Unified Semantics for Lexical Expressions

tiny.cc/streusle

SLIDE 8

Comprehensive MWE annotation in STREUSLE

[Schneider et al., LREC 2014]

8

SLIDE 9

Scene: An AMR design meeting in June 2012

9

We should annotate all kinds of MWEs in AMR! LOL good luck getting annotators to agree on what’s an MWE CHALLENGE ACCEPTED

SLIDE 10

MWE Definition

Multiword expression (MWE): 2 or more

• rthographic words that are tightly associated
• Strong MWEs: idiomatic = not fully predictable

in form and/or function

• non- or semi-compositional: 


ice cream, daddy longlegs, pay attention

• unusual morphosyntax: Me/*Him neither; 


by and large; plural of daddy longlegs?

• Weak MWEs: statistically collocated or formulaic
• p(heavy rain) > p(strong rain); 


highly recommended; no amount of … can …

10

SLIDE 11

MWE Challenges

• Not superficially apparent in text
• Syntactic variability
• Number/frequency
• Too many expressions to list all of them
• Individually rare, but frequent in aggregate
• Diversity
• Many different construction types
• Semantically unrestricted
• Can be gappy

11

SLIDE 12

12

Noam Chomsky daddy longlegs, hot dog dry out depend on, come across pay attention (to) put up with, give in (to) under the weather cut and dry in spite of pick up where __ left off easy as pie You’re welcome. To each his own. The structure of this paper is as follows. pay dry the clothes

• ut

close attention (to) they pick up where left off __ no attention was paid (to)

SLIDE 13

Some syntactically-focused English MWE datasets

• NNCs [Reddy et al. 2011]
• VNCs [Venkatapathy & Joshi 2005, Cook et al. 2008]
• LVCs [Fazly et al. 2007, Hwang al. 2010, Tu & Roth 2011]
• VPCs [Bannard et al. 2003, McCarthy et al. 2003, Baldwin 2005]
• VPCs + PVs [Tu & Roth 2012]
• PARSME VMWEs (LVC, VPC, VID, …) 


[Savary et al. 2017, Ramisch et al. 2018]

• Functional MWEs (complex prepositions, adverbs, …) 


[Shigeto et al. 2013]

13

SLIDE 14

English corpora with several kinds of MWEs

• SemCor [Miller et al. 1993]
• Lexical annotation with WordNet synsets
• NEs, compound nominals, some phrasal

verbs

• Prague CEDT [Hajič et al. 2012]
• NEs, light verb constructions, phrasal

idioms, multiword tlemmas

• Wiki50 [Vincze et al. 2011]
• NEs, compound nominals, LVCs, VPCs,

phrasal idioms

14

SLIDE 15

Comprehensive Approach

• Radical new approach
• 1. Teach annotators the concept of MWE (with

examples of many kinds)

• 2. Give them sentences
• 3. Ask for all the MWEs

15

[Schneider et al., LREC 2014]

SLIDE 16

16

Noam_Chomsky refused to give_in_to the vicious daddy_longlegs .

SLIDE 17

17

Noam_Chomsky refused to give_in_to the vicious daddy_longlegs .

SLIDE 18

18

Noam_Chomsky refused to give_in_to the vicious daddy_longlegs .

SLIDE 19

19

Noam_Chomsky refused to give_in_to the vicious daddy_longlegs .

SLIDE 20

20

Alan_Black refused to give_in_to the vicious daddy_longlegs .

Lexical segmentation

Noam_Chomsky refused to give_in_to the vicious daddy_longlegs .

SLIDE 21

Real Example, Gappy MWE

My wife had taken_ her '07_Ford_Fusion _in for a routine oil_change .

21

SLIDE 22

22

SLIDE 23

23

SLIDE 24

The corpus

• The entire Reviews subsection of the

English Web Treebank (Bies et al. 2012), fully annotated for MWEs

• 723 reviews
• 3,800 sentences
• 55,000 words
• found 3,500 MWE instances [original version]
• Every sentence: negotiated consensus

between at least 2 annotators

• IAA between pairs: ~77%

24

SLIDE 25

Remarks

• More on resulting guidelines shortly
• Annotators were not shown syntax
• But the sentences are treebanked, part of

English-EWT corpus which has gold Universal Dependencies (included in STREUSLE release)

• Recently added PARSEME VMWE subtypes

(single annotator)

• HAMSTER [Chen et al. 2017]

25

SLIDE 26

26

!

Supersense Tagged Repository of English with a Unified Semantics for Lexical Expressions

tiny.cc/streusle

✦ 55k words of English web reviews ✴ 3,000 strong MWE mentions

• 900 VMWEs with

PARSEME subtypes ✴ 700 weak MWE mentions

SLIDE 27

27

What a joy to stroll off historic Canyon_Road in Santa_Fe into a gallery with a gorgeous diversity of art

SLIDE 28

28

I googled restaurants in the area and Fuji_Sushi came_up and reviews were great so I made_ a carry_out _order

SLIDE 29

Kinds of MWEs & other notable constructions in STREUSLE

29

SLIDE 30

30

white-nosed coati

LONG TAIL!

SLIDE 31

31

SLIDE 32

32

POS MWEs pattern contig. gappy most frequent types (lowercased lemmas) and their counts

N_N

331 1 customer service: 31

• il change: 9

wait staff: 5 garage door: 4

ˆ_ˆ

325 1 santa fe: 4

• dr. shady: 4

V_P

217 44 work with: 27 deal with: 16 look for: 12 have to: 12 ask for: 8

V_T

149 42 pick up: 15 check out: 10 show up: 9 end up: 6 give up: 5

V_N

31 107 take time: 7 give chance: 5 waste time: 5 have experience: 5

A_N

133 3 front desk: 6 top notch: 6 last minute: 5

V_R

103 30 come in: 12 come out: 8 take in: 7 stop in: 6 call back: 5

D_N

83 1 a lot: 30 a bit: 13 a couple: 9

P_N

67 8

• n time: 10

in town: 9 in fact: 7

R_R

72 1 at least: 10 at best: 7 as well: 6

• f course: 5

at all: 5

V_D_N

46 21 take the time: 11 do a job: 8

V~N

7 56 do job: 9 waste time: 4

ˆ_ˆ_ˆ

63 home delivery service: 3 lake forest tots: 3

R~V

49 highly recommend: 43 well spend: 1 pleasantly surprise: 1

P_D_N

33 6

• ver the phone: 4
• n the side: 3

at this point: 2

• n a budget: 2

A_P

39 pleased with: 7 happy with: 6 interested in: 5

P_P

39

• ut of: 10

due to: 9 because of: 7

V_O

38 thank you: 26 get it: 2 trust me: 2

V_V

8 30 get do: 8 let know: 5 have do: 4

N~N

34 1 channel guide: 2 drug seeker: 2 room key: 1 bus route: 1

A~N

31 hidden gem: 3 great job: 2 physical address: 2 many thanks: 2 great guy: 1

V_N_P

16 15 take care of: 14 have problem with: 5

N_V

18 10 mind blow: 2 test drive: 2 home make: 2

ˆ_$

28 bj s: 2 fraiser ’s: 2 ham s: 2 alan ’s: 2 max ’s: 2

D_A

28 a few: 13 a little: 11

R_P

25 1 all over: 3 even though: 3 instead of: 2 even if: 2

V_A

19 6 make sure: 7 get busy: 3 get healthy: 2 play dumb: 1

V_P_N

14 6 go to school: 2 put at ease: 2 be in hands: 2 keep in mind: 1

SLIDE 33

Names, Dates, Values

33

Canyon_Road Dr._Lori_Levin Harry_,_Prince_of_Wales Santa_Fe~,~NM Fourth_of_July

• Jan. 1 , 1980

north_-_northeast north_east macOS_10.13.6 2002_Toyota_Camry 10 % A_+ 3 x the speed 3 x 4 = 12 #_1 5_star review 100 square_miles

SLIDE 34

Natural Kinds, Food Dishes

34

Indian_elephant my dog is a yellow_lab furcifer_pardalis brown dog ice_cream chicken_salad sandwich macaroni_and_cheese General_Tso_’s_chicken cheese~and~crackers spaghetti~with~meatballs turkey sandwich strawberry banana milkshake green_tea

SLIDE 35

Quantifiers, Determiners

35

Our shirts are the_same half_a million dollars half_a mile away half of a mile away a_lot of cats a_few cats several cats plenty of cats

SLIDE 36

Slogans

36

Just_Do_It

SLIDE 37

Constructions

Construction Grammar: Framework positing continuity between lexicon and grammar construction = conventionalized form/function pairing of any grammatical shape, level of abstractness constructicon = structured inventory of constructions characterizing knowledge of a language

37

LEXICAL GRAMMATICAL cats kick the bucket ice cream SVO the Xer, the Yer spill the beans

SLIDE 38

Constructions

Construction Grammar: Framework positing continuity between lexicon and grammar

• Many multiword expressions are partially productive,

thus partially lexical, partially syntactic

• the X-er the Y-er [Fillmore, Kay, O’Connor 1988]
• Some constructions 1 or 0 lexicalized elements, but

are nevertheless long-tail patterns that convey meaning

• We discovered many of these while looking for MWEs



 


38

SLIDE 39

The X-101 Construction

• FORM: X 101, where X is a concept or skill that can

be learned

• FUNCTION: name of the most introductory course on

the topic of X in an institution of higher learning (based on a naming convention common at U.S. universities)

• Idiomatic construction requires “101” even though

some universities count from 100

• STREUSLE does not annotate X+101 as an MWE

because only “101” is fixed, so misses this idiom.

39

These guys took Customer_Service 101 from a Neanderthal.

SLIDE 40

Possessives

40

Modell_’s is nearby

business named after person,

• fficially or unofficially referred

to in the possessive [Quirk and

Greenbaum, 1973, pp. 329–330]

Kroger_’s is nearby

[Blodgett & Schneider 2018]

SLIDE 41

Possessives

• possessed idioms [Bond et al. 2013, 2015]

where possessive pronoun agrees with subject
 


• other opaque possessive idioms



 
 


41

Johni is quick_on_ hisi _feet Ii tried_ myi _best youi are on_ youri _own I helped in her hour_of_need I helped in Mary ’s_hour_of_need

[Blodgett & Schneider 2018]

SLIDE 42

Time

42

that will take_ some _time youi should take_ youri _time he took_the_time to learn linguistics a long week/month/year/… she took_time_out_of her busy schedule to help you should take_ some _time_off to travel have/spend/save/waste~time 3 weeks/months/years/…_old

SLIDE 43

Verbal Domain (beyond classic VMWEs): go etc.

43

she suggested I go check_ it _out have_to

serial verb semi-auxiliary

• ught_to

be going_to

‘must’ ‘should’ ‘will’

I would_like this cookie to_go

polite ‘want’ ‘not to eat here’ Also challenging: light meanings of get 
 (passive, causative, inchoative)

something went_wrong

V+A idiom

wanted to go and test_ it _out for my_self

go and V

SLIDE 44

Prepositional

44

• utside as_long_as it ’s light out

hair as long as a horse ’s tail load the items one_by_one

complex P PP idiom V+P idiom (IAV) VPC

try not to pass_out pass_out the candy I came_across a nice restaurant

N+P idiom

in_front_of

A+P idiom

X is close_to Y the problem~with prepositions in_trouble at_least

SLIDE 45

• Actually several long tails
• MWE types for a particular syntactic pattern
• syntactic patterns underlying MWEs
• constructions that aren’t quite MWEs 


(<2 lexicalized elements)

• Comprehensive MWE annotation taught me

about minor syntactic patterns in English!

45

SLIDE 46

Supersenses in STREUSLE

46

[Schneider & Smith, NAACL 2015]

SLIDE 47

Adding Supersenses

• We then went through the whole corpus 2

more times to add noun and verb supersenses (classes from WordNet).

47

SLIDE 48

NATURAL OBJECT ARTIFACT LOCATION PERSON GROUP SUBSTANCE TIME RELATION QUANTITY FEELING MOTIVE COMMUNICATION COGNITION STATE ATTRIBUTE ACT EVENT PROCESS PHENOMENON SHAPE POSSESSION FOOD BODY PLANT ANIMAL OTHER BODY CHANGE COGNITION COMMUNICATION COMPETITION CONSUMPTION CONTACT CREATION EMOTION MOTION PERCEPTION POSSESSION SOCIAL STATIVE WEATHER

48

sewer

supersenses

noun verb

SLIDE 49

Supersenses

• A layer of semantic disambiguation
• That doesn’t require a sense lexicon
• General-purpose: transcends domains,

languages

• Related to NER

49

SLIDE 50

N/V Supersenses

• Corpus creation
• Can be inferred from WordNet annotations in

SemCor [Miller et al. 1993]

• …or annotated directly [Schneider et al. 2012: Arabic

Wikipedia; D. Hovy et al. 2014: English Twitter]

• Automatic tagging [Segond et al. 1997; Ciaramita &

Altun 2006; Paaß & Reichartz 2009; D. Hovy et al. 2014]

• also in Italian [Picca et al., 2008, 2009; Attardi et al.,

2010; Rossi et al., 2013], Chinese [Qiu et al., 2011],

Arabic [Schneider et al., 2013], Danish [Martínez

Alonso et al., 2015], Sanskrit [Hellwig 2017]

50

SLIDE 51

Adding Supersenses

• We went through the STREUSLE corpus 2

more times to add noun and verb supersenses.

• Single-word expressions + strong multiword

expressions

51

SLIDE 52

52

What a joy to stroll

• ff historic Canyon_Road

in Santa_Fe into a gallery with a gorgeous diversity

• f art

–

N.COGNITION V .MOTION N.LOCATION N.LOCATION N.GROUP N.COGNITION N.COGNITION

SLIDE 53

53

I googled restaurants in the area and Fuji_Sushi came_up and reviews were great so I made_ a carry_out _order

–

V .COMMUNICATION N.GROUP N.LOCATION N.GROUP V .COMMUNICATION N.COMMUNICATION V .COMMUNICATION N.POSSESSION

SLIDE 54

54

SLIDE 55

55

!

Supersense Tagged Repository of English with a Unified Semantics for Lexical Expressions

tiny.cc/streusle

✦ 55k words of English web reviews ✴ 3,000 strong MWE mentions ✴ 700 weak MWE mentions ✴ 9,000 noun mentions ✴ 8,000 verb mentions

SLIDE 56

MWE+N+V Tagging

• Joint sequence model of MWEs &

supersenses along the lines of Ciaramita & Altun’s (2006) supersense tagger

56

results on test set (oracle POS) MWE (F1) SST (F1) Tag Acc full model 62.7 70.7 82.5

[Schneider & Smith, NAACL 2015]

SLIDE 57

SemEval 2016 Task 10 SemEval 2016

Detection of Minimal Semantic Units and their Meanings 
 (DiMSUM)

Nathan Schneider Dirk Hovy Anders Johannsen Marine Carpuat

All data at: https://github.com/dimsum16/dimsum-data

[Schneider et al., SemEval 2016]

SLIDE 58

DiMSUM Data Servings

58

STREUSLE Trustpilot Ritter Lowlands Tweebank IWSLT 


(incl. NAIST-NTT)

90k words


5800 sentences

=

IAA: ≈75%–80%

[Schneider et al., SemEval 2016]

SLIDE 59

DiMSUM Results

• Scalability: Our test set annotator had ≈8 hours of

training; annotation speed of ≈90s/sentence

• IAA roughly 75%–80%.
• Difficult NLP task!

59

ICL-HD system Reviews Tweets TED Supersenses 58 56 60 MWEs 53 59 57 Combined 57 57 60

[Schneider et al., SemEval 2016]

SLIDE 60

Comprehensive preposition/possessive annotation in STREUSLE

[Schneider et al. ACL 2018]

SLIDE 61

“I study preposition semantics.”

61

SLIDE 62

62

SLIDE 63

With great frequency comes great polysemy.

63

leave for Paris

spatial: goal/ destination

go to Paris ate for hours temporal: duration ate over most of an hour a gift for mother

recipient

give the gift to mother go to the store for eggs

purpose

go to the store to buy eggs pay/search for the eggs

theme

spend money on the eggs

SLIDE 64

Tradition of Semantic Annotation/ Disambiguation of Prepositions

• Sense-based, e.g. The Preposition Project and

spinoffs [Litkowski and Hargraves, 2005, 2007; Litkowski

2014; Ye and Baldwin, 2007; Saint-Dizier 2006; Dahlmeier et al. 2009; Tratz and Hovy 2009; Hovy et al. 2010, 2011; Tratz and Hovy 2013]

• Class-based [Moldovan et al. 2004; Badulescu and

Moldovan 2009; O’Hara and Wiebe 2009; Srikumar and Roth 2011, 2013; Müller et al. 2012 for German]

• Our work is the first class-based approach

that is comprehensive w.r.t. tokens AND types

[Schneider et al. 2015, 2016, 2018; Hwang et al. 2017]

64

SLIDE 65

Semantic Network of Adposition and Case Supersenses (SNACS)

65

Circumstance Temporal Time StartTime EndTime Frequency Duration Interval Locus Source Goal Path Direction Extent Means Manner Explanation Purpose Participant Causer Agent Co-Agent Theme Co-Theme Topic Stimulus Experiencer Originator Recipient Cost Beneficiary Instrument Configuration Identity Species Gestalt Possessor Whole Characteristic Possession PartPortion Stuff Accompanier InsteadOf ComparisonRef RateUnit Quantity Approximator SocialRel OrgRole

[Schneider et al., ACL 2018]

(SNACS)

SLIDE 66

66

!

Supersense Tagged Repository of English with a Unified Semantics for Lexical Expressions

tiny.cc/streusle

✦ 55k words of English web reviews ✴ 3,000 strong MWE mentions ✴ 700 weak MWE mentions ✴ 9,000 noun mentions ✴ 8,000 verb mentions ✴ 4,000 prepositions ✴ 1,000 possessives

SLIDE 67

STREUSLE Examples

Three weeks ago, burglars tried to gain_entry 
 
 into the rear of my home. Mrs._Tolchin provided us with excellent service and 
 
 came with a_great_deal of knowledge and professionalism!

67

Characteristic Theme Quantity

[Schneider et al., ACL 2018] (simplified slightly)

Time Goal Whole Possessor

SLIDE 68

Prepositions

68

Other 2,503 Temporal 516 Spatial 1,148

P and PP tokens by scene role in web reviews (STREUSLE 4.1)

[Schneider et al., ACL 2018]

SLIDE 69

Interannotator Agreement: New Corpus & Genre

69

[Schneider et al., ACL 2018]

After a few rounds of pilot annotation on The Little Prince and minor additions to the guidelines: 78% on 216 unseen targets

SLIDE 70

Target Identification

• Main challenge: multiword expressions (some

prepositional, some not)

70

10 20 30 40 50 60 70 80 90 Prec. Rec. F1

Gold Syntax Automatic Syntax

[Schneider et al., ACL 2018]

SLIDE 71

Disambiguation

• Accuracy using gold targets & gold syntax

71

10 20 30 40 50 60 70 80 90 Role Fxn Full

Most Frequent Neural Feature-rich linear

[Schneider et al., ACL 2018]

SLIDE 72

Case and Adposition Representation for Multi-Lingual Semantics (CARMLS)

Can we use the supersenses for case markers and adpositions in other languages?

72

SLIDE 73

SLIDE 74

Takeaways

• It’s impossible to do lexical semantic

tagging without making MWE decisions

• Even for prepositions!
• MWE identification, supersenses as joint

statistical sequence tagging—even gappy MWEs 


[Diab & Bhutada 2009, Constant & Sigogne 2011, Schneider et al. 2014, Schneider & Smith 2015, DiMSUM task]

• MWE annotation sometimes crudely

approximates productive constructions

• Comprehensive annotation forces reckoning

with the long tail

74

SLIDE 75

Outlook

• We need a range of linguistic

representations and annotation methods

• Spans, syntax
• “Top-down”: building grammars/lexicons/

constructicons to apply to corpora—good for linguistic precision & detail

• “Bottom-up”: token-level annotation—good for

coverage/long tail

• Constructions!
• Multiple languages!

75

SLIDE 76

76

Many_thanks


(*Several thanks)


Thanks_a_million


(*Thanks a thousand)


Thanks_a_lot


(?Lots of thanks)

tiny.cc/streusle