Surface Construction Labeling Lori Levin Language Technologies - - PowerPoint PPT Presentation

Corpus Annotation for Surface Construction Labeling

Lori Levin Language Technologies Institute Carnegie Mellon University

Collaborators and acknowledgements

• Collaborators: Jesse Dunietz, Jaime Carbonell,

– Dunietz, Jesse, Lori Levin, and Jaime Carbonell. "Automatically Tagging Constructions of Causation and Their Slot-Fillers." Transactions of the Association for Computational Linguistics 5 (2017). 117-133 – Dunietz, Jesse, Lori Levin, and Jaime Carbonell. "The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations." Proceedings of LAW XI – The 11th Linguistic Annotation Workshop (2017). 95-104. – Dunietz, Jesse, Lori Levin, and Jaime Carbonell. "Annotating Causal Language Using Corpus Lexicography of Constructions." Proceedings of LAW IX – The 9th Linguistic Annotation Workshop(2015): 188-196.

– Dunietz, Jesse, Lori Levin, and Miriam R. L. Petruck. "Construction Detection in a Conventional NLP Pipeline." AAAI Spring Symposium Technical Report SS-17-02: Computational Construction Grammar and Natural Language Understanding (2017).

– Jesse Dunietz, Annotating and Automatically Tagging Constructions of Causal Language, Ph.D. Thesis, Carnegie Mellon University, 2018.

– Jesse Dunietz, Jaime Carbonell and Lori Levin, “DeepCx: A transition-based approach for shallow semantic parsing with complex constructional triggers”. EMNLP 2018

• Thank you: Nathan Schneider, Miriam Petruck, Alexis Palmer

Outline

• What is Surface Construction Labeling (SCL)?
• The causality task (Dunietz 2018)
• The Because onomasiological annotation scheme
• Making the task doable

– Constructicon as an annotation tool – Overlapping Relations

• Slow research musings about grammaticalization

and the contents of the constructicon

Surface Construction Labeling is a type of Frame Semantic Parsing (aka Shallow Semantic Parsing, Semantic Role Labeling)

http://www.cs.cmu.edu/~ark/SEMAFOR/

Surface Construction Labeling is a type of Frame Semantic Parsing (aka Shallow Semantic Parsing, Semantic Role Labeling)

http://www.cs.cmu.edu/~ark/SEMAFOR/

Names of semantic Frames from the FrameNet lexicon

Surface Construction Labeling is a type of Frame Semantic Parsing (aka Shallow Semantic Parsing, Semantic Role Labeling)

http://www.cs.cmu.edu/~ark/SEMAFOR/

Names of semantic Frames from the FrameNet lexicon Frame-Evoking Elements

Surface Construction Labeling is a type of Frame Semantic Parsing (aka Shallow Semantic Parsing, Semantic Role Labeling)

http://www.cs.cmu.edu/~ark/SEMAFOR/

Names of semantic Frames from the FrameNet lexicon Frame-Evoking Elements Each Frame-Evoking Element is a lexical item

• r MWE.

Surface Construction Labeling is a type of Frame Semantic Parsing (aka Shallow Semantic Parsing, Semantic Role Labeling)

http://www.cs.cmu.edu/~ark/SEMAFOR/

Frame Elements of the DESIRING frame: Experiencer and Event Each Frame Element has a label and an extent.

Example of Surface Construction Labeling (in a notation that is easier to format) She got so lonely that she decided to watch TV Cause Effect Motivation (enable)

Frame Name and Frame Evoking Elements

Example of Surface Construction Labeling (in a notation that is easier to format) She got so lonely that she decided to watch TV Cause Effect Motivation (enable)

Frame Name and Frame Evoking Elements The frame name is from the annotation scheme that I will be describing in this talk. The frame indicates that being excessively lonely was her motivation for watching TV.

Example of Surface Construction Labeling (in a notation that is easier to format) She got so lonely that she decided to watch TV Cause Effect Motivation (enable)

Frame Name and Frame Evoking Elements The Frame-Evoking Element(s) is not a single lexical item. It is a construction consisting of an adjective phrase with an intensifier and a finite clausal complement.

Example of Surface Construction Labeling

She got so lonely that she decided to watch TV Cause Effect Motivation (enable)

Constructions: (Fillmore, Kay, and O’Connor, 1988) pairings of form and meaning, where the form may be any constellation of morpho-syntactic or lexical material. A Constructicon is repository of constructions where the Frame Evoking Elements can an arbitrary constellation of forms and the meaning side evokes a frame. (Fillmore et al., 2011)

• Expand the scope of frame semantic parsing
• By allowing Frame Evoking Elements to be constructs,

arbitrary constellations of morpho-syntactic and lexical elements.

• Creating a unified approach to Frame Semantic Parsing,

integrating the FrameNet lexicon and the FrameNet Constructicon

• Resulting in broader coverage of more ways to express

each frame

Goal of Surface Construction Labeling

Why is it called Surface Construction Labeling? She got so lonely that she decided to watch TV.

Full construction labeling (aka “constructions all the way down”) would parse all of the constructions in the sentence and their composition into larger constructions.

• Definite referring expression

construction, intensified adjective construction, etc. SCL recognizes constructions shallowly

• n top of Stanford NLP parses.

Why is it called Surface Construction Labeling? She got so lonely that she decided to watch TV.

Full construction labeling (aka “constructions all the way down”) would parse all of the constructions in the sentence and their composition into larger constructions.

• Definite referring expression

construction, intensified adjective construction, etc. SCL recognizes constructions shallowly

• n top of Stanford NLP parses.

Outline

• What is Surface Construction Labeling (SCL)?
• The causality task (Dunietz 2018)
• The Because onomasiological annotation scheme
• Making the task doable

– Constructicon as an annotation tool – Overlapping Relations

• Slow research musings about grammaticalization

and the contents of the constructicon

CAUSE CAUSATION

(ENABLEMENT)

The hot summer

EFFECT

set the stage for the devastating fires . Why

CAUSATION

Annotating Causality

is California on fire ?

EFFECT

CAUSE CAUSATION

(ENABLEMENT)

The hot summer

EFFECT

set the stage for the devastating fires . Why

CAUSATION

Annotating Causality

is California on fire ?

EFFECT

Connective Argument spans

Causality is way too hard. Why did we choose it?

Variety in Frame Evoking Elements

Such swelling can impede breathing. They moved because of the schools. We’re running late, so let’s move quickly. Our success is contingent on your support. This opens the way for broader regulation. Judy’s comments were so offensive that I left. For markets to work, banks can’t expect bailouts. (Verbs) (Prepositions) (Conjunctions) (Adjectives) (Multi-word expr.s) (Complex) (Complex)

Variety in syntactic positions of FEs with respect to FEEs

For nominal FEEs, the EFFECT may not be a syntactic complement of the FEE The cause of her illness was dehydration. Her illness’ cause was dehydration. Her chart listed her illness’ cause as dehydration.

33% of explicit relations between French verbs (Conrath et al. 2011) 12% of explicit discourse connectives in Penn Discourse Treebank (Prasad et al., 2008)

Causality is not rare

>5% and among the most complex

• f questions asked to question-answering systems

(Verberne et al., 2010)

23

(Temporal) (Extremity) (Correlation) (Permission) (Temporal + (Correlation) After a drink, she felt much better. They’re too big to fail. The more I read his work, the less I like it. The police let his sister visit him briefly. As voters get to know Mr. Romney, his poll numbers will rise.

Causal language is difficult to disentangle from overlapping semantic domains.

85.2% 62.8% 63.6% 63.4% 65.4% 18.1% 46.2% 44.6% 55.8% 56.5% 29.5% 53.1% 52.3% 59.2% 60.5% Benchmark Benchmark + CW-S Benchmark + CW-L DeepCx DeepCx (known conn.s) Precision Recall F1

Automatic labeling of FEEs

Neural net won

Automatic labeling of FEs

22.3% 37.9% 38.5% 43.9% 44.7% 73.5% 24.7% 42.5% 43.5% 50.6% 51.0% 80.9% 15.4% 24.8% 30.6% 41.0% 42.8% 67.8% 22.4% 38.7% 40.7% 51.7% 52.6% 82.8% Benchmark Benchmark + CW-S Benchmark + CW-L DeepCx DeepCx (known conn.s) DeepCx (oracle conn.s) Cause (exact) Cause (50%) Effect (exact) Effect (50%) Neural net won

FrameNet

(Fillmore & Baker, 2010; Ruppenhofer et al., 2016)

Penn Discourse Treebank

(Prasad et al., 2008)

Each existing semantic parsing representation handles only a portion of this space.

CONTINGENCY:Cause ARG1

so Its products are simpler ,

ARG 2

customers need less assistance .

Conjunctions & adverbs only

(one) meaning

• ne

word

CAUSATION

EFFECT

EFFECT

CAUSER

made bow me He

PURPO

SE

to show his dominance Words or constituents

• nly

PropBank

(Palmer et al., 2005)

MAKE.02 ARG2 ARG1 ARG0

made bow me He

ARGM-PRP

to show his dominance . Verbs only

We did some analysis, and the neural net really won. So, unfortunately, that has to be presented in EMNLP. Jesse Dunietz, Jaime Carbonell and Lori Levin, “DeepCx: A transition-based approach for shallow semantic parsing with complex constructional triggers”. EMNLP 2018

Outline

• What is Surface Construction Labeling (SCL)?
• The causality task (Dunietz 2018)
• The Because onomasiological annotation scheme
• Making the task doable

– Constructicon as an annotation tool – Overlapping Relations

• Slow research musings about grammaticalization

and the contents of the constructicon

Annotation for Surface Construction Labeling

• Onomasiological

– “How do you express x?”

• https://en.wikipedia.org/wiki/Onomasiology
• Hasegawa et al.
• For any potential annotation unit:

– Is meaning X expressed? (semasiological) – Is there a form that expresses X?

• If you do this as a corpus study, you end up with a

collection of constructions that express X in the corpus.

The Because Corpus

Documents

Sentences Causal

New York Times Washington section

(Sandhaus, 2014)

59 1924 717 Penn TreeBank WSJ 47 1542 534 2014 NLP Unshared Task in PoliInformatics

(Smith et al., 2014)

3 772 324 Manually Annotated Sub- Corpus

(Ide et al., 2010)

12 629 228 Total 121 4790 1803

BECAUSE = Bank of Effects and Causes Stated Explicitly

The Because Annotation Scheme

• Bank of Effects and Causes Stated Explicitly

Causal language: a clause or phrase in which

• ne event, state, action, or entity

is explicitly presented as promoting or hindering another

Causal language: a clause or phrase in which

• ne event, state, action, or entity

is explicitly presented as promoting or hindering another

Annotators can’t do this.

We annotate three types of causation.

The system failed because of a loose screw.

CONSEQUENCE

Mary left because John was there.

MOTIVATION

Mary left in order to avoid John.

PURPOSE

For us to succeed, we all have to cooperate We all have to cooperate (cause) for us to succeed (effect)

Motivation and Purpose

• Motivation: the reason exists

– Mary left because John was there

• John being there (cause) motivated Mary to leave

(effect)

• Purpose: a desired state

– Mary left in order to avoid John – (Mary wants to avoid John) causes (Mary leaves) – (Mary leaves) may cause/enable (Mary avoids John)

For us to succeed, we all have to cooperate We all have to cooperate (cause) for us to succeed (effect)

Causation can be positive or negative.

This has often caused problems elsewhere.

FACILITATE

He kept the dog from leaping at her.

INHIBIT

Connective: fixed morphological or

lexical cue indicating a causal construction

John killed the dog because it was threatening his chickens. John prevented the dog from eating his chickens. Ice cream consumption causes drowning.

We do not annotate most agentive verbs. We

• nly annotate verbs that express causation as

their primary meaning.

Effect: presented as outcome Cause: presented as producing effect

John killed the dog because it was threatening his chickens. John prevented the dog from eating his chickens. Ice cream consumption causes drowning. She must have met him before, because she recognized him yesterday.

Means arguments for cases with an agent and an action

caused a commotion My dad by shattering a glass .

MEANS EFFECT CAUSE

By altering immune responses, inflammation can trigger depression.

We have not yet added the AFFECTED frame element (Rehbein and Ruppenhofer 2017).

We exclude language that does not encode pure, explicit causation:

Relationships with no lexical trigger

John killed the dog. It was threatening his chickens.

Connectives lexicalizing a means or result

John killed the dog.

Unspecified causal relationships

The treatment is linked to better outcomes.

Actual corpus examples can get quite complex.

“For market discipline to effectively constrain risk, financial institutions must be allowed to fail.” Average causal sentence length: 30 words “If properly done, a market sensitive regulatory authority not only prevents some of the problems, but is pro-market, because we have investors now who are unwilling to invest even in things they should.”

Complex corpus examples

Outline

• What is Surface Construction Labeling (SCL)?
• The causality task (Dunietz 2018)
• The Because onomasiological annotation scheme
• Making the task doable

– Constructicon as an annotation tool – Overlapping Relations

• Slow research musings about grammaticalization

and the contents of the constructicon

Decision Tree

Connective pattern <cause> prevents <effect> from <effect> <enough cause> for <effect> to <effect> Annotatable words prevent, from enough, for, to WordNet verb senses prevent.verb.01 prevent.verb.02 Type Verbal Complex Degree INHIBIT FACILITATE Type restrictions Not PURPOSE Example His actions prevented disaster.

There’s enough time for you to find a restroom.

Annotators were guided by a “constructicon.”

Inter-annotator agreement

Causal Overlapping Connective spans (F1) 0.77 0.89 Relation types (κ) 0.70 0.91 Degrees (κ) 0.92 (n/a) CAUSE/ARGC spans (%) 0.89 0.96 CAUSE/ARGC spans (Jaccard) 0.92 0.97 CAUSE/ARGC heads (%) 0.92 0.96 EFFECT/ARGE spans (%) 0.86 0.84 EFFECT/ARGE spans (Jaccard) 0.93 0.92 EFFECT/ARGE heads (%) 0.95 0.89

260 sentences; 98 causal instances; 82 overlapping relations

The Causal Language Constructicon

• 290 construction variants
• 192 lexically distinct connectives

– “prevent” and “prevent from” use the same primary connective word

New examples can be added

• Annotators recommend new connectives
• Do a quick corpus study to verify that that

connective frequently expresses causality

– In other words, its use for causality seems to be conventionalized

We annotate 7 different types

• f overlapping relations.

TEMPORAL CORRELATION HYPOTHETICAL OBLIGATION/PERMISSION CREATION/TERMINATION EXTREMITY/SUFFICIENCY CONTEXT

After; once; during As; the more…the more… If…then… Require; permit Generate; eliminate So…that…; sufficient…to… Without; when (circumstances where…)

Lingering difficulties with

• ther overlapping relations

Origin/destination:

toward that goal

Topic:

fuming over recent media reports

Component:

as part of the liquidation

Evidentiary basis:

went to war on bad intelligence

Having a role:

as an American citizen

Placing in a position:

puts us at risk

Overlapping Relation Examples

• Temporal

– Within minutes after the committee released its letter, Torricelli took the senate floor to apologize to the people of New Jersey.

• Correlation

– Auburn football players are reminded of last year’s losses every time they go into the weight room.

• Hypothetical

– Previously, he allowed increases in emissions as long as they did not exceed the rate of economic growth.

Overlapping Relation Examples

• Obligation/Permission

– He will roll back a provision known as a new source review that compels utilities to install modern pollution controls whenever the significantly upgrade

• lder plants.
• “whenever” is also a connective
• Creation/Termination

– Many expected synergies of financial service activities gave rise to conflicts and excessive risk taking.

• Context

– With Hamas controlling Gaza, it was not clear that Mr. Abbas had the power to carry out his decrees.

Annotators applied several tests to determine when an

• verlapping relation was also causal.
• Can the reader answer a “why” question?
• Does the cause precede the effect?
• Counterfactuality(Grivaz, 2010) : would the effect

have been just as probable without the cause?

Rules out: My bus will leave soon, I just finished my breakfast.

• Ontological asymmetry (Grivaz, 2010):

could the cause and effect be reversed?

– Rules out: It is a triangle. It has three sides.

• Can it be rephrased as “because?”

Temporal with and without causal interpretation After last year’s fiasco, everyone is being cautious.

ARGE ARGC MOTIVATION + TEMPORAL

After last year’s fiasco, they’ve rebounded this year.

ARGE ARGC TEMPORAL

Conditional hypotheticals don’t have to be causal, but most are.

84% carry causal meaning Non-causal: If he comes, he’ll bring his wife. Causal: If I told you, I’d have to kill you.

Overlapping relations

Causality has seeped into the temporal and hypothetical domains.

~7% are expressed as hypotheticals Of the causal expressions in the corpus: > 14% are piggybacked on temporal relations

Conditional hypotheticals don’t have to be causal, but most are.

84% carry causal meaning Non-causal: If he comes, he’ll bring his wife. Causal: If I told you, I’d have to kill you.

Outline

• What is Surface Construction Labeling (SCL)?
• The causality task (Dunietz 2018)
• The Because onomasiological annotation scheme
• Making the task doable

– Constructicon as an annotation tool – Overlapping Relations

• Slow research musings about grammaticalization

and the contents of the constructicon

Slow Research Musings

• Usefulness of SCL and onomasiological

annotation

– In NLP tasks – For linguistic discovery

Slow research musings

• Convention vs exploitation

– Patrick Hanks, Lexical Analysis: Norms and Exploitations, MIT Press.

• With respect to constructions is conventional the same

as grammaticalized?

• In 84% of if-then in the Because corpus, the sentence

seems to have the intent of expressing causality:

– Does that mean that if-then is a conventional/grammaticalized way of expression causality? – If so, does it lead us to extreme conclusions…….

Slow research musings

• Constructions whose meaning side is a frame:

e.g., causality

• Constructions whose meaning side is a

function: e.g., “to V-base”, “Det N”

Slow Research Musings

• Fusion constructions vs overlapping relations

– English Dative shift The child was hungry. So his mother gave him (recipient) a cookie (theme). So his mother gave a cookie (theme) to him (recipient).

• Animacy, information status, definiteness, NP weight, change of

possession/information/state

• Joan Bresnan, Anna Cueni, Tatiana Nikitina, and Harald Baayen. 2007. "Predicting the Dative

Alternation." In Cognitive Foundations of Interpretation, ed. by G. Boume, I. Kraemer, and J.

• Zwarts. Amsterdam: Royal Netherlands Academy of Science, pp. 69--94.
• Frishkoff et al. (2008) Principal components analysis reduced Bresnan’s 14 features to four,

roughly corresponding to verb class, NP weight, animacy, and information status.

Slow research musings

Fusion Constructions

– Chinese ba (aside from issues of the part of speech)

• proposition-level completedness, lexical aspect,

discourse information status of the direct object, affectedness of the direct object

Slow research musings

• What is the meaning side of fusional

constructions in a constructicon?

– An abstraction from which all the other functions follow?

• English Dative Shift: Topic worthiness of the recipient
• Ba: affectedness or completedness

– Or would fusional constructions be represented as a collection of overlapping relations?

Stop here

Lingering difficulties include

• ther overlapping relations

and bidirectional relationships.

For us to succeed, we all have to cooperate.

succeed cooperate

enables

succeed cooperate

necessitates

• But not every construction evokes a frame.

There could be a function instead of a frame, like information structure (Goldberg). (Fillmore, Lee-Goldman, and Rhodes, 2012). Chuck mentioned the need for interactional frames for pragmatics and some people have pursued it.

Foundation: Form and Meaning

• Constructions (pairs of form and meaning) (Fillmore):

– Meaning is not a discrete space

• However there are centroids of meaning that tend to get

grammaticalized

– Meaning is grammaticalized in arbitrary constellations of linguistic units – Meaning can be compositional or conventional (idiomatic)

• Languages have comparable categories

– Common centroids of form and meaning like “noun”

• Languages differ

– How they carve up the meaning space when they grammaticalize it – How they spread from the centroids – What constellations of linguistic units are used

Meaning is not a discrete space

• http://www.ecenglish.com/learnenglish/lessons/will-would-shall-should
• Desire, preference, choice or consent:

– Will you please be quiet? – He won’t wash the dishes

• Future:

– It will be a great party. – I will go to the market.

• Capability/capacity:

– The ship will take three hundred guests. – This bottle will hold two litres of wine.

• Other:

– Phone rings: That will be my son

Past Present Future Hypo- thetical Counter- factual Prohibitive Imperative

Ferdinand De Haan, “On Representing Semantic Maps” The ovals represent the points in semantic space. The outlines each represent the irrealis morpheme in one language, showing what part of the semantic space it covers. Discrete centroids tend to be grammaticalized, but not in the same way in each language.

Muyuw irrealis morpheme Sinaugoro irrealis morpheme Manam irrealis morpheme Susurunga irrealis morpheme

Related spaces are grammaticalized differently Anaphoricity, Specificity, Familiarity

Discourse-old

I met a student. The student was tall.

Discourse Predictable

I went to a wedding. The bride and groom looked great.

Specific

I’m looking for a

• student. Her name is

Chris.

Non Specific

I’m looking for a

• student. I need one

to help me with something.

Familiarity/Context

Hand me the pen on the desk.

Genericity

Uniqueness Abstract nouns h Mass nouns “The” and “a” are not

• meanings. They are

grammaticalizations of parts of this space.

Constructions: pairs of form and meaning (Goldberg)

• Morpheme

– -ed past time

• Word

– Student

• Phrase (productive and meaning is compositional)

– The student (an anchored instance of a student)

Constructions: pairs of form and meaning

• Phrase (productive but meaning is non-compositional)

– What a nice shirt! – Why not read a book? (suggestion, invitation, deontic)

• Phrase (not productive, Goldberg)

– To prison – To bed – To school – To hospital – On holiday – *to airport

– *To bath

• Idiom

– Out in left field

Constructions: pairs of form and meaning

• Arbitrary constellation of units of form, possibly discontinuous,

possibly non-compositional

– What is she doing going to the movies? (Fillmore et al.) (incongruity) – What do you mean you don’t know? (disbelief, incongruity) – It was too big to fail – No intellectual characteristic is too ineffable for assessment.

Task definition: connective discovery + argument identification

I worry because I care. I worry because I care.

Connective discovery

Find lexical triggers

• f causal relations

Argument identification

Identify cause & effect spans for each connective

• Improve shallow semantic parsing coverage using richer, more

flexible linguistic representations, leading to a unified approach to Frame Semantic Parsing, integrating the FrameNet Lexicon and the FrameNet Constructicon.

• Create a proof-of-concept for SCL:
• Design annotation guidelines & annotate a corpus

using these representations.

• Build automated machine learning taggers

for constructional realizations of semantic relations.

Goal of Surface Construction Labeling

Goal of the causal labeling project

Four Automatic Labelers

• Syntax-based

– Benchmark – CausewayS

• String-based

– CausewayL

• Neural: DeepCx

Syntax-based connective discovery: each construction is treated as a partially-fixed parse tree fragment

worry/VBP

nsubj advcl

I/PRP care/VBP

mark nsubj

I/PRP because/IN I worry because I care.

“head” of because I care

Syntax-based connective discovery: each construction is treated as a partially-fixed parse tree fragment.

advcl mark

because/IN

nsubj

I/PRP

nsubj

I/PRP worry/VBP care/VBP I worry because I care.

Syntax-based connective discovery: each construction is treated as a partially-fixed parse tree fragment.

advcl mark

because/IN

Benchmark: Syntax-based dependency path memorization heuristic.

Connective Parse paths to possible cause/effect heads Causal / Not causal prevent from nsubj, advcl 27/ 4 prevent from nsubj, advmod 0 / 8 because of case, case  nmod

14 / 1

… … …

The syntax-based benchmark system

• Benchmark

– For each causal tree fragment

• Count how many times it is causal in the training data
• Count how many times the same fragment is not causal in the

training data

• If that pattern is causal more often in the training data, then

whenever you see it in the test data, label it as causal.

• If that pattern is non-causal more often in the training data, then

whenever you see it in the test data, don’t label it as causal.

– High precision: If the benchmark system thinks it’s causal, it is usually right – Low recall: The benchmark system misses too many instances of causality

Syntax-based connective discovery: TRegex patterns are extracted in training, and matched at test time.

Training: Test: I worry because I care.

advcl mark

because/IN

(/^because_[0-9]+$/ <2 /^IN.*/ <1 mark > (/.*_[0-9]+/ <1 advcl > (/.*_[0-9]+/))) (/^because_[0-9]+$/ <2 /^IN.*/ <1 mark > (/.*_[0-9]+/ <1 advcl > (/.*_[0-9]+/)))

I worry because I love you.

TRegex 1

I worry because I love you.

+

1 Levy and Andrew, 2006

The syntax-based CausewayS system

• Fancier way of extracting tree fragments from

the training data

– Dreyfus-Wagner, minimum-weight sub-tree

• For each tree fragment extracted from the

training data, find all instances of it in the test data.

• CausewayS, at this point, has high recall (finds

a lot of instances of causality) but low precision (a lot of what it finds isn’t right).

• Apply a classifier to increase precision.

Syntax-based argument ID: Argument heads are expanded to include most dependents.

nsubj advcl

I/PRP

mark nsubj

I/PRP because/IN care/VBP worry/VBP

Syntax-based argument ID: Argument heads are expanded to include most dependents.

nsubj advcl

I/PRP

mark nsubj

because/IN care/VBP worry/VBP I/PRP

CausewayL: constructions are matched by regular expressions over word lemmas.

Training: Test: I worry because I care. I worry because I love you.

regex

I worry because I love you.

+

(ˆ | )([ \ S]+ )+?(because/IN) ([ \ S]+ )+? (ˆ | )([ \ S]+ )+?(because/IN) ([ \ S]+ )+?

CausewayL: Arguments are labeled by a conditional random field.

labels featurized words … … CAUSE EFFECT EFFECT

Features include information about:

• Word
• Connective
• Relationship between word & connective

𝑞 𝐳 𝐲 = 1 𝑎 𝐲 ෑ

𝑢=1 𝑈

exp ෍

𝑙=1 𝐿

𝜄𝑙𝑔

𝑙 𝑧𝑢, 𝑧𝑢−1, 𝐲𝑢

CausewayL

• Recall is high (finds a lot of matches)
• Precision is low (a lot of what it finds is wrong)
• Use a classifier to raise precision

DeepCx: Neural Net, LSTM

Analysis of results

• DeepCx is better than Causeway in these circumstances

– Ambiguity (entropy) of the causal connective.

• Eg., to has entropy of 1

– F1 for DeepCx: 40.6% – F1 for Causeway: 25.5%

– Connectives whose part of speech is Adverb

• Causeway missed all instances of “why” and “when”

– Design problem?

– Connectives whose part of speech is Noun

• Fewer instances to train on

– More words in the causal connective

• But CausewayL does pretty well too because it is matching a pattern over a

whole sentence whereas DeepCx is proceeding one word at a time. CausewayS is missing complex parse paths.

Analysis of results

• CausewayL and DeepCx do better than CausewayS on most
• verlapping relations
• Amount of training data

– DeepCx performs better at all amounts of training data – It appears to be better at generalizing across patterns (vs generalizing within patterns) – The gap between CausewayS and DeepCx remains constant across all amounts of training data – CausewayL increases fastest with more training data

• Length of Cause and Effect arguments

– DeepCx and CausewayL are better than CausewayS as argument length increases

• CausewayL uses CRF to label argument spans
• CausewayS uses the dependency trees

Next steps

• Reproduce with other meanings with large

constructicons such as comparatives

• Reproduce with multiple languages
• Apply where the constructicon is small and

quirky too such as incongruity

Related annotation schemes and labelers for causality

Previous projects have struggled to annotate real-world causality.

“A person infected with a <e1>flu</e1>

<e2>virus</e2> strain develops antibodies against

it.”

Cause-Effect(e2, e1) = "true"

SemEval 2007 Task 4

(Girju et al., 2007)

Richer Event Descriptions

(O’Gorman et al., 2016; Croft et al., 2016)

We’ve allocated a budget to equip the barrier with electronic detention equipment.

BEFORE-PRECONDITIONS

CaTeRS

(Mostafazadeh et al., 2016)

Existing shallow semantic parsing schemes include some elements of causal language.

…

Penn Discourse Treebank

(Prasad et al., 2008)

PropBank

(Palmer et al., 2005)

FrameNet

(Fillmore & Baker, 2010; Ruppenhofer et al., 2016)

CAUSATION

EFFECT EFFECT CAUSER

made bow me He

PURPOSE

to show his dominance .

Others have focused specifically on causality.

Causality in TempEval-3

(Mirza et al., 2014)

CAUSE EVENT BEFORE EVENT TLINK

HP acquired 730,070 common shares as a result of a stock purchase agreement.

BioCause

(Mihaila et al., 2013)

CaTeRS

(Mostafazadeh et al., 2016)

Richer Event Description

(O’Gorman et al., 2016)

We’ve allocated a budget to equip the barrier with electronic detention equipment.

BEFORE-PRECONDITIONS

A note on universals

• Categories in linguistics:

– Degrees of universality

• Describing a particular language on its own terms
• Using comparable categories for language typology
• Making a theory of universal categories for all

languages

A note on universals

• Universals in NLP

– Universal categories for cross-lingual training – Some new multi-lingual resources are going viral

• Google universal parts of speech
• Google Universal Dependencies (60 languages)
• Abstract Meaning Representation
• UniMorph

– They know not what they do. We need to be involved.

Acknowledgements

• Jesse Dunietz
• Jaime Carbonell
• David Mortensen
• Archna Bhatia
• Nathan Schneider
• Miriam Petruck
• Judith Klavans

Thank you