[PPT] - A coherence model based on syntactic patterns by Annie Louis and Ani PowerPoint Presentation

SLIDE 1

A coherence model based on syntactic patterns

by Annie Louis and Ani Nenkova

M.Sc. Seminar: Discourse Coherence Theories and Modeling Nikolina Koleva

Saarland University Department of Computational Linguistics

June 10, 2013

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 1 / 32

SLIDE 2

Overview

1 Motivation 2 Coherence models based on syntax

Evidence for syntactic coherence Representing syntax Local co-occurrence model Global model

3 Evaluation

Prediction on reports Prediction on academic articles

4 Conclusion

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 2 / 32

SLIDE 3

Motivation

Factors contributing to coherence

1 attentional structure (items under discussion) 2 organization of discourse segments 3 intentional structure (purpose of the discourse)

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 3 / 32

SLIDE 4

Motivation

Factors contributing to coherence

1 attentional structure: items under discussion

✧ entity approaches

2 organization of discourse segments

✧ content approaches

3 intentional structure: purpose of the discourse

✪ not much work

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 3 / 32

SLIDE 5

Motivation

Every discourse has a purpose

explaining a concept
narrating an event
critiquing an idea
...

✌ each sentence in a text has a communicative goal

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 4 / 32

SLIDE 6

Motivation

Example

1 An aqueduct is a water supply or navigable channel constructed

to convey water.

2 In modern engineering, the term is used for any system of pipes,

canals, tunnels, and other structures used for this purpose.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 5 / 32

SLIDE 7

Motivation

Example

1 An aqueduct is a water supply or navigable channel constructed

to convey water.

2 In modern engineering, the term is used for any system of pipes,

canals, tunnels, and other structures used for this purpose.

1 Cytokine receptors are receptors that bind cytokines. 2 In recent years, the cytokine receptors have come to demand

more attention because their deficiency has now been directly linked to certain debilitating immunodeficiency states.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 5 / 32

SLIDE 8

Motivation

Example

1 An aqueduct is a water supply or navigable channel

constructed to convey water.

2 In modern engineering, the termis used for any system of

pipes, canals, tunnels, and other structures used for this purpose.

1 Cytokine receptors are receptors that bind cytokines. 2

In recent years, the cytokine receptors have come to demand more attention because their deficiency has now been directly linked to certain debilitating immunodeficiency states.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 5 / 32

SLIDE 9

Motivation

Example

1 An aqueduct is a water supply or navigable channel

constructed to convey water.

2 In modern engineering, the term is used for any system of

pipes, canals, tunnels, and other structures used for this purpose.

1 Cytokine receptors are receptors that bind cytokines. 2 In recent years, the cytokine receptors have come to demand

more attention because their deficiency has now been directly linked to certain debilitating immunodeficiency states.

✌ unique syntactic structure of definitions, questions etc. ✌ syntax as proxy for the communicative goal

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 5 / 32

SLIDE 10

Coherence models based on syntax

Coherence model based on syntax

Underlying assumptions:

1 Sentences with similar syntax are likely to have the same

communicative goal.

2 Regularities in intentional structure will be manifested in

syntactic regularities between adjacent sentences.

✌ supported by recent related work

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 6 / 32

SLIDE 11

Coherence models based on syntax Evidence for syntactic coherence

Pilot study for the validation of assumption No: 2

Material: gold standard parse trees from the Penn Treebank
Unit of analysis: two adjacent sentences, a pair (S1,S2)

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 7 / 32

SLIDE 12

Coherence models based on syntax Evidence for syntactic coherence

Pilot study for the validation of assumption No: 2

Material: gold standard parse trees from the Penn Treebank
Unit of analysis: two adjacent sentences, a pair (S1,S2)

Steps:

1 enumerate all productions = 197 unique productions

productions with frequency < 25 are removed

2 for all ordered pairs (p1,p2) compute

c(p1,p2) ,c(p1,¬p2), c(¬p1,p2) and c(¬p1,¬p2)

c(p1,p2): # of sentence pairs where p1 ∈ S1 and p2 ∈ S2

3 perform chi-square test to

prove significance of the count c(p1,p2)
check independence of the occurrences of p1 and p2

where, p1: production 1 and p2: production 2

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 7 / 32

SLIDE 13

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

small fraction of repetitions (5%)

p1: VP → VBD SBAR p2: VP → VBD SBAR

1

S1: Documents filed with the Securities and Exchange Commission on the pending spinoff [[disclosed]VBD [that Cray Research Inc. will withdraw the almost $ 100 million in financing it is providing the new firm if Mr. Cray leaves

r if the product-design project, he heads, is scrapped]SBAR]VP.

2

S2: The documents also [[said]VBD [that although the 64-year-old Mr. Cray has been working on the project for more than six years , the Cray-3 machine is at least another year away from a fully operational prototype]SBAR]VP.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 8 / 32

SLIDE 14

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

finance domain-specific

p1: NP → NP NP-ADV p2: QP → CD CD

1

S1: The two concerns said they entered into a definitive merger agreement under which Ratners will begin a tender offer for all of Weisfield’s common shares for [$57.50 each]NP.

2

S2: Also on the takeover front, Jaguar’s ADRs rose 1/4 to 13 7/8 on turnover of [4.4 million]QP.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 9 / 32

SLIDE 15

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

neither repetitions nor domain dependent

p1: VP → VB VP p2: NP-SBJ → NNP NNP

1

S1: "The refund pool may not [be held hostage through another round of appeals]VP, " Judge Curry said.

2

S2: [Commonwealth Edison]NP−SBJ said it is already appealing the underlying commission order and is considering appealing Judge Curry’s order.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 10 / 32

SLIDE 16

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

neither repetitions nor domain dependent

p1: VP → VB VP p2: NP-SBJ → NNP NNP

1

S1: "The refund pool may not [be held hostage through another round of appeals]VP, " Judge Curry said.

2

S2: [Commonwealth Edison]NP−SBJ said it is already appealing the underlying commission order and is considering appealing Judge Curry’s order.

S1 present hypothesis or speculation
S2 introduces an entity (PERS, ORG) that gives explanation or opinion
n the statement
intentional structure: SPECULATE , ENDORSE

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 10 / 32

SLIDE 17

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

neither repetitions nor domain dependent

p1: NP-LOC → NNP p2: S-TPC-1 → NP-SBJ VP

1

S1: "It has to be considered as an additional risk for the investor," said Gary P . Smaby of Smaby Group Inc., [Minneapolis]NP−LOC.

2

S2: ["Cray Computer will be a concept stock,"]S−TPC−1 he said.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 11 / 32

SLIDE 18

Coherence models based on syntax Evidence for syntactic coherence

Outcome of the study

neither repetitions nor domain dependent

p1: NP-LOC → NNP p2: S-TPC-1 → NP-SBJ VP

1

S1: "It has to be considered as an additional risk for the investor," said Gary P . Smaby of Smaby Group Inc., [Minneapolis]NP−LOC.

2

S2: ["Cray Computer will be a concept stock,"]S−TPC−1 he said.

S1 introduces location name associated with an entity
S2 contains quote from that entity
intentional structure: INTRODUCE X , STATEMENT BY X

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 11 / 32

SLIDE 19

Coherence models based on syntax Representing syntax

Representing syntax

1 productions

sentence as set of grammatical productions (LHS →RHS)
RHS could be very long and thus rather specific
available information only about nodes of the same constituent

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 12 / 32

SLIDE 20

Coherence models based on syntax Representing syntax

Representing syntax

1 productions

sentence as set of grammatical productions (LHS →RHS)
RHS could be very long and thus rather specific
available information only about nodes of the same constituent

2 d-sequence

cut the parse tree at level d
sentence as sequence of leaf nodes (of the cut tree)
for each node in the sequence augmented the tag of the left most child

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 12 / 32

SLIDE 21

Coherence models based on syntax Representing syntax

d-sequence example

depth-2 sequence: " S:dt , " NP:nnp VP:vbd .

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 13 / 32

SLIDE 22

Coherence models based on syntax Representing syntax

d-sequence example

Please, write down the depth-3 sequence.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 13 / 32

SLIDE 23

Coherence models based on syntax Representing syntax

d-sequence example

Please, write down the depth-3 sequence.

depth-3 sequence: " NP:dt VP:vbz , " NNP NNP VBD .

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 13 / 32

SLIDE 24

Coherence models based on syntax Local co-occurrence model

Local co-occurrence model idea

to test assumption No 2: Regularities in intentional structure will be manifested in syntactic regularities between adjacent sentences. Steps:

1 estimate probabilities of pairs of syntactic items (from the training set) 2 use these probabilities to compute the coherence of a new text

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 14 / 32

SLIDE 25

Coherence models based on syntax Local co-occurrence model

Local co-occurrence model implementation

n: number of sentences
Sy

x the yth item of the xth sentence

δC: smoothing constant
|V|: size of the vocabulary

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 15 / 32

SLIDE 26

Coherence models based on syntax Local co-occurrence model

Local co-occurrence model example

s1:

1 S → NP VP 2 NP → DT N 3 VP → VBD NP 4 NP → DT N

s2:

1 S → NP VP 2 NP → DT N 3 VP → VBD PP 4 PP → P NP 5 NP → DT N

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 16 / 32

SLIDE 27

Coherence models based on syntax Local co-occurrence model

Local co-occurrence model example

s1:

1 S → NP VP 2 NP → DT N 3 VP → VBD NP 4 NP → DT N

s2:

1 S → NP VP 2 NP → DT N 3 VP → VBD PP 4 PP → P NP 5 NP → DT N

[ p (S → NP VP | S → NP VP ) + p (S → NP VP | NP → DT N ) + p (S → NP VP | VP →

VBD NP) + p (S → NP VP | NP → DT N ) ] * [ p (NP → DT N | S → NP VP ) + p (NP → DT N | NP → DT N ) + p (NP → DT N | VP → VBD NP) + p (NP → DT N | NP → DT N ) ] * ...

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 16 / 32

SLIDE 28

Coherence models based on syntax Local co-occurrence model

Global coherence model idea

to test assumption No 1: Sentences with similar syntax are likely to have the same communicative goal.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 17 / 32

SLIDE 29

Coherence models based on syntax Local co-occurrence model

Clusters from abstracts of journal articles

Cluster a: VP → VBZ ADJP ; ADJP → JJ PP

1 This method [is [capable of sequence-specific detection of DNA with

high accuracy]ADJP]VP

2 The same [is [true for synthetic polyamines such as

polyallylamine]ADJP]VP

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 18 / 32

SLIDE 30

Coherence models based on syntax Local co-occurrence model

Clusters from abstracts of journal articles

Cluster a: VP → VBZ ADJP ; ADJP → JJ PP

1 This method [is [capable of sequence-specific detection of DNA with

high accuracy]ADJP]VP

2 The same [is [true for synthetic polyamines such as

polyallylamine]ADJP]VP Cluster b: VP → MD VP ; VP → VB VP

1 Our results for the difference in reactivity [can [be linked to

experimental observations]VP]VP

2 These phenomena taken together [can [be considered as the signature

f the gelation process]VP ]VP

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 18 / 32

SLIDE 31

Coherence models based on syntax Local co-occurrence model

Clusters from abstracts of journal articles

Cluster a: VP → VBZ ADJP ; ADJP → JJ PP

1 This method [is [capable of sequence-specific detection of DNA with

high accuracy]ADJP]VP

2 The same [is [true for synthetic polyamines such as

polyallylamine]ADJP]VP

✌ captures descriptive sentences

Cluster b: VP → MD VP ; VP → VB VP

1 Our results for the difference in reactivity [can [be linked to

experimental observations]VP]VP

2 These phenomena taken together [can [be considered as the signature

f the gelation process]VP ]VP

✌ captures speculative sentences

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 18 / 32

SLIDE 32

Coherence models based on syntax Global model

Global coherence model idea

and to capture the common patterns in the intentional structure for the domain (by using HMM)

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 19 / 32

SLIDE 33

Coherence models based on syntax Global model

Global coherence model idea

and to capture the common patterns in the intentional structure for the domain (by using HMM) Steps:

1 cluster sentences of different documents by syntactic similarity

features :

productions: frequency of production in a parse tree
d-sequence: n-grams of size one to four

2 estimate emission and transition probabilities (from the training set) 3 use these probabilities to compute the coherence of a new text

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 19 / 32

SLIDE 34

Coherence models based on syntax Global model

Global coherence model

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 20 / 32

SLIDE 35

Coherence models based on syntax Global model

Global coherence model implementation

n: number of sentences, St: the tth sentence
ht: the tth state
d(ht): # of docs whose sentences appear in ht
d(ht,ht−1) # of docs where subsequent sentences in subsequent states
δM: smoothing constant, C: # of clusters

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 21 / 32

SLIDE 36

Evaluation

Evaluating syntactic coherence

1 prediction on reports

use pairs of articles: (original article, random permutation)
testing on identifying the original article
compare with content and entity approaches

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 22 / 32

SLIDE 37

Evaluation

Evaluating syntactic coherence

1 prediction on reports

use pairs of articles: (original article, random permutation)
testing on identifying the original article
compare with content and entity approaches

2 prediction on academic articles

original vs. permuted sections
conference vs. workshop papers

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 22 / 32

SLIDE 38

Evaluation Prediction on reports

Prediction on reports

Model Airplane Accidents Earthquake Parameters Accuracy Parameters Accuracy baseline 50.0 50.0 Local co-occurrence model Prod 72.8 55.0 d-seq depth MVP + 2 71.8 depth MVP + 1 65.1 PoS 61.3 42.6 HMM-syntax Prod

clus. 37

74.6

clus. 5

93.8 d-seq depth MVP + 8, clus. 8 82.2 depth MVP + 9, clus. 45 86.5 Other approaches Egrid history 1 67.6 history 1 82.2 Content clus.48 71.4

clus. 23

84.5

✌ the articles of each corpus have the same intentional structure

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 23 / 32

SLIDE 39

Evaluation Prediction on reports

Combining predictions of different models

Accuracy Model Airplane Accidents Earthquake Content + Egrid 76.8 90.7 Content + HMM-prod 74.2 95.3 Content + HMM-d-seq 82.1 90.3 Egrid + HMM-prod 79.6 93.9 Egrid + HMM-d-seq 84.2 91.1 Egrid + Content + HMM-prod 79.5 95.0 Egrid + Content + HMM-d-seq 84.1 92.3 Egrid + Content + HMM-prod + HMM-d-seq 83.6 95.7

✌

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 24 / 32

SLIDE 40

Evaluation Prediction on reports

Combining predictions of different models

Accuracy Model Airplane Accidents Earthquake Content + Egrid 76.8 90.7 Content + HMM-prod 74.2 95.3 Content + HMM-d-seq 82.1 90.3 Egrid + HMM-prod 79.6 93.9 Egrid + HMM-d-seq 84.2 91.1 Egrid + Content + HMM-prod 79.5 95.0 Egrid + Content + HMM-d-seq 84.1 92.3 Egrid + Content + HMM-prod + HMM-d-seq 83.6 95.7

✌ syntax supplements content and entity grid methods

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 24 / 32

SLIDE 41

Evaluation Prediction on academic articles

Corpora of academic articles

1 ART Corpus

225 Chemistry journal articles
manually annotated for intentional structure

2 ACL Anthology Network (AAN)

500 ACL-NAACL conference articles
500 ACL-sponsored workshop articles

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 25 / 32

SLIDE 42

Evaluation Prediction on academic articles

Detected clusters vs. manually annotated zones

manually annotated zones in ART

1

Motivation

2

Background

3

Hypothesis

4

Objective

5

...

compare to detected clusters
compute c(Ci,Zj): # of sentences that are annotated as Zj and are

contained in Ci

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 26 / 32

SLIDE 43

Evaluation Prediction on academic articles

Original vs. permuted sections

Table : Accuracy in %

Data Section Test Pairs Local-prod Local-d-seq HMM-prod HMM-d-seq Oracle zones ART Corpus Abstract 1633 57.0 52.9 64.1 55.0 80.8 Introduction 1640 44.5 54.6 58.1 64.6 94.0 ACL Abstract 8815 44.0 47.2 58.2 63.7 Introduction 9966 54.5 53.0 64.4 74.0

Rel. work

10,000 54.6 54.4 57.3 67.3 Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 27 / 32

SLIDE 44

Evaluation Prediction on academic articles

Distinguish conference and workshop articles

conference articles: complete work
workshop articles: preliminary studies

✌ information presentation differ in abstracts and introductions

used features
indicating perplexity of the local and global models
fine-grained taken from the local model
most significant 30 pairs

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 28 / 32

SLIDE 45

Evaluation Prediction on academic articles

Distinguish conference and workshop articles

conference articles: complete work
workshop articles: preliminary studies

✌ information presentation differ in abstracts and introductions

used features
indicating perplexity of the local and global models
fine-grained taken from the local model
most significant 30 pairs

Table : Accuracy above confidence level

Conf Abstract Introduction

Rel. work

>= 0.5 59.3 50.3 55.4

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 28 / 32

SLIDE 46

Conclusion

Summary

Syntactic patterns are reliable clues for intentional structure

detection

Possible syntactic representations
productions
d-sequence
Local coherence model: exploring pairs of adjacent sentences
Global coherence model: clustering sentences based on syntactic

similarity

High accuracy on distinguishing coherent and incoherent news

articles

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 29 / 32

SLIDE 47

Conclusion

Thank you for your attention! Any questions?

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 30 / 32

SLIDE 48

Conclusion

Discussion

Would this approach work for languages with free word order?

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 31 / 32

SLIDE 49

Conclusion

References

Annie Louis and Ani Nenkova. A coherence model based on syntactic patterns. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL ’12, pages 1157–1168, Stroudsburg, PA, USA,

2012. Association for Computational Linguistics.

URL http:

//dl.acm.org/citation.cfm?id=2390948.2391078.

Manfred Stede. Discourse Processing. Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers, 2011.

Nikolina Koleva (CoLi Saarland) Syntactic Approach to Modeling Coherence June 10, 2013 32 / 32