Advanced Natural Language Processing Guest Lecture: Modeling Human - - PowerPoint PPT Presentation

Introduction Prediction and Grammar Predictive Parsing Evaluation

Advanced Natural Language Processing Guest Lecture: Modeling Human Parsing

Frank Keller

Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh http://homepages.inf.ed.ac.uk/keller/

October 30, 2013

Frank Keller ANLP Guest Lecture 1

Introduction Prediction and Grammar Predictive Parsing Evaluation

1

Introduction Incrementality Prediction

2

Prediction and Grammar Conceptual Issues Formalism Comparison with TAG Modeling Prediction

3

Predictive Parsing Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

4

Evaluation Parsing Performance Cognitive Plausibility

Frank Keller ANLP Guest Lecture 2

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

1

Introduction Incrementality Prediction

2

Prediction and Grammar Conceptual Issues Formalism Comparison with TAG Modeling Prediction

3

Predictive Parsing Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

4

Evaluation Parsing Performance Cognitive Plausibility

Frank Keller ANLP Guest Lecture 3

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

Incrementality

Text and speech are perceived serially. Human language processing is adapted to this: sentence comprehension proceeds incrementally: the interpretation of a sentence is built word by word; each new word is integrated as fully as possible into a representation of the sentence thus far; processing effort depends on the properties of the word and its relationship to the preceding context.

Frank Keller ANLP Guest Lecture 4

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

Structural Prediction

Staub and Clifton (2006) show that the sentence processor can also make structural predictions: (1) Peter read either a book or an essay in the school magazine. (2) Peter read a book or an essay in the school magazine. The presence of either leads to shorter reading times on or and on the NP that follows it (eye-tracking study). The word either makes it possible to anticipate an upcoming NP conjunction (rather than VP conjunction).

Frank Keller ANLP Guest Lecture 5

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

Semantic Prediction

Visual world paradigm: image and speech presented synchronously; eye-movements reflect listeners’ interpretation of input; they can also indicate predictions about upcoming input. Altmann and Kamide (1999) use this paradigm to provided evidence for semantic prediction. They presented sentences such as: (3)

• a. The boy will eat . . .
• b. The boy will move . . .

together with a scene that contained one edible but several movable objects.

Frank Keller ANLP Guest Lecture 6

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

Semantic Prediction

When participants heard eat, they looked more at the cake. Evidence for prediction induced by semantic restrictions of the verb.

Frank Keller ANLP Guest Lecture 7

Introduction Prediction and Grammar Predictive Parsing Evaluation Incrementality Prediction

Granularity of Prediction

What is the granularity of prediction? We saw predictions can be triggered by: specific collocations (either . . . or); semantic restrictions of a lexical item. Recent evidence also points to lexically specific syntactic prediction (Arai and Keller 2013): the subcategorization frame of a verb is predicted; verb morphology (e.g., tense information) is used for prediction.

Frank Keller ANLP Guest Lecture 8

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

1

Introduction Incrementality Prediction

2

Prediction and Grammar Conceptual Issues Formalism Comparison with TAG Modeling Prediction

3

Predictive Parsing Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

4

Evaluation Parsing Performance Cognitive Plausibility

Frank Keller ANLP Guest Lecture 9

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Conceptual Issues

Challenge: develop a model of prediction in sentence processing that accounts for these experimental results. Assumptions: structures are built incrementally (word by word); partial structures do not contain unconnected nodes; upcoming syntactic material is predicted. Evidence for connectedness: Sturt and Lombardo (2005). Existing incremental parsers don’t build fully connected structures. Approach: devise a grammar formalism that supports incrementality and connectedness; prediction then follows (Demberg et al. 2013).

Frank Keller ANLP Guest Lecture 10

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Implementing Prediction

Experimental results inform the model regarding the granularity of

• prediction. The model predicts:

lexical items when they are syntactically required (e.g., either . . . or, pick . . . up); syntactic structure when required by subcat frames; syntactic structure when required by connectedness.

Frank Keller ANLP Guest Lecture 11

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Demberg et al. (2013) propose Psycholinguistically Motivated TAG (PLTAG), a variant of tree-adjoining grammar: in standard TAG, the lexicon consists of initial trees and auxiliary trees (both are lexicalized); we add unlexicalized predictive trees to achieve connectivity; the standard TAG operations are substitution and adjunction; we add verification to verify predictive trees; we use TAG’s extended domain of locality for lexical prediction. PLTAG supports parsing with incremental, fully connected structures.

Frank Keller ANLP Guest Lecture 12

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG)

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG) Example

Initial Tree: NP Peter S NP↓ VP sleeps Auxiliary Tree: VP AP

• ften

VP*

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG) Example

NP Peter substitutes into S NP↓ VP sleeps resulting in S NP Peter VP sleeps

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG) Example

VP AP

• ften

VP* adjoins to S NP Peter VP sleeps resulting in S NP Peter VP AP

• ften

VP sleeps

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG) Example

Prediction Tree: Sk NPk↓ VPk

k

Index k marks predicted node.

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Formalism

Lexicon: Standard TAG lexicon Predictive lexicon (PLTAG) Operations: Substitution Adjunction Verification (PLTAG) Example

S1 NP1 Peter VP1 AP

• ften

VP1 is verified by S NP↓ VP sleeps resulting in S NP Peter VP AP

• ften

VP sleeps All nodes indexed with k have to be verified.

Frank Keller ANLP Guest Lecture 13

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Comparison with and TAG

TAG derivations are not always incremental. Example

NP ↓ S VP sleeps subst S VP sleeps NP Peter adj VP AP

• ften

VP S NP Peter sleeps

Frank Keller ANLP Guest Lecture 14

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Comparison with and TAG

PLTAG derivation are always incremental and fully connected. Example

NP Peter S VP NP 1 Peter VP AP

• ften

VP S NP Peter VP AP

• ften

VP S NP Peter sleeps subst adj verif 1 1 1 1 1 1 1 Frank Keller ANLP Guest Lecture 14

Introduction Prediction and Grammar Predictive Parsing Evaluation Conceptual Issues Formalism Comparison with TAG Modeling Prediction

Modeling Prediction

PLTAG assumes three types of prediction: predictive nodes (required by connectivity);

• pen substitution nodes (subcategorization);

lexical prediction (e.g., either . . . or). Connectedness and prediction interact closely: in order to achieve incrementality with full connectedness, upcoming nodes have to be predicted; in a fully connected structure, predictions can be read off straightforwardly (all open prediction and substitution nodes).

Frank Keller ANLP Guest Lecture 15

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

1

Introduction Incrementality Prediction

2

Prediction and Grammar Conceptual Issues Formalism Comparison with TAG Modeling Prediction

3

Predictive Parsing Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

4

Evaluation Parsing Performance Cognitive Plausibility

Frank Keller ANLP Guest Lecture 16

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

An Incremental Parser for PLTAG

In order to construct an incremental parser for PLTAG, we need to:

1 convert the Penn Treebank into PLTAG format; 2 induce a lexicon from it; 3 develop an incremental parsing algorithm; 4 devise a probability model; 5 formulate a linking theory. Frank Keller ANLP Guest Lecture 17

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 1: Treebank Conversion

Convert Penn Treebank into TAG format (Xia et al. 2000) using: head percolation table for determining how to cut up a tree into elementary trees; Propbank for distinguishing arguments and modifiers; noun phrase annotation to derive NP-internal structure. The resulting trees are less flat, contain head information, and argument/modifier distinction.

Frank Keller ANLP Guest Lecture 18

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

A standard TAG lexicon can be derived from the TAG Treebank by cutting up the trees into initial trees and adjunction trees. For the predictive lexicon, we need the notion of connection path. Connection Path The connection path of w1 is the minimal amount of structure needed to connect words w1 . . . wi under one node (Sturt et al. 2003). Essentially, we determine which parts of the tree we need to predict to achieve connectivity.

Frank Keller ANLP Guest Lecture 19

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

The DET Italian ADJ N people N NP vote V VP The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP Berlusconi NP

• ften

ADVP VP S

none

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

vote V VP vote V VP Berlusconi NP

• ften

ADVP VP S people N NP DET NP N

s s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S DET NP N

s s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

Berlusconi NP

no new predictive entry

S Berlusconi NP vote V VP vote V VP

• ften

ADVP VP

• ften

ADVP VP DET NP N

s s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

The DET Italian ADJ N people N NP vote V VP The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP Berlusconi NP

• ften

ADVP VP S DET NP N

s s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

vote V VP vote V VP Berlusconi NP Berlusconi NP S DET NP N S VP NP s

s s s s s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 2: Lexicon Induction

predictive lexicon entries generated from tree

The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S The DET Italian ADJ N people N NP vote V VP Berlusconi NP

• ften

ADVP VP S DET NP N

s s s s

S VP NP s

s s s

Frank Keller ANLP Guest Lecture 20

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 3: Parsing Algorithm

Properties: incrementally builds fully connected partial structures;

• nly allows valid partial PLTAG structures;

constructs all possible structures in parallel. At word wi, retrieve elementary tree ǫ for wi and connect it to the prefix tree β for w1 . . . wi−1: parsing operations: substitution, adjunction, verification; dependent on status of β and ǫ: standard or predictive tree.

Frank Keller ANLP Guest Lecture 21

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 4: Probability Model

The following properties need to hold (Chiang 2000): Substitution:

• ǫ

P(ǫ|ηβ) = 1 Adjunction:

• ǫ

P(ǫ|ηβ) + P(NONE|ηβ) = 1 Verification:

• ǫ

P(ǫ|πβ) = 1 where P(ǫ|ηβ) = P(τǫ|ηβ)P(λǫ|τǫ, λβ) and P(ǫ|πβ) = P(τǫ|πβ)P(λǫ|τǫ)

elementary tree ǫ prefix tree β prediction tree π tree structure τ integration point node η a tree’s head leaf λ

Frank Keller ANLP Guest Lecture 22

Introduction Prediction and Grammar Predictive Parsing Evaluation Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

Step 5: Linking Theory

The linking theory translates parser states into processing difficulty: elementary tree ǫwi is integrated with prefix tree βw1...wi−1; processing difficulty proportional to change in distribution P(β) from wi−1 to wi; each predicted tree π has a time-stamp t; at verification, decay d is calculated based on t (recently accessed structures are easier to integrate). Surprisal Dwi =

• − log
• βw1...wi

P(βw1...wi) + log

• βw1...wi−1

P(βw1...wi−1) − log

• π

P(π)(1−dtπ )

• Verification Cost

Frank Keller ANLP Guest Lecture 23

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

1

Introduction Incrementality Prediction

2

Prediction and Grammar Conceptual Issues Formalism Comparison with TAG Modeling Prediction

3

Predictive Parsing Treebank Conversion and Lexicon Induction Parsing Algorithm and Probability Model Linking Theory

4

Evaluation Parsing Performance Cognitive Plausibility

Frank Keller ANLP Guest Lecture 24

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Parsing Performance

Computational evaluation of PLTAG parser: train and test on standard Penn Treebank data (converted to PLTAG), with sentences of length 40 or less; assume gold-standard POS tags; use a supertagger to choose prediction trees (one word lookahead); coverage on the test set is not perfect: beam search; missing lexical entries.

Frank Keller ANLP Guest Lecture 25

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Results

Model Precision Recall F-score Coverage Baseline 44.39 52.38 48.06 85.10 PLTAG Parser 79.43 79.39 79.41 98.09 Prediction Tree Oracle 81.15 81.13 81.14 96.18 Baseline: pick most frequent tree (highest combined frequency of all subtrees). Oracle: assume correct prediction tree (instead of supertagging).

Frank Keller ANLP Guest Lecture 26

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Cognitive Plausibility

Psycholinguistic evaluation of PLTAG parser: train on Penn Treebank; take experimental materials from psycholinguistic experiments; parse them using the PLTAG parser, compute processing difficulty values for each sentence; compare to published reading time results. Baseline: standard surprisal model (PLTAG without prediction and verification component).

Frank Keller ANLP Guest Lecture 27

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Either . . . or Constructions

PLTAG model predicts difficulty in either . . . or constructions: (4) Peter read either a book or an essay in the school magazine. (5) Peter read a book or an essay in the school magazine.

Frank Keller ANLP Guest Lecture 28

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Relative Clause Asymmetry

Classic result in psycholinguistics: subject relative clauses are easier to process than object relative clauses:

(6) SRC: The reporter that attacked the senator admitted the error. (7) ORC: The reporter that the senator attacked admitted the error.

To be modeled: reading time differences on the relative clause verb and noun phrase (Staub 2010).

Frank Keller ANLP Guest Lecture 29

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Relative Clause Asymmetry

rel_pron src_vb det noun

• rc_vb

main_vb SRC ORC

Empirical Data (Staub 2010, Expt 1)

Word Reading Time in ms 100 200 300 400 500 600

*** *** ***

Frank Keller ANLP Guest Lecture 30

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Relative Clause Asymmetry

PLTAG model predicts difficulty profile correctly:

rel_pron src_vb det noun

• rc_vb

main_vb SRC ORC

Empirical Data (Staub 2010, Expt 1)

Word Reading Time in ms 100 200 300 400 500 600

*** *** ***

rel_pron src_vb det noun

• rc_vb

main_vb SRC ORC

Prediction Theory

Word Prediction Theory Difficulty Estimates 2 4 6 8 10 12 14

*** ***

Frank Keller ANLP Guest Lecture 30

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Relative Clause Asymmetry

Surprisal (Hale 2001) is not able to model this data correctly:

rel_pron src_vb det noun

• rc_vb

main_vb SRC ORC

Empirical Data (Staub 2010, Expt 1)

Word Reading Time in ms 100 200 300 400 500 600

*** *** ***

rel_pron src_vb det noun

• rc_vb

main_vb SRC ORC

Surprisal Predictions

Word Surprisal 2 4 6 8 10 12 14

***

Frank Keller ANLP Guest Lecture 31

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

Summary

Human sentence processing is incremental and predictive; evidence for lexical syntactic prediction (subcat frames); we presented a version of TAG that models these properties; the model comes with a parser, a probability model, and a linking theory; performance comparable to parsers with similar properties in the TAG literature; cognitive evaluation using experimental data: either . . . or prediction and relative clauses; also models reading time on the Dundee eyetracking corpus.

Frank Keller ANLP Guest Lecture 32

Introduction Prediction and Grammar Predictive Parsing Evaluation Parsing Performance Cognitive Plausibility

References

Altmann, Gerry T. M., and Yuki Kamide. 1999. Incremental interpretation at verbs: Restricting the domain of subsequent reference. Cognition 73: 247–264. Arai, Manabu, and Frank Keller. 2013. The use of verb-specific information for prediction in sentence processing. Language and Cognitive Processes 28: 525–560. Chiang, David. 2000. Statistical parsing with an automatically-extracted tree adjoining grammar. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, 463–470. Hong Kong. Demberg, Vera, Frank Keller, and Alexander Koller. 2013. Incremental, predictive parsing with psycholinguistically motivated tree-adjoining grammar. Computational Linguistics 39(4). In press. Hale, John. 2001. A probabilistic Earley parser as a psycholinguistic model. In Proceedings of the 2nd Conference

• f the North American Chapter of the Association for Computational Linguistics, vol. 2, 159–166. Pittsburgh,

PA: Association for Computational Linguistics. Staub, Adrian. 2010. Eye movements and processing difficulty in object relative clauses. Cognition 116: 71–86. Staub, Adrian, and Charles Clifton. 2006. Syntactic prediction in language comprehension: Evidence from either . . . or. Journal of Experimental Psychology: Learning, Memory, and Cognition 32: 425–436. Sturt, Patrick, Fabrizio Costa, Vincenzo Lombardo, and Paolo Frasconi. 2003. Learning first-pass structural attachment preferences with dynamic grammars and recursive neural networks. Cognition 88: 133–169. Sturt, Patrick, and Vincenzo Lombardo. 2005. Processing coordinated structures: Incrementality and

• connectedness. Cognitive Science 29(2): 291–305.

Xia, Fei, Martha Palmer, and Aravind Joshi. 2000. A uniform method of grammar extraction and its applications. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, 53–62. Hong Kong. Frank Keller ANLP Guest Lecture 33