Thoughts on Learner Data and Motivation Learner Language - - PowerPoint PPT Presentation

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Universit¨ at T¨ ubingen SFB 833, Project A4 Second T¨ ubingen-Berlin Meeting on Analyzing Learner Language T¨ ubingen, 5./6. Dezember 2011

1 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Overview

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example Conclusion

2 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

General Motivation

Why dependency parsing?

◮ Focus on lexical dependency structure as an interface

to interpretation. → CoMiC project compares meaning

• f answers to reading comprehension questions

◮ At the same time, to characterize the nature of learner

language, capturing morphosyntactic dependencies can also be an important goal (→ SLA research)

3 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Dependency Parsing in the CoMiC Project (I)

Comparing Meaning in Context

◮ The CoMiC project investigates how the meaning of

student answers can be compared to the meaning of target answers in reading comprehension exercises.

◮ Data: Corpus from German classes in the US, Ohio

State University (Prof. Kathryn Corl), University of Kansas (prof. Nina Vyatkina).

◮ Target answers and student answers are compared with

respect to meaning, not form.

◮ Trying to detect automatically: Did the student answer

the question correctly or not?

◮ We want to parse German learner language

automatically with dependency parsers.

◮ These data are not annotated with errors or target

hypotheses.

4 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Dependency Parsing in the CoMiC Project (II)

Comparing Meaning in Context

◮ Our experimental system CoMiC-DE performs meaning

comparison on many levels, beginning from simple token overlap.

◮ So far, our most sophisticated level of linguistic

representation is based on Lexical Resource Semantics (LRS, Richter & Sailer 2003).

◮ Hahn & Meurers (2011) present an approach to the

construction LRS representation from dependency structures.

◮ Naturally, we need well-behaved dependency structures

• f our learner data in order to construct good LRS

representations.

◮ Furthermore, we use dependency triples directly in the

system.

5 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Learner Language and Dependency Annotation

◮ Ott & Ziai (2010) trained a statistical dependency parser

• n a dependency-converted version of the T¨

uba-D/Z treebank and used it to parse learner language.

◮ CREG109, data set with 109 manually student answers ◮ We are currently working on an extended data set

containing more data and questions and target answers.

◮ Annotation scheme used: the one described by Foth

(2006).

6 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Abusing Annotation Schemes

◮ The dependency annotation scheme by Foth (2006)

has not been designed for learner language.

◮ Hence, we are using an annotation scheme that simply

cannot handle many constructions in the learner data.

◮ What are possible solutions to this issue?

• 1. Annotating interlanguage as a system in its own right

using a special annotation scheme (Dickinson & Ragheb 2009).

• 2. Annotating target hypotheses that map to well-formed

language and annotate these (or parse: Hirschmann et al. 2010).

7 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Annotating/Parsing Interlanguage

◮ So, if we would stick to interlanguage as a system in its

• wn right?

◮ Interlanguage is influenced by many learner-dependent

factors (stage of acquisition, L1, background, etc). ➥ Difficult to capture in a general parsing model.

8 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Aside: Isn’t it only a robustness issue?

◮ Arguably, robust tools should be able to deal with

learner language to some extent.

◮ Foster (2007) automatically ‘damaged’ the Penn

Treebank with simulated learner errors and trained a parser on it to achieve more error-tolerance.

◮ Still, this does not solve the problem of abusing an

annotation scheme.

◮ Possibly, there is a difference between learner levels

◮ Very advanced learners will be close to native speakers,

so using native language categories might still be OK.

◮ In our data, we have many beginners and intermediate

learners that produce language that often is impossible to treat with native language categories.

◮ Robustness is good for us but robustness alone does

not help us.

9 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Annotating/Parsing Target Hypotheses

◮ Using target hypotheses seems appealing in our

situation.

◮ Standard NLP tools could be used in the tool chain,

since we would have well-behaved language back again.

◮ Still, we would have an explicit mapping back to the

• riginal learner data.

◮ However, we do not want to annotate target hypotheses.

◮ Our corpus is large, it would be a lot of work. ◮ It would not be applicable to tutoring systems that aim

at giving feedback on unseen learner data.

◮ Target hypotheses in the sense of Falko’s ZH1

(Reznicek et al. 2010) would be great, but also they would provide more than we need.

10 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Let’s do it Automatically?!

Can we create target hypotheses automatically?

◮ It seems quite impossible, since humans put a lot of

thinking into creating these.

◮ But perhaps we can create approximated target

hypotheses, that provide enough wellformedness for

• ur tools.

11 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Example: Missing Verbs (I)

(From Ott & Ziai 2010) 33,9 33.9 Prozent, percent, die who ¨ uber

• ver

25 25 Jahre years alt

• ld

[sind], [are], sind are M¨ anner. men.

→ Without the verb as a functor, it is hard for annotators and

parsers to attach the dependency relations to a token for which it makes sense.

12 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Example: Missing Verbs (II)

◮ Missing verbs let us lose the main functor-argument

relations in the sentence.

◮ Detecting missing verbs and inserting a dummy verb

could help.

◮ This dummy verb would have no lexical form and no

meaning.

◮ Parsers would have to be trained to work properly with

such underspecified tokens.

◮ A dummy verb would still allow for an almost-complete

analysis, also in the automatically constructed LRS representation!

13 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsing with Approximated Target Hypotheses

◮ A component for constructing approximated target

hypotheses needs to be invented.

◮ As a starting point, we can focus on missing tokens, this

will be difficult enough. ➥ Insertion of dummy tokens that provide function, but not meaning.

◮ Inspired by Foster (2007), we could systematically

sprinkle these dummy tokens into a treebank such as T¨ uba-D/Z in order to train a robust parser to work on it.

➥ Parsing approximated target hypotheses with a native

language dependency scheme, combined with robustness.

14 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Comparing Rule-based and Data-Driven Parsing of Learner Language

(Krivanek & Meurers 2011)

◮ Two dependency parsing approaches

◮ Rule-based: WCDG Parser (Foth & Menzel 2006) ◮ Data-driven: MaltParser (Nivre et al. 2007) 15 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Comparing Rule-based and Data-Driven Parsing of Learner Language

(Krivanek & Meurers 2011)

◮ Two dependency parsing approaches

◮ Rule-based: WCDG Parser (Foth & Menzel 2006) ◮ Data-driven: MaltParser (Nivre et al. 2007)

◮ Two test corpora:

◮ Native language: German dependency treebank

derived from T¨ uBa-D/Z (Telljohann et al. 2004)

◮ Learner language: CREG-109 (Ott & Ziai 2010) 15 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Comparing Rule-based and Data-Driven Parsing of Learner Language

(Krivanek & Meurers 2011)

◮ Two dependency parsing approaches

◮ Rule-based: WCDG Parser (Foth & Menzel 2006) ◮ Data-driven: MaltParser (Nivre et al. 2007)

◮ Two test corpora:

◮ Native language: German dependency treebank

derived from T¨ uBa-D/Z (Telljohann et al. 2004)

◮ Learner language: CREG-109 (Ott & Ziai 2010)

◮ Two types of grammatical relations:

◮ Argument relations: obligatory, linguistic knowledge ◮ Adjunct relations: optional, relevance of world knowledge 15 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Hypothesis about impact of parsing method

◮ Rule-based WCDG relies on a hand-written lexicon.

◮ Hand-written, information-rich lexicon encodes

knowledge about linguistic relations.

16 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Hypothesis about impact of parsing method

◮ Rule-based WCDG relies on a hand-written lexicon.

◮ Hand-written, information-rich lexicon encodes

knowledge about linguistic relations. → Rule-based approach will fare better in identifying the core functor-argument relations.

16 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Hypothesis about impact of parsing method

◮ Rule-based WCDG relies on a hand-written lexicon.

◮ Hand-written, information-rich lexicon encodes

knowledge about linguistic relations. → Rule-based approach will fare better in identifying the core functor-argument relations.

◮ The statistical MaltParser is trained on annotated corpora.

◮ Corpora encode a combination of language competence,

language use, and facts about the world.

16 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Hypothesis about impact of parsing method

◮ Rule-based WCDG relies on a hand-written lexicon.

◮ Hand-written, information-rich lexicon encodes

knowledge about linguistic relations. → Rule-based approach will fare better in identifying the core functor-argument relations.

◮ The statistical MaltParser is trained on annotated corpora.

◮ Corpora encode a combination of language competence,

language use, and facts about the world. → Data-driven approach will fare better in identifying adjunct relations.

➥ Test hypotheses in parsing experiments.

16 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsing experiments

Corpora used: Learner corpus CREG (Meurers, Ott & Ziai 2010)

◮ Corpus of Reading Comprehension Exercises in German

◮ answers to reading comprehension questions ◮ written by US college students at the beginner and

intermediate levels of German programs

17 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsing experiments

Corpora used: Learner corpus CREG (Meurers, Ott & Ziai 2010)

◮ Corpus of Reading Comprehension Exercises in German

◮ answers to reading comprehension questions ◮ written by US college students at the beginner and

intermediate levels of German programs

◮ CREG-109 is a subset manually annotated using

dependency annotation scheme of Foth (2006).

◮ 109 sentences (sentence length: avg. 8,26, max. 17) ◮ 17 of those are ungrammatical ◮ errors in word order, agreement, and case government ◮ dependencies were annotated on a grammatical target

hypothesis (with lexical mappings to the learner tokens)

17 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

CREG Example

Was sind die Kritikpunkte, die Leute ¨ uber Hamburg ¨ außern? ‘What are the objections people have about Hamburg?’ TA: (Reading comprehension text) Der The Gestank stink von

• f

Fisch fish und and Schiffsdiesel fuel an

• n

den the Kais quays . . SA: Der The Geruch smell zon

• ferr

Fish fisherr und and Schiffsdiesel fuel beim at the Hafen port . . Q: T:

18 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsing experiments

Corpora used: Native language corpus

◮ T¨

uBa-D/Z treebank (Ver. 5; Telljohann et al. 2004)

◮ newspaper text ◮ converted to dependency annotation of Foth (2006)

using script from Versley (2005)

◮ 10 % (4142 sentences) as test set ◮ 90 % used as training set for MaltParser 19 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsing experiments

Corpora used: Native language corpus

◮ T¨

uBa-D/Z treebank (Ver. 5; Telljohann et al. 2004)

◮ newspaper text ◮ converted to dependency annotation of Foth (2006)

using script from Versley (2005)

◮ 10 % (4142 sentences) as test set ◮ 90 % used as training set for MaltParser

◮ For comparability, both CREG-109 and T¨

uBa-D/Z test corpora were automatically pos-tagged with TnT tagger

(Brants 2000) using the STTS tagset (Thielen et al. 1999)

19 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsers used

◮ MaltParser: state-of-the-art system for transition-based

dependency parsing (Nivre et al. 2007)

20 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Parsers used

◮ MaltParser: state-of-the-art system for transition-based

dependency parsing (Nivre et al. 2007)

◮ WCDG: Weighted Constraint Dependency parsing for

German (Foth et al. 2005; Foth & Menzel 2006)

◮ hand-written grammar, with some other components: ◮ heuristic search option (frobbing) ◮ some stochastic predictor components (chunker,

supertagger, probabilistic shift-reduce parser)

◮ efficiency remains an issue 20 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Overall Scores (using eval.pl, Buchholz & Marsi 2006)

WCDG LAS UAS T¨ uBa-D/Z 81.42% 85.71% CREG-109 79.28% 86.36% MaltParser LAS UAS T¨ uBa-D/Z 84.04% 87.25% CREG-109 78.12% 84.56%

◮ overall results quite similar for both parsers

21 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Overall Scores (using eval.pl, Buchholz & Marsi 2006)

WCDG LAS UAS T¨ uBa-D/Z 81.42% 85.71% CREG-109 79.28% 86.36% MaltParser LAS UAS T¨ uBa-D/Z 84.04% 87.25% CREG-109 78.12% 84.56%

◮ overall results quite similar for both parsers

◮ WCDG: robust parsing of learner language 21 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Overall Scores (using eval.pl, Buchholz & Marsi 2006)

WCDG LAS UAS T¨ uBa-D/Z 81.42% 85.71% CREG-109 79.28% 86.36% MaltParser LAS UAS T¨ uBa-D/Z 84.04% 87.25% CREG-109 78.12% 84.56%

◮ overall results quite similar for both parsers

◮ WCDG: robust parsing of learner language ◮ MALT better for native language (which it was trained on) 21 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

◮ parallel observation also holds for Maltparser results 22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

◮ parallel observation also holds for Maltparser results

◮ Hypothesis: This 2.79% higher drop for CREG-109 may

result from ungrammatical learner sentences.

22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

◮ parallel observation also holds for Maltparser results

◮ Hypothesis: This 2.79% higher drop for CREG-109 may

result from ungrammatical learner sentences.

◮ Manually inspected the 53 cases (7.08%) where the

parser assigned correct relations but false labels

22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

◮ parallel observation also holds for Maltparser results

◮ Hypothesis: This 2.79% higher drop for CREG-109 may

result from ungrammatical learner sentences.

◮ Manually inspected the 53 cases (7.08%) where the

parser assigned correct relations but false labels

◮ 21 resulted from ungrammaticality 22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop from UAS to LAS

WCDG LAS UAS UAS − LAS T¨ uBa-D/Z 81.42% 85.71% 4.29% CREG-109 79.28% 86.36% 7.08% = 2.79% diff.

◮ 7.08% drop from UAS to LAS for CREG-109 (learner),

but only 4.29% drop for T¨ uBa-D/Z (native)

◮ parallel observation also holds for Maltparser results

◮ Hypothesis: This 2.79% higher drop for CREG-109 may

result from ungrammatical learner sentences.

◮ Manually inspected the 53 cases (7.08%) where the

parser assigned correct relations but false labels

◮ 21 resulted from ungrammaticality = 2.8% ! 22 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Investigating the drop between UAS and LAS

Example: mixed up subject & object due to agreement error (1) Seine hispl Eltern parentspl hat hassg BA BAsg geholfen. helped

intended: His parents have helped BA.

[WCDG parse]

23 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

CREG-109 results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 95 84.21 80.00 87.37 86.46 OBJA 52 65.38 70.83 75.00 75.00 PRED 26 61.54 69.57 57.69 83.33 AUX 23 60.87 87.50 73.91 94.44 Modifier relations ADV 44 65.91 56.86 65.91 48.33 PP 32 75.00 55.81 71.88 43.40

24 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

CREG-109 results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 95 84.21 80.00 87.37 86.46 OBJA 52 65.38 70.83 75.00 75.00 PRED 26 61.54 69.57 57.69 83.33 AUX 23 60.87 87.50 73.91 94.44 Modifier relations ADV 44 65.91 56.86 65.91 48.33 PP 32 75.00 55.81 71.88 43.40

24 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

CREG-109 results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 95 84.21 80.00 87.37 86.46 OBJA 52 65.38 70.83 75.00 75.00 PRED 26 61.54 69.57 57.69 83.33 AUX 23 60.87 87.50 73.91 94.44 Modifier relations ADV 44 65.91 56.86 65.91 48.33 PP 32 75.00 55.81 71.88 43.40

24 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

T¨ uBa-D/Z results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 5408 83.54 87.05 89.00 89.64 OBJA 2658 75.43 72.96 79.83 82.15 PRED 1044 66.48 71.77 60.82 76.51 AUX 2236 85.73 89.41 91.77 96.11 Modifier relations ADV 5115 78.92 77.78 69.72 64.13 PP 5562 71.88 72.26 69.67 62.92

25 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

T¨ uBa-D/Z results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 5408 83.54 87.05 89.00 89.64 OBJA 2658 75.43 72.96 79.83 82.15 PRED 1044 66.48 71.77 60.82 76.51 AUX 2236 85.73 89.41 91.77 96.11 Modifier relations ADV 5115 78.92 77.78 69.72 64.13 PP 5562 71.88 72.26 69.67 62.92

25 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

T¨ uBa-D/Z results by dependency types

MaltParser WCDG Label # Recall Prec. Recall Prec. Argument relations SUBJ 5408 83.54 87.05 89.00 89.64 OBJA 2658 75.43 72.96 79.83 82.15 PRED 1044 66.48 71.77 60.82 76.51 AUX 2236 85.73 89.41 91.77 96.11 Modifier relations ADV 5115 78.92 77.78 69.72 64.13 PP 5562 71.88 72.26 69.67 62.92

25 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

WCDG: robust parsing of subjectless sentences

(2) Vielleicht perhaps adoptieren adoptplur ein a Kind. childsing

26 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

WCDG: robust parsing of subjectless sentences

(2) Vielleicht perhaps adoptieren adoptplur ein a Kind. childsing

MaltParser: object attached as subject

26 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

WCDG: robust parsing of subjectless sentences

(2) Vielleicht perhaps adoptieren adoptplur ein a Kind. childsing

MaltParser: object attached as subject WCDG parser: subjectless analysis

26 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Conclusion I

◮ Strengths of different information included in parser:

◮ hand-written lexical information (WCDG) helpful in

identifying obligatory functor-argument relations

◮ world knowledge in corpora helps data-driven parsers

perform well on optional adjunct relations

◮ For parsing learner language, several levels of

dependency analysis are probably needed

◮ robust analysis glossing over learner language

specifics, as a step towards meaning How can one make the target hypotheses explicit that underlies this, in an automated process from data to interpretation?

◮ analysis identifying specific learner language evidence 27 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

Conclusion II

Concerning robust parsing of learner language, we will

◮ annotate a sample of CREG with target hypotheses ◮ investigate to what extent an approximated target

hypothesis could be generated automatically

◮ ask the Berlin crowd how annotation on manual target

hypotheses can be mapped back to learner language.

28 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

References I

Brants, T. (2000). TnT – A Statistical Part-of-Speech Tagger. In Proceedings of the Sixth Conference on Applied Natural Language Processing. Association for Computational Linguistics, pp. 224–231. Buchholz, S. & E. Marsi (2006). CoNLL-X shared task on multilingual dependency parsing. In CoNLL-X ’06: Proceedings of the Tenth Conference on Computational Natural Language Learning. Morristown, NJ, USA: Association for Computational Linguistics, pp. 149–164. Dickinson, M. & M. Ragheb (2009). Dependency Annotation for Learner Corpora. In Proceedings of the Eighth Workshop on Treebanks and Linguistic Theories (TLT-8). Milan, Italy. Foster, J. (2007). Treebanks gone bad: Parser evaluation and retraining using a treebank of ungrammatical

• sentences. International Journal on Document Analysis and Recognition 10(3–4), 129–145.

Foth, K. (2006). Eine umfassende Constraint-Dependenz-Grammatik des Deutschen. Tech. rep., Universit¨ at Hamburg. Foth, K., W. Menzel & I. Schr¨

• der (2005). Robust parsing with weighted constraints. Natural Language

Engineering 11(01), 1–25. Foth, K. A. & W. Menzel (2006). Hybrid parsing: using probabilistic models as predictors for a symbolic parser. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Stroudsburg, PA, USA: Association for Computational Linguistics, ACL-44, pp. 321–328. Hahn, M. & D. Meurers (2011). On deriving semantic representations from dependencies: A practical approach for evaluating meaning in learner corpora. In Proceedings of the Intern. Conference on Dependency Linguistics (DEPLING 2011). Barcelona. Hirschmann, H., A. L¨ udeling, I. Rehbein, M. Reznicek & A. Zeldes (2010). Syntactic Overuse and Underuse: A Study of a Parsed Learner Corpus and its Target Hypothesis. Presentation given at the Treebanks and Linguistic Theory Workshop. Krivanek, J. & D. Meurers (2011). Comparing Rule-Based and Data-Driven Dependency Parsing of Learner

• Language. In Proceedings of the Intern. Conference on Dependency Linguistics (DEPLING 2011).

Barcelona. Meurers, D., N. Ott & R. Ziai (2010). Compiling a Task-Based Corpus for the Analysis of Learner Language in

• Context. In Pre-Proceedings of Linguistic Evidence. T¨

ubingen, pp. 214–217. 29 / 30

Thoughts on Learner Data and Dependency Parsing

Niels Ott, Ramon Ziai, Julia Krivanek, Detmar Meurers

Introduction and Motivation Learner Language and Dependency Annotation Approximated Target Hypotheses Rule-Based vs. Data-Driven

Hypothesis Corpora used Parsers used Overall Results Drop between UAS & LAS Results by dependency type A subjectless example

Conclusion References

SFB 833

References II

Nivre, J., J. Nilsson, J. Hall, A. Chanev, G. Eryigit, S. K¨ ubler, S. Marinov & E. Marsi (2007). MaltParser: A Language-Independent System for Data-Driven Dependency Parsing. Natural Language Engineering 13(1), 1–41. Ott, N. & R. Ziai (2010). Evaluating Dependency Parsing Performance on German Learner Language. In

• M. Dickinson, K. M¨

u¨ urisep & M. Passarotti (eds.), Proceedings of the Ninth International Workshop on Treebanks and Linguistic Theories. vol. 9 of NEALT Proceeding Series, pp. 175–186. Reznicek, M., M. Walter, K. Schmid, A. L¨ udeling, H. Hirschmann & C. Krummes (2010). Das Falko-Handbuch. Korpusaufbau und Annotationen Version 1.0. Richter, F. & M. Sailer (2003). Basic Concepts of Lexical Resource Semantics. In A. Beckmann & N. Preining (eds.), ESSLLI 2003 – Course Material I. Wien: Kurt G¨

• del Society, vol. 5 of Collegium Logicum, pp.

87–143. Telljohann, H., E. Hinrichs & S. K¨ ubler (2004). The T¨ uBa-D/Z Treebank: Annotating German with a Context-Free Backbone. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004). Lissabon. Telljohann, H., E. W. Hinrichs, S. K¨ ubler, H. Zinsmeister & K. Beck (2009). Stylebook for the T¨ ubingen Treebank of Written German (T¨ uBa-D/Z). Tech. rep., Seminar f¨ ur Sprachwissenschaft, Universit¨ at T¨ ubingen, Germany. Thielen, C., A. Schiller, S. Teufel & C. St¨

• ckert (1999). Guidelines f¨

ur das Tagging deutscher Textkorpora mit

• STTS. Tech. rep., Institut f¨

ur Maschinelle Sprachverarbeitung Stuttgart and Seminar f¨ ur Sprachwissenschaft T¨ ubingen. Versley, Y. (2005). Parser Evaluation across Text Types. In Proceedings of the Fourth Workshop on Treebanks and Linguistic Theories (TLT). Barcelona, Spain. 30 / 30