Aut utom omat atic ic Cor orrecti ection on of of Adv dver - - PowerPoint PPT Presentation

Marie Garnier

CAS/IRIT, Université de Toulouse, France EuroCALL 2012, 22-25 August, Gothenburg

Aut utom

• mat

atic ic Cor

• rrecti

ection

• n of
• f Adv

dver erb Pl Plac acem emen ent t Er Error

• rs

s for

• r CAL

ALL

2

1.

Background



Focus on French speakers



Aspects of the project

2.

Predicting adverb placement



Grammaticality judgment tests

3.

Implementation



Evaluation: methods and results



Limits and strengths

4.

Corrective feedback



Design and implementation

1. . Ba Back ckground

• und

 Persistence of adverb placement errors at an intermediate

to advanced level

 Adverbs in V-Adv-O order are markedly more frequent, mostly

with learners with L1 French, Italian and Spanish

 Survey of 11 common grammar checkers: adverb placement

errors not detected

 Tricky grammatical issue

3

Osborne (2008), "Adverb placement in post-intermediate learner English: a contrastive study of learner corpora"

Focus on French speakers

 French/English: partial overlap between canonical positions for

adverbs as modifiers/adjuncts

 Possible positions for adverb lentement (Fr.) / slowly (Eng.):

 , elle  ouvrit  la porte .  , she  opened  the door .

 V-Adv-O accepted/necessary in some cases in English

(collocations, weight of object NP, syntax of AdvP): I understood very clearly the meaning of the speech you gave yesterday.

 May

result in negative transfer (Odlin, 1989) and

• vergeneralisation (Osborne, 2008)

4

Aspects of the project

 Objective: design automatic detection and correction strategies

for adverb placement errors in the written productions of intermediate to advanced users of English with L1 French

6

Manner adverbs as modifiers in the VP or adjuncts in the clause. Modelling of adverb placement Patterns and rewriting rules programmed in Prolog, using <TextCoop> Generation of tailored corrective feedback messages

• 2. Pred

edicting icting correct rect adverb erb pl placement cement

 Detecting errors and predicting correct placement means

understanding adverb placement in English

 Lack of definitive rules:  Multifactorial syntactic phenomenon (Garnier, 2012):

 semantic type of adverb and scope (VP/clause)  weight and composition of the AdvP (long/short adverb, very/too/more)  sentence structure (transitive/intransitive VP, information packaging

choices)

 prosody (prosodically integrated/detached adverbs)

7

Only rather broad and approximate flexible generalisations about adverb placement and sequence can be made. There is a great deal of variation in use, and features of context, style, prosody, and euphony play a role in some decisions.

Huddleston and Pullum (2002). CGEL, p. 576

Grammaticality judgment tests (1)

 Indicative study: 3 native English speakers (2 AmEng, 1 BrEng)  Objective: assess the influence of specific syntactic parameters

• n the placement of manner adverbs

 Parameters:  Weight and structure of the AdvP (long/short adverb, modification by

very, too, more, etc.)

 Presence and weight of verb complements (direct and indirect

• bjects)

 Verbs followed by prep. phrases  Presence of auxiliaries

8

Grammaticality judgment tests (2)

9

 56 sentences organized in 13 sets of 3 to 5 sentences

representing all possible positions for an adverb in a test sentence

 NS asked to identify all correct placements + choose the most

natural-sounding one

 Results:

"Correct": complete agreement reached in only 36% of cases "Best": complete agreement reached in 69% of cases

3.

• 3. Im

Implem emen enta tati tion

• n

10

 Creation of detection patterns and rewriting rules  22 detection patterns associated with 1 to 3 rewriting rules

System can issue several propositions when more than

• ne correction is possible

 Resources:

 Lexicons: adverbs, aux., verbs, prep., adjectives, nouns, determ.  Grammars: Noun Phrase, Adjective Phrase

11

Error She must remove carefully the bandage. Detection pattern NP1 (Aux) (Aux) Vlex (Prep) ADV NP2 Rewriting rules

• 1. NP1 (Aux) (Aux) ADV Vlex (Prep) NP2
• 2. NP1 (Aux) (Aux) Vlex (Prep) NP2 ADV

Corrected sentence 1. She must carefully remove the bandage.

• 2. She must remove the bandage carefully.

Table 1. Schematized example of a pattern/rewriting rule

12

 Patterns and rules implemented within <TextCoop>  <TextCoop>: discourse analysis platform programmed in

Prolog (Saint-Dizier, 2011)

 Logic-based programming means model can be implemented

directly

forme(corr-adva, [plus(aux(AUX,_)), verb(V,_), opt(prep(P1,_)), adv(ADV,manner),

• pt(prep(P2,_)), np(NP,_)],

[conc(VE1,E2,E1), conc(NP11,S,E5), card(NP11, T), (T < 5)], ['<erreur3a>', AUX, VE1, P1, ADV, P2, NP11, '</erreur>',' <correct>', AUX, ADV, VE1, P1, P2, NP11, '</correct>']). forme(corr-advb, [plus(aux(AUX,_)), verb(V,_), opt(prep(P1,_)), adv(ADV,manner),

• pt(prep(P2,_)), np(NP,_)],

[conc(VE1,E2,E1), conc(NP11,S,E5), card(NP11, T), (T < 5)], ['<erreur3b>', AUX, VE1, P1, ADV, P2, NP11, '</erreur>',' <correct>', AUX, VE1, P1, P2, NP11, ADV,'</correct>']).

Table 2. Example of a pattern/rewriting rule in <TextCoop>

Evaluation: Method (1)

14

 Evaluation on an 80,000-word corpus of native English (British

and American online newspapers, British and American blog posts, scientific publications)

 279 occurrences of manner adverbs  False positives = < 4%  Main cause: words belonging to several grammatical categories

(ex. check n./v., use n./v.)

Evaluation: Method (2)

15

 Evaluation on a modified 20,000-word corpus of learner and

user English (learner productions from ICLE and personal emails)

 The availability of appropriate learner/user corpora for testing is

• ne of the challenges of research on automatic grammar-

checking (Foster and Andersen, 2009): A proficient English user with L1 French was asked to introduce manner adverbs in authentic texts, producing correct and incorrect sentences

Evaluation: Results

16

 2nd proposition: 70% precision  Simplest patterns (i.e. adverb without modification in finite sentences

with NPs of less than 5 words) are used in 83% of all cases

 Among them, V-Adv-O patterns used in 48% of all cases  Patterns are mutually exclusive (no conflicts)

• 1. Detection

precision = 96% recall = 83%

• 2. Correction

precision = 87% recall = 80%

Limits of the system

17

 Number of syntactic configurations that can be recognized:

e.g. embedded that-clauses, ditransitive VPs

 Words belonging to more than one category:

e.g. check, use, purchase, being, etc.

 Adverb with more than one semantic type:

e.g. He looked at her sadly. (= manner) Sadly, they were arguing about money again. (= act-related)

 Requires mostly grammatical input, but:

 Intermediate users/learners: overall sentence structure is usually

mastered

 Patterns use categories, which allows for a margin of error in the

input text

Strengths of the system

18

 Directly portable to other romance languages (Italian, Spanish)  Architecture is adaptable to any kind of L1, with specific

alterations

 Creation of new patterns and new feedback messages is easy

and can be done by linguists or teachers with little to no programming skills

 Design of specific feedback for each error type is simplified

4.

• 4. Cor
• rrec

ecti tive e fee eedb dbac ack

19

 Five key steps:

• 1. Error marking
• 2. Error diagnosis
• 3. Metalinguistic feedback
• 4. Remediation
• 5. Illustration

Heift (2004), Heift and Schulze (2007): metalinguistic feedback combined with highlighting has positive effects on learning Garnier (2011): review of feedback messages included in commercial/research grammar checkers; found to be often inconsistent and not adaptable to users

20

In this sentence, the manner adverb [separately] is placed between the verb [tested] and the object [all the parameters]. In English, manner adverbs are not generally found between a verb and a following object. There are 2 ways to correct this sentence:

• By placing the adverb after the object;
• By placing the adverb between the subject

and the verb. Examples:

• She unwrapped the present carefully
• She carefully unwrapped the present.
• 1. Error marking
• 2. Error diagnosis
• 3. Metalinguistic FB
• 4. Remediation
• 5. Illustration

We have tested separately all the parameters

21

 Can be adapted to the profile of the user:

 Learner: looking for help with her/his learning  steps 1, 2, 3, 4, 5 without the correction, in order to elicit

self-correction

 Curious: looking for more information on the error or the

construction in general

 steps 1, 2, 3, 4, 5  Careful: wants to check why the segment was flagged as an

error

 steps 1, 2, 5  Rushed: only wants to see the error and its correction(s)  step 1

Implementation of feedback messages

22

 Profile is selected in advance by the user of the system  Feedback is displayed on an html page when user clicks on the

error segment

 Specific messages for each pattern + include the actual adverb,

NP or verb used in the text

Futu uture e work

• rk

23

 Complete implementation of feedback messages and conduct

functional evaluation

 Evaluate usability and adequacy of feedback system with

English learners

 Creation of a resource for adverbs: identification of major

semantic types associated with a particular adverb, using WordNet and other resources

 Extension of the system to other errors :

 Use of also  word-order errors in the NP, i.e. N+N constructions

 Integration in a tutorial/grammar-checking system

Ref eferenc erences es

24

Foster, Jennifer and Oistein Andersen (2009). "GenERRate: Generating errors for use in grammatical error detection". Actes du workshop ACL Use of NLP for building educational applications. Garnier, Marie. (2011). "Explanation and corrective feedback in grammar checking systems". Proceedings of the 6th International ExACt workshop at IJCAI'11, 81-90. Garnier, Marie. (2012). "Automatically correcting adverb placement errors in the writings of French users of English". Procedia – Social and Behavioral Sciences 34, 59-63. Amsterdam: Elsevier. Granger, Sylviane and Estelle Dagneaux, Fanny Meunier and Magali Paquot, éds. (2009). International Corpus of Learner English v.2. Louvain La Neuve : Presses Universitaires de Louvain. Heift, Trude. (2004). "Corrective feedback and learner uptake in CALL". ReCALL, 16:2, 416- 431. Heift, Trude and Mathias Schulze. (2007). Errors and Intelligence in Computer-Assisted Language Learning: Parsers and Pedagogues. Routledge, New York. Huddleston, Rodney and Geoffrey Pullum (2002). The Cambridge Grammar of the English

• Language. Camdridge : CUP.

Meurers, Detmar and Vanessa Metcalf (2006), "Towards a treatment of word-order errors in CALL: When to use deep processing and when not to". Large-Scale Grammar Development and Grammar Engineering Research Workshop of the Israel Science Foundation Odlin, Terence (1989). Language Transfer: Cross-linguistic influence in language learning. New York: Cambridge University Press. Osborne, John (2008). "Adverb placement in post-intermediate learner English: a contrastive study of learner corpora". Linking up Contrastive Linguistics and Interlanguage Research, Gaëtanelle Gilquin, Szilvia Papp, María Belén Díez-Bedmar. Londres : Rodopi. Saint-Dizier, Patrick (2011). "TextCoop : un analyseur de discours basé sur des grammaires logiques". Actes du colloque TALN.