using measures of linguistic complexity to assess german
play

Using Measures of Linguistic Complexity to Assess German L2 - PowerPoint PPT Presentation

Dec 07, 2023 •283 likes •1.31k views

Using Measures of Linguistic Complexity to Assess German L2 Proficiency in Learner Corpora under Consideration of Task-Effects Zarah Wei Eberhard Karls Universitt Tbingen Kolloquium Korpuslinguistik und Phonetik (SS17) Humboldt

  1. Merlin Tasks Task Test � A1 A2 B1 B2 C1 C2 Going swimming A1 56 8 45 3 0 0 0 Apartment search A1 77 11 50 0 0 0 16 Child birth A1 74 25 41 8 0 0 0 Ticket offer A2 66 5 28 31 2 0 0 Pet sitting A2 72 0 32 40 0 0 0 Housing office A2 70 4 43 22 1 0 0 Announce visit B1 67 2 31 29 5 0 0 Happy birthday B1 70 0 24 38 8 0 0 Happy new year B1 73 2 10 54 7 0 0 Application B2 69 0 1 22 42 4 0 Work complaint B2 70 0 1 20 47 2 0 Information request B2 65 0 0 24 41 0 0 Housing situation C1 72 0 0 7 52 13 1 Learning German C1 42 0 0 1 26 15 2 Traditions & Assimilation C1 90 0 0 16 62 12 1 Table: Mapping of tasks to test levels, task frequency, and their distribution across overall proficiency scores (A1 to C2).

  2. Falko Georgetown In a Nutshell • Partially longitudinal corpus of 209 German L2 writings by 123 students • Elicited in curricular writing courses at Georgetown University by Falko Georgetown Dokumentation 2007; Reznicek et al. 2007 • Course levels 1 to 4 for intermediate to advanced learners of German • No external validation of proficiency besides course levels Figure: Texts written by learners who contributed multiple writings to Falko

  3. Falko Georgetown Tasks � Task Level 1 Level 2 Level 3 Level 4 Write a letter 21 21 0 0 0 Continue a novel 28 0 28 0 0 Write an article 28 0 0 28 0 Write a speech 16 0 0 0 16 Book review 116 19 25 23 49 Table: Frequency of tasks across course levels in the Falko Georgetown L2 corpus.

  4. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  5. Overview • Annotate 15 cognitive and functional task factors • Goal: disentangle correlation of course / test level and task • Based on task descriptions in supplementary material Falko Georgetown Dokumentation 2007; Merlin project 2014a,b,c,d,e,f,g,h,i,j,k,l,m,n,o • Follow approach by Alexopoulou et al. 2017

  6. Operationalizations Cognitive Factors Code complexity: instructions provided no, few, or detailed language material to draw from Cognitive complexity: require reasoning about writing structure vs. outline or refer to known structure Shared context temporal and spatial dislocation (here/there; now/then) Reasoning demands: quantity and elaborateness of spatial reasoning, i.e. referencing a location without extra-linguistic support, and reasoning about other people’s intentions, beliefs, desires, or relationships Referenced elements: number of discourse referents minimally required in solution Perspective requires perspective of i) self, ii) psomeone else; iii) multiple other people.

  7. Operationalizations Functional Factors Genre text category; cf. task descriptions Audience recipient; cf. task descriptions (partially grouped) Formality tone; cf. task descriptions or inferred from genre and audience Task theme general topic; professional/occupational interests, public social affairs, small talk, or (by extension) goal-oriented personal matters ( demand ) Task type determined by a combination of functional needs and genre; argumentative, narrative, descriptive/expositional, and instructional

  8. Tasks in Merlin Task Test Time Expected Genre Audience Formality Theme Task Type Code Cognitive Shared Reasoning Referenced Perspective Level Words Complexity Complexity Context Elements Going swimming A1 45 Min. 30 Email Friend Informal Demand Descriptive High Low T & T Low Few Own Apartment search A1 45 Min. 30 Email Friend Informal Demand Instructive High Low H & N Low Few Own Child birth A1 45 Min. 30 Letter Friend Informal Small talk Descriptive High Low H & N Low Few Own Ticket offer A2 50 Min. 40 Letter Friend Informal Demand Instructional High Low H & N Medium Few Own Pet sitting A2 50 Min. 40 Email Friend Informal Demand Instructional High Low H & N Medium Few Own Housing office A2 50 Min. 40 Letter Agency Informal Demand Descriptive High Low H & N Low Few Own Announce visit B1 30 Min. 128 Letter Friend Informal Small talk Narrative Low Low H & N Low Many Own Happy birthday B1 30 Min. 128 Letter Friend Informal Small talk Narrative Low Low H & N Medium Many Own Happy new year B1 30 Min. 128 Letter Friend Informal Small talk Narrative Low Low T & T Medium Many Own Application B2 30 Min. 150 Letter Agency Formal Profession Argumentative High Medium H & N Medium Few Own Work complaint B2 30 Min. 150 Letter Agency Formal Profession Argumentative High Medium T & T Medium Many Own Information request B2 30 Min. 150 Letter Agency Formal Profession Descriptive High Low H & N Medium Few Own Housing situation C1 60 Min. 200 Essay Public Formal Society Descriptive High High H & N Medium Open Own & others Learning German C1 60 Min. 200 Essay Public Formal Society Argumentative High High H & N High Open Own & others Traditions & C1 60 Min. 200 Essay Public Formal Society Argumentative High High H & N High Open Own & others Assimilation Table: Properties and task factors annotated for Merlin tasks.

  9. Tasks in Falko Georgetown Task Test Audience Genre Formality Theme Task Type Code Cognitive Shared Reasoning Referenced Perspective Level Complexity Complexity Context Elements Write a letter 1 Friend Letter Informal Small talk Instructional High Low T & T Low Few Own Continue a novel 2 Public Novel Informal Mystery Narrative Low Low T & T Medium Few Other Write an article 3 Public Article Formal Society Descriptive Low High T & T Medium Many Own & others Write a speech 4 Public Speech Formal Society Argumentative Low Low T & T High Open Own & others Book review 1-4 Public Review Formal Society Argumentative High High T & T High Open Own & others Table: Properties and task factors annotated for Falko Georgetown L2 tasks.

  10. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  11. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  12. Overview • 398 measures of elaborateness and variedness of various domains • Extracted automatically using elaborate NLP tool chain • Written by Galasso 2014; Hancke 2013; Weiß 2015, 2017 • Domains: 1 Language use 2 Human language processing 3 Discourse & encoding of meaning 4 Theoretical linguistics (lexico-semantics, syntax, morphology)

  13. Pipeline Figure: System pipeline from plain text corpus to feature analysis.

  14. Resources Task Component Version Model � Tokenization and OpenNLP 1.6.0 default sentence segmentation  POS tagging   Lemmatization  3.6.0 default Mate tools Morphological analysis   Dependeny parsing  Compound splitting JWordSplitter 3.4.0 default Constituency parsing Stanford PCFG parser 3.6.0 default Topological field parsing Berkeley parser 1.7.0 cf. Ramon Ziai Table: NLP components used in the complexity analysis pipeline.

  15. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  16. Language Use Domains: • Corpus and psycho-linguistics Measures: • Word frequencies: less frequent = more sophisticated/complex • Age of acquisition: later AoA = more sophisticated/complex Implemented: • Frequency data bases: dlexDB, SUBTLEX-DE, Google Books 2000 • AoA approximation based on KCT (Lavalley, Berkling, and Stüker 2015)

  17. Human Language Processing Domains: • Cognitive science, psycho-linguistics and information theory Measures: • Cognitive processing costs as identified by processing time, reading time, etc. • Storage and integration of discourse referents consumes cognitive resources • Long distances between referents increase these costs Implemented: • Dependency Locality Theory (DLT) by Gibson 2000; Shain et al. 2016 • Verb argument distances in syllables (Weiß 2015)

  18. Discourse & Encoding of Meaning Domains: • Psychology, psycho-linguistics Measures: • Propositional idea density (Brown et al. 2008): more propositions = more complex encoding of meaning • Connectives • Co-referential expressions • Grammatical transitions → cause more cohesive writing and complex discourse Implemented: • PID (Louwerse et al. 2004) • Connectives as listed by Duden (Gr) 2009 • Local and global overlap of linguistic material • Co-referential expressions (pronouns, articles, etc.) • Local transitions of grammatical roles (Barzilay and Lapata 2008; Galasso 2014; Todirascu et al. 2013)

  19. Theoretical Linguistics Lexio-Semantic • Measures: concreteness, relatedness, diversity, and variation • Implemented: TTR, lexical TTR, GermaNet semantic relations Syntax: • Measures: clausal complexity (subordination), phrasal complexity (modification) • Implemented: dependent clause ratios, modifier ratios, complex NP, periphrastic constructions, etc. Morphology: • Measures: inflection, derivation, composition • Implemented: nominalization, tense, compound depth, etc.

  20. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  21. Overview • Plots of measures with 95% confidence intervals • Compare proficiency trajectory across corpora and task profiles • Sample of over 100 complexity measures • Grouped under theoretical considerations into concepts • Selected to represent at least one concept per domain • All 398 measures at http://www.sfs.uni-tuebingen.de/~zweiss/ma-thesis/ supplementary-material/complexity-plots/

  22. Human Language Processing DLT-V and syllable distance measures 4.0 adjHighICAreas/Vfin 3.5 adjHighICAreas/Vfin totalICAtVfin/Vfin 0.002 totalICAtVfin/Vfin ● 3.5 0.001 3.0 ● 0.001 ● ● ● 3.0 0.000 2.5 ● ● 0.000 ● 2.5 −0.001 a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 proficiency score proficiency score course course 2.4 syllablesInMF/MF maxTotalIC/Vfin syllablesInMF/MF 10 maxTotalIC/Vfin 7.5 2.5 ● ● 2.2 ● 8 5.0 2.0 ● 2.0 ● ● 1.8 6 ● ● 2.5 1.5 a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 proficiency score proficiency score course course dist1stArgToVerb/VerbWithDistArg dist1stArgToVerb/VerbWithDistArg 12 8 ● 10 6 ● 8 4 ● ● 6 2 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  23. Discourse & Encoding of Meaning Overlap of linguistic material globalNounOverlapsPerSentence localNounOverlapsPerSentence globalNounOverlapsPerSentence localArgOverlapsPerSentence localNounOverlapsPerSentence localArgOverlapsPerSentence 0.3 0.7 0.6 0.4 ● ● 0.20 ● ● ● ● ● ● 0.6 0.5 ● 0.2 0.15 0.3 0.2 ● ● ● 0.4 0.5 0.10 0.2 0.3 0.1 0.1 0.05 0.2 0.4 0.1 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score globalContentOverlapsPerSentence course localContentOverlapsPerSentence course course globalContentOverlapsPerSentence localContentOverlapsPerSentence globalArgOverlapsPerSentence globalArgOverlapsPerSentence 0.45 0.12 0.08 0.06 0.07 0.40 ● 0.4 ● ● ● 0.10 ● 0.05 ● 0.06 0.35 0.06 ● ● ● 0.3 ● 0.04 0.08 0.05 ● 0.30 ● 0.03 0.04 0.04 0.2 0.06 0.25 0.03 0.02 0.1 0.04 0.20 0.02 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score course course course globalStemOverlapsPerSentence localStemOverlapsPerSentence globalStemOverlapsPerSentence localStemOverlapsPerSentence 0.5 ● 0.3 ● ● ● ● 0.4 ● ● 0.3 ● 0.2 0.2 0.3 0.2 0.1 0.2 0.1 0.1 0.1 a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 proficiency score proficiency score course course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  24. Syntactic Complexity Complex NPs 0.025 2.0 0.03 npDeps/npWithDeps 0.6 2.0 4.0 npDeps/npWithDeps attrParticiples/np 4.0 0.020 0.6 ● attrParticiples/np 1.9 1.9 npMods/np 0.5 words/np ● npMods/np words/np ● ● ● 0.02 0.015 ● ● 3.5 1.8 0.5 3.6 1.8 ● 0.4 ● 0.010 ● ● 1.7 1.7 0.4 ● 0.01 3.2 0.3 3.0 ● 0.005 1.6 ● ● 1.6 ● 0.3 0.2 0.000 2.8 2.5 1.5 0.00 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course comparativeNounMods/np possessiveNounMods/np 0.15 0.05 comparativeNounMods/np possessiveNounMods/np clausalNounMods/np 0.04 0.4 0.10 clausalNounMods/np 0.0075 ● 0.0075 determiners/np 0.04 ● ● determiners/np 0.5 0.09 ● 0.03 ● ● 0.0050 0.10 0.3 0.0050 0.08 0.03 ● 0.4 ● ● ● ● ● 0.02 0.07 ● 0.02 0.0025 0.0025 ● 0.2 0.3 0.06 0.05 ● ● 0.01 0.01 0.0000 0.0000 0.05 0.2 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course 0.25 0.30 1.0 coverageModifierTypes postnominalMods/np prenominalMods/np 0.7 coverageModifierTypes 0.20 postnominalMods/np 0.200 prenominalMods/np ● 0.25 0.9 0.20 0.6 0.175 ● ● 0.15 0.20 ● 0.8 ● 0.150 0.5 ● ● ● 0.15 0.15 0.7 0.10 ● 0.125 ● ● 0.4 ● 0.10 0.6 0.100 0.05 0.10 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score course course course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  25. Morphological Complexity Inflection measures nominatives/noun accusatives/noun 0.6 0.15 genitives/noun 0.30 nominatives/noun 0.50 accusatives/noun 0.35 datives/noun ● 0.35 genitives/noun datives/noun 0.45 0.06 0.25 ● ● 0.5 0.30 0.10 0.3 ● 0.40 0.30 0.20 0.04 ● 0.4 ● ● 0.25 0.35 ● ● ● ● 0.15 0.05 0.2 ● 0.25 ● ● 0.30 0.3 0.02 ● ● 0.20 0.10 0.25 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course participleVerbs/verb infiniteVerbs/verb participleVerbs/verb 0.150 0.3 0.10 imperatives/vfin infiniteVerbs/verb 0.006 imperatives/vfin 0.008 0.80 ● 0.25 vfin/verb 0.8 0.08 0.004 vfin/verb ● 0.125 0.006 ● 0.2 0.75 0.20 ● ● 0.004 0.002 ● 0.100 ● 0.06 0.70 0.7 ● ● ● ● 0.15 0.1 0.002 0.000 ● 0.04 0.65 0.075 ● 0.000 ● ● ● 0.10 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course 2ndPersonInfl/vfin subjunctives/vfin 1stPersonInfl/vfin 0.6 2ndPersonInfl/vfin indicatives/vfin 0.975 0.5 subjunctives/vfin 0.100 1stPersonInfl/vfin indicatives/vfin ● ● ● 0.975 0.09 0.950 0.075 0.4 0.075 ● 0.04 0.4 0.925 0.050 0.950 0.06 0.3 0.050 0.900 0.02 0.025 ● 0.875 0.925 0.2 0.03 0.2 ● 0.025 ● ● ● ● ● ● ● 0.850 0.000 ● ● ● 0.00 0.900 0.0 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course 3rdPersonInfl/vfin 3rdPersonInfl/vfin ● ● ● 0.8 ● 0.8 0.7 0.6 0.6 0.5 0.4 0.4 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  26. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  27. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  28. Overview • Regression analysis to predict proficiency from complexity and task factors • Use ordinal generative additive regression models (GAMs) • 2 studies on Merlin: i) task effects; ii) performance effects • Studies on Falko Georgetown not reported here

  29. Ordinal Generative Additive Regression Models Overview GAMs • Extension of linear regression models • Use splines as smooths for controlled introduction of non-linear relations • Highly interpretable, yet similar predictive power as ML techniques like SVM • Share requirements of regression models: normal, uncorrelated predictors • Support 1 predictor per 15 to 20 data points Ordinal Regression • Link function to non-exponential distribution by Wood 2006 • Estimates boundaries between classes → keeps precedence without introducing quantity

  30. Model Design Iterative, data-driven model approach 1 Rank measures by information gain using WEKA 2 Test most informative measure for normality; normalize if necessary 3 Test for correlation of predictors a. If < ± 0 . 70 Pearson correlation: add measure to model b. Else: remove correlated measures, add measure to model 4 Smooth measures unless they are linear 5 If changes lead to significant model improvement ( χ 2 test), keep them 6 Do until 20 iterations did not yield better model or model contains 15/n measures

  31. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  32. Study 1: Task Effects Set Up Figure: Model formula of Merlin interaction model predicting overall CEFR scores from scaled and transformed complexity measures.

  33. Study 1: Task Effects Model Fit • R 2 = 0 . 7660 • Approximately homescedastic residual errors with µ = 0 . 04 ; sd = 7 . 26 after outlier removal • Severe outliers across all model variants: Assumption of idiosyncratic properties in texts • Outliers systematically include learners who performed above or below test level → Prompted performance effect analysis in Study 2

  34. Study 1: Task Effects Model Fit χ 2 Pr ( > χ 2 ) Model AIC Df REML Edf Compared with Edf difference Complexity 1315.05 30.37 658.56 19 Reference 1287.08 28.41 642.77 20 Complexity 15.790 1 1.914e-08 Interaction 1281.00 39.27 628.84 31 Complexity 29.717 12 2.861e-08 Reference 13.928 11 0.003 Table: Model comparison for complexity, reference, and interaction model build on the Merlin data.

  35. Study 1: Task Effects Model Discussion A. parametric coefficients Estimate Std. Error t-value p-value (Intercept) 8.3759 0.3833 21.8509 < 0.0001 hasTransitionsFromSubjectToNot[TRUE] -0.5349 0.2387 -2.2408 0.0250 has3rdPersPossessivePronouns[TRUE] -0.8906 0.2030 -4.3873 < 0.0001 containsToInfinitives[TRUE] -0.5541 0.2282 -2.4284 0.0152 halfModalClusterPerVP 0.1831 0.1011 1.8113 0.0701 logSumNonTerminalNodesPerSentence 1.9714 0.1785 11.0435 < 0.0001 avgVTotalIntegrationCostAtFiniteVerb 0.3705 0.1059 3.4968 0.0005 lexTypesFoundInDlexPerLexType 0.8840 0.0942 9.3858 < 0.0001 Table: Interaction model: linear measures.

  36. Study 1: Task Effects Model Discussion A. parametric coefficients Estimate Std. Error t-value p-value (Intercept) 8.3759 0.3833 21.8509 < 0.0001 usesConjunctionalClauses[TRUE] -0.6051 0.3173 -1.9074 0.0565 logATFBand2PerTypesFoundInDlex -0.3003 0.1091 -2.7528 0.0059 typeTokenRato 1.2853 0.2038 6.3068 < 0.0001 logSumNonTerminalNodesPerWord -0.7130 0.1598 -4.4619 < 0.0001 TaskTheme[Society] 0.4921 0.7085 0.6947 0.4873 TaskTheme[Profession] 1.0774 0.5508 1.9560 0.0505 TaskTheme[Smalltalk] -0.8117 0.3529 -2.3004 0.0214 usesConjunctionalClauses:TaskTheme[Society] 2.1839 0.9603 2.2742 0.0230 usesConjunctionalClauses:TaskTheme[Profession] -0.4185 0.5417 -0.7726 0.4398 usesConjunctionalClauses:TaskTheme[Smalltalk] 0.5155 0.4714 1.0937 0.2741 logATFBand2PerTypesFoundInDlex:TaskTheme[Society] -0.1827 0.4194 -0.4357 0.6631 logATFBand2PerTypesFoundInDlex:TaskTheme[Profession] 0.5517 0.3530 1.5628 0.1181 logATFBand2PerTypesFoundInDlex:TaskTheme[Smalltalk] 0.5392 0.2197 2.4539 0.0141 typeTokenRato:TaskTheme[Society] -0.4750 0.3634 -1.3072 0.1912 typeTokenRato:TaskTheme[Profession] -0.5975 0.3998 -1.4947 0.1350 typeTokenRato:TaskTheme[Smalltalk] -0.8335 0.2925 -2.8494 0.0044 logSumNonTerminalNodesPerWord:TaskTheme[Society] -0.9369 0.4216 -2.2224 0.0263 logSumNonTerminalNodesPerWord:TaskTheme[Profession] -0.1522 0.3409 -0.4465 0.6552 logSumNonTerminalNodesPerWord:TaskTheme[Smalltalk] 0.2680 0.2344 1.1429 0.2531 Table: Interaction model: interactions measures.

  37. Study 1: Task Effects Model Discussion B. smooth terms edf Ref.df F-value p-value s(charactersPerWord) 2.7714 3.5484 18.5670 0.0007 s(numberOfSentencesSquared) 4.6262 5.7193 254.0399 < 0.0001 Table: Interaction model: smoothed measures. Figure: Smooths of Merlin interaction model.

  38. Study 1: Task Effects Classification Experiment Model µ F1 ± SD µ Recall ± SD µ Precision ± SD Majority Baseline 7.37 11.59 7.44 11.33 7.37 11.37 Complexity 70.97 4.25 71.63 4.74 72.30 4.09 Reference 71.32 4.33 71.78 4.87 72.74 4.10 Interaction 72.17 4.43 72.69 4.94 73.39 4.15 Table: Weighted average precision, recall, and f1 score for complexity, reference, and interaction model for 10 iterations of 10-folds cross-validation.

  39. Study 1: Task Effects Classification Experiment Predicted ↓ / Observed → A1 A2 B1 B2 C A1 25.5 10.1 0.0 0.0 0.0 A2 29.5 241.3 45.4 0.0 0.0 B1 0.0 52.6 233.5 37.8 0.0 B2 0.0 0.0 49.1 243.1 37.9 C 0.0 0.0 0.0 10.1 8.1 Table: Averaged confusion matrix for classification of L2 proficiency in Merlin using the interaction model.

  40. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  41. Study 2: Performance Effects Set Up Figure: Model formula of Merlin success model predicting overall CEFR scores from scaled and transformed complexity measures

  42. Study 2: Performance Effects Model Fit • R 2 = 0 . 9000 • Approximately homescedastic residual errors with µ = − 0 . 14 ; sd = 12 . 91 after outlier removal • Same outliers as before, except for under-performing learners • Outliers still systematically include learners who performed above or below test level

  43. Study 2: Performance Effects Model Fit χ 2 Pr ( > χ 2 ) Model AIC Df REML Edf Comparison with Edf diff. Complexity 1315.05 30.37 658.56 19 Reference model 1287.08 28.41 642.77 20 Interaction model 1281.00 39.27 628.84 31 Success model 821.11 35.76 401.19 26 Complexity 257.36 7 < 2 e − 16 Success model Reference 241.573 6 < 2 e − 16 Success model Interaction 227.65 -5 Table: Model comparison for reference, complexity, interaction, and success GAMs modeling L2 proficiency from complexity measures and task theme on the Merlin data.

  44. Study 2: Performance Effects Model Discussion A. parametric coefficients Estimate Std. Error t-value p-value (Intercept) 3.8610 0.5186 7.4455 < 0.0001 hasTransitionsFromSubjectToNot[TRUE] -0.8565 0.2732 -3.1350 0.0017 has3rdPersPossessivePronouns[TRUE] -1.3267 0.2556 -5.1903 < 0.0001 containsToInfinitives[TRUE] -0.7246 0.2701 -2.6825 0.0073 usesConjunctionalClauses[TRUE] -0.5514 0.2698 -2.0434 0.0410 logATFBand2PerTypesFoundInDlex -0.3972 0.1202 -3.3054 0.0009 avgVTotalIntegrationCostAtFiniteVerb 0.4765 0.1380 3.4522 0.0006 lexTypesFoundInDlexPerLexType 0.9649 0.1132 8.5218 < 0.0001 typeTokenRato 1.2797 0.1877 6.8176 < 0.0001 sumNonTerminalNodesPerWord -0.8316 0.1398 -5.9464 < 0.0001 logSumNonTerminalNodesPerSentence 2.4829 0.2093 11.8655 < 0.0001 Passed[TRUE] 6.6843 0.3018 22.1510 < 0.0001 TaskTheme[Society] 11.5649 0.6668 17.3437 < 0.0001 TaskTheme[Profession] 7.2479 0.5982 12.1158 < 0.0001 TaskTheme[Smalltalk] 0.9101 0.2796 3.2550 0.0011 logATFBand2PerTypesFoundInDlex:TaskTheme[Society] 0.4881 0.5980 0.8163 0.4144 logATFBand2PerTypesFoundInDlex:TaskTheme[Profession] 1.1930 0.4812 2.4795 0.0132 logATFBand2PerTypesFoundInDlex:TaskTheme[Smalltalk] 0.3248 0.2428 1.3376 0.1810 s(charactersPerWord):Passed[FALSE] 2.5964 3.2239 5.3214 0.1503 s(charactersPerWord):Passed[TRUE] 1.3498 1.6284 6.5613 0.0297 s(numberOfSentencesSquared):Passed[FALSE] 3.6322 4.5433 69.6021 < 0.0001 s(numberOfSentencesSquared):Passed[TRUE] 4.3517 5.3657 306.2779 < 0.0001 Table: Summary of success model predicting Merlin overall CEFR scores from scaled and transformed complexity measures in Merlin. Uses ’demand’ as

  45. Study 2: Performance Effects Model Discussion Figure: Smooths of Merlin success model.

  46. Study 2: Performance Effects Model Discussion • Most task theme interactions become uninformative • Still significantly different slopes, but not enough new variance explained • Especially: texts about society heavily confounded with failed tests • Unclear relationship between performance and task theme

  47. Study 2: Performance Effects Classification Experiment Model µ F1 ± SD µ Recall ± SD µ Precision ± SD Majority Baseline 7.37 11.59 7.44 11.33 7.37 11.37 Complexity 71.20 4.25 71.89 4.71 72.53 4.03 Reference 71.32 4.33 71.78 4.87 72.74 4.10 Interaction 72.17 4.43 72.69 4.94 73.39 4.15 Success 84.98 2.75 85.60 2.80 85.28 2.74 Table: Weighted average precision, recall, and F1 score for complexity, reference, interaction and success model for 10 iterations of 10-folds cross-validation.

  48. Study 2: Performance Effects Classification Experiment Pred. ↓ / Obs. → A1 A2 B1 B2 C A1 12.7 0.0 0.0 0.0 28.1 A2 26.9 260.9 34.5 0.0 0.0 B1 0.0 30.4 271.9 18.2 0.0 B2 0.0 0.0 21.6 271.9 6.0 C 0.0 0.0 0.0 0.4 40.0 Table: Averaged confusion matrix for classification of L2 proficiency in Merlin using the success model.

  49. 1 Introduction 2 Theoretical Background Complexity in SLA Task Effects in SLA 3 Data Corpora Task Annotations 4 Measuring Complexity Automatic System Complexity Measures 5 Descriptive Cross-Corpus Analysis 6 Inferential Regression Modeling Set Up Study 1: Task Effects Study 2: Performance Effects 7 Conclusion

  50. Conclusion Findings How do measures of complexity model German L2 proficiency? • Most indices of the same concept tend to develop homogeneously and stable across corpora • Most indices develop homogeneously across corpora • Data-driven feature selection approaches yield diverse set of measures • GAMs are highly interpretable, yet show considerable predictive power

  51. Conclusion Findings To which extend is this influenced by cognitive or functional task-effects? • Some measures are more stable across heterogeneous task backgrounds (human language processing, complex NPs) • Other measures are less stable • Stable measures especially promising for systems evaluating diverse task backgrounds • Task factors seem to predominantly effect local measures of structural complexity • Further research on this required

  52. Conclusion Findings Does a retrospective analysis of German learner corpora with diverse task backgrounds improve complexity-based L2 proficiency modeling? • Post-hoc annotation straight forward if task documentation available • Suited to decrease confound of tasks and course levels • Task factors improve model fit significantly and decrease non-linearity • Interactions seem unstable, models suffer from wide standard deviation • Results lack interpretability due to skewed distribution • Analysis improves situation, but idiosyncratic distributional properties of data remain problematic

  53. Future Work Next: 1 Investigation of task and performance effects on more balanced data sets 2 Study of adequacy of L2 complexity by comparing results on comparable L1 productions (Falko) 3 Make complexity code used here publicly available in CTAP (Chen and Meurers 2016) Also interesting: • Analysis of task type interactions in data • Cross-corpus testing of success model • Systematically assess sensitivity of system and structure complexity to task effects • Systematic validation of measure validity on L2 data

  54. Thank you for your attention! Questions?

  55. References I Abel, Andrea et al. (2013). merlin: A Trilingual Learner Corpus illustrating European Reference Levels . LRC 2013. Bergen, Norway. Alexopoulou, Theodora et al. (2017). “Task Effects on Linguistic Complexity and Accuracy: A Large-Scale Learner Corpus Analysis Employing Natural Language Processing Techniques”. In: Language Learning , pp. 1–29. Barzilay, Regina and Mirella Lapata (2008). “Modeling local coherence: An entity-based approach”. In: Computational Linguistics 34, pp. 1–34. Biber, Douglas, Bethany Gray, and Kornwepa Poonpon (2011). “Should we use characteristics of conversation to measure grammatical complexity in L2 writing development?” In: Tesol Quarterly 45, pp. 5–35. Brown, Cati et al. (2008). “Automatic measurement of propositional idea density from part-of-speech tagging”. In: Behavior research methods 40.2, pp. 540–545.

  56. References II Bulté, Bram and Alex Housen (2014). “Conceptualizing and measuring short-term changes in L2 writing complexity”. In: Journal of Second Language Writing 26, pp. 42–65. Chen, Xiaobin and Detmar Meurers (2016). “CTAP: A Web-Based Tool Supporting Automatic Complexity Analysis”. In: Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity , pp. 113–119. Crossley, Scott A. and Danielle S. McNamara (2011). “Shared features of L2 writing: Intergroup homogeneity and text classification”. In: Journal of Second Language Writing 20, pp. 271–285. Duden (Gr) (2009). Deutsche Grammatik . Ed. by Ursula Hoberg and Rudolf Hoberg. 4th ed. Vol. 4. Der kleine Duden. Berlin, Germany: Dudenverlag. Ellis, R. and G. Barkhuizen (2005). Analysing learner language . Oxford: Oxford University Press. Falko Georgetown Dokumentation (2007). Humnoldt-Universität zu Berlin.

  57. References III Frogner, Ellen (1933). “Problems of sentence structure in pupils’ themes”. In: English Journal 22, pp. 742–749. Galasso, Sabrina (2014). Exploring Textual Cohesion Characteristics for German Readability Classification . B.A. Thesis. Gibson, Edward (2000). “The dependency locality theory: A distance-based theory of linguistic complexity”. In: Image, language, brain , pp. 95–126. Hancke, Julia (2013). “Automatic Prediction of CERF Proficiency Levels Based on Linguistic Features of Learner Language”. MA thesis. Eberhard Karls Universität Tübingen. Hancke, Julia, Sowmya Vajjala, and Detmar Meurers (2012). “Readability Classification for German using lexical, syntactic and morphological features”. In: Proceedings of COLING . Mumbai, pp. 1063–1080. Housen, Alex, Ineke Vedder, and Folkert Kuiken (2012). “Document Viewing Options: Title: Dimensions of L2 Performance and Proficiency : Complexity, Accuracy and Fluency in SLA”. In: vol. 32. Language Learning & Language Teaching. Amsterdam, Philadelphia: John Benjamins Publishing. Chap. 1–2.

  58. References IV Jarvis, Scott et al. (2003). “Exploring multiple profiles of highly rated learner compositions”. In: Journal of Second Language Writing 12, pp. 377–403. Kyle, Kristopher (2016). “Measuring Syntactic Development in L2 Writing: Fine Grained Indices of Syntactic Complexity and Usage-Based Indices of Syntactic Sophistication”. PhD thesis. Georgia State University. Lavalley, Rémi, Kay Berkling, and Sebastian Stüker (2015). “Preparing Children’s Writing Database for Automated Processing”. In: Workshop on L1 Teaching, Learning and Technology (L1TLT) . Leipzig, Germany, pp. 9–15. Louwerse, Max M. et al. (2004). “Variation in language and cohesion across writ- ten and spoken registers”. In: Proceedings of the 26th Annual Meeting of the Cognitive Science Society , pp. 843–848. Lu, Xiaofei (2010). “Automatic analysis of syntactic complexity in second language writing”. In: International Journal of Corpus Linguistics 15.4, pp. 474–496.

  59. References V McNamara, Danielle S. et al. (2014). Automated evaluation of text and discourse with Coh-Metrix. Camebridge University Press. Merlin project (2014a). task desciption: Essay: why it’s of value to learn German . http://merlin-platform.eu/ . Merlin project (2014b). task desciption: Formal letter: apply for internship in sales department . http://merlin-platform.eu/ . Merlin project (2014c). task desciption: Formal letter: ask for information at Au pair Agency . http://merlin-platform.eu/ . Merlin project (2014d). task desciption: Formal letter: Au pair writes letter of complaint to Agency . http://merlin-platform.eu/ . Merlin project (2014e). task desciption: Formal letter to housing office . http://merlin-platform.eu/ . Merlin project (2014f). task desciption: Informal e-mail: arrange an appointment with a friend to go swimming together . http://merlin-platform.eu/ . Merlin project (2014g). task desciption: Informal e-mail: ask a friend for help with finding an apartment . http://merlin-platform.eu/ .

  60. References VI Merlin project (2014h). task desciption: Informal letter: ask friend to take care of pet . http://merlin-platform.eu/ . Merlin project (2014i). task desciption: Informal letter: birthday congratulations . http://merlin-platform.eu/ . Merlin project (2014j). task desciption: Informal letter: congratulate to birth of a child . http://merlin-platform.eu/ . Merlin project (2014k). task desciption: Informal letter for New Year to a friend . http://merlin-platform.eu/ . Merlin project (2014l). task desciption: Informal letter: offer a ticket not used to a friend . http://merlin-platform.eu/ . Merlin project (2014m). task desciption: Informal letter to a friend announcing a visit . http://merlin-platform.eu/ . Merlin project (2014n). task desciption: Online article: about sticking to one’s traditions and "assimilation" in a new environment . http://merlin-platform.eu/ . Merlin project (2014o). task desciption: Report: about the housing situation . http://merlin-platform.eu/ .

  61. References VII Ortega, Lourdes (2003). “Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing”. In: Applied Linguistics 24, pp. 492–518. Pallotti, G. and S. Ferrari (2008). “Lavariabilità situazionale dell’interlingua: Implicazioni per la ricerca acquisizionale e il testing linguistico”. In: Competenze Lessicali e Discorsive nell’Acquisizione di Lingue Seconde . Pallotti, Gabrielle (2009). “CAF: Defining, Refining and Differentiating Constructs”. In: Applied Linguistics 30.4, pp. 590–601. Pallotti, Gabrielle (2015). “A simple view of linguistic complexity”. In: Second Language Research 31.1, pp. 117–134. Polio, Charlene and J.-H. Park (2016). “Language development in second language writing”. In: Handbook of second and foreign language writing . Ed. by R. Manchón and P. K. Matsuda. Mouton de Gruyter. Rescher, Nicholas (1998). Complexity: A philosophical overview . Transaction Publishers. Reznicek, Marc et al. Das Falko-Handbuch Korpusaufbau und Annotationen . Humnoldt-Universität zu Berlin.

  62. References VIII Robinson, Peter (2001). “Task Complexity, Task Difficulty, and Task Production: Exploring Interactions in a Componential Framework”. In: Applied Linguistics 22.1, pp. 27–57. Shain, Cory et al. (2016). “Memory access during incremental sentence processing causes reading time latency”. In: Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity , pp. 49–58. Skehan, Peter (1996). “A Framework for the Implementation of Task-based Instruction”. In: Applied Linguistics 17.1, pp. 38–62. Thorndike, E. L. (1921). “Word Knowledge in the Elementary School”. In: Teachers College Record 28.5, pp. 334–370. Todirascu, Amalia et al. (2013). “Coherence and cohesion for the assessment of text readability”. In: Natural Language Processing and Cognitive Science 11, pp. 11–19. Tracy-Ventura, Nicole and Florence Myles (2015). “The importance of task variability in the design of learner corpora for SLA research”. In: International Journal of Learner Corpus Research 1.1, pp. 58–95.

  63. References IX von der Brück, Tim and Sven Hartrumpf (2007). “A Semantically Oriented Readability Checker for German”. In: Proceedings of the 3rd Language & Technology Conference , pp. 270–274. von der Brück, Tim, Sven Hartrumpf, and Hermann Helbig (2008). “A Readability Checker with Supervised Learning Using Deep Indicators”. In: Informatica 32, pp. 429–435. Weiß, Zarah Leonie (2015). More Linguistically Motivated Features of Language Complexity in Readability Classification of German Textbooks: Implementation and Evaluation . B.A. Thesis. Tübingen, Germany. Weiß, Zarah Leonie (2017). “Using Measures of Linguistic Complexity to Assess German L2 Proficiency in Learner Corpora under Consideration of Task-Effects”. MA thesis. Eberhard Karls Universität Tübingen. Wolfe-Quintero, Kate, Shunji Inagaki, and Hae-Young Kim (1998). Second Language Development in Writing: Measures of Fluency, Accuracy & Complexity . Second Language Teaching & Curriculum Center.

  64. References X Wood, Simon N. (2006). Generalized additive models: an introduction with R . CRC press. Yoon, Hyung-Jo and Charlene Polio (2016). “The Linguistic Development of Students of English as a Second Language in Two Written Genres”. In: Tesol Quarterly .

  65. Robinson’s Cognition Hypothesis Figure: Task complexity, condition, and difficulty (Robinson 2001, p. 30, Figure 1).

  66. Language Use DlexDB frequencies lemmaFreq/LTD 40000 30000 3.7 typeFreq/LTD 60000 40000 lemmaFreq/LTD 80000 ATFreq/LTD 4.00 logATF/LTD 3.6 typeFreq/LTD ● 25000 ATFreq/LTD ● ● logATF/LTD ● 35000 ● 25000 ● ● 35000 ● ● ● 3.5 ● ● 70000 3.95 50000 ● ● 20000 3.4 ● 30000 3.90 20000 30000 ● 60000 3.3 40000 3.85 15000 25000 15000 3.2 25000 50000 3.80 3.1 20000 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course logLemmaFreq/LTD logATFBand1/LTD logATFBand2/LTD logTypeFreq/LTD logLemmaFreq/LTD 4.55 0.15 logTypeFreq/LTD logATFBand1/LTD logATFBand2/LTD 4.1 0.015 3.7 4.1 ● 0.03 0.075 ● 4.50 ● ● 0.12 ● 4.0 ● ● ● 3.6 4.45 ● 0.010 0.050 4.0 0.02 ● 3.5 3.9 0.09 4.40 ● ● 4.35 ● 3.8 0.025 0.005 ● 3.4 0.06 0.01 ● 3.9 ● 4.30 3.7 3.3 4.25 0.000 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course logATFBand3/LTD logATFBand4/LTD logATFBand5/LTD logATFBand6/LTD 0.28 logATFBand3/LTD logATFBand4/LTD 0.48 logATFBand5/LTD 0.52 logATFBand6/LTD 0.12 0.40 0.06 0.07 0.34 0.48 ● 0.24 0.10 ● ● 0.44 ● 0.35 0.32 ● 0.06 0.04 ● ● ● 0.44 0.20 0.30 0.08 ● ● ● 0.40 0.05 0.30 ● 0.02 ● 0.28 ● ● ● 0.40 0.04 0.16 0.06 0.25 0.36 0.26 0.03 0.36 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course typesNotInDlex/LT typesNotInDlex/LT 0.55 0.5 0.50 ● ● 0.4 ● ● 0.45 0.3 0.40 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  67. Discourse & Encoding of Meaning Pronouns, articles and names 3PPersPron/TIS 0.25 0.06 pronouns/TIS persPron/TIS 0.12 possPron/TIS 0.16 3PPersPron/TIS possPron/TIS 0.03 0.016 pronouns/TIS persPron/TIS 0.200 ● 0.10 0.03 0.20 ● 0.12 0.04 ● ● 0.012 ● 0.175 0.02 0.08 ● ● 0.15 ● ● 0.02 0.08 ● ● ● 0.150 0.008 ● ● 0.06 ● 0.02 ● 0.01 0.10 0.04 0.01 0.125 0.004 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course 3PPers&PossPron/TIS 3PPers&PossPron/TIS 3PPossPron/TIS 1PPersPron/TIS 1PPossPron/TIS 0.08 0.125 0.030 3PPossPron/TIS 1PPersPron/TIS 1PPossPron/TIS 0.020 0.015 0.015 ● 0.06 ● 0.025 ● 0.100 0.02 0.015 0.06 ● ● 0.010 0.010 ● 0.075 0.020 0.04 0.010 ● 0.015 0.04 0.01 0.050 0.005 ● 0.005 0.02 0.005 0.010 0.025 ● ● ● ● ● ● ● ● 0.000 0.00 0.000 0.000 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course 1PPers&PossPron/TIS 2PPers&PossPron/TIS 1PPers&PossPron/TIS 2PPers&PossPron/TIS 2PPossPron/TIS 2PPersPron/TIS 0.020 2PPersPron/TIS 2PPossPron/TIS 0.03 0.04 0.015 0.0015 0.08 0.010 0.015 0.03 0.10 0.02 0.0010 0.010 0.02 0.010 0.06 0.005 ● 0.01 0.0005 0.05 ● 0.01 0.005 0.005 0.04 ● ● ● ● ● ● ● ● 0.0000 0.00 0.000 0.00 ● ● ● ● ● ● 0.000 0.000 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course properNamesPerSentence properNamesPerSentence 1.25 0.7 indefArt/TIS 0.7 defArt/TIS 1.5 0.80 indefArt/TIS 0.80 defArt/TIS ● ● ● 1.00 ● 0.6 0.6 ● 0.75 0.75 ● 1.0 0.5 0.5 0.75 0.70 0.70 ● ● ● ● ● ● 0.4 0.4 0.5 0.65 0.65 0.50 0.3 0.3 0.60 0.60 0.25 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score course course course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  68. Lexical Complexity Lexical Variation lexVerbTypes/lexToken lexVerbTypes/lexVerbs lexVerbTypes/lexToken lexVerbTypes/lexVerbs lexTypes/lexToken lexTypes/Token 0.625 lexTypes/lexToken 0.92 0.75 ● lexTypes/Token 0.54 ● 0.24 0.85 0.90 0.600 0.22 ● 0.88 0.70 0.52 ● ● ● ● 0.575 ● 0.80 ● ● ● 0.20 0.20 0.85 0.84 ● ● ● 0.550 0.65 0.50 ● ● 0.75 0.16 0.18 0.80 0.525 0.80 0.48 0.60 0.70 0.500 0.12 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course (LexVerbTypes/lexVerbs)^2 corrLexVerbTypes/lexVerb (LexVerbTypes/lexVerbs)^2 corrLexVerbTypes/lexVerb nouns/lexToken 0.6 lexVerbs/token 6 nouns/lexToken 20 lexVerbs/token ● 3.0 0.35 0.13 60 0.150 15 2.5 0.5 5 ● ● ● 0.30 0.12 2.0 40 0.125 10 4 ● ● 0.4 ● ● ● ● 1.5 ● ● ● 0.11 ● 0.25 5 0.100 3 20 ● ● 1.0 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course adjectives/lexToken adverbs/lexToken adjectives/lexToken 1.4 0.16 0.40 0.175 0.10 adverbs/lexToken nouns/token verbs/noun 0.275 ● 0.8 nouns/token ● ● ● ● verbs/noun 1.2 0.35 0.150 0.08 0.14 0.08 0.250 0.7 1.0 ● ● ● ● ● 0.30 0.125 ● 0.6 0.225 0.06 0.06 0.8 0.12 ● ● ● 0.25 0.100 ● 0.5 0.200 ● 0.6 0.04 0.04 0.10 0.175 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course adj+adv/lexToken 0.23 adj+adv/lexToken ● 0.24 0.21 ● 0.20 ● 0.19 ● 0.16 0.17 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  69. Syntactic Complexity Periphrastic grammatical measures eventivePassive/finClause quasiPassives/finClause eventivePassive/finClause quasiPassives/finClause passives/finClause passives/finClause 0.20 0.125 0.04 0.04 0.30 ● 0.06 sein/verbs 0.15 ● 0.100 sein/verbs 0.03 0.04 0.15 ● 0.03 ● 0.25 0.075 0.04 ● ● 0.02 ● ● 0.12 ● ● 0.02 0.050 0.20 0.10 ● 0.02 ● ● 0.01 0.02 ● ● 0.025 0.09 0.01 0.15 ● 0.00 0.00 0.00 0.00 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course simplePresent/vfin presentPerfect/vfin 0.15 presentPerfect/vfin simplePast/vfin 0.16 simplePresent/vfin 0.85 0.10 0.6 haben/verbs ● simplePast/vfin ● ● haben/verbs ● 0.12 0.80 0.100 0.14 0.6 0.08 ● 0.10 0.75 0.12 0.4 ● ● 0.09 0.075 0.06 ● ● ● 0.70 0.10 ● 0.4 0.050 0.04 ● 0.06 0.2 0.05 0.65 0.08 ● ● ● ● 0.02 0.2 0.06 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score course course course course pastPerfect/vfin 0.50 coverageTenses 0.08 0.05 coverageTenses pastPerfect/vfin 0.0100 future1/vfin future2/vfin 0.5 0.003 0.8 0.03 0.25 0.04 future1/vfin 0.06 future2/vfin 0.0075 0.02 0.4 0.03 0.002 0.7 0.0050 0.00 0.04 0.02 ● 0.001 ● 0.0025 0.01 0.3 0.6 ● − 0.25 ● ● ● 0.01 ● 0.02 ● ● ● ● 0.0000 ● ● ● ● ● 0.00 0.000 − 0.50 0.2 0.00 0.5 0.00 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score proficiency score coveragePeriphrasticTenses course course course course coveragePeriphrasticTenses 0.7 0.3 0.6 0.2 0.5 ● ● 0.1 0.4 ● ● 0.3 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  70. Syntactic Complexity Dependent clause measures 0.4 0.6 depClauses/sentence conjClauses/sentence depClauses/sentence conjClauses/sentence 0.3 clauses/sentence ● 2.00 clauses/sentence ● 0.6 2.2 ● 0.4 ● 0.5 1.75 0.2 0.3 ● ● ● ● 2.0 ● 0.4 1.50 0.2 ● 0.1 1.8 0.3 ● 0.2 ● 1.25 0.0 0.0 0.2 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score course course course depClausesWithoutConj/sentence depClausesWithConj/sentence interrogativeClauses/sentence depClausesWithoutConj/sentence depClausesWithConj/sentence interrogativeClauses/sentence 0.5 0.6 0.0100 0.06 0.06 0.4 ● ● 0.0075 0.5 ● 0.02 ● ● 0.3 0.04 ● 0.0050 0.04 0.4 ● 0.2 ● 0.02 0.0025 0.01 0.3 0.02 0.1 ● ● 0.0000 ● ● 0.00 0.2 0.0 0.00 0.00 a1 a2 b1 b2 c a1 a2 b1 b2 c a1 a2 b1 b2 c 1 2 3 4 1 2 3 4 1 2 3 4 proficiency score proficiency score proficiency score course course course relativeClauses/sentence relativeClauses/sentence 0.20 0.09 ● ● 0.15 0.06 ● 0.10 0.03 ● 0.05 0.00 0.00 a1 a2 b1 b2 c 1 2 3 4 proficiency score course Figure: Merlin . Figure: Falko GT L2 ; △ : curricular tasks, ◦ : book reviews.

  71. Generative Additive Regression Models From Linear to Additive Models I y = η + ǫ, where ǫ ∼ N ( 0 , σ 2 ) and η = β 0 + � ˆ (1) x i β i i = 1 I � g (ˆ y ) = η + ǫ, where η = β 0 + (2) x i β i i = 1 I � g (ˆ y ) = η + ǫ, where η = β 0 + s i ( x i ) (3) i = 1

  72. Generative Additive Regression Models From Linear to Additive Models I � g (ˆ y ) = η + ǫ, where η = β 0 + s i ( x i ) (4) i = 1 K � s ( x ) = b k ( x ) β k , (5) k = 1 C + 1 � x c − 1 β c (6) s ( x ) = c = 1

  73. Generative Additive Regression Models Regression Splines Figure: Single cubic basis function (left) and full cubic regression spline (right), cf. Wood 2006, p. 147, Figure 4.1.

  74. Generative Additive Regression Models Regression Splines Figure: A rank 7 thin plate regression spline preceded by its weighted basis functions, cf. Wood 2006, p. 153, Figure 4.5.

  75. Generative Additive Regression Models Ordinal Models I � u = η + ǫ, where η = β 0 + s i ( x i ) , and u ∈ [ ±∞ ] (7) i = 1 • Ordinal data neither numeric nor nominal • Ordinal distribution not covered in exponential link functions g () • Solution by Wood 2006: partition ±∞ into K bins using K-1 boundaries • Estimate latent variable u with regression model • Assign ordinal category based on interval in which u falls

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Health Care Quality Measures IBHC Measures Atlas The Problem Finding quality measures to assess

Health Care Quality Measures IBHC Measures Atlas The Problem Finding quality measures to assess

Funded under contract #HHSA290-2010-00002i by the Agency for Healthcare Research and Quality Atlas of Integrated Behavioral Health Care Quality Measures IBHC Measures Atlas The Problem Finding quality measures to assess the degree to which

479 views • 8 slides
On the difference between the Federal German and the Austrian German discourse particle eh An

On the difference between the Federal German and the Austrian German discourse particle eh An

Linguistic Evidence 2018 On the difference between the Federal German and the Austrian German discourse particle eh An experimental investigation Sarah Zobel Feb 17, 2018 Introduction FG and AG eh Experiment Discussion Conclusion The

775 views • 65 slides
Semantic Complexity and Linguistic Distributions Jakub Szymanik Institute for Logic, Language and

Semantic Complexity and Linguistic Distributions Jakub Szymanik Institute for Logic, Language and

Semantic Complexity and Linguistic Distributions Jakub Szymanik Institute for Logic, Language and Computation University of Amsterdam LEGO, 21 February 2014 Outline Motivation Semantic Complexity Inferential meaning Referential meaning

1.02k views • 58 slides
Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation

Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation

Complexity Measures for Parallel Computation Complexity Measures for Parallel Computation Problem parameters: n index of problem size p number of processors Algorithm parameters: t p running time on p processors t 1 time on 1

311 views • 16 slides
Application of Life Cycle Thinking Application of Life Cycle Thinking to Assess Local Measures

Application of Life Cycle Thinking Application of Life Cycle Thinking to Assess Local Measures

Research Institute of Science for Safety and Sustainability Application of Life Cycle Thinking Application of Life Cycle Thinking to Assess Local Measures and Activities to Assess Local Measures and Activities to Assess Local Measures and

601 views • 41 slides
Broad Linguistic Modeling is Beneficial for German L2 Proficiency Assessment Zarah Wei and

Broad Linguistic Modeling is Beneficial for German L2 Proficiency Assessment Zarah Wei and

Broad Linguistic Modeling is Beneficial for German L2 Proficiency Assessment Zarah Wei and Detmar Meurers Eberhard Karls University, T ubingen Learner Corpus Research Conference Bolzano/Bozen, 5-7 October 2017 October 6th, 2017 1

1.32k views • 47 slides
Translation and Linguistic Validation of the ASCOT measures ASCOT Reception ILPN Conference,

Translation and Linguistic Validation of the ASCOT measures ASCOT Reception ILPN Conference,

Translation and Linguistic Validation of the ASCOT measures ASCOT Reception ILPN Conference, London 1 st September 2014 Speaker: Rebecca Two 1 Welcome Introduction to PharmaQuest Overview of translation and linguistic validation of

358 views • 11 slides
The new outcome measures in MS: possible better ways to assess disability that overcome the

The new outcome measures in MS: possible better ways to assess disability that overcome the

The new outcome measures in MS: possible better ways to assess disability that overcome the limitations of the EDSS Gilmore ONeill MB MRCPI MMSc VP, Neurology Clinical Development Biogen Idec Inc. 1 Think-Tank 29 November 2011 AGENDA

212 views • 17 slides
Plan Parallelism Complexity Measures 1 Multithreaded Parallelism and Performance Measures cilk

Plan Parallelism Complexity Measures 1 Multithreaded Parallelism and Performance Measures cilk

Plan Parallelism Complexity Measures 1 Multithreaded Parallelism and Performance Measures cilk for Loops 2 Marc Moreno Maza Scheduling Theory and Implementation 3 University of Western Ontario, London, Ontario (Canada) Measuring Parallelism

531 views • 16 slides
How to assess the fit of multilevel logit models with Stata? Meeting of the German Stata User

How to assess the fit of multilevel logit models with Stata? Meeting of the German Stata User

How to assess the fit of multilevel logit models with Stata? Meeting of the German Stata User Group at the Humboldt University Berlin, June 23rd, 2017 ? Models should not be true but it is important that they are applicable. John W. Tukey

835 views • 37 slides
Classification Complexity Measures and Their Relationship to Feature Selection J. L. Solka and

Classification Complexity Measures and Their Relationship to Feature Selection J. L. Solka and

Classification Complexity Measures and Their Relationship to Feature Selection J. L. Solka and D. A. Johannsen solkajl@nswc.navy.mil;johannsen@nswc.navy.mil NSWCDD Interface04 p.1/40 Agenda Minimal spanning tree complexity measures. An

439 views • 40 slides
Cognitive Complexity of Linguistic Patterns Artificial Grammar Learning Workshop Max Planck

Cognitive Complexity of Linguistic Patterns Artificial Grammar Learning Workshop Max Planck

AGL Workshop 1 Cognitive Complexity of Linguistic Patterns Artificial Grammar Learning Workshop Max Planck Institute for Psycholinguistics 2324 November 2010 Slide 1 James Rogers Dept. of Computer Science Earlham College

523 views • 37 slides
Exploring Measures of Readability for Spoken Language Introduction Analyzing linguistic

Exploring Measures of Readability for Spoken Language Introduction Analyzing linguistic

Readability for Spoken Language Sowmya Vajjala and Detmar Meurers Exploring Measures of Readability for Spoken Language Introduction Analyzing linguistic features of subtitles to identify age-specific TV programs Our Approach The

353 views • 21 slides
Beyond Power Capping Coping with the Complexity of the German Electricity Market Dagstuhl Seminar

Beyond Power Capping Coping with the Complexity of the German Electricity Market Dagstuhl Seminar

Beyond Power Capping Coping with the Complexity of the German Electricity Market Dagstuhl Seminar Dark Silicon: From Embedded to HPC Systems Axel Auweter Leibniz Supercomputing Centre of the Bavarian Academy of Sciences and Humanities

363 views • 23 slides
Overview Grammars, or: how to specify linguistic knowledge Towards more complex grammar

Overview Grammars, or: how to specify linguistic knowledge Towards more complex grammar

Overview Grammars, or: how to specify linguistic knowledge Towards more complex grammar systems Some basic formal language theory Automata, or: how to process with linguistic knowledge Levels of complexity in grammars and automata:

735 views • 4 slides
Using Strengths Based Measures to Assess and Manage Risk of Future Negative outcomes Simone

Using Strengths Based Measures to Assess and Manage Risk of Future Negative outcomes Simone

Using Strengths Based Measures to Assess and Manage Risk of Future Negative outcomes Simone Viljoen, M.Sc. Indian Health Service Clinical Rounds 02 June, 2014 Objectives 1. Discuss common negative outcomes present in AI/AN youth. 2.

623 views • 35 slides
Robot Motion Planning Barbara Frank, Cyrill Stachniss, Rdiger Schmedding, Matthias Teschner,

Robot Motion Planning Barbara Frank, Cyrill Stachniss, Rdiger Schmedding, Matthias Teschner,

Learning Object Deformation Models for Robot Motion Planning Barbara Frank, Cyrill Stachniss, Rdiger Schmedding, Matthias Teschner, Wolfram Burgard European Workshop on Deformable Object Manipulation March 20, 2014 Lyon, France 1

862 views • 26 slides
Day 9 Optimization of Cloud Data Centre Energy Consumption

Day 9 Optimization of Cloud Data Centre Energy Consumption

2/8/2018 Day 9 Optimization of Cloud Data Centre Energy Consumption https://www.ncbi.nlm.nih.gov/pmc/articles/P MC4446568/ Agenda for Today Why energy is an issue in Cloud data center? How is energy consumed in Cloud data center ?

470 views • 20 slides
Reproducing Kernel Hilbert Spaces for Classification Katarina Domijan and Simon P. Wilson

Reproducing Kernel Hilbert Spaces for Classification Katarina Domijan and Simon P. Wilson

Reproducing Kernel Hilbert Spaces for Classification Katarina Domijan and Simon P. Wilson Department of Statistics, University of Dublin, Trinity College, Ireland November 1, 2005 1 Working Group on Statistical Learning General problem

428 views • 21 slides
Final Evaluation Results of The Milwaukee Community Literacy Project/ SPARK Early Literacy Curtis

Final Evaluation Results of The Milwaukee Community Literacy Project/ SPARK Early Literacy Curtis

Final Evaluation Results of The Milwaukee Community Literacy Project/ SPARK Early Literacy Curtis Jones, SREed, University of Wisconsin, Milwaukee Michael Christian, Education Analytics Andrew Rice, Education Analytics March, 3 rd , 2016

839 views • 37 slides
Lake Creek Instream Flow Study Status Update Fish/Aquatics Meeting June 2007 INSTREAM FLOW

Lake Creek Instream Flow Study Status Update Fish/Aquatics Meeting June 2007 INSTREAM FLOW

Lake Creek Instream Flow Study Status Update Fish/Aquatics Meeting June 2007 INSTREAM FLOW STUDIES Habitat-Based Model Physical Component Physical Habitat Surveys Study Site and Transect Weighting Biological Component

1.14k views • 69 slides
The Potential for Beneficial Use The Potential for Beneficial Use of Stormwater Stormwater in

The Potential for Beneficial Use The Potential for Beneficial Use of Stormwater Stormwater in

The Potential for Beneficial Use The Potential for Beneficial Use of Stormwater Stormwater in Pima County in Pima County of Findings of the City/County Findings of the City/County Study on the Use of Stormwater Stormwater Study on the Use

724 views • 25 slides
Detection and Attribution of Climate Change and Human Activities Impacts on Water Resources in

Detection and Attribution of Climate Change and Human Activities Impacts on Water Resources in

Detection and Attribution of Climate Change and Human Activities Impacts on Water Resources in the Haihe River Basin of China YANGWEN JIA, XIANGYI DING, HAO WANG, YAQING QIU &amp; ZUHAO ZHOU China Institute of Water Resources and Hydropower

632 views • 32 slides
Data to Examine Consumption Poverty and I Inequality in the U.S.: 1960-2008 lit i th U S 1960

Data to Examine Consumption Poverty and I Inequality in the U.S.: 1960-2008 lit i th U S 1960

Data to Examine Consumption Poverty and I Inequality in the U.S.: 1960-2008 lit i th U S 1960 2008 CE Survey Data Users Needs Forum y June 22, 2010 Bruce D. Meyer University of Chicago and NBER Based on work with J James X. Sullivan

762 views • 34 slides

More recommend