On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
On Measures of Text Complexity
Sowmya V.B. Detmar Meurers
Universit¨ at T¨ ubingen
Second T¨ ubingen-Berlin Meeting on Analyzing Learner Language T¨ ubingen, December 5-6, 2011
1 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
Background
What do we mean by “measures of text complexity” ?
◮ Measuring how difficult it is to read a text, ◮ given a purpose, e.g.,
◮ general comprehension of key ideas of text ◮ identification of specific information looked for
◮ based on properties of the text using criteria which are
◮ theory-driven (e.g., difficult syntactic constructions) ◮ data-induced (e.g., corpora with graded texts), and a lot ◮ in-between (e.g., derived frequency information for words)
◮ and information about the user (e.g., language ability,
age, working memory)
◮ obtained directly (e.g., questionnaire), or ◮ indirectly (e.g, inferred from nature of a query) 2 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
Background
Why would anyone want to do this?
◮ To evaluate the quality of (manually written) texts, e.g.,
◮ for articles, manuals, books to be accessible to the
intended readership
◮ for reading and writing assessment in language teaching
◮ To evaluate the quality of natural language generation
systems (e.g., in text summarization)
◮ To track first and second language acquisition and
language attrition
◮ Analysis of complexity in Kobalt-DaF network ◮ Criterial features for language development (MERLIN) 3 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
A concrete “real-life” challenge
◮ Develop a search engine ranking web-search results
based on complexity.
◮ support a range of complexity features ◮ first prototype of a Language-Aware Search Engine
(Ott & Meurers 2010)
◮ Which measures of complexity should we use? ◮ Which gold-standards can the resulting approach be
evaluated against?
4 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
How do we measure text complexity?
Traditional readability formulas and lexical measures
◮ Clearly different aspects of linguistic complexity play a
role in determining the readability of a text.
◮ But traditional readability formulas use only shallow
quantiative features (e.g., lengths of words, sentences).
◮ (e.g., Flesch 1948; Coleman & Liau 1975; Kincaid et al.
1975; DuBay 2004)
◮ Others are exclusively based on lexical measures,
such as occurrence in specific word lists (Dale & Chall
1948; Chall & Dale 1995; Coxhead 2000; Bauer & Nation 1993).
5 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
Some recent CL research on text complexity
◮ Language n-gram models (Collins-Thompson & Callan
2004; Si & Callan 2001)
◮ Machine learning approaches, with several lexical and
syntactic features (Heilman et al. 2007; Petersen &
Ostendorf 2009; Lijun Feng & Elhadad 2010)
◮ What kind of features are relevant here? Insights from
◮ Language Acquisition ◮ Psycholinguistics 6 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
Complexity in language acquisition
Automated L1 acquisition measures:
◮ Measures based on identifying specific syntactic patterns:
◮ Index of Productive Syntax (IPSyn, Scarborough 1990;
Sagae et al. 2005)
◮ Developmental Level (D-Level, Rosenberg & Abbeduto
1987; Covington et al. 2006; Lu 2009)
◮ Some others include (cf., Cheung & Kemper 1992):
◮ Developmental Sentence Scoring (DSS) ◮ Directional Complexity (D-Complexity) ◮ Frazier’s node count, Yngve’s depth
Automated Second-Language Acquisition measures:
◮ Lexical richness (Lu 2011b) ◮ Syntactic complexity in second language writing
(Lu 2010, 2011a; Vyatkina 2012)
◮ Measures mostly based on general counts of phrases,
T-units, clauses, . . .
7 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
Complexity and psycholinguistics
◮ long and colorful history, cf. Derivational Theory of
Complexity (DTC, Fodor et al. 1974)
◮ meaning: propositional idea density (Kintsch 1974;
Turner & Greene 1977; Brown et al. 2008)
◮ form: complexity in human sentence processing (e.g.,
surprisal Boston et al. 2008, 2011)
◮ discourse: text coherence and cohension (Coh-Metrix
project, McNamara et al. 2002)
◮ Link to cognition also relevant for applications:
◮ Cognitively motivated readability assessment for adults
with intellectual disabilities (Feng et al. 2009; Feng 2010)
◮ Using Syntactic Complexity measures to detecting
cognitive impairment (Roark et al. 2007)
8 / 12 On Measures of Text Complexity
Sowmya V.B., Detmar MeurersBackground
What is it? Why is it relevant?Real-life challenge Some measures
Traditional formulas and lexical measures Some recent CL approaches Language acquisition PsycholinguisticsHow to evaluate Work in progress References
How do we evaluate complexity measures?
◮ Measures of readability typically are evaluated against
a gold standard classification of graded readers, which are written with the traditional measures in mind.
◮ What can serve as independently motivated gold
standard for evaluating complexity?
◮ Correlating complexity with cognitive measures
◮ online eye tracking measures identifying processing
difficulty in human sentence processing
◮ working memory decrease in language attrition
(Cheung & Kemper 1992)
◮ Analyzing complexity of the language produced at
different times in first language acquisition
9 / 12