Selection of irrelevant navigation paths during digital reading: - - PowerPoint PPT Presentation

Selection of irrelevant navigation paths during digital reading: Underlying cognitive processes

Ladislao Salmerón ERI Lectura, University of Valencia, Spain

ProcessDataConf2019

Overview

• Navigation paths

– Indicators – Interpretation – Context dependencies

• Study
• Conclusions

ProcessDataConf2019

Navigation paths

ProcessDataConf2019

· Major difference between print and digital reading is the presence of hyperlinks that create non-sequential page structures (OECD, 2011, 2015) · Strong association between quality of students’ navigation and digital reading … … at country (OECD, 2015) and at individual (e.g.

Naumann & Salmerón, 2016) level.

… even after accounting for performance in print reading. In digital reading, process data is already (partially) accounted in performance scores.

Navigation paths

ProcessDataConf2019

(OECD, 2015)

Navigation paths

ProcessDataConf2019

· “Savvy digital readers are expected to navigate across pages that contain information relevant for the question to be answered, or the task to be solved” · Process indicators of navigation quality (Naumann, 2015)

• 1. Number of relevant pages visited
• 2. Total number of pages visited
• 3. Derived indicators (% relevant pages)

1 2 3 4 A page is Relevant if information for the response must be:

• located in p. 4
• related p. 1 to p.4
• related 1-2-3-4

Navigation paths

ProcessDataConf2019

A page is irrelevant if it doesn’t contain information useful to answer the question, or is not in the path guiding to a relevant one

Navigation paths

ProcessDataConf2019

· Cognitive processes underlying navigation: a) To relevant pages > inferential process · Print reading comprehension (Hahnel, Goldhammer, Kröhne,

& Naumann, 2017, 2018; Naumann, Richter, Christmann, & Groeben, 2008; Naumann & Salmerón, 2016; OECD, 2011, 2015; Salmerón & García, 2011)

· Progression across high-school years (Keil & Kominsky,

2013; Salmerón, García & Vidal-Abarca, 2018)

Navigation paths

ProcessDataConf2019

· Cognitive processes underlying navigation: b) To irrelevant pages: · Generally interpreted as ‘Lostness’ (Shapiro &

Niederhauser, 2004)

· Its isolated effect has been less studied in the past

Navigation paths

ProcessDataConf2019

· Scenarios in the Digital reading assessment in PISA include highly differentiated sections.

Navigation paths

ProcessDataConf2019

· Can visits to irrelevant pages also represent students’ attempts to capture a complete view of the theme?

• Self-regulated vs. cue dependent navigation

(Balcytiene, 1999)

· This aspect should be more obvious in digital texts that are highly topically interconnected, such as Wikipedia texts, than in scenarios with highly differentiated sections, such as web portals.

Goal

ProcessDataConf2019

· Test if visits to irrelevant pages differ across digital contexts (less / more interconnected):

• Relationship with print reading skills
• Relationship with digital reading scores
• Potential mediation between the former

Study

ProcessDataConf2019

Participants 555 students (12-16 years old) · 5 schools (2 rural, 3 urban) · 40 intact classes Exclusion criteria: learning disability, curricular adaptation In class participation. Two sessions of 50’

Grade 7th 154 8th 141 9th 143 10th 137

ProcessDataConf2019

• Students’ print comprehension skills (reading literacy)

were assessed using a standardized paper and pencil test

(Llorens et al., 2011).

• Texts:

– a textbook excerpt about global warming, – an article from a popular science magazine about the language

• f the bees,

– a pamphlet about the postural ergonomics of work chairs, – two letters to the editor about the pros and cons of nuclear energy, – a newspaper diagram about traffic accidents.

• Each unit included 3-5 questions covering the different

aspects suggested by the PISA framework

• Cronbach’s alpha = .80.

Method

Method

ProcessDataConf2019

· Two web reading scenarios were designed following the PISA (OECD, 2009) scheme. · Participants were given one question at a time, and were told to use the information from the web pages provided to answer the question. · Questions were constructed following three aspects (OECD, 2009): access and retrieve, integrate and interprete, and reflect and evaluate. · Heterogeneous relationship between question and relevant hyperlinks: word match, paraphrases, irrelevant word matches (Cerdán, Gilabert, & Vidal-Abarca, 2011)

Method

ProcessDataConf2019

· Web portal scenario:

• Portal dedicated to young people
• Divided into five categories (sports, courses, technology,

nature and health), with three subcategories each.

• 12 questions required that participants clicked on at

least two (maximum seven) links to access relevant information to answer the question, and involved both textual and graphic information, such as tables or maps.

Method

#19. You would like to learn how to sing, so you decide to enroll in a course offered by the Youth Portal. Search the address for the center. a) St. Del Padre Vitoria b) St. Mayor 25 c) St. De Donoso Cortés d) St. Goya

ProcessDataConf2019

Example question from Web portal scenario

ProcessDataConf2019

Method

ProcessDataConf2019

· Wikipedia scenario:

• Main document on the topic of the French Revolution
• Included 10 subsections and 24 embedded hyperlinks to

additional Wikipedia pages.

• 6 questions required that participants clicked on at

least one (maximum two) embedded hyperlink(s) to access relevant information to answer the question.

Method

#13. Why the policy of the Antient Regime was against the new social ideas from the French Revolution? Because… a) … contrary to the Ancient Regime, the new society valued only daily life problems. b) …social ideas from the Ancient Regime were based on the power of the army while new ideas were based on reason. c)… in the new regime the philosophers governed society and in the old regime monarchs and the Church ruled. d) … the new social ideas were based on reason, and the old in divine law.

ProcessDataConf2019

Example question from Wikipedia scenario

ProcessDataConf2019

Method

• Log-files indicators:

– Number of unique pages visited – Relevant and irrelevant to the task – By scenario (Wikipedia and Web portal)

ProcessDataConf2019

Results

ProcessDataConf2019

Min Max Mean SD Asymmet ry Skewness

Unique Relevant visits Wikipedia 1.67 .37 .33 .41

• .74

Unique Irrelevant visits Wikipedia .33 .06 .10 1.28 .72 Unique Relevant visits Web Portal 3.33 2.0 .74

• 1.17

.63 Unique Irrelevant visits Web Portal .33 .10 .40 .39

• .59

Descriptive scores for navigation indicators, averaged by question

ProcessDataConf2019

Print reading comprehension Relevant visits Digital reading comprehension Irrelevant visits

Results

r = .45* r = -.15* r = .53* r = -.13*

Web portal scenario: Pearson correlations

Results

Web portal scenario

Visits Visits

Print Reading comprehension

Digital Reading comprehension

Unique relevant visits Unique irrelevant visits

ProcessDataConf2019

ProcessDataConf2019

Print reading comprehension Relevant visits Digital reading comprehension Irrelevant visits

Results

r = .40* r = .15* r = .61* r = .31*

Wikipedia scenario: Pearson correlations

Results

Wikipedia scenario

Unique relevant visits Unique irrelevant visits

Visits Visits

Print Reading comprehension

Digital Reading comprehension

ProcessDataConf2019

ProcessDataConf2019

Print reading comprehension Relevant visits Digital reading comprehension Irrelevant visits

Results

.45*

• .15*

.42*

• .17*

Web portal scenario: Mediation analysis (Bootstraping, Preacher & Hayes (2008))

c’= .34* Indirect effects:

• relevant visits: .18 [.14, .24]
• irrelevant visits: .03 [.01, .05]

R2

adj = .43*

ProcessDataConf2019

Print reading comprehension Relevant visits Digital reading comprehension Irrelevant visits

Results

.40* .15* .56*

• .02, n.s.

Wikipedia scenario: Mediation analysis (Bootstraping, Preacher & Hayes (2008))

c’= .22* Indirect effects:

• relevant visits: .22 [.17, .27]
• irrelevant visits: -.01 [-.02, .01]

R2

adj = .44*

Conclusions

• Irrelevant visits in the Web portal scenario,

with highly differenciated topic structure:

– Negative correlations with print reading comprehension and digital reading performance – Indirect effect of print reading comprehension to digital reading performance.

ProcessDataConf2019

Conclusions

• Irrelevant visits in the Wikipedia scenario,

with highly interconnected topic structure:

– Positive correlations with print reading comprehension and digital reading performance – It adds no variance to digital reading performance.

ProcessDataConf2019

Conclusions

• Underlying cognitive processes?

– In loosely interconnected texts: Distraction? (must be independent of inferential processes, accounted for by relevant visits effects). – In highly interconnected texts: Byproduct of inferential processes (exploration, curiosity,

Balcytiene, 1999)

• Need to complement log-files with

qualitative data (verbal protocols, interviews)

ProcessDataConf2019

Conclusions

• Practical implications:

– Consider ‘irrelevant visits’ with caution when building indices of ‘navigation quality’. – Navigation indices may be dependent on the interconnectedness of the scenarios – ‘Unique relevant visits’ alone are a strong predictor of performance across scenarios

ProcessDataConf2019

Thanks for your attention

www.uv.es/lectura www.uv.es/lasalgon Twitter: LSalmeron_UV

ProcessDataConf2019

Unique relevant visits Unique irrelevant visits

Materials

• Q: at your school garden your class is

cultivating some plants on danger. Do you think you could cultivate a ‘Clover water’?

Spanish plants on danger

One of these [extingueshed plants] is the ‘Clover water’. Scientists have found spores of it, which allows its use in the lab and its reintroduction in nature.

Materials

• Q: The Youth Portal just posted a report on

the extinction danger of mammals. What is the extinction level in Canada?

Animals in danger of extinction across the planet

Extinction risk varies according to the continent