How to Define, Assess and Develop Critical Thinking in - - PowerPoint PPT Presentation

How to Define, Assess and Develop Critical Thinking in Professional Development Contexts - Reflections and Empirical Evidence

Antonella Poce Associate Professor in Experimental Pedagogy at Roma Tre University Antonella.poce@uniroma3.it

Aims for today today…

The unexamined life is not worth living Socrates

• Understanding better

the role of Critical Thinking and Information Literacy in the 21st century society;

• Presenting some ideas

regarding how to improve these skills;

• Showing the results of

my research on the automatization of CT assessment in open- ended answers;

The Centre for Museum Studies (CDM)

CDM was set up in 1994 to encourage a more conscious and targeted use of museum heritage, bearing in mind:

the insufficient use made of the huge wealth of heritage assets available in Italy for educational aims the gap between Italy and other countries as regards cumulative knowledge and definition

• f models of intervention

research in the specific field of education in

• rder to work out new strategies through

museum heritage

CDM

Antonella Poce Francesco Agrusti Maria Rosaria Re Francesca Amenduni Carlo De Medio Mara Valente

CD CDM M Nation

• nal and Intern

rnation

• nal Pr

Projects

National Project

Inclusive Memory Interact

European Projects

CrithinkEDU OpenVM DigiCulture

Post- graduate courses

Advanced Studies in Museum Education Museum Education. Theoretical aspects

The leitmotif of

• ur research

Promote and assess thinking skills and a critical use of digital technology by learning through cultural heritage, arts and literature enhanced by technologies.

Why Critical Thinking is so critical today?

Educational policy makers have recognized from many years Critical Thinking as an essential driver for progress and knowledge growth in any field and in the broad society (World Economic Forum; UNESCO; OECD).

New (unpredictable?) challenges for Critical Thinkers

• We know that the current pandemic is related in different ways

to technological progress, for better and for worse.

• As a researcher and professional in the field of education, I am

concerned about a specific kind of relation between the Covid- 19 and another well-known effect of technological progress, also known as information overload.

• When during a pandemic a big amount of new information is

produced, we face what the World Health Organization call infodemic. Infodemic was defined as an over-abundance of information – some accurate and some not – that makes it hard for people to find trustworthy sources and reliable guidance when they need it (World Health Organization, 2019). WHO (2019) Novel Coronavirus(2019-nCoV) Situation Report –

• 13. Retrieved from: https://www.who.int/docs/default-

source/coronaviruse/situation-reports/20200202-sitrep-13-ncov- v3.pdf?sfvrsn=195f4010_6

The WHO was interested in infodemic because of the danger of misinformation during the management of virus

• utbreaks, since it could

even speed up the epidemic process by influencing and fragmenting social response.

The different type of mis and disinformation https://firstdraftnews.org/latest/fake-news-complicated/

• One poll by YouGov and the

Economist in March 2020 found 13% of Americans believed the Covid-19 crisis was a hoax while a whopping 49% believed the epidemic might be man-made. https://docs.cdn.yougov.com/1ghnp qhhpu/econToplines.pdf

• The New York Times reported that

several videos featuring the Gates conspiracy had been viewed over 160,000 times on TikTok (The New York Times, 6th February 2020) https://www.nytimes.com/2020/02/0 6/health/coronavirus- misinformation-social-media.html

What should we learn?

• We need to develop in

citizens the anti-bodies to be prepared to autonomous interpret and assess the information that they receive in complex digital eco- systems and to be responsible for the information they share;

• These antibodies are Critical

Thinking and Information Literacy.

Source: World Economic Forum https://www.weforum.org/agenda/2020/05/children- digital-risks-cybersecurity-screentime-covid19/

What is Critical Thinking?

Skills: e.g. analyse, evaluate, make inference, argument; Dispositions: open-minded; skeptical; inquisitive

According to McPeck (1981) “Critical Thinking is a process where an interaction occurs between individuals and the interpretation of knowledge which they contribute to create”. In his definition

• f CT, McPeck not only includes internal cognitive

activities, but also a disposition to act in a critical, social and interactive way in order to interpret and create knowledge.

Nowadays, a great amount of information exchange happens online.

From Critical Thinking to Information Literacy

While some similar skills are required to read both online and printed documents, online texts

• ften pose greater challenges to

readers than printed texts (OCSE, 2015):

• multimodality;
• The presence of hyper-links (non

sequential texts); Text-processing + navigation

PISA Students, Computers and Learning Making the Connection: Making the Connection. Retrieved from http://www.oecd.org/publications/students-computers-and-learning-9789264239555-en.htm

Digital / Information Literacy

Calvani, Fini and Ranieri (2009) summarize digital literacy as a combination of concrete and unquantifiable skills

Covello, S., & Lei, J. (2010). A review of digital literacy assessment

• instruments. Syracuse University, 1-31.

Information Literacy Skills

Intra-personal Inter-personal

Katz, I. R. (2005). Beyond technical competence: Literacy in information and communication technology. Educational Technology, 45(6), 44-47.

Putting here the poll!

Three open ended questions. Time 1 minute and 30 seconds. Q1) Emily’s father has three daughters. The first two are named April and

• May. What is the third daughter’s name?

Q2) If you’re running a race and you pass the person in second place, what place are you in? Q3) Go to the DATA section of the OECD official website http://www.oecd.org/ and look for data relating to employment in the ICT sector: "ICT employment". Which country has a percentage of 4.7?

Q1) Emily’s father has three daughters. The first two are named April and May. What is the third daughter’s name? Emily Q2) If you’re running a race and you pass the person in second place, what place are you in? Second Q3) Go to the DATA section of the OECD official website http://www.oecd.org/ and look for data relating to employment in the ICT sector: "ICT employment". Which country has a percentage of 4.7? Japan

Q1 + Q2 Cognitive Reflection Test (Sirota

• et. al., 2018) – CT

dispositions Q3 Webtrotter (2019) Information Literacy (Goal oriented navigation)

How can we develop IL and CT skills?

Research highlights that social interaction and language (verbal, mathematical, figurative) are fundamental tools for the development of higher order cognitive skills:

• deep “reading” (e.g. literary texts; pieces
• f art; museum objects and artifacts);
• writing and building knowledge artefacts

(e.g. essays; object-based learning; digital story telling);

• dialogic interactions, both online and
• ffline. (From Socratic tradition);

Source: World Economic Forum Blog: https://www.weforum.org/agenda/2020/06/c

• vid-19-museum-kids-action-plan/

Online dialogic interactions

Research suggests that asynchronous online discussions can support critical thinking and knowledge co-construction even more than synchronous face-to-face discussions, due to the available time for reflection (Wang et al., 2009; Meyer, 2003) and more critical, constructive and explicit contributions. (Wang et al., 2009).

Meyer, K. A. (2003). Face-to-face versus threaded discussions: the role of time and higher order thinking. Journal of Asynchronous Learning Networks, 7(3), 55-65. Wang, Q., Woo, H. L., & Zhao, J. (2009). Investigating critical thinking and knowledge construction in an interactive learning

• environment. Interactive learning environments, 17(1), 95-104.

The assessment challenge

Although we know that online discussions and activities could be designed to promote CT and IL, there are still some issues:

• How to monitor these activities?
• How could we assess the quality of

dialogic interactions, especially in discussion forums of Massive Open Online Courses or big classroom?

How to assess CT in online discussion?

In recent years, to face the challenge of the analysis of big amount of educational new research field have been developed named Learning Analytics and Educational Data mining (Liñán, & Pérez, 2015).

• Learning analytics focuses on extracting meaning from large

amounts of data in educational digital environments.

• Educational data mining adapts statistical, machine-learning

and data-mining methods to study educational data generated basically by students and instructors.

• Recent research describe the

development and the validation of automatic tools for the assessment of CT sub-skills, such as reasoning (Mao et al., 2018) or argumentation (Song, Heilman, Klebanov, & Deane, 2014).

• Having said that, the authors

suggest that there are still

• pen-challenges in terms of

validity and reliability of the measures.

Go Goal al of re researc rch in CT T au automatic tic as asses essmen ent

• For the last years, I have been working with my research group on

the development of a prototype aimed at automatic assessing six sub-skills of CT (Poce, 2017): use of language, argumentation,

relevance, importance, critical evaluation and novelty.

• Even though different tools have been developed to assess one or

more of these sub-skills of CT, this prototype is designed all of them together, based on the use of Natural Language Processing (NLP) techniques.

• The primary research question of this work is: how reliable it is

when using NLP approach to evaluate and score CT related subskills, and whether this reliability is comparable to the case when human judgement is employed.

Macro-indicators Indicators

Descriptors Marks

Use of language Language ability (punctuation, spelling, morphosyntax, lexicon)

• a. rich and original
• b. appropriate
• c. mainly correct
• d. not precise
• e. not correct and improper

Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Justification / Argumentation Elaboration ability (thesis definition and elements of reasoning)

• a. rich and articulate
• b. clear and ordered
• c. too synthetic
• d. quite consistent
• e. inconsistent

Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Relevance Consistency (the topic under issue is mentioned)

• a. complete, deep and original
• b. complete and correct
• c. generic
• d. partial
• e. out of line

Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Importance Knowledge of the topic (main issues related to the topic are mentioned)

• a. deep and critical
• b. complete
• c. appropriate
• d. superficial
• e. not sufficient

Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Critical evaluation Personal and critical elaboration of sources and background

• a. critical and well sounded
• b. wide and adequate
• c. essential and simple
• d. partial
• e. contradictory

Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Novelty New information, ideas and solutions are added to discuss the issues raised in the questions

• a. widely, critically and originally;
• b. in detail;
• c. correctly;
• d. simply and or partially
• e. no new information and solutions are

added Excellent Very good Good Insufficient Clearly insufficient 5 4 3 2 1

Data collection

Data were collected in two steps.

• In 2019, a group of 64 college/university

teachers were invited to participate in workshops in the USA, Italy, and

• Belgium. In these workshops,

participants were asked to answer different kinds of constructed-response questions .

• The second step (in progress) included

200 university students from Italy. Each was asked to read an extract from Galilei’s book “Dialogue on the Two Chief World Systems” and write an essay based on 5 questions.

Data analysis

• Two human assessors rated

these responses collected in both steps and provided ratings on each of the six subskills (Poce, 2017) from a minimum of 1 to a maximum

• f 5.
• The same open-ended

answers were assessed by the prototype through different kinds of methodologies.

Indicators Features 2019 version Features implemented in 2020 version Use of language Misspellings and grammatical errors i) misspelling and grammatical errors, ii) frequency of words and iii) lexical diversity Argumentation NO The prototype is trained at distinguishing discourse categories, checking: tense verbs; polarity, and arguing lexicon. Relevance It is obtained by crossing the concepts extracted from the answers with the concepts defined a priori. It is evaluated using Latent Semantic Analysis (LSA) Importance It is assessed by extracting the concepts contained in the texts of the question and in the answer and calculating the numbr

• f intersectios between those sets of

word. It is obtained by means of Intelligent Essay Assessor. Critical Evaluation NO NO Novelty Itis obtained by crossing the successors extracted from the answers with the successors defined a priori. Novelty is assessed through LSA and TF-IDF (Term Frequency-Inverse Document Frequency).

Results – correlation between macro-indicators and total score

The macro-indicators total correlation in the human assessment is presented on the

• right. All the macro-indicators showed high

correlation with total critical thinking score and therefore they all had good discriminating power. The table on the right presents the prototype’s macro-indicators total

• correlation. The macro-indicators showed

from moderate to high correlation with total score.

Macro- indicator Macro-indicators-Total Correlation Sign. UOL 0.860** 0.000 ARG 0.887** 0.000 REL 0.878** 0.000 IMP 0.873** 0.000 CE 0.903** 0.000 NOV 0.845** 0.000 Macro- indicators Macro-indicators-Total Correlation Sign. UOL 0.575** 0.003 IMP 0.495* 0.14 UOL 0.595* 0.03 REL 0.478* 0.018

Results – comparison of average scores given by the prototype and human rater

in paraphrase the prototype provides higher score than human raters for the macro-indicators Use of Language and

• Relevance. On the other hand, the

average score for the indicator Importance is slightly higher for human raters than in the prototype. In the commentary, there is a general trend of the prototype to provide lower scores comparing to the human raters. However, it is possible to see that the differences between the average scores for the Use of Language scores and Novelty in the commentary is quite low.

Results – inter-rater realiability

• The rubric for CT assessment

shows good properties, with satisfactory reliability between two human.

• The best correlation among human

raters and CT prototype 1.0 were

• btained for the macro-indicators

Relevance (r = 0,47) in the commentary and Importance, both in the paraphrase (r= 0,45) and commentary (0,45). However, the

• verall reliability could be not

considered satisfactory yet.

Macro- indicator H-H Correlation H-H Quadratic Weighted Kappa Paraphrase_Us e of Language 0,911* 0,83* Commentary_ Use

• f

Language 0,745* 0,618* Paraphrase_Re levance 0,75* 0,682* Commentary_R elevance 0,881** 0,811* Paraphrase_Im portance 1,000** 1,000* Commentary_I mportance 0,642 0,571

Conclusive remarks

• In line with previous research (Liu et al., 2014), human raters tended to assign higher scores than our

CT assessment prototype 1.0 in the commentary. On the other hand, in the paraphrase the prototype assigned higher scores than human raters on the macro-indicators Relevance and Importance. This result could be explained because the prototype is designed to infer concepts from the questions and answers texts. In the paraphrase, the participants are required to report all the text’s topics. In this condition, the prototype easily identifies all the concepts, without the need of further analysis. For these reasons, in paraphrase exercise the macro-indicators Relevance and Importance could obtain higher scores than the other macro-indicators and, more in general, than commentary or argumentation texts. This data leads us to think that it may be necessary to apply changes to the evaluation of the macro-indicators based on the type of stimulus given to the participants (paraphrase, argumentation, commentary, poetry).

• We are continuing the analysis of the data collected in the 2nd step. Though early findings of this

study suggest that the NLP approach appears to have a lower level of rating quality than human raters, more research seems necessary to explore features and possibilities to improve such rating quality in the future.

References

• Cinelli, M., Quattrociocchi, W., Galeazzi, A., Valensise, C. M., Brugnoli, E., Schmidt, A. L., ... & Scala, A. (2020). The covid-19 social media infodemic. arXiv preprint arXiv:2003.05004. Retrieved from:

https://arxiv.org/pdf/2003.05004.pdf

• Covello, S., & Lei, J. (2010). A review of digital literacy assessment instruments. Syracuse University, 1-31.
• Liñán, L. C., & Pérez, Á. A. J. (2015). Educational Data Mining and Learning Analytics: differences, similarities, and time evolution. International Journal of Educational Technology in Higher Education, 12(3), 98-112
• Katz, I. R. (2005). Beyond technical competence: Literacy in information and communication technology. Educational Technology, 45(6), 44-47
• Katz, I. R. (2007). Testing information literacy in digital environments: ETS's iSkills assessment. Information technology and Libraries, 26(3), 3-12.
• Mercier, H., & Sperber, D. (2011). Argumentation: its adaptiveness and efficacy. Behavioral and Brain Sciences, 34(2), 94.
• Meyer, K. A. (2003). Face-to-face versus threaded discussions: the role of time and higher order thinking. Journal of Asynchronous Learning Networks, 7(3), 55-65.
• OCSE (2015): PISA Students, Computers and Learning Making the Connection: Making the Connection. Retrieved from http://www.oecd.org/publications/students-computers-and-learning-9789264239555-en.htm
• Pennycook, G., McPhetres, J., Zhang, Y., & Rand, D. (2020). Fighting COVID-19 misinformation on social media: Experimental evidence for a scalable accuracy nudge intervention.
• Poce, A., Amenduni, F., De Medio, C., & Re, M. R. (2019). Road to Critical Thinking automatic assessment: a pilot study. Form@ re-Open Journal per la formazione in rete, 19(3), 60-72.
• Sirota, M., Kostovicova, L., Juanchich, M., Dewberry, C., & Marshall, A. C. (2018). Measuring cognitive reflection without maths: Developing and validating the verbal cognitive reflection test.

https://www.researchgate.net/publication/329463465_Measuring_cognitive_reflection_without_maths_Developing_and_validating_the_Verbal_Cognitive_Reflection_Test/figures

• Tremblay, K., Lalancette, D., & Roseveare, D. (2012). AHELO: Feasibility study report: Volume 1: Design and implementation. Paris, France: Organisation for Economic Cooperation and Development.
• Wang, Q., Woo, H. L., & Zhao, J. (2009). Investigating critical thinking and knowledge construction in an interactive learning environment. Interactive learning environments, 17(1), 95-104.
• WHO (2019) Novel Coronavirus(2019-nCoV) Situation Report – 13. Retrieved from: https://www.who.int/docs/default-source/coronaviruse/situation-reports/20200202-sitrep-13-ncov-v3.pdf?sfvrsn=195f4010_6