Formation and Learning Analytics? Simon Buckingham Shum Knowledge - - PowerPoint PPT Presentation
Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 July 2, 2014 https://futures.georgetown.edu/formation Learning Emergence Formation and Learning Analytics? Simon Buckingham Shum Knowledge Media
Formation and Learning Analytics?
Simon Buckingham Shum
Knowledge Media Institute, The Open University Visiting Fellow, Grad. School of Education, University of Bristol From August: Connected Intelligence Centre, University of Technology Sydney http://simon.buckinghamshum.net http://twitter.com/sbskmi Formation by Design (FxD) Symposium Georgetown University, Washington, June 30 – July 2, 2014 https://futures.georgetown.edu/formation
1
Emergence ¡ Learning ¡
It’s out of the labs and into products: every learning tool now has an “analytics dashboard” (a Google image search)
2
https://twitter.com/Wiswijzer2/status/414055472451575808
3
https://twitter.com/Wiswijzer2/status/414055472451575808
“Note: ¡check ¡the ¡huge ¡ difference ¡between ¡ knowing ¡and ¡ measuring…” ¡
4
5
c.f. ¡“Data ¡is ¡the ¡new ¡oil” Scaleable ¡ Precise ¡ Quan?fiable ¡ Reprocessable ¡
For Morozov, analytics is where technological solutionism hits education:
6
“This flight from thinking and the urge to replace human judgments with timeless truths produced by algorithms is the underlying driving force of solutionism.”
If something like this is the pedagogical architecture for (trans)formational learning…
7
Structuring Knowledge Performing Generating Learning Power Purposing
If something like this is the pedagogical architecture for (trans)formational learning…
8
Structuring Knowledge Performing Generating Learning Power Purposing
…what roles can computational learning analytics play?
The ¡worldview ¡clash ¡ is ¡what ¡makes ¡this ¡ fascina?ng! ¡
For discussion on how analytics shape educational worldviews
9
http://people.kmi.open.ac.uk/sbs/2014/06/edmedia2014-keynote
10
= opportunity for transformation
11
Liminal space = opportunity for transformation, or withdrawal
12
So while we initially strive to make our students
feel comfortable on campus, we then must help them balance
their desire for security with the need to take risks and explore new ideas and possibilities. Rather than attempting to resolve the tension, a college should help students find their place on this
precarious threshold, in the liminal space between the familiar and strange, the old and the new.
Johansson & Felten, 2014
Liminal space = opportunity for transformation, or withdrawal
13
“It is when you have left the tried and true, but have not yet been able to replace it with anything else. [...] It is when you are between your old comfort zone and any possible new answer. If you
are not trained in how to hold anxiety, how to live with ambiguity, how to entrust and wait, you will run... anything to flee this terrible cloud of unknowing.”
Richard Rohr O.F .M. on the spirituality of liminal space
¡
14
15
Social Network Analysis (SNAPP)
16
What’s going on in these discussion forums?
Social Network Analysis to gain insight into peer-peer and peer-mentor dynamics (SNAPP tool)
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation Bakharia A and Dawson S. (2011) SNAPP: a bird's-eye view of temporal participant interaction. Proceedings of the 1st International Conference
Social Network Analysis (SNAPP)
18
http://www.slideshare.net/aneeshabakharia/snapp-20minute-presentation
2 learners connect
clusters tutor only engaging with active students, ignoring disengaged
Visualizing and filtering online social ties: by topic and type of tie
Schreurs B., Teplovs C., Ferguson R., De Laat M. and Buckingham Shum S. (2013). Visualizing Social Learning Ties by Type and Topic: Rationale and Concept
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
Visualizing and filtering online social ties: by topic and type of tie
24
Dispositional Learning Analytics Workshop
25
http://learningemergence.net/events/lasi-dla-wkshp http://learningemergence.net/2014/03/01/ assessing-learning-dispositions-academic-mindsets
Observation
informal and formal
Self-Diagnostic
informal and formal
Behavioural Analytics
Assessing Learning Dispositions/Mindsets
Future sweetspot... multiple lenses to provoke self-reflection
http://people.kmi.open.ac.uk/sbs/2014/02/assessing-learning-dispositions-academic-mindsets
Self-report through reflective blogging
9-10 yr old EnquiryBloggers • Bushfield School, Wolverton, UK
EnquiryBlogger Wordpress Multisite plugins http://learningemergence.net/tools/enquiryblogger
27
Masters level EnquiryBloggers
Graduate School of Education, University of Bristol
EnquiryBlogger: blogging for Learning Power & Authentic Enquiry http://learningemergence.net/2012/06/20/enquiryblogger-for-learning-power-authentic-enquiry
28
29
EnquiryBlogger teacher’s dashboard – direct navigation to learners’ blogs from the visual analytic
We are quantifying our lives
http://learningemergence.net/2014/03/01/assessing-learning-dispositions-academic-mindsets
Dispositional profile from behavioural traces, to complement self-report?
Social network patterns, teamwork effectiveness and initiation of relationships Questioning, arguing and search behaviours reveal intrinsic curiosity and epistemic commitments Tagging/sharing/ blogging/social patterns reveal how you see connections between ideas Behavioural and somatic traces associated with perseverance, grit, tenacity; overcoming panic/stress when stretched
32
The ¡Storymaking ¡Project: ¡PhD ¡by ¡Joanna ¡Kwiat ¡ hLp://projects.kmi.open.ac.uk/storymaking ¡
1st International Workshop on
Discourse-Centric Learning Analytics
analytics that look beneath the surface, and quantify linguistic proxies for ‘deeper learning’
Beyond number / size / frequency of posts; ‘hottest thread’
http://www.glennsasscer.com/wordpress/wp-content/uploads/2011/10/iceberg.jpghttp://solaresearch.org/events/lak/lak13/dcla13
Why are stories so important to formation?
34
“Fundamentally different to logical argument, the story positively demands those things that the other either denies or restricts: emotion, agency, character, perspective, and so on. To understand a story on some level is to become engaged in its telling and this will normally be achieved by allowing oneself to empathise or identify with the story’s characters or its narrator.”
Discourse analytics on webinar textchat
Ferguson, R. and Buckingham Shum, S., Learning analytics to identify exploratory dialogue within synchronous text chat. In: 1st International Conference
Can we spot the
quality learning conversations in a
2.5 hr webinar?
20 40 60 80 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory
Discourse analytics on webinar textchat
Sheffield, UK not as sunny as yesterday - still warm Greetings from Hong Kong Morning from Wiltshire, sunny here! See you! bye for now! bye, and thank you Bye all for now
Given a 2.5 hour webinar, where in the live textchat were the most effective learning conversations? Not at the start and end of a webinar…
Ferguson, R., Wei, Z., He, Y . and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
20 40 60 80 9:28 9:32 9:36 9:40 9:41 9:46 9:50 9:53 9:56 10:00 10:05 10:07 10:07 10:09 10:13 10:17 10:23 10:27 10:31 10:35 10:40 10:45 10:52 10:55 11:04 11:08 11:11 11:17 11:20 11:24 11:26 11:28 11:31 11:32 11:35 11:36 11:38 11:39 11:41 11:44 11:46 11:48 11:52 11:54 12:00 12:03 12:04 12:05 Average Exploratory
Discourse analytics on webinar textchat
…but if we zoom in on a peak…
Ferguson, R., Wei, Z., He, Y . and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
Discourse analytics on webinar textchat
100 9:28 9:40 9:50 10:00 10:07 10:17 10:31 10:45 11:04 11:17 11:26 11:32 11:38 11:44 11:52 12:03 Averag Classified as “exploratory talk” (more substantive for learning)
“non- exploratory”
…language is used in a manner more akin to “Exploratory Talk” (Neil Mercer)
Ferguson, R., Wei, Z., He, Y . and Buckingham Shum, S., An Evaluation of Learning Analytics to Identify Exploratory Dialogue in Online Discussions. In: Proc. 3rd International Conference on Learning Analytics & Knowledge (Leuven, BE, 8-12 April, 2013). ACM. http://oro.open.ac.uk/36664
Rhetorical discourse analytics
41
OPEN QUESTION: “… little is known …” “… role … has been elusive” “Current data is insufficient …” CONTRASTING IDEAS: “… unorthodox view resolves …” “In contrast with previous hypotheses ...” “... inconsistent with past findings ...” SURPRISE: “We have recently observed ... surprisingly” “We have identified ... unusual” “The recent discovery ... suggests intriguing roles”
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Rhetorical discourse analytics
42
Human analyst
Rhetorical discourse analytics
43
Human analyst Computational analyst
http://technologies.kmi.open.ac.uk/cohere/2012/01/09/cohere-plus-automated-rhetorical-annotation De Liddo, A., Sándor, Á. and Buckingham Shum, S., Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation
Rhetorical discourse analytics
44
Duygu Simsek’s PhD: http://people.kmi.open.ac.uk/simsek/research/
Glimpses of analytics capable of detecting higher order thinking. But humans will always read differently to machines Can we correlate this with “academic writing”, and can such analytics be used as formative feedback on drafts?
Rhetorical discourse analytics
45
Simsek D, Buckingham Shum S, Sándor Á, De Liddo A and Ferguson R. (2013) XIP Dashboard: http://oro.open.ac.uk/37391
CONTRAST SUMMARY & CONTRIBUTION
Evidence Hub: semantic storytelling for students, practitioners and researchers
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
A wizard guides the user through the submission of a structured story:
addressing?
supports/challenges this?
themes
Evidence Hub: Argument Maps
Systems Learning & Leadership Evidence Hub: http://sysll.evidence-hub.net
The wizard then generates a structured Knowledge Tree showing evidence-based claims (and disagreements)
http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
An alternative graphical network rendering can be generated to show the current Argument Map of supporting and challenging evidence around an Issue
Evidence Hub: Argument Maps
Evidence Hub: professional development
http://learningemergence.net/2013/07/17/deed-elli-ai-ci-systemic-school-learning
Issue ¡ Poten?al ¡ Solu?on ¡ Suppor?ng ¡ Evidence ¡ (prac??oner ¡ story) ¡
50
De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-centric learning analytics. 1st Int. Conf. Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr). ACM: New York. Eprint: http://oro.open.ac.uk/25829
Argumentation analytics (in Debate Hub & Cohere apps)
what epistemic contributions are learners making in the network?
51
Rebecca is playing the role of broker, connecting different peers’ contributions in meaningful ways We now have the basis for recommending that you engage with people NOT like you…
Generates novel visualizations to summarise community arguments on a given issue
http://consider.it
Structured Deliberation/Argument Maps
Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
What is it to ‘know’ when we search? http://sjgknight.com/finding-knowledge/2014/02/knowledge-in-search Danish exams permit Net: http://sjgknight.com/finding-knowledge/2013/07/danish-use-of-internet-in-exams-epistemology-pedagogy-assessment Epistemic networks for epistemic commitments: http://oro.open.ac.uk/39254
Learning sciences research into Epistemic Commitments/ Cognition combined with ‘researching search’ methodologies.
Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
Dimensions of Epistemic Belief
Certainty The degree to which knowledge is conceived as stable or changing, ranging from absolute to tentative and evolving knowledge Simplicity The degree to which knowledge is conceived as compartmentalised or interrelated, ranging from knowledge as made up of discrete and simple facts to knowledge as complex and comprising interrelated concepts Source The relationship between knower and known, ranging from the belief that knowledge resides outside the self and is transmitted, to the belief that it is constructed by the self Justification What makes a sufficient knowledge claim, ranging from the belief in observation
Knight, Simon; Buckingham Shum, Simon and Littleton, Karen (2014). Epistemology, assessment, pedagogy: where learning meets analytics in the middle space. Journal of Learning Analytics (In press). http://oro.open.ac.uk/39226
Shifts in epistemic commitments?
(Simon Knight, KMi PhD research)
Behavioural Proxies of Epistemic Belief?
Certainty
Frequency and density of positive and negative polarity links (e.g. supports; builds on, is consistent with, versus challenges; refutes; is inconsistent with). Presence of stability markers – (e.g. current sources, bibliographic references, geographical spread). Frequency and density of meta-discourse markers in the text of nodes, which moderate assertions in scholarly ways (e.g. in contrast to; remains an unresolved issue)Simplicity
Standard network analytics describing the topography of the structure Semantic analysis of node and link types describing the variety of types used by a learner, the balance in link polarity (positive/neutral/negative), and the balance of propositional (causes; refutes; solves) versus analogical thinking (is a metaphor of; is analogous to). Semantic analysis of tags on nodes and links Encapsulation of node clusters or sub-networks within a broader category expressing a higher order construct The use of different argumentation schemes, which reflect different sources of authority (argument from expertise; argument by analogy; argument from precedent) In collaborative maps, the social network view showing who is connected to whom, but what kinds of linksSource
Presence within the text of nodes of ‘I think’ or other restatements of fact Challenges to claims through negative link types and meta-discourse markers in node text Few additional nodes made other than those created as quotations.Justification
Judgements of relevance, and supporting or explanatory notes (‘this evidences/ explains x’). Ties to method ‘ideas’. Use of well known Argumentation Schemes, especially evidence and arguments that grounded in the attribution of authority to people (‘experts’) Deakin Crick, R., Barr, S., Green, H. and Pedder, D. (2013). Evaluating the Wider Outcomes of Schools: Complex Systems Modelling. Centre for Systems Learning & Leadership, Graduate School of Education, University of Bristol, UK. http://learningemergence.net/library/reports/
Providing a more complete analytic on the health of the educational community
57
“Test grades are just ONE of a dozen indicators in
intelligence • “wellbeing”…
Hierarchical Process Modelling (Univ. Bristol Perimeta tool)
58
Seeing a learning community as a complex adaptive system requires the voices of learners, teachers, leaders, parents and ‘external’ stakeholders
The core mission and values of a network of school academies
60
Hierarchical Process Modelling transforms qualitative and quantitative inputs into a multi-level visual analytic
‘Italian Flag’ visual analytic
61
Degree of green / white / red reflects current (un)certainty over availability of evidence: supporting / unknown / challenging
1 2 3 4 5
Envisioning a social learning analytics dashboard
Ferguson R and Buckingham Shum S. (2012) Social Learning Analytics: Five Approaches. Proc. 2nd International Conference on Learning Analytics &
63
Use of computational techniques to reflect back to learners apparent shifts in • how they engage in a community of enquiry/practice • how they narrate their learning journeys • how they negotiate meaning • how their worldview seems to have shifted • how they respond cognitively (even somatically) when challenged.