Personal Analytics: Getting off the deficit path Professor Gregor - - PowerPoint PPT Presentation

Personal Analytics: Getting off the deficit path

Professor Gregor Kennedy The University of Melbourne

What are Personal Analytics?

http://birdsontheblog.co.uk/getting-fitter-and-healthier-with-the-fitbit/

What are Personal Analytics?

https://gigaom.com/2011/11/07/is-klout-crossing-the-line-when-it-comes-to-privacy/

What are Personal Analytics?

Big Data = Analytics

https://www.linkedin.com/today/post/article/20140312180810-246665791-the-future-of-big-data-and-analytics

Learning Analytics is all the Rage

With Great Promise

• Detect potential “at risk” students
• Formative and summative feedback to students on their

learning processes and outcomes

• Assist with evidence-based resource allocation
• Improve institutional decision-making and responsiveness to

known challenges

• Promote a shared understanding of institutional

successes and challenges

• Academic research and development

(Long & Siemens, 2011)

Defining Analytics

Society for Learning Analytics Research (2011)

Learning Analytics is the measurement, collection, analysis and reporting of data about learners and their contexts for the purposes of understanding and optimizing learning, and the environments in which it occurs. Academic Analytics is the improvement of organizational processes, workflows, resource allocation, and institutional measurement through the use of learner, academic, and institutional data. Managers, Administrators, Funders Learners, Educators, Teachers

THIS BIT IS NOT NEW

Two Traditions

Intelligent Tutoring Systems Interactivity Research

Two Traditions

Interactivity Research

Taxonomies of Interaction

Taxonomies and Classifications e.g. Thompson & Jorgenson (1989)

Reactive Interactive Proactive

Interactivity Research

Taxonomies and Classifications e.g. Schwier & Misanchuk (1993)

Reactive Proactive Mutual

Interactivity Research

Taxonomies of Interaction

Interactivity Research

Concerns about the past

• Often use fairly raw metrics, simple

student measures and inputs (e.g. MCQs, simple access counts).

• Largely descriptive (useful) but
• ften fails to complete the feedback

loop to students and/or teachers.

Two Traditions

Intelligent Tutoring Systems Interactivity Research

Two Traditions

Intelligent Tutoring Systems

Student Model Pedagogical Model Domain Knowledge Feedback

Two Traditions

Intelligent Tutoring Systems

Flag the Error Explain the Error Give a Hint Show a worked example

(Mike Timms)

Intelligent Tutoring Systems

• “ITS were recognised as narrow and brittle” (Cumming & McDougall, 2000)
• … heavily reliant on educational programs and applications that

had defined or discrete stages and steps.

• They were often tied to a program and were not generalisable.

Concerns about the past

Interactivity Research

Student Smart System Assess Diagnose Recognise Personal Adaptive

Intelligent Tutoring Systems

Two Traditions Combined

Two Traditions

Intelligent Tutoring Systems Interactivity Research Drill and Practice Procedural Simulation Conceptual Simulation

Drill and Practice

Student Path A B C Content Content Content Content Content A B C A B C A B C A B C Feedback Content

X

Procedural Simulation

Implicit Feedback Explicit Feedback

X

Student Path

Conceptual Simulation

X

Student Path Implicit Feedback Explicit Feedback

Back to (Today’s) Analytics

Detect “At Risk” Students for Retention Teaching & Learning Research, Evaluation & QA Personalised or Adaptive Feedback for Learning

How Today’s Analytics are Used

✓

✓

✓

• Purdue University’s “Signals”
• Used to predict students who are “at risk”
• Individual student risk is predicted using an algorithm

based on data from four sources: − Performance … “points earned in a course to date” − Effort … interaction with the learning management system as compared to peers − Academic history … e.g. GPA, prior academic history − Student characteristics … e.g. residency, age

(Arnold & Pistilli, 2012)

#1 … “At Risk” Analytics

• Post to students’ LMS
• Email or text students
• Refer them to an advisor
• Call for a chat

(Arnold & Pistilli, 2012)

“At Risk” Analytics

How Today’s Analytics are Used

Analytics = Diagnosing Deficit Path

Preferred Parameters or Pathway Student Path Feedback Assess, diagnose and recognise “deficit”

The field of educational technology has always been interested in using students’ digital traces to assess and diagnose when they move away from preferred learning pathways.

X

This approach has useful pedagogical applications … but

Macro: Attrition Micro: Drill & Practice

Deficit Pathways

So what?

Is this a problem?

The Promise of Learning Analytics

How can we …

• harness different data analysis techniques
• for the provision of more meaningful feedback
• to students on their learning processes
• in real time
• for genuinely personlised learning environments?

A core promise of learning analytics is to improving students’ micro learning processes in order to enhance their learning outcomes

Getting off the Deficit Path

From Personal Deficit Analytics … … to Personal Learning Analytics

Example 1: Surgical Skills Simulation

James Bailey Professor, Computing & Information Systems Ioanna Ioannou Research Fellow, Otolaryngology Stephen O'Leary Professor, Otolaryngology Patorn Piromchai PhD Student, Otolaryngology Sudathi Wijewickrema Research Fellow, Otolaryngology Yun Zhou PhD Student, Computing & Information Systems

Example 1: Surgical Skills Simulation

O'Leary, S., et al. (2008). Validation of a networked virtual reality simulation of temporal bone surgery. The Laryngoscope, 118(6), 1040-1046.

Metrics from the Simulator

• Tool position, orientation and force metrics
• e.g. current force applied by the drill
• Burr metrics
• e.g. radius of the current burr
• Anatomical structure metrics
• e.g. distance of the drill tip to the closest point of one of

three key anatomical structures

• Bone specimen metrics
• e.g. rotation of the bone
• ---- 15 records of 48 metrics generated per second -----

• A sequence of points containing a continuous drilling motion
• The end of a stroke is reached
• when drilling ceases; or
• when there is an abrupt change in the direction of drilling
• Once a way of identifying strokes has been determined a

range of “stroke metrics” can be calculated from the data stream output by the simulator (e.g. stroke duration, stroke length, average stroke speed, minimum distance of stroke to structures, etc.)

A Key Metric: Stroke

(Hall, Rathod, et al., 2008)

Data Mining for Personal Feedback

• We needed to provide personalised feedback to trainees

across multiple dimensions or features in an open, complex, procedural simulation.

• Not just deficit feedback about manifest error or procedural stage
• “You hit the facial nerve”
• “You should have completed X before Y”
• For example:
• force used
• stroke length
• stroke smoothness
• distance to critical structures, etc.

• Prototype 1: Hidden Markov Models built to discriminate patterns
• f novice and expert behaviour on a single association rule.
• Prototype 2: A range of analysis techniques used to develop

models to provide feedback on multiple features:

• A random forest model to determine expert/novice behaviour
• Nearest neighbour techniques along with a random forest

model to generate feedback in the case of novice behaviour

• An independent feedback system (application) was built

Data Mining for Personal Feedback

Feedback Parser Feedback Generator Simulator Stroke Detector

Simulator Metrics Proximity Triggers Feedback Stroke Metrics Stroke Metrics Technique Feedback

A Personal Feedback System

A Personal Feedback System

• 24 medical students
• 12 were provided with automated feedback
• 12 were not
• Knowledge of anatomy but not surgery;

video tutorial of surgery and simulator familiarisation.

• Two group comparison of students’ performance on a

cortical mastoidectomy

• Effectiveness of technique feedback
• Accuracy of feedback
• Usability of system

Feedback System Test

Effectiveness of Technique

% of Expert Stokes With Feedback Without Feedback M (SD) M (SD) F p 61.59 (16.19) 38.86 (13.11) 14.29 <.001

Effectiveness of Technique

• A surgeon undertook a post hoc analysis of the feedback

provided by the system

• False Positives: feedback was provided when stroke

technique was acceptable

• False Negatives: feedback was not provided when

technique was unacceptable.

• Wrong Feedback: participants’ technique was accurately

classified as “trainee” but the content of the feedback was inaccurate.

Accuracy of Feedback

# of Feedback Messages Percentage (of Total) False Positives 39 6.8% False Negatives 69 11.4% Wrong Feedback 52 9.0% Total Feedback 576

Accuracy of Feedback

Overwhelmingly positive feedback from participants

• n the basis of post-test interviews

“it reminded me to be gentle near structures” “particularly helpful was changing burr size and whether or not to zoom in” “it gave me the confidence to go faster”

Usability

Example 2: Cognition and Interaction

Barney Dalgarno Charles Sturt University Sue Bennett University of Wollongong

Dalgarno, B., Kennedy, G., & Bennett, S. (in press). The impact of students’ exploration strategies on discovery learning using computer-based simulations.Educational Media International. Accepted Oct 2014.

Example 2: Cognition and Interaction

How does the design of interactive multimedia impact on students’ learning strategies and cognition? Study

• Two learning conditions (observation & exploration)
• Two content areas (global warming & blood alcohol concentration)
• Each participant (n=158) completed:

– the observation condition in one content area – the exploration condition in the other – a pre and post-test of knowledge in each content area

• Students’ actions were logged.

Exploration Condition

• Content screens providing background to the area and

some terminology explanation, but no explanation of key concepts.

• A series of screens allowing students to manipulate the

simulation parameters and asking them to “predict, observe, explain”. Observation Condition

• The same series of background content screens.
• A series of simulation output screens each showing the

effect of pre-set manipulations of input parameters.

Multimedia Design

Blood Alcohol Concentration

• No effect of learning condition for Global Warming
• Modest main effect of learning condition for Blood Alcohol

Mean Post-Test Observation Mean Post-Test Exploration F (1,155) p Global Warming 1.42 1.72 2.40 0.13 Blood Alcohol 3.42 3.93 5.52 0.02

*ANCOVA using pre-test as a covariate

Results: Observation v Exploration

• We noticed that the variance in post-test scores for exploration

participants was quite high.

• Eyeballing the logs showed some students seemed more

systematic in their exploration of the simulation than others.

End of Story?

Students’ learning behaviours, strategies and approaches were characterised using various heuristics as well as cluster analysis.

Cluster Analysis

[not shown here!]

• time spent on the background material before simulation
• total time spent on the simulation
• number of cycles in which exactly one variable was changed

from the previous cycle

• number of cycles in which exactly one variable was changed

from the provided base values

• number of cycles where at least one variable was changed

from the previous cycle

• the sum of the number of variables changed per cycle across

all cycles

Classifying Students Approaches

Systematic Exploration

• Students who completed 4 or more simulation cycles with
• nly one variable changed from previous cycle; or
• Students who completed 4 or more simulation cycles with
• nly one variable changed from pre-set values

Non-Systematic Exploration

• All other “exploration” students

Observation

• Students who completed the Observation learning condition

(no simulation)

Classifying Students Approaches

• Significant main effect of condition in both content domains

Post-Test Observation Non-Systematic Post-Test Exploration Systematic Post-Test Exploration F p Global Warming 1.42 (1.29) 1.33 (1.52) 2.48 (2.20) 4.17 .017 Blood Alcohol 3.42 (1.31) 3.51 (1.30) 4.56 (1.33) 8.69 <.001

Observation = Non Systematic < Systematic

Outcome by Approach

Getting Off the Deficit Path

From Personal Deficit Analytics … … to Personal Learning Analytics

Implications …

• Each study presents a simulation-based, digital learning

environment: one procedural, one conceptual.

• Each employs analytic approaches which are very much

framed by the learning design of these environments and the pedagogical intent of the task: – what we wanted students to do and learn.

• But the approaches attempt to move away from a narrow

“deficit” model of analytics based on – assessing what students know; – how much they fail to participate; and/or – how much they are “at risk” of disengaging from the task (or dropping out of the course)

Implications …

• These studies show how learning analytics can be used to

uncover complex patterns of students’ on-task learning behaviour which are: – indicative of distinct approaches to learning; – correspond to adaptive (and maladaptive) learning or “thinking” processes; and – are associated with good (or poor) learning outcomes.

Analytics that are used to determine students’ adaptive learning patterns and processes with digital learning tasks, which can be used as the basis for individualised, personal feedback, to improve their learning processes and, ultimately, their learning outcomes.

What are Personal Learning Analytics?

PLA : Simply a Matter of Perspective?

http://www.marquette.edu/magazine/recent.php?subaction=showfull&id=1357063200

Deficit Personal

PLA : Simply a Matter of Perspective?

A Work in Progress

Deficit Analytics in T&L Personal Analytics in T&L Assessing what you know Determining how you are coming to know Determining how much you don’t participate and access Determining the way in which you participate Macro: often multidimensional Macro: does not speak to macro Micro: based on simple knowledge assessments Micro: based on approaches and responses to the learning context Micro: profiles of simple access to learning resources and activities Patterns of interactions with learning activities and tasks

What are Personal Analytics?

Is there a role for Fit Bit for students?

• Tracking engagement with University
• “macro” level interactions with

learning activities and resources

Conclusion

• The holy grail … and the somewhat elusive promise of learning

analytics is to create genuinely adaptive and personalised online environments to improve individual student’s learning.

• Identifying when students’ transgress and step off the defined

learning path gets us some of the way …

• … but understanding how we can use students’ adaptive

patterns of engagement with specific learning tasks is an important next step …

PBC Directions : eLearning Incubator

PBC Bid Director of eLearning

eLI

CSHE Support ITS Academics (F&GS) Learning Environments Business Development

Thanks gek@unimelb.edu.au