Adaptive Learning Meets Crowdsourcing Towards Development of - - PowerPoint PPT Presentation

Adaptive Learning Meets Crowdsourcing

Adaptive Learning Meets Crowdsourcing

Towards Development of Cost-Effective Adaptive Educational Systems

Dr Hassan Khosravi The University of Queensland Brisbane, QLD, Australia h.khosravi@uq.edu.au

Adaptive Learning Meets Crowdsourcing

2

Overview

Challenges in delivering learner-centered learning at scale Adaptive Educational systems provide a potential solution. But they are expensive to purchase or develop. RiPPLE: A discipline-agnostic cost-effective crowdsourced adaptive educational system Reflections and intellectual challenges

Adaptive Learning Meets Crowdsourcing Development and Adoption of an Adaptive Learning System 3

Developing Cost-Effective AESs Adaptive Educational Systems (AESs) Conclusion The RiPPLE Platform Reflections and Intellectual Challenges

Adaptive Learning Meets Crowdsourcing

4

On Overview of Adaptive Educational Systems

Adaptive Learning Meets Crowdsourcing

5

Adaptive Educational Systems

Adaptive Educational Systems make use of data about students, learning processes, and learning products to provide an efficient, effective and customised learning experience for students by dynamically adapting learning content to suit their individual abilities or preferences.

Domain Model Learner Model Content Repository Adaptation Engine

Adaptive Learning Meets Crowdsourcing

6

• Design-loop adaptivity: data-driven decisions made by course designers before

and between iterations of system design

• Task-loop adaptivity: data-driven decisions the system makes to select

instructional tasks for an individual learner

• Step-loop adaptivity: data-driven decisions the system makes in response to

individual actions a student takes within an instructional task

How and When to Adapt?

Adaptive Learning Meets Crowdsourcing

7

• Knowledge state: This includes prior knowledge and knowledge growth
• Students’ path through a problem: This includes solution strategy, specific errors,

requests for help

• Affect, motivation: This includes mind-wondering, emotion and cognitive load.
• Metacognition: This includes self-regulation strategies
• Learning Styles

What to adapt to?

Adaptive Learning Meets Crowdsourcing

8

The Adaptivity Grid

Design Loop Task Loop Step Loop Knowledge Using erroneous examples to improve mathematics learning with a web-based tutoring system (Adam etal., 2014) Personalised adaptive task selection in air traffic control: Effects on training efficiency and transfer (Saldenet al., 2010) The effect of positive feedback in a constraint based intelligent tutoring system (Mitrovicet al., 2013) Students’ path Example problems that improve student learning in algebra: Differentiating between correct and incorrect examples (Booth et al., 2013) The invention lab: Using a hybrid of model tracing and constraint- based modeling to offer intelligent support in inquiry environments (Roll et al., 2010) Does supporting multiple student strategies lead to greater learning and motivation? Investigating a source of complexity in the architecture of intelligent tutoring systems (Waalkenset al., 2013) Affect, motivation Confusion can be beneficial for learning (D’Melloet al., 2014) Using adaptive learning technologies to personalize instruction: The impact of interest-based scenarios on performance in algebra. (Walkington& Sherman, 2012) Gaze tutor: A gaze-reactive intelligent tutoring system (D’Mello et al., 2014) Metacognition Limitations of student control: Do students know when they need help? (Aleven& Koedinger, 2000) Supporting students’ self-regulated learning with an open learner model in a linear equation tutor (Long & Aleven, 2013) Motivation matters: Interactions between achievement goals and agent scaffolding for self-regulated learning within an intelligent tutoring system(Duffy & Azevedo, 2015)

Adaptive Learning Meets Crowdsourcing

9

Effectiveness of Adaptive Educational Systems

This review found that the effect size of human tutoring is d = 0.79 and the effect size of intelligent tutoring systems was 0.76, so they are nearly as effective as human tutoring A meta-analysis of 107 studies on ITSs involving 14,321 participants found that: ITS were associated with higher achievement relative to teacher-led large-group instruction, non-ITS computer based- instruction, and texbooks or workbooks. This paper conducted a study on 3422 students from 198 offerings that have used ALEKS reporting significantly higher pass rates amongst students using ALEKS. Yilmaz The meta-analysis indicated that Intelligent Tutoring Systems produced a large effect size on reading comprehension when compared to traditional instruction (0.86)

Adaptive Learning Meets Crowdsourcing

10

Developing Adaptive Educational Systems

Publisher Model: designed with pre-existing content.

• Examples: Pearsons MyLabs, McGraw-Hills

LearnSmart and ALEKS

• Successful in K12 where content is standardized.
• Expensive to use

Platform Model: provides a content-agnostic system infrastructure that enables instructors to develop content.

• Examples: Smart Sparrow, Desire2Learn

and edX incorporate adaptive functionalities

• Introduce significant overhead for

instructors

• 25 hours of an expert's time for each hour of adaptive instruction (Aleven et al., 2006).
• Both types are very expensive to develop /purchase and challenging to scale across

different domains.

Adaptive Learning Meets Crowdsourcing Development and Adoption of an Adaptive Learning System 7

Adaptive Educational Systems (AESs) Reflections and Intellectual Challenges Conclusion The RiPPLE Platform Developing Cost-Effective AESs

Adaptive Learning Meets Crowdsourcing

12

Developing Cost-Effective Adaptive Learning Systems

Adaptive Learning Meets Crowdsourcing

13

Successful Crowdsourcing Stories Outside of Education

Crowdsourcing Knowledge Crowdsourcing information Crowdsourcing Service Crowdsourcing micro tasks Crowdsourcing answers Crowdfunding

Adaptive Learning Meets Crowdsourcing

14

Successful Crowdsourcing Stories in Education

PeerWise: Crowdsourcing learning resources Peer grading and evaluation system Crowdy: Interactive, Collaborative, Crowd- powered Video Learning Piazza: crowdsources answers in discussion forums

Adaptive Learning Meets Crowdsourcing

15

Developing Cost-Effective Adaptive Learning Systems

• Would Students Benefit from Creating/Evaluating Resources?
• Can Students Create High-Quality Resources?
• Can Students Accurately Evaluate the Quality of Resources?

Adaptive Learning Meets Crowdsourcing

16

Would Students Benefit from Creating/Evaluating Resources?

Question writing frequency correlated most strongly with summative performance (Spearman's rank: 0.24, p=<0.001). Only two questions of the 300 'most-answered' questions analysed had an unacceptable discriminatory value (<0.2)

Adaptive Learning Meets Crowdsourcing

17

Can Students Create High-Quality Resources?

“60% of all explanations classified as being of high or outstanding

• quality. Overall, 75% of questions met combined quality criteria”

“People with greater expertise tend to make assumptions about student learning that turn out to be in conflict with students’ actual performance and developmental propensities.” “Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. ”

Adaptive Learning Meets Crowdsourcing

18

Can Students Accurately Evaluate the Quality of Resources?

1. Students’ subjective rating of the quality of learning resources strongly correlates with that

• f domain experts.

2. Using a hybrid human-machine intelligence crowdsourcing consensus approach can increase the accuracy of the results

Under review by the Journal of IEEE Transaction in Educational Technology

1. Aggregated subjective ratings are highly (and stat. sig.) predictive of the resources’ average learning gains, with Pearson correlation of 0.78.

Adaptive Learning Meets Crowdsourcing

19

• For this vision to be successful, educational technologies need to devise effective

mechanisms to:

Integrating Human/Machine Intelligence for Development of AESs

Harness the creativity and evaluation power of students to create high-quality learning resources. Enable knowledgeable and time-poor academics to facilitate students in content creation and moderation while providing minimal oversight. Utilize AI algorithms to suggest spot-checking, dynamically adapt instruction, learning content and activities to suit students’ individual abilities or preferences.

Adaptive Learning Meets Crowdsourcing Development and Adoption of an Adaptive Learning System 13

Adaptive Learning Systems (ALSs) Reflections and Intellectual Challenges Conclusion Developing Cost-Effective ALSs The RiPPLE Platform

Adaptive Learning Meets Crowdsourcing

21

The RiPPLE Platform

Content creation Content moderation Adaptive practice Peer study recommendations Clicker-based in-class activity See ripplelearning.org

Adaptive Learning Meets Crowdsourcing

22

Content Creation

MCQs Worked Examples Notes

Adaptive Learning Meets Crowdsourcing

23

Content Moderation

Adaptive Learning Meets Crowdsourcing

24

Content Moderation Feedback

Adaptive Learning Meets Crowdsourcing

25

Spot Checking

Developing algorithms that can utilize limited efforts in spot-checking to increase the accuracy of the moderation results

Adaptive Learning Meets Crowdsourcing

26

Delivering an Adaptive Learning Experience

• 1. The Learner model visualises the current

knowledge stage of a student

• 2. The recommender system suggests effective

resources on topics that the student is developing at an appropriate level of difficulty

Adaptive Learning Meets Crowdsourcing

27

RiPPLE Demo

Adaptive Learning Meets Crowdsourcing

28

Support Ethical Low Cost Empirical Educational research

Develop guidelines referring to consent, transparency and benevolence Develop mechanisms for instructors to run controlled experiments. Provide access to rich analytics to support

• bservational studies.

Develop mechanisms for instructors to run quasi- experiments.

Adaptive Learning Meets Crowdsourcing Development and Adoption of an Adaptive Learning System 13

Adaptive Learning Systems (ALSs) The RiPPLE Platform Conclusion Developing Cost-Effective ALSs Reflections and Intellectual Challenges

Adaptive Learning Meets Crowdsourcing

30

Reflections and Intellectual Challenges

Quality Control

How can AESs use student judgements to accurately judge the quality of a learning resource?

Incentives

How can crowdsourced AESs incentivize students to engage with content creation and moderation?

Reliability Systems

How can AESs accurately, transparently and fairly rate the reliability of each of the students?

Training and Support

How can AESs help students actively develop their creativity and evaluative judgment skills?

Optimal Spot Checking

How can AESs optimally utilize the minimal availability of instructors in moderation?

Benchmark and Metrics

What benchmarking metrics can be used to measure the effectiveness of a crowdsourced AES?

Adaptive Learning Meets Crowdsourcing Development and Adoption of an Adaptive Learning System 13

Adaptive Learning Systems (ALSs) The RiPPLE Platform Reflections and Lessons Learned Developing Cost-Effective ALSs Conclusion

Adaptive Learning Meets Crowdsourcing

32

Conclusion

Educational technologies can help with delivering learner-centered learning at scale

RiPPLE: A discipline-agnostic cost-effective crowdsourced adaptive learning system

Dr Hassan Khosravi The University of Queensland Brisbane, QLD, Australia h.khosravi@uq.edu.au

See ripplelearning.org