From Predictive Models to Instructional Policies Joseph Rollinson - - PowerPoint PPT Presentation

▶
from predictive models to instructional policies
SMART_READER_LITE
LIVE PREVIEW

From Predictive Models to Instructional Policies Joseph Rollinson - - PowerPoint PPT Presentation

Sep 06, 2023 151 likes •552 views

From Predictive Models to Instructional Policies Joseph Rollinson (jtrollinson@gmail.com) Emma Brunskill (ebrun@cs.cmu.edu) Carnegie Mellon 1 Student models are a representation of the student Predictions Observations Student Beliefs

slide-1
SLIDE 1

From Predictive Models to Instructional Policies

Joseph Rollinson (jtrollinson@gmail.com) Emma Brunskill (ebrun@cs.cmu.edu)

Carnegie Mellon

1

slide-2
SLIDE 2

Student models are a representation of the student

Observations Predictions Beliefs

…

Student Model

2

Corbett et al. 1994, Cen et al. 2006, Pavlik et al. 2009, Chi et al. 2011, Khajah 2014, Gong 2010, Pardos 2010, Falakmasir 2013

slide-3
SLIDE 3

Student models are a representation of the student

Observations Predictions Beliefs

…

Much prior work building student models for predicting future student performance Student Model

2

Corbett et al. 1994, Cen et al. 2006, Pavlik et al. 2009, Chi et al. 2011, Khajah 2014, Gong 2010, Pardos 2010, Falakmasir 2013

slide-4
SLIDE 4

Student models are also used with

  • uter-loop instructional policies

3

Outer Loop

Response Activity

slide-5
SLIDE 5

Student models are also used with

  • uter-loop instructional policies

4

Student Model Instructional Policy Response Activity

slide-6
SLIDE 6

Many predictive student models cannot be used with any existing instructional policy

Student Model Instructional Policy

5

Response Activity

slide-7
SLIDE 7

Contribution

Model agnostic instructional policy for the when-to-stop decision problem

6

slide-8
SLIDE 8

Background Bayesian Knowledge Tracing

start Non-Mastery Mastery

1 − P(L0) P(L0) 1 − P(T) P(T ) 1 − P(G) P(G) 1 P(S) 1 − P(S)

Corbett, A. T., & Anderson, J. R. (1995). Knowledge tracing: modeling the acquisition of procedural

  • knowledge. User Modeling and User-Adapted Interaction, 4, 253–278

7

slide-9
SLIDE 9

Background Performance Factors Model (PFM)

Logistic model for predicting student performance Features

  • Student (i)
  • Skill (k)
  • # Correct responses for skill (s)
  • # Incorrect responses for skill (f)

8

Cen et al. 2006, Pavlik et al. 2009, Chi et al. 2011

slide-10
SLIDE 10

Background Performance Factors Model (PFM)

Logistic model for predicting student performance Features

  • Student (i)
  • Skill (k)
  • # Correct responses for skill (s)
  • # Incorrect responses for skill (f)

8

Cen et al. 2006, Pavlik et al. 2009, Chi et al. 2011

slide-11
SLIDE 11

When-To-Stop Decision Problem

Situation: Teaching single skill with indistinguishable activities Observations: Correctness of student responses Decision: When to stop providing activities to student

9

slide-12
SLIDE 12

Prior Work Mastery Threshold Policy

Stop if we are confident that the student has mastered the skill

10

slide-13
SLIDE 13

Prior Work Mastery Threshold Policy

Stop if we are confident that the student has mastered the skill

P (M ) > ∆

10

slide-14
SLIDE 14

Issues with the Mastery Threshold Policy

  • 1. Requires student model with concept of mastery
  • 2. Will not stop if student cannot progress with given

instruction (wheel-spinning)

11

Beck, Joseph E., and Yue Gong. "Wheel-spinning: Students who fail to master a skill." Artificial Intelligence in Education. Springer Berlin Heidelberg, 2013.

slide-15
SLIDE 15

New Policy Predictive Similarity Policy

Stop if we are confident that the student model’s prediction of the student’s performance will not change very much if the student is given another question

12

slide-16
SLIDE 16

New Policy Predictive Similarity Policy

Stop if we are confident that the student model’s prediction of the student’s performance will not change very much if the student is given another question

12

Pr

Pt+1(C) − Pt(C) <

  • >
slide-17
SLIDE 17

3 Stopping Conditions:

13

Pr

Pt+1(C) − Pt(C) <

  • >
slide-18
SLIDE 18

3 Stopping Conditions:

13

Pr

Pt+1(C) − Pt(C) <

  • >

Pt(C) > δ | Pt+1(C) - Pt(C | Ct) | < 𝜁 Confident that student will respond correctly. Prediction does not change much if student responds correctly.

slide-19
SLIDE 19

3 Stopping Conditions:

13

Pr

Pt+1(C) − Pt(C) <

  • >

Pt(C) > δ | Pt+1(C) - Pt(C | Ct) | < 𝜁 Confident that student will respond correctly. Prediction does not change much if student responds correctly. Pt(¬C) > δ | Pt+1(C) - Pt(C | ¬Ct) | < 𝜁 Confident that student will respond incorrectly. Prediction does not change much if student responds incorrectly.

slide-20
SLIDE 20

3 Stopping Conditions:

13

Pr

Pt+1(C) − Pt(C) <

  • >

Pt(C) > δ | Pt+1(C) - Pt(C | Ct) | < 𝜁 Confident that student will respond correctly. Prediction does not change much if student responds correctly. Pt(¬C) > δ | Pt+1(C) - Pt(C | ¬Ct) | < 𝜁 Confident that student will respond incorrectly. Prediction does not change much if student responds incorrectly. | Pt+1(C) - Pt(C | Ct) | < 𝜁 | Pt+1(C) - Pt(C | ¬Ct) | < 𝜁 Prediction does not change much no matter how the student’s

  • bservation.
slide-21
SLIDE 21

Experiments Methodology

  • 1. Train student models on data set
  • 2. Calculate expected amount of practice for each

skill in dataset using instructional policy and student model

  • 3. Compare expected amount of practice per skill

14

slide-22
SLIDE 22

Dataset KDD Cup Algebra I

> 3000 students 505 skills BKT and PFM have similar predictive accuracy

15

  • J. Stamper, A. Niculescu-Mizil, S. Ritter, G. Gordon, and K. Koedinger. Algebra 1 2008-2009. challenge data set from

kdd cup 2010 educational data mining challenge. find it at http://pslcdatashop.web.cmu.edu/kddcup/downloads.jsp.

slide-23
SLIDE 23

Expected Amount of Practice (ExpOps)

Metric of the number of questions given to students by a policy with a given student model.

16

  • J. I. Lee and E. Brunskill. The impact on individualizing student models on necessary practice opportunities. In EDM, 2012.
slide-24
SLIDE 24

Expected Amount of Practice (ExpOps)

Metric of the number of questions given to students by a policy with a given student model. Comparison, not a measure of quality

16

  • J. I. Lee and E. Brunskill. The impact on individualizing student models on necessary practice opportunities. In EDM, 2012.
slide-25
SLIDE 25

Experiment 1 Predictive Similarity vs. Mastery Threshold

  • 1. Train BKT with EM for each skill in dataset
  • 2. For each skill, calculate expected amount of

practice using Predictive Similarity and Mastery Threshold policies with trained BKTs

  • 3. Compare expected amount of practice on skills

with non-degenerate BKTs

17

slide-26
SLIDE 26

5 10 15 20

Mastery Threshold Policy with BKT (Expops)

5 10 15 20

Predctive Similarity Policy with BKT (Expops)

Experiment 1 Results

18

slide-27
SLIDE 27

5 10 15 20

Mastery Threshold Policy with BKT (Expops)

5 10 15 20

Predctive Similarity Policy with BKT (Expops)

Experiment 1 Results

Predictive similarity policy makes similar decisions to mastery threshold policy 
 (coef 0.95)

18

slide-28
SLIDE 28

Experiment 2 BKT vs. PFM

  • 1. Train PFM on KDD Cup dataset using logistic

regression

  • 2. Calculate expected amount of practice using

Predictive Similarity policy with underlying BKT and PFM for each skill

  • 3. Compare expected amount of practice values

19

slide-29
SLIDE 29

PFM vs. BKT

20

10 20 30 40 50 60

Predictive Similarity with PFM (ExpOps)

10 20 30 40 50 60

Predictive Similarity with BKT (ExpOps)

slide-30
SLIDE 30

PFM vs. BKT

20

PFM based policy either:

10 20 30 40 50 60

Predictive Similarity with PFM (ExpOps)

10 20 30 40 50 60

Predictive Similarity with BKT (ExpOps)

slide-31
SLIDE 31

PFM vs. BKT

20

PFM based policy either:

  • Stops immediately

10 20 30 40 50 60

Predictive Similarity with PFM (ExpOps)

10 20 30 40 50 60

Predictive Similarity with BKT (ExpOps)

slide-32
SLIDE 32

PFM vs. BKT

20

PFM based policy either:

  • Stops immediately
  • Longer than BKT

based policy

10 20 30 40 50 60

Predictive Similarity with PFM (ExpOps)

10 20 30 40 50 60

Predictive Similarity with BKT (ExpOps)

slide-33
SLIDE 33

Diving In Comparing BKT and PFM by skill

Calculate student model predictions for skill if:

  • simulated student always responds correctly
  • simulated student always responds incorrectly

21

slide-34
SLIDE 34

5 10 15 20 25

Number of questions

0.0 0.2 0.4 0.6 0.8 1.0

P(Correct)

BKT always correct BKT always incorrect

22

Skill: PFM Immediately stops

slide-35
SLIDE 35

5 10 15 20 25

Number of questions

0.0 0.2 0.4 0.6 0.8 1.0

P(Correct)

BKT always correct BKT always incorrect PFM always correct PFM always incorrect

23

PFM predictions change very slowly.

Skill: PFM Immediately stops

slide-36
SLIDE 36

5 10 15 20 25

Number of questions

0.0 0.2 0.4 0.6 0.8 1.0

P(Correct)

BKT always correct BKT always incorrect

24

Skill: PFM longer than BKT

slide-37
SLIDE 37

5 10 15 20 25

Number of questions

0.0 0.2 0.4 0.6 0.8 1.0

P(Correct)

BKT always correct BKT always incorrect PFM always correct PFM always incorrect

25

PFM predictions asymptote much later than BKT predictions

Skill: PFM longer than BKT

slide-38
SLIDE 38

Discussion / Summary

  • Contribution: a model-agnostic when-to-stop

instructional policy called predictive similarity

  • Predictive similarity policy acts like the 


mastery threshold policy when used with a BKT

  • Models with similar predictive accuracies may lead

to very different instructional behavior

26

slide-39
SLIDE 39

Future Work

  • Perform experiments on another dataset
  • Incorporating other observations into the predictive

similarity policy

  • Expanding predictive similarity policy to longer

horizons

  • Model agnostic instructional policies for more

complicated instructional decisions (e.g. multiple skills)

  • Method for evaluating policies

27

slide-40
SLIDE 40

Questions?

28