Joint Modeling of Feedback-Use and Time Data Advances in Bayesian - - PowerPoint PPT Presentation

Joint Modeling of Feedback-Use and Time Data

Advances in Bayesian Item Response Modeling Jean-Paul Fox

University of Twente Department of Research Methodology, Measurement and Data Analysis Faculty of Behavioural Sciences Enschede, Netherlands

J.-P. Fox Advances in Bayesian Item Response Modeling

Outline

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

J.-P. Fox Advances in Bayesian Item Response Modeling

Outline

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

2 Complex Multivariate Count Data

Multivariate Zero-Inflated Poisson Modeling

Results

Feedback Behavior Study: Use (Latent) Predictors

Results

J.-P. Fox Advances in Bayesian Item Response Modeling

Outline

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

2 Complex Multivariate Count Data

Multivariate Zero-Inflated Poisson Modeling

Results

Feedback Behavior Study: Use (Latent) Predictors

Results

3 Discussion

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

2 Complex Multivariate Count Data

Multivariate Zero-Inflated Poisson Modeling

Results

Feedback Behavior Study: Use (Latent) Predictors

Results

3 Discussion

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Formative Computer-Based Assessment

◮ Two-stage testing: Ability - feedback use ◮ Observe response times (speed) and feedback times (reading) ◮ Dutch study: Differential use of feedback in test assessment

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Formative Computer-Based Assessment

◮ Two-stage testing: Ability - feedback use ◮ Observe response times (speed) and feedback times (reading) ◮ Dutch study: Differential use of feedback in test assessment

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Formative Computer-Based Assessment

◮ Two-stage testing: Ability - feedback use ◮ Observe response times (speed) and feedback times (reading) ◮ Dutch study: Differential use of feedback in test assessment

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Bayesian Modeling of Multivariate Count Data

A Bayesian Modeling Approach:

◮ Hierarchical Structured Data, uncertainty/sampling error at

different levels

◮ Use Powerful Simulation Techniques ◮ Use Prior Knowledge

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Bayesian Modeling of Multivariate Count Data

A Bayesian Modeling Approach:

◮ Hierarchical Structured Data, uncertainty/sampling error at

different levels

◮ Use Powerful Simulation Techniques ◮ Use Prior Knowledge

J.-P. Fox Advances in Bayesian Item Response Modeling

Introduction Feedback Behavior Study

Bayesian Modeling of Multivariate Count Data

A Bayesian Modeling Approach:

◮ Hierarchical Structured Data, uncertainty/sampling error at

different levels

◮ Use Powerful Simulation Techniques ◮ Use Prior Knowledge

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data

Feedback-Use and Feedback-Time Data

2 4 6 8 10 50 100 150 200 250 1 3 5 7 9 11 Feedback Use Number of Subjects 10 20 30 40 10 20 30 40 50 Percentage Subjects Feedback Time (Seconds)

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data

Feedback-Use and Feedback-Time Data

20 40 60 80 100 10 20 30 40

Zero Pages One Page

10 20 30 40

Two Pages Three Pages Four Pages

20 40 60 80 100

Five Pages

20 40 60 80 100

Six Pages Seven Pages Eight Pages Nine Pages

10 20 30 40

Ten Pages

20 40 60 80 100

Eleven Pages

Feedback Time | Feedback Use

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data

Modeling Multivariate Count Data

Count Data Subjects

• No. Pages

Total Times 2 7 . . . . . . yf

i

yt

i

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data

Modeling Multivariate Count Data

Count Data Subjects

• No. Pages

Total Times 2 7 . . . . . . yf

i

yt

i

Summary Statistics Mean SD % Zeros Mean | No Zeros Feedback Use 2.35 5.35 .43 4.11 Feedback Times 2.75 6.19 .43 9.35

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

2 Complex Multivariate Count Data

Multivariate Zero-Inflated Poisson Modeling

Results

Feedback Behavior Study: Use (Latent) Predictors

Results

3 Discussion

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback-Use No. Pages

The idea is to model feedback use (yes or no), feedback pages (count pages), feedback times (count seconds) Mixture of Observed Feedback Pages Y f

i

∼

• 0,

with probability 1 − φi Poisson

• λ(f)

i

• ,

with probability φi,

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback-Use No. Pages

The idea is to model feedback use (yes or no), feedback pages (count pages), feedback times (count seconds) Mixture of Observed Feedback Pages Y f

i

∼

• 0,

with probability 1 − φi Poisson

• λ(f)

i

• ,

with probability φi, Model Feedback Count Data P

• Y f

i = 0 | λi = λ(f) i

• =

(1 − φi) + φie−λi P

• Y f

i = j | λi = λ(f) i

• =

φi e−λiλj

i

j! ,

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback Times

Mixture of Observed Feedback Times T f

i

∼

• 0,

with probability 1 − φi Poisson

• λ(t)

i

• ,

with probability φi,

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback Times

Mixture of Observed Feedback Times T f

i

∼

• 0,

with probability 1 − φi Poisson

• λ(t)

i

• ,

with probability φi, Model Feedback Time Count Data P

• T f

i = 0 | λi = λ(t) i

• =

(1 − φi) + φie−λi P

• T f

i = j | λi = λ(t) i

• =

φi e−λiλj

i

j! ,

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback Use

Identify (non-)users of feedback pages using explanatory subject information Observed Feedback Use Zi | λ(t)

i , λ(f) i

∼    0, with probability (1 − φi)P

• Y f

i = 0, T f i = 0

• 1,

with probability φi

• 1 − P
• Y f

i = 0, T f i = 0

• J.-P. Fox

Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Feedback Use

Identify (non-)users of feedback pages using explanatory subject information Observed Feedback Use Zi | λ(t)

i , λ(f) i

∼    0, with probability (1 − φi)P

• Y f

i = 0, T f i = 0

• 1,

with probability φi

• 1 − P
• Y f

i = 0, T f i = 0

• Feedback Use

φi = P (Zi = 1) = exp

• xt

iα

• 1 + exp (xt

iα)

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Population Model Subjects

Respondents are sampled independently and identically distributed. Stage 2: Prior Expected Counts log λ(f)

i

= xt

iβf

log λ(t)

i

= xt

iβt

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Population Model Subjects

Respondents are sampled independently and identically distributed. Stage 2: Prior Expected Counts log λ(f)

i

= xt

iβf

log λ(t)

i

= xt

iβt

Stage 2: Multivariate Prior Expected Counts

• log λ(f)

i

log λ(t)

i

• ∼

N (xβ, Σλ)

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Multivariate Zero-Inflated Poisson Modeling

Population Results

Joint Model (No Predictors) Component Parameter Mean HPD

Feedback Use (Bernoulli part)

Use Feedback Intercept, α0 .30 (.13,.45) No Feedback 1 − φ .43 (.38,.46)

Feedback Behavior (Poisson part)

• No. Pages

Intercept, µ1 3.06 (2.69,3.46) Time Intercept, µ2 7.09 (6.35,7.92) Correlation,Σ12 .20 (.13,.27)

– HPD: 95% Highest Posterior Density interval J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Overview

1 Introduction

Feedback Behavior Study

Bayesian Response Modeling

2 Complex Multivariate Count Data

Multivariate Zero-Inflated Poisson Modeling

Results

Feedback Behavior Study: Use (Latent) Predictors

Results

3 Discussion

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Ability-Speed Model

Collection of Responses and Response Times, N persons and K items

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Ability-Speed Model

Collection of Responses and Response Times, N persons and K items Measuring Ability P (Y a

ik = 1 | θi, ak, bk)

= Φ(akθi − bk) IRT Model

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Ability-Speed Model

Collection of Responses and Response Times, N persons and K items Measuring Ability P (Y a

ik = 1 | θi, ak, bk)

= Φ(akθi − bk) IRT Model Measuring Speed of Working log T a

ik | ζi, ck, dk

∼ N

• dk − ckζi, σ2

ǫ

• RT Model

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Joint Model Results

Joint Model (Latent Predictors Speed and Ability) Component Parameter Mean HPD

Feedback Use (Bernoulli part)

Intercept, α0 .32 (.15,.47) 1 − φ .42 (.36,.48) Ability, α1 .68 (.33,1.00) Speed, α2

• .95

(-1.32,-.50)

– Latent predictors are grand-mean centered J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Feedback-Use

• 2
• 1

1 2

Student ability

0.0 0.2 0.4 0.6 0.8

Probability of using feedback

• 2
• 1

1 2

Student speed of working

0.0 0.2 0.4 0.6 0.8

Probability of using feedback

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Joint Model Results

Joint Model (latent Predictors Speed and Ability) Component Parameter Mean HPD

Feedback Behavior (Poisson part)

Feedback Intercept, β0 1.13 (3.09) (1.00,1.25) Ability, β1

• .40

(-.69,-.11) Speed, β2

• .16

(-.52,.16) Feedback-Time Intercept, β0 1.97 (7.17) (1.85,2.08) Ability, β1

• .33

(-.59,-.07) Speed, β2

• .32

(-.63,-.03) Correlation Σ12 .18 (.11,.24)

– Latent predictors are grand-mean centered J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Feedback Page Counts

• 2
• 1

1 2

Ability

1 2 3 4 5

Expected Number of Feedback-Pages

• 2
• 1

1 2 1 2 3 4 5

Expected Number of Feedback-Pages Speed

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Feedback Times

• 2
• 1

1 2

Ability

2 4 6 8 10 12 14

Expected Total Feedback-Time

• 2
• 1

1 2

Speed

2 4 6 8 10 12 14

Expected Total Feedback-Time

J.-P. Fox Advances in Bayesian Item Response Modeling

Complex Multivariate Count Data Feedback Behavior Study: Use (Latent) Predictors

Model Fit

2 4 6 8 10 12 Observed Feedback Pages 2 4 6 8 10 12 Fitted Number of Feedback-Pages 10 20 30 40 Observed Feedback Time 10 20 30 40 Fitted Total Feedback-Time

J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

◮ Use (higher-level) latent predictor variables

J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

◮ Use (higher-level) latent predictor variables ◮ Feedback Behavior Study

J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

◮ Use (higher-level) latent predictor variables ◮ Feedback Behavior Study

◮ Heterogeneity in feedback-use versus feedback to improve learning J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

◮ Use (higher-level) latent predictor variables ◮ Feedback Behavior Study

◮ Heterogeneity in feedback-use versus feedback to improve learning ◮ Ability positively and speed negatively related to feedback use J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Discussion

◮ Flexible joint model for multivariate zero-inflated discrete count

data

◮ Use (higher-level) latent predictor variables ◮ Feedback Behavior Study

◮ Heterogeneity in feedback-use versus feedback to improve learning ◮ Ability positively and speed negatively related to feedback use ◮ Ability and speed negatively related to feedback counts and times J.-P. Fox Advances in Bayesian Item Response Modeling

Discussion

Some References

◮ Jean-Paul Fox (2010) Bayesian Item Response Modeling, Springer-

Science, New-York.

◮ Fox, J.-P., Klein Entink, R.H., van der Linden, W.J. (2007).

Modeling of responses and response times with the package cirt. Journal of Statistical Software, 20, issue 7.

◮ Klein Entink, R.H., Fox, J.-P., van der Linden, W.J. (2009). A

multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21-48

◮ www.Jean-PaulFox.com

J.-P. Fox Advances in Bayesian Item Response Modeling