A new statistical method for detecting Differential Item Functioning - - PowerPoint PPT Presentation

A new statistical method for detecting Differential Item Functioning in the Rasch-Model

Julia Kopf Achim Zeileis Carolin Strobl LMU

Gist of this presentation

Main idea: Method to detect parameter instability in the Rasch-model Usage of model-based recursive partitioning algorithm Application of the method to detect DIF

2/12

Usage of Rasch-Trees

Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states

3/12

Usage of Rasch-Trees

Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states

Possible reason: Differential Item Functioning in the Rasch-Model

3/12

Usage of Rasch-Trees

Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states

Possible reason: Differential Item Functioning in the Rasch-Model

A_levels 1

• ther

{Hessen, Rheinl.−Pfalz} Node 2 (n = 8321)

• ●
• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 A_levels 3 Hessen Rheinl.−Pfalz Node 4 (n = 721)

• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 Node 5 (n = 400)

• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03

• Politics

History Economics Culture Natural Science

3/12

Usage of Rasch-Trees

Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states

Possible reason: Differential Item Functioning in the Rasch-Model

A_levels 1

• ther

{Hessen, Rheinl.−Pfalz} Node 2 (n = 8321)

• ●
• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 A_levels 3 Hessen Rheinl.−Pfalz Node 4 (n = 721)

• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 Node 5 (n = 400)

• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03

• Politics

History Economics Culture Natural Science

Item 4: Find Hesse on the German map! Item 5: What’s the capital of Rhineland-Palatinate?

3/12

Usage of Rasch-Trees

Surprising result: Higher general knowledge in Rhineland-Palatinate comparing to other German Federal states

Possible reason: Differential Item Functioning in the Rasch-Model

A_levels 1

• ther

{Hessen, Rheinl.−Pfalz} Node 2 (n = 8321)

• ●
• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 A_levels 3 Hessen Rheinl.−Pfalz Node 4 (n = 721)

• ●
• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03 Node 5 (n = 400)

• ●

1 2 3 4 5 6 7 8 9 −2.26 4.03

• Politics

History Economics Culture Natural Science

Item 4: Find Hesse on the German map! Item 5: What’s the capital of Rhineland-Palatinate? Obtained result: ⇒ The questions in the survey do not lead to fair comparisons.

3/12

Overview: The Rasch-model

Objective of the Rasch-model: Measurement of latent variables Obtain at least interval scaled person parameters These are monotone transformation of raw scores Examples: Intelligence and attainment tests Extensions: 2-pl (Birnbaum), 3-pl models Essential data:

i1 i2 i3 i4 i5 i6 i7 i8 i9 i10 i11 i12 i13 i14 sex* domicile* 1 1 man west 5 1 1 woman west 7 1 1 1 1 1 1 man west 8 1 1 1 1 1 1 man west 10 1 1 1 1 1 man west 11 1 1 1 1 woman west

4/12

Overview: The Rasch-model

Assumptions of the Rasch-model (Rasch, 1960): Influence of latent variable Assumptions about Item Characteristic Curves (ICC) Unidimensionality Local stochastic independence Invariance of Item parameters ,,The importance of the property of invariance of item and ability parameters cannot be overstated. This property is the cornerstone

• f item response theory and makes possible such important

applications as equating, item banking, investigation of item bias, and adaptive testing” (Hambleton, Swaminathan and Rogers, 1991: 25).

5/12

Item Characteristic Curves and estimation process

Assumptions about the ICCs: Probability of solving or agreeing as a function of

latent variable item difficulty

monotone, logistic form

−10 −5 5 10 0.0 0.2 0.4 0.6 0.8 1.0

Itemcharakteristiken

Latente Dimension Wahrscheinlichkeit für richtige Antwort Item i1 Item i2

Estimation via Conditional Maximum Likelihood (CML): Probability for person i (= 1 . . . , n) solving item j (= 1 . . . , k) is: P (Uij = uij | θi, βj) = exp[(θi − βj) · uij] 1 + exp(θi − βj) , βj denotes the item parameter of item j θi is the person parameter of individual i uij ∈ {0; 1} symbolizes the answer of person i to item j ri = k

j=1 uij and sj := n i=1 uij

6/12

CML estimation and subject-wise score functions

New parametrisation (Fischer und Molenaar, 1995): ξi = exp(θi) and εj = exp(−βj) Individual Loglikelihoods: Ψ(yi, ε) =

k

• j=1

uij log(εj) − log γri with γri =

• k

j=1 uij=ri

k

• j=1

ε

uij j

. Elementary symmetric functions (Liou, 1994): γ0 = 1 γ1 = ε1 + ε2 + . . . + εk γ2 = ε1 · ε2 + ε1 · ε3 + . . . + εk−1 · εk . . . γk = ε1 · ε2 · . . . · εk Individual Scores: ψ(yi, ε⋆) = ∂Ψ(yi, ε) ∂ε⋆ = uij⋆ ε⋆ − γ(j⋆)

ri−1

γri .

7/12

Model-based recursive partitioning of Rasch-models

Implementation of Achim Zeileis in psychotree The code conversation can be summarized in the following way:

1 Hand-off formula like item1 + item2 + ...

+ itemk˜X1 + X2 + ... + Xl, arguments, data

2 Model class RaschModel including RaschModel.fit 3 Data sanity checks 4 Passing to mob() from package party (Zeileis et al., 2008)

Available functions in updated package psychotree: summary() plot() coef() worth()

8/12

Identifying parameter instability

Ways of identifying violation of parameter invariance: Graphical model test according to item raw scores and sex

−3 −2 −1 1 2 3 −3 −2 −1 1 2 3

According to ability

Difficulty for group low raw score Difficulty for group with high raw score

1 2 3 4 5 6 7 89 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 −3 −2 −1 1 2 3 −3 −2 −1 1 2 3

According to sex

Difficulty for women Difficulty for men

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

Likelihood Ratio tests Problem: Which groups may influence the item parameters?

9/12

Results of the new method

Ways of identifying parameter variance: New method: Rasch trees

AlterKat 1 ≤ 23−25 > 23−25 Tat 2 {Student, Doktorand} {alle anderen} Geschlecht 3 Frau Mann Node 4 (n = 2978)

• Node 5 (n = 3907)
• Geschlecht

6 Frau Mann AlterKat 7 ≤ bis 19 > bis 19 Node 8 (n = 2014)

• Node 9 (n = 1822)
• AlterKat

10 ≤ bis 19 > bis 19 Node 11 (n = 1893)

• Node 12 (n = 2085)
• Geschlecht

13 Frau Mann Node 14 (n = 2367)

• AlterKat

15 ≤ 30−39 > 30−39 OnlSpiegel 16 ≤ 2−3d > 2−3d Node 17 (n = 2682)

• Node 18 (n = 1652)
• Node 19 (n = 1521)
• Politik

Geschichte Wirtschaft Kultur Naturwissenschaften

10/12

Discussion of the results

Advantages: Groups are found automatically Statistical influence is tested Promising simulation results Open questions and possible topics: Extended simulations, e.g. combination of covariate types Post-hoc tests: Which items have significant DIF? Extensions of Item Response Theory Criteria of tree stability

11/12

Literature

Fischer, G. und Molenaar, I. (1995): Rasch Models - Foundations, Recent Developements, and Applications. New York: Springer. Hambleton, R., Swaminathan, H. und Rogers, H. (1991): Fundamentals of Item Response Theory. Newbury Park: Sage Publications. Liou, M. (1994): More on the Computation of Higher-Order Derivatives on the Elementary Symmetric Functions in the Rasch Model. Applied Psychological Measurement, 18 (1), 53–62. Rasch, G. (1960): Probabilistic Models for some Intelligence and Attainment

• Tests. Chicago, London: The University of Chicago Press.

Zeileis, A., Hothorn, T. und Hornik, K. (2008): Model-Based Recursive

• Partitioning. Journal of Computational and Graphical Statistics, 17 (2),

492–514.

12/12