Quasi-Exact Tests for the dichotomous Rasch Model conditional - - PowerPoint PPT Presentation

Quasi-Exact Tests for the dichotomous Rasch Model

Ingrid Koller, Vienna University Reinhold Hatzinger, WU-Vienna

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 1 Introduction

Rasch model Properties ❨ unidimensionality/homogenous items ❨ conditional independence (local independence) ❨ specific objectivity/sample independence ❨ strictly monotone increasing item characteristic function ❨ sufficient statistics

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 2 Introduction

Why quasi-exact tests? Parametric methods need large samples because of ❨ consistency and unbiasedness of parameter estimates ❨ assumption of asymptotic distribution of test statistics ❨ higher power of test-statistics Small samples in practise ❨ large samples often not available (e.g., clinical studies) ❨ complex study designs (e.g., experiments) ❨ smaller costs and less time-consuming ❨ possibility to test the quality of items also in small samples (e.g., stepwise test-construction)

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 3 Quasi-exact tests

Quasi-exact tests? with quasi-exact tests it is possible to test the Rasch-model (RM) also with small samples Sampling binary matrices description of MCMC method: Kathrin Gruber development of test-statistics (T) for the dichotomous RM ❨ Ponocny (1996, 2001) ❨ Chen & Small (2005) ❨ Verhelst (2008) ❨ Koller & Hatzinger (in prep.)

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 4

Procedure

General procedure for the T-statistics ❨ A0 is the observed matrix with the margins rv and ci where rv Pi xvi (person score) and ci Pv xvi (item score) ❨ Σrc is the set of all matrices with fixed r and c (sample space) Algorithm ❨ sample s 1,...,S matrices As from Σrc ❨ calculate T0 for the observed matrix A0 ❨ calulate T1,...,TS for all sampled matrices A1,...,AS ❨ determine your p-value by p

S

◗

s1

ts⑦S where ts ➣ ➝ ➝ ➛ ➝ ➝ ↕ 1, Ts❼As➁ ❈ T0❼A0➁ 0, else

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 5 Conditional dependence

Conditional dependence

T11: large inter-item correlations

T11❼A➁ ◗

ij

❙rij ✏ ➬ rij❙ where ➬ rij PS

s1 rij

S rij . . . the inter-item-correlation for item i and item j ➬ rij . . . mean of rij from all simulated matrices p

S

◗

s1

ts⑦S where ts ➣ ➝ ➝ ➛ ➝ ➝ ↕ 1, Ts❼As➁ ❈ T0❼A0➁ 0, else if rij in A0 is large, then the difference rij ✏ ➬ rij is also large

• nly a few Ts show the same or a higher difference than T0

highly correlated items indicate violation of conditional indepen- dence

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 6 Multidimensionality

Multidimensionality

T11m: small inter-item-correlations

same equation as for T11, but modified test: p

S

◗

s1

ts⑦S where ts ➣ ➝ ➝ ➛ ➝ ➝ ↕ 1, Ts❼As➁ ❇ T0❼A0➁ 0, else if rij in A0 is small, then the difference rij ✏ ➬ rij is also small

• nly a few Ts show the same or a smaller difference than T0

small correlations between items indicate multidimensionality

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 7 Conditional dependence & Multidimensionality

Conditional dependence

T1: many equal responses

❨ count the number of ➌00➑ and ➌11➑ patterns in items i and j ❨ how many Ts have same or a higher value than T0 T1❼A➁ ◗

v

δij where δij ➣ ➝ ➝ ➛ ➝ ➝ ↕ 1, xvi xvj 0, xvi ① xvj many equal responses indicate violation of conditional indepen- dence Multidimensionality:

T1m: few equal responses

❨ how many Ts have same or a lower value than T0 few equal responses indicate that the correlation between items is too small, unidimensionality assumption may be violated

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 8

Conditional dependence & Multidimensionality

Learning

T1ℓ : many ➌11➑ patterns (e.g.,Koller & Hatzinger)

❨ count only ➌11➑ patterns as opposed to T1 T1l❼A➁ ◗

v

δij where δij ➣ ➝ ➝ ➛ ➝ ➝ ↕ 1, xvi xvj 1 else ❨ how many Ts have same or a higher value than T0 if person has learned from one item (xvi 1) then the probability p❼xvj 1➁ is increased for a positive reponse to another item j

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 9 Conditional dependence & Multidimensionality

Conditional dependence

T2 : high dispersion of rawscore rv for a set of items

❨ if items are dependent, the variance of rv is large ❨ because of var❼z➁ var❼x➁ ✔ var❼y➁ ✔ 2 ❻ cov❼x,y➁ ❨ define a set of items I and calculate r❼I➁

v

❨ count how many Ts❈T0 T2❼A➁ varv❼r❼I➁

v ➁

where r❼I➁

v

◗

i❃I

xvi

• ther possibilities: range, mean absolute deviation, median ab-

solute deviation. Multidimensionality:

T2m : low dispersion of rawscore rv for a set of items

❨ count how many Ts❇T0

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 10 Multidimensionality

Multidimensionality

TMU: correlation of rawscore for item subsets

(Koller & Hatzinger) ❨ if two sets of items I are unidimensional, rI

v of set I and rJ v

• f set J should be positiv correlated

❨ with increasing rI

v also rJ v should be increasing

❨ count the number of correlations Ts❇T0 TMU❼A➁ cor❼rI

v,rJ v ➁

• rI

v ◗ i❃I

xvi

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 11 Subgroup-invariance

Subgroup-invariance

T10: based on counts on certain item responses

❨ nij⑦nji is proportional to the ratio of exp❼βi➁⑦exp❼βj➁ ❨ no parameter differences for focal-group foc and reference- group ref: nref

ij ⑦nref ji

nfoc

ij ⑦nfoc ji

❨ sum of differences for all pairs of items ❨ counts of Ts❈T0 T10❼A➁ ◗

ij

❙nref

ij nfoc ji

✏ nref

ji nfoc ij ❙

❨ if the parameter differ across groups, the difference should be increasing

Note: ❨ external criterion (e.g., gender): uniform DIF ❨ internal criterion (e.g., rawscore-median): discrimination, guessing, fal- sity ❨ split on specified item: conditional dependence Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 12

Subgroup-invariance

Subgroup-invariance

T4:counts of positive responses in person subgroups

❨ assumption: in one group of persons G one or more items are easier/more difficult as expected in the RM ❨ count the number of persons who solved these items ❨ easier: counts of Ts❈T0 ❨ more difficult:counts of Ts❇T0 T4❼A➁ ◗

v❃G

xvi

Note: ❨ tests the same assumptions as T10 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 13 Subgroup-invariance

Subgroup-invariance

TDTF : based on item differences on item sumscores

(Koller & Hatzinger) ❨ similar to T4, but with the possibility to test DTF (all items in a test shows subgroup-invariance) ❨ calculate the sumscores (c) for one item (or a group of items) for the reference group cref

i

and for the focal group cfoc

i

❨ calculate the difference of c between focal and reference group. ❨ easier: counts of Ts❈T0 ❨ more difficult:counts of Ts❇T0 TDTF❼A➁ ◗

i❃I

❼cref

i

✏ cfoc

i

➁2

Note: ❨ tests the same assumptions as T10 & T4 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 14 Unfolding response structure

Unfolding response structure - monotonicity

T6: responses in three person subgroups

❨ similar to T4. ❨ split the sample in three rawscore groups and count the number of positive responses only in the middle group Gm ❨ easier (reversed U-shape): counts of Ts❈T0 ❨ more difficult (U-shape): counts of Ts❇T0 T6❼A➁ ◗

v❃Gm

xvi

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 15 Item discrimination

Item discrimination

T5: rawscore for persons with xvi 0 for a certain item

❨ include persons who answer with 0 to a certain item ❨ sum rv of the remaining items for group xvi 0 ❨ counts of Ts❈T0 T5 ❼A➁ ◗

v❙xvi0

rv ❨ if persons with high ability (rv high) fail to solve a certain item, this item may show too low discrimination, falsity, or indicate multidimensionality

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 16

New test statistics

Constructing new test statistics ❨ based on substantive considerations. ❨ based on statistics, where the approximation to the asymp- totic distribution is questionable. – monotone transformations example: point-biserial correlation – simplification example: Mantel-Haenszel statistic

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 17 Monotone transformation

Example: point-biserial correlation rpbis r0 ✏ r1 sr ➽ n0n1 n❼n ✏ 1➁ ➀ ❿Pr0 n0 ✏ Pr1 n1 ➄n0n1 remove sr and n (constant) remove ➸ is a monotone function (n0,n1 ❈ 0) Tpbis❼A➁ n1 Pr0 ✏ n0 Pr1

✘✘✘✘✘

n0n1

✘✘✘✘✘

n0n1

• n1 ◗r0 ✏ n0 ◗r1

❨ counts Ts❈T0 ❨ if persons with high ability (rv high) fail to solve a certain item, this item may show too low discrimination, falsity, or indicate multidimensionality

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 18 Simplification

Example: Mantel-Haenszel statistic tests for conditional independence of two nominal variables across several strata (e.g., 2 ✕ 2 ✕ C tables) MH ❽Pc N11c ✏ Pc E❼N11c❙nc➁➂2 V ar❽Pc N11c❙nc➁➂

as.

✂ χ2

d f1

can be used to test various RM violations. Verguts & DeBoeck (2001): sufficiency, unidimensionality, item dependence Mantel-Haenszel statistic may be simplified to TMH ❽◗

c

n11c ✏ ◗

c

˜ n11c➂2 where ˜ n11c

S

◗

s1

n11c⑦S

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 19 Implementation in R: eRm & RaschSampler

Implementation in R: eRm & RaschSampler

some statistics in eRm

❨ subgroup-invariance – T10 based on counts on certain item responses (global test) – T4 counts of positive responses in person subgroups (test on item level) ❨ conditional independence – T11 large inter-item correlations (global test) – T1 many equal responses (test on item level) – T2 high dispersion of rawscore rv for a set of items (test on item level)

RaschSampler

❨ supply user defined function for arbitrary T-statistics Verhelst, Hatzinger, & Mair (2007)

Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 20

Application in eRm

Example: conditional dependence

T1: many equal responses (test on item level)

> library(eRm) > library(RaschSampler) > t1 <- NPtest(raschdat1,n=500,burn_in=500,step=32,seed=123,method="T1") > print(t1,alpha=0.05) Nonparametric RM model test: T1 (local dependence - increased inter-item correlations) (counting cases with equal responses on both items) Number of sampled matrices: 500 Number of Item-Pairs tested: 435 Item-Pairs with one-sided p < 0.05 (1,9) (1,26) (4,12) (4,26) (4,28) (8,13) (9,16) (10,24) 0.030 0.042 0.036 0.036 0.018 0.014 0.010 0.032 (18,28) (21,22) (25,29) 0.032 0.004 0.002 Psychoco 2012, Ingrid Koller & Reinhold Hatzinger 21