Relationship between test item cognitive complexity, content, and - - PowerPoint PPT Presentation

Relationship between test item cognitive complexity, content, and instructional sensitivity

Tanya Longabach Jie Chen, Ph.D. Neal Kingston, Ph.D.

Presentation outline

n Significance of instructional sensitivity in today’s education

environment

n Study purpose n Methods and data analysis n Results: factors that may influence reading and math item

instructional sensitivity

n Results: difference in instructional sensitivity of reading and math

items

n Implications and direction for future research

Instructional sensitivity in today’s education environment

n Increased interest in instructional sensitivity as a means to

determine appropriateness of assessment

n Instructional sensitivity is the responsiveness of a test or test item to

instruction

n A test is only as instructionally sensitive as the items on the test n However, few studies have been done to determine what it is that

makes test items instructionally sensitive

Issues associated with determining instructional sensitivity

n Critical thinking items and instructional sensitivity (Wiliam, 2007;

Embretson, 2010; Muthen, Kao, & Burstein, 1988)

n Critical thinking items tend to be less instructionally sensitive

n Content subjects (e.g. math vs. reading) (Nye et al., 2004; Boscardin

et al., 2005)

n Math items tend to be more instructionally sensitive than reading

items

Study purpose

n Address the gaps in knowledge about characteristics of

instructionally sensitive items

n Examine the impact of the following test item characteristics on

instructional sensitivity:

n Use of formula to answer the question (math only) n Presence of a chart or graph in the item (math only) n Use of special vocabulary in the item (math only) n Cognitive complexity as defined by Bloom’s taxonomy (math and reading) n Standard to which the items were written (math and reading) n Content subject (math vs. reading) n Grade level

Bloom’s taxonomy

Kansas Curricular Standards example: mathematics

Standard description Indicator number Indicator descriptions Number of items in each interim assessment Number and computation M.5.1.1.K1 Represents and explains whole numbers and non-negative rational numbers from 0 to 1,000,000. 2 M.5.1.3.A4 . Uses a variety of computational methods to solve problems with exact or approximate answers 2 M.5.1.3.K2 Uses various strategies to estimate non- negative whole and rational quantities. 2 M.5.1.4.A1 Solves one- and two-step problems using a variety of computational procedures. 2 M.5.1.4.K4 Determines greatest common factor and least common multiple of two whole numbers. 2 Algebra M.5.2.2.K1 Represents and relates unknown quantities from 0 to 1,000 using variables and symbols. 2 M.5.2.2.K2 Solves one-step equations with whole number solutions using addition, subtraction, or multiplication. 2 M.5.2.3.K4 Uses a function table to identify, plot, and label points in the first quadrant of the coordinate plane. 2

Kansas Curricular Standards example: reading

Standard description Indicator number Indicator descriptions Number of items in each interim assessment Reading, Vocabulary R.5.1.3.1 Determines the meaning of words or phrases by using context clues (e.g., definitions, restatements, examples, descriptions) from sentences or paragraphs. 2 R.5.1.3.4 . Determines meaning of words through knowledge of word structure (e.g., contractions, root words, prefixes, suffixes). 2 Literature, Literary Concepts R.5.2.1.1 Identifies and describes characters' physical traits, personality traits, and feelings, and explains reasons for characters' actions and the consequences of those actions. 2 R.5.2.1.2 Identifies and describes the setting (e.g., environment, time of day or year, historical period, situation, place) and explains the importance of the setting to the story or literary text. 2 R.5.2.1.3 Identifies and describes the major conflict in a story and major events related to the conflict (e.g., problem or conflict, climax, resolution). 2

Methods

n Data from 2011-2012 Kansas Interim Assessment, window 2

(November 11, 2011, to January 13, 2012)

n Multi-stage computer adaptive test, two testlets

Mathematics Grade level 3

r d grade 4 th grade 5 th grade 6 th grade 7 th grade 8 th grade Total

N items 24 28 30 28 30 28 168 N students 1307 778 724 936 1220 1387 6350 Reading N items 18 21 21 n/a n/a n/a 60 N students 1194 807 838 n/a n/a n/a 2839

Data analysis

n Conducted logistic regression to identify instructionally sensitive

items (Chen & Kingston, 2012) Z = β0 + β1θ + β2G + β3(θ*G)

n Instructional sensitivity was measured as the difference between

Nagelkerke’s R2 value of the logistic regression model in step 3 and that of the model in step 1 (Chen, 2012), converted to z score ∆R2 = R2 (M3) - R2 (M1)

n Conducted correlation and stepwise regression to examine

relationship of instructional sensitivity with predicting factors

Results: instructionally sensitive items

Correlation results: mathematics

*Instructional sensitivity from here on was measured as the difference between Nagelkerke’s R2 value of the logistic regression model in step 3 and that of the model in step 1 (Chen, 2012), converted to z score. Bolded correlation values are significant; * indicates p<.05, ** indicates p<.01. Instructional sensitivity* S_1 S_2 S_3 Bloom’s taxonomy graph vocabulary formula 3M

• .10
• .25

.22

• .35
• .35

.88

**

.52

**

4M

• .32
• .21

.12

• .18

.37 .33 .13 5M

• .08
• .21

.12

• .19
• .25
• .10

.52

**

6M

• .41

*

• .22

.60

**

• .47

*

.40

*

.41

*

• .14

7M .07 .14

• .03
• .18
• .38

*

.03 .40

*

8M

• .23

.03 .05

• .26
• .08

.16

• .16

Regression results: mathematics

grade variable 1 β weight variable 2 β weight R

2

3rd vocabulary .86 graph

• .23

.82 4th

• 5th

formula .52

• .28

6th standard .60

• .36

7th formula .47 graph

• .46

.36 8th

Results: instructionally sensitive items

Correlation and regression results: reading

• Regression: no variables were entered in the equation.

Instructional sensitivity S_1 S_2 Bloom’s taxonomy 3R .02

• .30
• .11

4R

• .35

.30 .25 5R

• .21

.13 .35

Combined grades by subject

Correlation and regression results: grades combined by subject

Regression Correlation

grade variable 1 β weight variable 2 β weight R

2

All grades math Bloom’s taxonomy

• .24

vocabulary .21 .12 All grades reading

• Instructional

sensitivity Bloom’s taxonomy graph vocabulary formula grade All grades math

• .29**
• .03

.26** .09

• .08

All grades reading .13 n/a n/a n/a .15

Discussion

n More instructionally sensitive items in math assessment than in

reading assessment

n Reading skills are acquired in a variety of settings, while math skills

are acquired mainly in school

n Math and reading assessments may be testing constructs at

different granularity levels

n Non-linear relationship of instructional sensitivity with grade variable

Implications and future directions

n OTL information should be included in test information dataset for

the purposes of research on instructional sensitivity

n Further research is needed on how to make test items more

instructionally sensitive

n Controlling for confounding factors, such as previous exposure to

content, will increase the precision of instructional sensitivity measurement

n Standard clarity should be examined further n Using more precise measurement of OTL n Nested information needs to be taken into account

Questions? Comments?

For additional information, please contact: Tanya Longabach tlongabach@ku.edu