PARCC Diagnostic Assessments for Mathematics Comprehension: A - - PowerPoint PPT Presentation

PARCC Diagnostic Assessments for Mathematics Comprehension: A Diagnostic Classification Model Approach

Laine P. Bradshaw

The University of Georgia

June 24, 2015

• A brief introduction to diagnostic classification models

(DCMs)

» How are they different from the models that are typically used in assessment? » Why are they particularly useful for diagnostic assessments?

• How did we use the DCM framework to develop PARCC’s

diagnostic assessment system for mathematics comprehension in Grades 2-8?

• What new challenges in reporting exist when

transitioning to a new psychometric framework?

Overview

Big Picture: Math Content to be Measured

Typical End of Grade Summative Test

• Traditional testing procedures measure an overall

ability in an area with a continuous latent variable

7th grade Math Ability

Cluster 4 Item

Math ability is continuous Responses to items are observed The more math ability a person has the more likely he or she is to answer an item correctly

Traditional Measurement Models for Summative Tests

Cluster 2 Item Cluster 1 Item Cluster 3 Item … Cluster 2 Item

Traditional Testing and Classification Methods

• Information from Continuum:

» Spencer has more math ability than Sue » Hugh scored a 240 on the test » Juan scored in the 70th percentile

• Diagnosis from Cut Score:

» Sue scored below the cut score » Sue is not proficient in 7th grade math Not Proficient

7th Grade Math Ability

Proficient

HIGH LOW

Question difficult to answer: What are Spencer’s weaknesses?

7th grade Math Ability

Cluster 2 Item Cluster 6 Item Cluster 1 Item Cluster 3 Item

…

Diagnostic Approach

Instead of measuring an overall math ability in 7th grade, we can break “math” down into a set of skills or attributes: Cluster 1 Cluster 2 Cluster 3 Cluster 4

Cluster 1

Diagnostic Approach

Cluster 4 Item Cluster 2 Item Cluster 1 Item Cluster 3 Item … Cluster 2 Item

Cluster 2 Cluster 3 Cluster 4

Each cluster has two levels A student can be in one of two groups, or levels, for each cluster

• Higher group (masters; on-track)
• Lower group (non-masters; needs attention)

Masters are more likely than non-masters to answer items correctly.

Diagnostic Classification Models

Subtract Add Multiply Divide Masters Non-masters

• DCMs uses responses to items to place students into groups

according to multiple skills

–No cut score is used to put students into groups; the model is built to do that

Cluster 1 Cluster 2 Cluster 3 Cluster 4

Diagnostic Classification Models

Subtract Multiply Divide Add

• Students receive attribute-specific feedback
• The model provides a probability each attribute is mastered
• Notice there is no “score” in a traditional or grading sense

Spencer has mastered Cluster 1 and 2, but should improve his understanding

• f Cluster 3 and 4.

.84 .76 .35 .28

Cluster 1 Cluster 2 Cluster 3 Cluster 4

• Aligns with a standards-based view of achievement

where mastery of all standards is monitored

» Modeling multiple “dimensions” or traits » Instead of answering: Did this student meet the standards? » Answer: Which standards has the student met?

• DCMs provide valuable information with fewer data

demands

» Higher reliability per dimension than IRT/MIRT models » Potential to drastically reduce testing time » Quick and dirty categorical feedback

Benefits of Using DCMs

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 3 13 23 33 43 53 63 73 83 93 Reliability Number of Items

DCM IRT

DCM Reliability

8 Item Reliability

Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275.

Psychometric Approach for Mathematics Comprehension

Non-summative Assessments

Mathematics Comprehension

Example: Cluster Level Attributes in Math

Number Systems Learning Progression Attribute 1: Divide Fractions Attribute 2: Understand Rational Numbers Attribute 3: Operate with Rational Numbers

Example: Cluster Level Attributes in Math

Number Systems Learning Progression

• Create a separate diagnostic test for each individual

cluster

… … …

Divide Fractions Understand Rational Numbers Operate with Rational Numbers

Diagnostic Feedback to Students

Divide Fractions Understand Rational Numbers Operate with Rational Numbers

Number System Results Diagnostics

Example Student A Example Student B Example Student C

On-track Needs Improvement

Diagnostic Key:

Mastery Probability

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways Individual Cluster Test 1 attribute ~8 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways

Grade Level Test 6 attributes ~48 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways

Learning Progression Test 6 attributes ~48 items

Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways

Grade Level Test 6 attributes ~48 items Learning Progression Test 6 attributes ~48 items Individual Cluster Test 1 attribute ~8 items

• DCMs do not provide a score!

» They provide classifications

• How will teachers and students interpret classifications?
• How will they interpret the probabilities of mastery?
• What is the best way to present the results so that

interpretations are clear and accurate?

Challenges in Reporting

• Teachers can monitor the percentage of students who

have mastered each cluster for a given grade level

» Aggregate feedback

At the class level

• Can show the multidimensional mastery profiles for

each student in the class

• Who has mastered each cluster?

At the class level

• Should the

probability of mastery be provided? Where?

» Analogous to decisions for where to put standard errors on score reports for IRT-based assessments

At the student level

• DCMs are parametric, latent class models

» Other examples of these models exist » For example, Bayes Nets

• Models in the same family produce the same student

estimates—mastery classifications and probabilities of mastery

» These models face similar challenges in reporting

Classification-based Reporting

Please feel free to contact me with any questions or comments: laineb@uga.edu