Empirical Methods for Evaluating Maps: Illustrations and Results W. - - PowerPoint PPT Presentation

▶
empirical methods for evaluating maps illustrations and
SMART_READER_LITE
LIVE PREVIEW

Empirical Methods for Evaluating Maps: Illustrations and Results W. - - PowerPoint PPT Presentation

Sep 05, 2023 342 likes •496 views

Empirical Methods for Evaluating Maps: Illustrations and Results W. Jake Thompson & Brooke Nash Methods for Evaluating Map Structure External outcomes Classical item statistics Unidimensional models 2 A Framework for Map

slide-1
SLIDE 1

Empirical Methods for Evaluating Maps: Illustrations and Results

  • W. Jake Thompson & Brooke Nash
slide-2
SLIDE 2

2

Methods for Evaluating Map Structure

  • External outcomes
  • Classical item statistics
  • Unidimensional models
slide-3
SLIDE 3

3

A Framework for Map Evaluation

  • Diagnostic Classification Models (DCMs)
  • Mastery profiles on the set of assessed skills
  • Three methods

–Patterns of Mastery Profiles –Patterns of Mastery Assignment –Patterns of Attribute Difficulty

slide-4
SLIDE 4

4

An Illustrative Example

  • 3 attribute assessment
  • Linear map structure

Initial Precursor Target

slide-5
SLIDE 5

5

Map Structure in a DCM Context

slide-6
SLIDE 6

6

  • Estimate two models

– Saturated model with all profiles – Reduced model with only hypothesized profiles

  • Assess model fit

– Posterior predictive model checks – Model comparisons

Patterns of Mastery Profiles

Initial Precursor Target 1 1 1 1 1 1 1 1 1 1 1 1

slide-7
SLIDE 7

7

Patterns of Attribute Mastery

  • Estimate each attribute as a separate 1-attribute DCM

(equivalent to LCA)

  • Set mastery threshold (0.8)

Student Initial Precursor Target 1 .97 .85 .43 2 .86 .52 .13 3 .92 .89 .83 4 .88 .65 .85 5 .55 .70 .33 … … … … Student Initial Precursor Target 1 1 1 2 1 3 1 1 1 4 1 1 5 … … … … Student Initial Precursor Target 1 1 1 2 1 3 1 1 1 4 1 1 5 … … … …

slide-8
SLIDE 8

8

  • Measure attribute difficulty using

classical p-values

  • Group similar respondents a priori
  • Calculate the weighted average p-

value for each attribute and group

Patterns of Attribute Difficulty

slide-9
SLIDE 9

9

Case Study: Dynamic Learning Maps

  • Each Essential Element (EE) available at multiple

levels of depth, breadth, and complexity –5 levels in ELA and mathematics –3 levels in science

  • Linkage levels are assumed to follow a linear

progression

  • Students test on only one linkage level for each EE

during the operational assessment

slide-10
SLIDE 10

10

  • Patterns of Profile Mastery

– Models fail to converge due to missing data

  • Patterns of Attribute Mastery

– The majority of flags were in ELA – More flags for higher linkage level reversals than lower

Case Study: Dynamic Learning Maps

slide-11
SLIDE 11

11

Case Study: Dynamic Learning Maps

  • Patterns of Attribute Difficulty

–Flags by subject

  • 28 ELA EEs
  • 35 mathematics EEs
  • 0 science EEs
slide-12
SLIDE 12

12

Summary

  • Benefits and limitations of each method within the

framework

  • Wide breadth of methods provides complementary

information

  • Application to DLM shows insights that can be

applied to future test and map development

slide-13
SLIDE 13

13

Ongoing Research

  • Continue to refine methods

–Alternative modeling strategies for Patterns of Mastery Profiles –Simulation studies to inform empirical flagging criteria

  • Expanding beyond the progression of linkage levels

within EEs to the more fine-grained map structure

slide-14
SLIDE 14

More Information