SLIDE 1
Reports for supporting full-cycle assessment Rebecca Freund and Amy - - PowerPoint PPT Presentation
Reports for supporting full-cycle assessment Rebecca Freund and Amy - - PowerPoint PPT Presentation
Reports for supporting full-cycle assessment Rebecca Freund and Amy Arneson University of California at Berkeley Overview NRC ASSESSMENT TRIANGLE Observation Cognition Reports Interpretation 3 PURPOSES OF REPORTS Informational
SLIDE 2
SLIDE 3
3
NRC ASSESSMENT TRIANGLE
Observation Interpretation Cognition Reports
SLIDE 4
4
PURPOSES OF REPORTS
- Informational
– Present results in an accessible way
- Interpretable
– Help users understand the information
- Educational
– Guide a deeper understanding of a concept
SLIDE 5
5
ASSESSMENT DEVELOPMENT CYCLE
Four building blocks
Construct Definition Item Design Outcome Space Definition Measurement Model
SLIDE 6
6
ASSESSMENT DEVELOPMENT CYCLE
Creation of scoring guides
Construct Definition Item Design Outcome Space Definition Measurement Model
BASS scoring collection begins
SLIDE 7
7
ASSESSMENT DEVELOPMENT CYCLE
Reports
Construct Definition Item Design Outcome Space Definition Measurement Model
Pre-Psychometric Analysis Supporting Psychometric Analysis Post-Psychometric Analysis
User facing Researcher facing User facing
SLIDE 8
Pre-analysis reports
SLIDE 9
9
MOTIVATION
- Provides useful information to teachers
- Encourages continued use of the application
- Supports conceptual understanding to prepare for
later, more complex reports
SLIDE 10
10
TYPES OF REPORTS
- Group report
– How many students responded at each level?
- Individual report
– How did this student respond to each item?
SLIDE 11
11
GROUP REPORT
- Size of bars =
proportion of students at each level of item
SLIDE 12
12
GROUP REPORT
- Size of bars =
proportion of students at each level of item
- Hover to see level
and student names
SLIDE 13
13
GROUP REPORT
- Size of bars =
proportion of students at each level of item
- Hover to see level
and student names
- Organized by
construct
SLIDE 14
14
INDIVIDUAL REPORT
- Key showing all
levels of construct
SLIDE 15
15
INDIVIDUAL REPORT
- Key showing all
levels of construct
- Shapes show
available levels for each item
SLIDE 16
16
INDIVIDUAL REPORT
- Key showing all
levels of construct
- Shapes show
available levels for each item
- Colors show
location of that student’s response
SLIDE 17
Psychometric reports
SLIDE 18
18
PRINCIPLES
- Show persons and items on the same scale
- Check for coverage at different levels of ability
- Identify items that show misfit
SLIDE 19
19
GROUP REPORT: WRIGHT MAP
- Left panel shows
location of students
- Right panel shows
location of item levels
- Identify difficulty
bands and outlier items and levels
SLIDE 20
20
INDIVIDUAL REPORT: KIDMAP
- Horizontal line
shows student location
- Left side shows
levels reached (scored at at least that level), right side not reached
- Yellow regions are
unexpected
SLIDE 21
Post-analysis reports
SLIDE 22
22
GOALS
- Show overall performance in a construct
- Present student results in terms of psychometrics in
accessible ways
- Bring together all the information in the system
SLIDE 23
23
INDIVIDUAL REPORT
- Width of levels suggests ability increase needed to advance
SLIDE 24
24
INDIVIDUAL REPORT
- Width of levels suggests ability increase needed to advance
- Student location in terms of construct levels, with error bars
SLIDE 25
25
INDIVIDUAL REPORT
- Width of levels suggests ability increase needed to advance
- Student location in terms of construct levels
- Hover to see names of levels
SLIDE 26
26
GROUP REPORT
- Location of all
students in terms of construct levels
- Assess the
proficiency level of the class as a whole
- Identify and
compare individual students
SLIDE 27
27
OVERALL REPORT PRINCIPLES
- Informational
– How students performed on the items and in the construct
- Interpretable
– Connecting responses to levels of the construct
- Educational
– Strengthen teachers’ understanding of performance in terms of construct levels, vs. right/wrong or arbitrary number of points
SLIDE 28