Improving Technology Enhanced Items Eric E. Zilbert National - - PowerPoint PPT Presentation

Improving Technology Enhanced Items

Eric E. Zilbert National Conference on Student Assessment June 24–26, 2019 Orlando, Florida

Presentation Overview

• What we know about Technology Enhanced Items

(TEI).

• What we at the California Department of Education

(CDE) have done to try and improve TEIs.

• What we need to think about to continue to improve

TEIs.

What we know

• There are many types.
• Most tend to be harder than single select multiple

choice items.

• Item type can impact test delivery, scoring, and

standard setting.

• Item type can impact use of accessibility features.
• Little information is provided to item writers about

when to select what type of item and why.

Many types of Items

• Overall CDE has identified 16 possible item types for

use on the science assessments and alternate assessments.

• SBAC uses some additional item types, e.g. text

select.

• 1. Multiple Choice, Single Select

MCSS

• 2. Multiple Choice, Multiple Select

MCMS

• 3. Zone, Single Select

ZoneSS

• 4. Zone, Multiple Select

ZoneMS

• 5. Grid Multiple Select

GridMS

• 6. Inline Choice List Single Select

InlineChoiceListSS

• 7. Inline Choice List Multiple Select

InlineChoiceListMS

• 8. Match, Single Select

MatchSS

• 9. Match, Multiple Select

MatchMS

More item types …

• 10. Extended Text
• 11. Fill-in-blank
• 12. Multiple Fill-in-blank
• 13. Composite
• 14. Composite CR
• 15. Interactive
• 16. Interactive CR

Subject & Grade Item Type Score Points N P-value EL3 TextExtractionMS - Member 0,1 4,839 0.03 EL6 FillInBlank - Discrete 0,1,2* 825 0.34 EL7 MCMS - Member 0,1 569 0.04 E11 Composite - Member 0,1 2,742 0.04 MA5 MatchMS - Discrete 0,1 507 0.04 MA5 GridMS - Discrete 0,1 919 0.03 MA5 ExtendedText - Discrete 0,1 525 0.04 MA7 TextEntry - Discrete 0,1 3,425 0.04 MA7 FillInBlank - Member 0,1 925 0.03 MA8 TextEntry - Discrete 0,1 919 0.02 MA8 TextEntry - Discrete 0,1 999 0.02 M11 TextEntry - Member 0,1 4,138 0.04 M11 ExtendedText - Discrete 0,1 3,829 0.03 M11 TextEntry - Discrete 0,1 751 0.04 M11 MCMS - Discrete 0,1 530 0.04

Of 41 flagged items

• 11 multiple choice multiple select
• 8 text extraction multiple select
• 6 fill in blank (1 too easy)
• 5 text entry items
• 3 extended text
• 3 grid multiple select
• 3 composite
• 1 match multiple select
• 1 multiple choice single select (too easy)

Purpose

• 1. Describe the item types that will be used on the

CAST;

• 2. Define the features of each item type and the

possible functions;

• 3. Provide item writing and accessibility

considerations; and

• 4. Provide the California Department of Education

(CDE) with a reference guide to use while reviewing items.

Contents

Table 1: Description of Item Types Table 2: Examples of Item Types

MCSS (Multiple Choice, Single Select) This item type generally consists of a stem and list of choices; the test taker can select only one choice to

• respond. The options use a radio button, but the test

taker can select text or an image to respond to the item by clicking in the option space rather than exactly in the radio button. Item should have four options: one key and three distractors. A text-only version of this item type is accessible. MCMS (Multiple Choice, Multiple Select) This item type generally consists of a stem and a list of choices; the test taker can select one or more choices (options) to respond. The options use check boxes or the test taker can select text or an image to respond to the item by clicking in the option space rather than exactly in the check boxes. The stem should specify how many selections should be made. A two-selection item should have four or five options, and a three- selection item should have five options. A text-only version of this item type is accessible.

Description of Item Types

Item Type Description of Item Functionality and Other Characteristics Item Writing and Accessibility Considerations

Some item types are more accessible than others …

Item Type Description of Item Functionality and Other Characteristics Item Writing and Accessibility Considerations Interactive Unscored simulations require the test taker to select inputs that provide certain outputs to answer one or more

• questions. Scored

simulations require the test taker to select inputs that are scored. This item type is used with performance tasks and not with discrete items. This item type is not accessible.

Example Items

InlineChoiceListSS

The test taker selects one response from a drop- down menu to fill in the blank. The keys are numerical values given in the same

• rder as the available
• selections. For example, a

1 represents the first choice in the list as the correct answer.

Item Type Test Taker Interaction and Keys Example

Enhancing Technology Enhanced Items

• Add information about the relationship between

specific content and the various item types.

• Add information about the function/design of

distractors for selected response items.

• Create guidelines for rubric development for short

constructed response items.

• Provide information on content that should not be

included in specific item types.