Evaluating Potential Benefits and Risks of Technology Enhanced (TE) - - PowerPoint PPT Presentation

This work is licensed under a Creative Commons Attribution 4.0 International License.

Evaluating Potential Benefits and Risks of Technology Enhanced (TE) Items in Early Literacy and Mathematics Assessments: Results and Lessons Learned

Julie Murphy K-5 Education Assessment Specialist Michigan Department of Education June 26, 2019 CCSSO National Conference on Student Assessment Orlando, FL Kyle Ward Mathematics Assessment Consultant Michigan Department of Education Nichole Mosser English Language Arts Assessment Consultant Michigan Department of Education Joseph Martineau Senior Associate Center for Assessment

Rationale

“Hey Napoleon, what did you again last summer?” —Napoleon Dynamite

www.nci ea.org

2

Considerations in Moving Toward TEIs

• The primary reason to use TEIs is to improve the directness with

which items measure the intended standard(s)

§ This is particularly so for alignment to cognitive complexity, practices, and processes. § Several verbs used in content standards are problematic for traditional items, for example

• Draw
• Graph
• Explain
• Write
• Demonstrate fluency...

§ When an examinee has to apply their knowledge by building a response, it is arguably a more direct measure than selecting a response from among a few of the possible response

3

Considerations in Moving Toward TEIs

• TEIs can reduce indirectness of measuring more complex skills,

but...

§They do not necessarily reduce indirectness of measuring complex skills. §It is generally more difficult to write items that directly measure higher level skills, no matter the item type.

• In my experience, many TEIs written to measure higher complexity

content are de facto multiple choice items.

§ This may be problematic if the affordances of TEIs could have been used to more directly measure the intended content. § This may not be problematic if the intended content can be appropriately measured by a multiple choice question, or at least as directly as is possible with a TEI

4

Considerations in Moving Toward TEIs

• There are two additional possibilities when comparing TEI item

types with traditional item types

§A TEI can improve on traditional item types in other ways than improving the directness of measuring complex content, for example...

• Increasing examinee engagement
• Consistency with increased classroom use of electronic devices
• Use of color in graphics
• Faster results

§A TEI can introduce construct irrelevant variance

• Requiring a drag and drop when a simple click would do
• Unnecessary use of graphics or animation
• Lack of attention to colorblindness
• Anxiety on the part of stakeholder because of disrupting the status quo

5

Considerations in Moving Toward TEIs

• It is important to evaluate whether the generally increased cost of

TEIs be evaluated against the benefits and risks of using TEIs

• When an assessment is already in place, this can be an anxiety-

inducing task

§ There are many questions we don’t ask because we are not sure we want to know the answers § Do I want to identify a potential problem with an existing assessment. The implications are considerable § Think of how we do bias and sensitivity review vs. content review

• We do data free review for both content review and bias/sensitivity review
• We do data review for both content review and bias/sensitivity review
• We generally do post-administration analysis only for content issues

6

What the Items Look Like

“[They] will ride eternal, shiny and chrome”

—Mad Max, Fury Road

www.nci ea.org

7

Explanation of Technology Enhanced (TE) Items

• According to the Smarter Balanced Assessment Consortium, a

Technology-Enhanced Item is a computer-delivered item which includes specialized interactions for collecting response data. These include interactions and responses beyond traditional selected- response or constructed-response.

Choice

Choice

Hot Spot

Hot Spot

Hot Text

Drag and Drop

Multiple Drag and Drop

Multiple Drag and Drop and Hot Spot

Methodology

“Orthogonalization algorithm. Yes, sir. I prefer it over Euclidean coordinates.”

—Hidden Figures

www.nci ea.org

17

Analysis of Questions Asked About TE Items

About 800 ELA TE items and 640 Math TE items were reviewed considering four

• questions. The questions were:
• Is the TE item a dressed-up Multiple Choice (MC) item?
• If so, is the TE item OK as a dressed-up MC item (based on alignment to content

standards and Depth of Knowledge (DOK)?

• Excluding issues of alignment, does being a TE item introduce other helpful

features beyond what would be possible using paper & pencil MC items?

• Excluding issues of alignment, does being a TE item introduce features likely to

create construct irrelevant variance that would not exist with paper & pencil MC items?

Analysis of Questions Asked About TE Items

The possible options for each category were:

• Definitely Not
• Probably Not
• Unsure
• Probably Yes
• Definitely Yes

Examples: Multiple Drag and Drop

Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Examples: Multiple Drag and Drop

Not a dressed up MC item (requires constructing a response rather than selecting correct responses)

Examples: Multiple Connection Matching

Match each animal to the words that tell how it moves. Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Examples: Multiple Matching Table

Not a dressed up MC item (requires multiple matching rather than selecting correct responses)

Order

Not a dressed up MC item (requires reordering text rather than selecting correct responses)

Multiple Sequence Drag and Drop

A student is writing a story about Mike Mouse. The student wants to add more details. Move 2 sentences into the paragraph that add the best details about Mike Mouse.

A dressed up MC item (to select two responses, the student has to drag them to a certain spot)

Hot Text

The sentence below has a mistake. Choose the word in the sentence that should begin with a capital letter.

Not a dressed-up MC item (the answer is given in context rather than naked as a response option)

Line Plot Input

Not a dressed up MC item (requires examinee to build a frequency line plot)

Hot Spot-Count

Not a dressed up MC item (requires one-to-one correspondence as evidence by mouse clicks)

TEIs Introducing Features Likely to Create Construct Irrelevant Variance

• There were some different reasons why TE items introduce features

to create construct irrelevant variance that would not exist with paper and pencil multiple-choice items. Here are a few reasons.

• Connection matching is not intuitive to the student.
• Some students may not have the computer skills to interact with this

type of TE item.

• Some students may not know how to use technology to match the

items.

TEIs Introducing Other Helpful Features

• This assessment does not have a Paper/Pencil option. Therefore, our

answer choices are based on students not transferring answer choices.

• Consider- Paper/Pencil assessments include answer documents that

students must use to transfer selections. For younger students, fine motor/vision transfer may be difficult.

• Color and graphics play an important role in kindergarten students'

ability to understand texts. Paper/Pencil assessments do not offer color as an option.

TEIs Introducing Other Helpful Features

• ELA

§There were 45 different reasons why the TE items introduce

• ther helpful features beyond what would be possible using

paper and pencil MC items. §Here are a few reasons. §Grade 2

• Drag and Drop Items

§ In context—paragraph, sentence. For example, it is helpful to move and then see a word, letter, picture, phrase, punctuation, etc. within the context. § Ordering—sequencing events. For example, it is helpful to move and then see the sentences in the correct order.

TEIs Introducing Other Helpful Features

• Grade 2 (continued)

§Matching Items—picture to text, text to text. For example, it would be helpful because the student can see the line going to the answer and eliminate options. §Hot Text Items—too many options. For example, there are too many words, phrases, sentence to create a multiple- choice question.

TEIs Introducing Other Helpful Features

• Grade 1 has many of the same helpful features as in grade 2.
• Here are some unique features.

§ Drag and Drop Items

• In context—paragraph, sentence. For example, it is helpful to move and then see a

punctuation at the end of a sentence or a capital letter at the beginning.

• Graphic with text—paragraph, sentence, word. For example, it is helpful to move

and then see the description next to the picture or sequence the pictures.

§ Hot Text

• When the word is read to the student, the item is more user friendly and the

pronunciation is consistent.

• It is helpful to be able to click on the first word in the sentence as opposed to

multiple choice in paper and pencil.

TEIs Introducing Other Helpful Features

§Grade K has many of the same helpful features as in grades 1 and

• 2. Unique features are the following:

§The use of color with both drag and drop and hot spot-select enhances the interaction. For example, it is helpful to have the color clue that matches the text. (red apple, gray sky)

• Drag and Drop Items

§ Viewing letters within a word is helpful so the student can see the completed word. § Hearing sounds within a word is helpful because it make the sounds consistent. § Graphic with text or another graphic—paragraph, sentence, word. For example, it is helpful to move and then see the sentence under the picture.

TEIs Introducing Other Helpful Features §Hot Spot-Select

• When the word is read to the student, students listen to

the letter sounds being read, which offers consistency.

• Students visually see the sentence laid out and can

easily click on a specified word.

• Clicking on the word while in context horizontally helps

the student with tracking and ease.

TEIs Introducing Other Helpful Features

• Math

§Hot Spot Count

• Items with more than 3 objects to count is not practical

to make as a multiple choice

TEIs Introducing Other Helpful Features

§ Multiple Drag and Drop

• Items with multiple numbers to be dragged—later choices depend on earlier
• choices. For example, number sequence with more than one missing number.
• Items with base ten blocks—creating a number is much different than selecting a
• number. For example, show the number 27 with base then blocks.
• Composing shapes—creating a shape is much more difficult than choosing a shape.

For example, create a hexagon given two trapezoids is different than selecting the trapezoid from a variety of shapes.

• Determining area—using squares to cover the shape is different from choosing the

number of squares.

• Picture Graph—dragging the pictures to correct category is different from selecting

from static picture graph.

TEIs Introducing Other Helpful Features

§ Multiple Drag and Drop (continued)

• Ordering by length—putting objects in order. For example, put the objects in order

from shortest to longest.

• Sorting shapes—putting shapes in the correct categories by shape name or

characteristics of shape. For example, place the triangles in the box marked triangles and the squares in the box marked squares.

• Measuring objects with draggable graphic. For example, how many stamps long is

the envelope?

• Dragging multiple graphics to give a number. For example, move 5 frogs to the pond.

TEIs Introducing Other Helpful Features

§ Line Plot Input

• Creating a line plot by adding the X’s above the values.

Kindergarten Teacher:

• “Most of my kids enjoyed

taking the assessment. They thought it was fun to do on the computer.”

First Grade Teacher:

• “It coved concepts that we

taught and used electronic tools that my students were familiar with in real like.”

Comments about the Assessment from Teachers

Second Grade Teacher:

• “My students were able to

log in and guide themselves through the test.”

First Grade Teacher:

• “The questions I saw were

content that we had taught during the year and seemed appropriate.”

Comments about the Assessment from Teachers

First Grade Student:

• “That was fun! It’s like

playing a game.”

Second Grade Student:

• “Wow—I can’t believe they

let us take a test on the computer!”

Comments about the Assessment from Students

Results of Judgmental Item Ratings

43

“What percentage of a plan do you have.”

—Guardians of the Galaxy

Ratings of “MC-ness” for grades K-2

44

Ratings of “MC-ness” for Kindergarten

45

Ratings of “MC-ness” for Grade 1

46

Ratings for Grade 2

47

Ratings of “MC-ness” for items of operational maturity

48

Ratings of “MC-ness” for items awaiting data review

49

Rating of whether de facto MC items are appropriate as such

50

Results of whether TE features improve items beyond alignment

51

Ratings of whether TE features introduce potential construct irrelevant variance

52

Lessons Learned

“I hope at least you kids have learned something from this.”

—National Lampoon’s Vacation

www.nci ea.org

53

Lessons Learned

• Kudos: MDE is the only organization I know of to critically examine a complete item bank composed of TEIs
• Need to refine the questions

54

Question Used in this Study Refined Questions for Follow-up Is the item a de facto multiple choice (MC) item? Is the TEI a de facto MC item (when the examinee responds to the item, does the method of response represent selecting one or more responses from a set of possible responses)? For de facto MC items, is the TEI OK as a de facto MC item (based on alignment to content standards and DOK)? For all items, could an item of...

• this TEI type more directly measure the intended content at the

intended level of cognitive complexity?

• another TEI type more directly measure the intended ...?

Excluding improving alignment to DOK, does being a TEI introduce other helpful features beyond what would be possible using paper & pencil MC items? For all items, beyond directness of measurement, does the technology enhancement of this item introduce features that improve upon what could be done MC items? Excluding improving alignment to DOK, does being a TEI introduce features likely to create construct irrelevant variance that would not exist with paper & pencil MC items? For all items, does technology enhancement introduce features likely to create construct irrelevant variance that would not exist with MC items?

Lessons Learned

• Kudos: MDE is the only organization I know of to critically examine a complete item bank composed of TEIs
• Need to refine the questions

55

Question Used in this Study Refined Questions for Follow-up Is the TEI a de facto MC item?

• Are the interactions a student with the item a method of selecting one or more correct

answers? Could an item of this/another TEI type more directly measure the intended content at the intended level of cognitive complexity?

• At what level of directness are the verbs in the intended content standards

represented in the interactions an examinee has with the item?

• At what level of directness are the concepts in the content standards represented in

the item? For all items, beyond directness of measurement, does the technology enhancement of this item introduce features that improve upon what could be done MC items?

• To what degree are there technology enhancements that that make the item more

engaging?

• To what degree are there technology enhancements that make the item easier to

interact with? For all items, does technology enhancement introduce features likely to create construct irrelevant variance that would not exist with MC items?

• To what degree do the technology enhancements introduce unnecessary complexity in

student interactions with the item?

• To what degree do the technology enhancements introduce novel and potentially

problematic ways of interacting with the item?

www.nciea.org jmartineau@nciea.org

This work is licensed under a Creative Commons Attribution 4.0 International License.

www.michigan.gov/mde mossern@michigan.gov murphyj12@michigan.gov wardk2@michigan.gov