Virginia Modified Achievement Standards Test (VMAST): Lessons - - PowerPoint PPT Presentation

▶
virginia modified achievement standards test vmast
SMART_READER_LITE
LIVE PREVIEW

Virginia Modified Achievement Standards Test (VMAST): Lessons - - PowerPoint PPT Presentation

Mar 08, 2023 219 likes •459 views

Virginia Modified Achievement Standards Test (VMAST): Lessons Learned Shelley Loving-Ryder Virginia Department of Education June 20, 2011 2011 National Conference on Student Assessment Orlando, Florida VMAST: Overview GSEG to develop

slide-1
SLIDE 1

2011 National Conference on Student Assessment Orlando, Florida

Virginia Modified Achievement Standards Test (VMAST): Lessons Learned

Shelley Loving-Ryder Virginia Department of Education June 20, 2011

slide-2
SLIDE 2

2011 National Conference on Student Assessment Orlando, Florida

VMAST: Overview

  • GSEG to develop assessment for Grade 8

reading and mathematics

  • Adds supports and simplifications recommended

by educators to existing online test items

slide-3
SLIDE 3

2011 National Conference on Student Assessment Orlando, Florida

  • Conducted focus groups and online surveys with

teachers and special education directors on learning characteristics and support/ simplifications used with identified students in instruction

  • In fall 2008 and spring 2009 conducted 2 pilots

using cognitive interviews with students from urban and rural school districts who were identified as likely VMAST participants

  • 3 -

Activities to Date

slide-4
SLIDE 4

2011 National Conference on Student Assessment Orlando, Florida

Field Test

  • Field test with approximately 500 students in

May-June, 2010 for VMAST grade 8 reading and mathematics.

  • Students participating in field test were students

with disabilities who met the participation criteria for VMAST

  • 47 field test students were administered

unsupported items first and were then administered the supported items

slide-5
SLIDE 5

2011 National Conference on Student Assessment Orlando, Florida

Results: Reliability

Subject N-Count Alpha VMAST Alpha SOL Math 235 0.85 0.89 - 0.93 Reading 257 0.84 0.88 – 0.90

slide-6
SLIDE 6

2011 National Conference on Student Assessment Orlando, Florida

Results of the Mathematics Field Test

slide-7
SLIDE 7

2011 National Conference on Student Assessment Orlando, Florida

Math Supports and Simplifications

  • Provided hint boxes with formulas,

strategies, and reminders,

  • Color coded important information
  • Provided online manipulatives
  • Simplified numbers
  • Simplified language in stems
  • Added or altered graphics
  • Reduced answer options from 4 to 3
slide-8
SLIDE 8

2011 National Conference on Student Assessment Orlando, Florida

Overall Comparison of VMAST vs SOL for Grade 8 Mathematics

VMAST is 40 MC items (vs 50 on the SOL test) measuring:

  • RC1: Number and Number Sense;
  • RC2: Computation and Estimation;
  • RC3: Measurement and Geometry;
  • RC4: Probability and Statistics; and
  • RC5: Patterns, Functions and Algebra.

Test N Average Percent Correct Std Dev Min Max VMAST 47 59 17.52 8 95 SOL 47 48 15.28 20 80

slide-9
SLIDE 9

2011 National Conference on Student Assessment Orlando, Florida

Mathematics Item-Level Analyses

  • Change in p-values for supported items
  • 5 items had an increase in p-value

from .20 to .34

  • 21 items had an increase in p-value

from.10 to .19

  • 8 items either had no change in p-value
  • r the increase was less than .10
  • 6 items had a decrease in p-value
slide-10
SLIDE 10

2011 National Conference on Student Assessment Orlando, Florida

Item-Level Analyses: Overall Results

  • P-Value
  • Ranged from

.34 to .90

  • Mean of .65
  • Point biserial
  • Ranged from

.11 to .59

slide-11
SLIDE 11

2011 National Conference on Student Assessment Orlando, Florida

Recap of Mathematics Results

  • Reliability and Validity look good
  • Students have higher percent correct on VMAST

than on SOL

  • No atypical patterns in gender/ethnicity results
  • Nice range in item difficulty and item

discrimination

  • Some supported items became more difficult
  • Consider number of supports added
  • Were supports used by students?
slide-12
SLIDE 12

2011 National Conference on Student Assessment Orlando, Florida

Results of the Reading Field Test

slide-13
SLIDE 13

2011 National Conference on Student Assessment Orlando, Florida

Reading Supports and Simplifications

  • Shortened reading passages
  • Excerpted relevant sections of reading passages

and presented them with item

  • Provided hint boxes with reminder to return to

passage when item dealt with passage as whole

  • Used graphic organizers to present passage

information

  • Simplified language in stems
  • Highlighted key words
  • Reduced answer options from 4 to 3
slide-14
SLIDE 14

2011 National Conference on Student Assessment Orlando, Florida

Overall Comparison of VMAST vs SOL for Grade 8 Reading

VMAST is 36 MC items (vs 45 on the SOL test) measuring:

  • RC1: Use word analysis strategies and information resources; and
  • RC2: Demonstrate comprehension of printed materials.

Test N Average Percent Correct Std Dev Min Max VMAST 47 60 17.93 22 94 SOL 47 44 16.76 11 84

slide-15
SLIDE 15

2011 National Conference on Student Assessment Orlando, Florida

Item-Level Analyses: Overall Results

  • P-Value
  • Ranged from

.29 to .90

  • Mean of .62
  • Point Biserial
  • Ranged from .22 to .57
slide-16
SLIDE 16

2011 National Conference on Student Assessment Orlando, Florida

Reading Item-Level Analyses

  • Impact of supports and simplifications on

reading VMAST items – 10 items had an increase in p-value from .20-.38 – 16 items had an increase in p-value from .10-.19 – 10 items had p-values that were either unchanged or had an increase of less than .10

slide-17
SLIDE 17

2011 National Conference on Student Assessment Orlando, Florida

Recap of Reading Results

  • Reliability and Validity look good
  • Students have higher percent correct on

VMAST than on SOL

  • No atypical patterns in gender/ethnicity results
  • Nice range in item difficulty and item

discrimination

  • Passage type may play a role in effectiveness
  • f the supports and simplifications
slide-18
SLIDE 18

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned: Effective Supports

  • Hint boxes
  • Simplifying language
  • Simplifying presentation of item
  • Drag and drop tool to fill in equations or

expressions

  • Color coding when used judiciously
  • Highlighting
  • Excerpted text
  • Reminder to return to passage when item

addresses passage as a whole

slide-19
SLIDE 19

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned

Supports are most helpful when they are consistently displayed across items

  • Hint boxes always in the same location
  • Components that can be dragged and

dropped identified consistently

  • Same “icon” to activate animation
  • 19 -
slide-20
SLIDE 20

2011 National Conference on Student Assessment Orlando, Florida

Lessons Learned

  • Must be judicious in use of supports:

“oversupporting” can be confusing to students

  • Supports and simplifications not helpful if

students don’t know content

  • Providing practice with items is crucial. Practice

items and guides for teachers may be found at www.doe.virginia.gov/testing/alternative_assessments/

vmast_va_mod_achievement_stds_test/practice_items/ index.shtml

slide-21
SLIDE 21

2011 National Conference on Student Assessment Orlando, Florida

Next Steps

  • Moving toward implementation of VMAST in

reading and mathematics in grades 3-8, Algebra I and end-of-course reading

  • Field tested mathematics in spring 2011
  • First operational administration of

mathematics in spring 2012

  • Field test reading in spring 2012
  • First operational administration of reading in

spring 2013

slide-22
SLIDE 22

2011 National Conference on Student Assessment Orlando, Florida

Application to Next Generation Assessments

  • Continue to study the characteristics of

students to ensure that participation criteria are clear and appropriate

  • Use principles of universal design in item

development

  • Reduce language load
  • Simplify presentation of items
  • Maximize the use of technology to develop

additional supports and simplifications

slide-23
SLIDE 23

2011 National Conference on Student Assessment Orlando, Florida

Contact Information:

Shelley Loving-Ryder

Virginia Department of Education

student_assessment@doe.virginia.gov

(804) 225-2102