Considerations for Transitioning Students from Operational 2% - - PowerPoint PPT Presentation

National Center on Educational Outcomes (NCEO)

Considerations for Transitioning Students from Operational 2% Assessments to General Assessments: Challenges for States

Moderator: Martha Thurlow (National Center on Educational Outcomes) Presenters: Sheryl Lazarus (National Center on Educational Outcomes) Wendy Stoica (Ohio Department of Education) Trinell Bowman (Maryland Department of Education) Susan Weigert (U.S. Department of Education, Office of Special Education Programs) Discussant: Marianne Perie (CETE, University of Kansas)

National Conference on Student Assessment

National Harbor, MD June 20, 2013

Lessons Learned Chapter Authors

• Parker, Gorin, and Bechard
• Perie, Fincher, Payne, and Swaffield
• Elliott, Kettler, Zigmond, & Kurz
• Lazarus & Thurlow**
• Elliott, Rodriguez, Roach, Beddow, Kettler, and Kurz
• Cohen, Danielson, Stoica, Wothke, and Zhang**
• Bechard
• Bowman**
• Dean and Roberts
• Loving-Ryder and Siler
• Zigmond, Kloo, Lemons & Lupp
• Nagle and Cameto

National Center on Educational Outcomes (NCEO)

.

• Projects funded by the U.S. Department of

Education in 2006-2007 under three funding sources (General Supervision Enhancement Grants, Enhanced Assessment Grants, and Supplemental Funding) compiled their findings in chapters that discussed the lessons learned from their studies on alternate assessments based on modified achievement standards (AA-MAS).

• This presentation highlights lessons from

three chapters, plus perspectives from OSEP

Lessons Learned in Federally Funded Projects That Can Improve the Instruction and Assessment of Low Performing Students with Disabilities (Thurlow, M., Lazarus, S., & Bechard, S. (Eds.))

Available at: www.nceo.info

National Center on Educational Outcomes (NCEO)

Sheryl Lazarus

.

The Characteristics of Low Performing Students with Disabilities

Sheryl Lazarus National Center on Educational Outcomes (NCEO) Email: laza0019@umn.edu

Demographic Characteristics of Low Performing Students with Disabilities

• 4 projects looked at demographic

characteristics of low performing students. More likely to be: – Male – From a racial/ethnic minority – From a low socio-economic background – Have ELL status

• Both students with and without disabilities

were low performing.

National Center on Educational Outcomes (NCEO)

Disability Categories

• 4 projects looked at the disabilities categories of students

who might be candidates for an AA-MAS.

• All found that a majority of the students had specific

learning disabilities.

• Other categories:
• Intellectual disabilities
• Speech and language impairments
• Other health impairments
• Emotional/behavioral disabilities
• Autism
• Other disability categories

National Center on Educational Outcomes (NCEO)

Opportunity to Learn

4 projects looked at whether students who were candidates for the AA-MAS had the opportunity to learn the content. All found that some students may not have had the opportunity to learn.

National Center on Educational Outcomes (NCEO)

Learning Characteristics/Barriers

10 projects found one or more of these learning characteristics/barriers:

• Difficulty interacting with print
• Difficulty solving problems that require multi-step solutions
• Easily distracted
• High vocabulary load
• Lack of availability of needed accommodations
• Limited meta cognition
• Need clarification of instructions
• Poor organizational skills
• Self-monitoring skills
• Slower work pace
• Text structure (passage length and formatting)
• Working memory capacity

National Center on Educational Outcomes (NCEO)

Wendy Stoica

National Center on Educational Outcomes (NCEO)

AA-MAS Test Development: Item Modifications

Wendy Stoica June 20, 2013

Research Questions

• 1. Do Target and GenEd group students

differ on the cognitive variables?

• 2. Are the cognitive variables accessible via

teacher judgment?

Item Modifications Fall 2008

Modifications of Reading Items Bold/Underlined Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students. Boxed Questions are interspersed within passages and offset with a box. Simplified Language Language is simplified beyond that which is typical for universal design. Thought Questions Additional questions designed to focus students’ attention on particular aspects of a passage are interspersed with the passage. These questions are not answered but serve to help students focus attention. Modifications of Mathematics Items Bold/Underlined (1 level) Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students. Modified Graphics Graphics may have been enlarged or simplified in order to increase readability. Relevant Pictures Information in the stem of the item is given as a picture or table to help students

• rganize and understand the information necessary to answer the question.

Scaffolding Complex items are decomposed into simpler parts. Simplified Language Language is simplified beyond that which is typical for universal design.

Spring 2009

Modifications of Reading Items Bold or Underlined Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students. Primed Items A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item. Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load. Graphic Organizers Graphic organizers provide a graphical structure to help students organize their thoughts. Passage Primed Thought questions are introduced before a reading passage to help the students engage in the content of the passage. Modifications of Mathematics Items Bold or Underlined Important elements of the problem are bolded or underlined. This will facilitate structured recall

• f the content passage by AA-MAS students.

Primed Items A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item. Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load. Relevant Pictures Added icons help students visualize the problem at hand.

Results of interaction of item modifications with cognitive traits for reading

Working Memory Executive Function/ Planning Focused Attention Sustained Attention Bold Increases Influence Decreases Influence Cue Increases Influence Graphic Organizers Decreases Influence Prime Passage Prime

Results of interaction of item modifications with cognitive traits for mathematics

Working Memory Executive Function/ Planning Focused Attention Sustained Attention Bold Increases Influence Increases Influence Cue Decreases Influence Decreases Influence Increases Influence Increases Influence Graphic Organizers Decreases Influence Increases Influence Prime Decreases Influence

More Research Questions

• 1. Do the revised items improve access?
• 2. Does performance exceed chance?
• 3. Do the revisions improve performance and

is the discrepancy reduced between the Target and GenEd group?

• 4. How do the revisions relate to the

student’s cognitive traits?

Conclusions/Recommendations

1. 17% of general education students not identified as having disabilities actually had significant deficits in sustained attention and 11% of the general education students had significant deficits in executive functioning. 2. Students often had deficits only on one, or perhaps two, cognitive traits. 3. Question the ability of classroom teachers to identify those students with disabilities who have specific cognitive processing deficits which has implications for how appropriate data might be collected to inform IEP team decision making. 4. IEP teams require additional training to help them understand that the indiscriminate use of accommodations may actually make tests less accessible for some students with disabilities.

education.ohio.gov

Social Media

@OHEducation

• hio-department-of-education

Ohio Families and Education Ohio Teachers’ Homeroom OhioEdDept storify.com/ohioEdDept

National Center on Educational Outcomes (NCEO)

Trinell Bowman

Maryland’s Alternate Assessment Based

• n Modified Achievement Standards (AA-MAS)

What did we learn?

2013 National Conference on Student Assessment National Harbor, Maryland

June 2013

Trinell Bowman Maryland State Department of Education

22

Lessons Learned from the Development of the AA-MAS

• Cognitive Labs were critical to the development process

and for the first time, students were asked to describe the “why” and “how” as they answered test questions.

• Online testing offers more standardization of certain

accommodations such as the use of the audio test and calculator.

• Online testing may offer additional support to students,

which they normally may not receive when taking a paper and pencil test. Examples include: eliminating a choice, highlighting critical information or marking a test question for review.

23

24

How does the student’s disability affect progress in the general curriculum? The student has difficulty with the following:

Percentage

Comprehending written materials

51.1%

Difficulty following multiple steps

51.1%

Overwhelmed with many sentences/words

• n a page

48.8%

Difficulty completing written tasks

44.1%

Maintaining time on tasks

39.5%

How are methods of instruction different for this student compared to special education students taking the regular assessments?

Percentage

Requires repetition of directions 62.7% Extended time 58.1% Scaffolding questions 55.8% Slower pace of instruction 53.5% Decreased level of content complexity 46.5% Increased wait time 44.2% More visual cues 32.5% Greater use of assistive technology 30.2%

25

Lessons Learned from the Monitoring Process

1. Special educators must be included in professional development training to fully understand state standards. 2. All teachers must understand how to write and implement standards based IEPs. 3. All students must receive appropriate research-based and/or evidenced-based interventions in order to address any gaps or deficit documented in an IEP.

Contact Information

Trinell Bowman

Program Manager Division of Curriculum and Assessment (410) 767-2498

tbowman@msde.state.md.us

27

National Center on Educational Outcomes (NCEO)

Susan Weigert

Perspectives of U.S. ED on Transition challenges and opportunities:

Students currently eligible for an AA-MAS

Transitioning to Improved Assessments

• AA-MAS Flexibility to states in 2007
• Students seen as unable to attain grade level

proficiency

• Era of accountability reflected in NCLB State

percent proficient with add-on flexibilities.

• We can do better

Changes in Accountability

• Permitted States to incorporate student growth
• For students performing below grade level this

changed what was required of an assessment—

• Features we asked for became part of the vision

for next generation assessment.

• Moved from a simply easier test
• To an assessment system more sensitive to lower

levels of performance, and capable of measuring progress toward grade level proficiency.

Improved Design of Test Items

• AA-MAS research identified a key problem in

accessibility of items: non-construct cognitive load.

• Evidence Centered Design has led to improvement in

measurement.

• Item accessibility features that do not undermine

construct validity

• Accommodations that are linked to student access

profiles.

Instructionally-Embedded Assessment

Next Generation Formative assessment Offer better evaluation of student’s problem-solving approaches Feedback to guide the student correctly along the instructional path. Instructors can get a detailed analysis not just of whether the student reached the final answer correctly, but how they solved the problem.

Focus on Continual Progress

Assessments sensitive to student performance range and growth toward standards. Provide students and their teachers with a far better understanding of where they are, where they need to go next Will Provide empirical evidence of how far they can actually go with targeted instruction.

National Center on Educational Outcomes (NCEO)

Marianne Perie