BEYOND ITEM REVIEW AND RANGEFINDING: INNOVATIVE WAYS STATES ENGAGE - - PowerPoint PPT Presentation

BEYOND ITEM REVIEW AND RANGEFINDING: INNOVATIVE WAYS STATES ENGAGE EDUCATORS TO DEVELOP AND IMPLEMENT ASSESSMENTS

Charity Flores, Ed.D. Indiana Department of Education Jan Reyes, Ed.D Georgia Department of Education Lisa Sireno Missouri Department of Elementary & Secondary Education Holly Baker, PhD Data Recognition Corporation Corey Palermo, PhD Measurement Incorporated

Introduction

• Welcome
• Background
• Presenters
• Feedback
• socrative.com OR
• download socrative student app
• Select login – student login
• Room name = PALERMO805

Engaging Educators in Assessment Development

June 2019

Lisa Sireno Standards and Assessment Administrator

Why engage Missouri educators?

Show-Me Success

Assessment Implementation Schedule

22

2016-17 2017-18 2018-19 2019-20 ELA & Math English Language Arts & Mathematics Science Science Field Test Science Social Studies Social Studies Personal Finance Personal Finance Assessments Aligned to the Previous Expectations Assessments Aligned to the New Expectations Social Studies Field Test

What did they do?

Building Missouri Assessments

Educators and Missouri Assessments

Educators representing

• ver 250 Missouri school

districts and charter LEAs participated in the development of Missouri’s newest statewide tests.

Recruiting

• Content-area collaboratives and work groups
• Listservs
• Nominations from area supervisors
• Nominations from Missouri educator and

administrator organizations

• Mix of participants
• Competing activities
• Logistics
• Paperwork!

Challenges

• Access to classroom experiences
• Diverse perspectives
• Educator ownership
• Professional learning
• Standards implementation
• Peer review evidence

Benefits

What did we learn?

• Establish the purpose
• Limit jargon
• Be flexible
• Expect the unexpected
• Some tasks are more difficult, but worth the

effort…

Questions? Lisa.Sireno@dese.mo.gov 573-751-3545

IN: Why engage educators?

• Educator contributions to assessment development have

a historical practice in Indiana

• Blueprint Development
• Content and Fairness Review
• Standard Setting
• Recent legislation and assessment changes shifted

increased engagement

• Legislation: scoring of open ended items
• Requires the use of IN educators to the extent possible
• Contributions to promote stakeholder involvement
• Specifications
• Data Review
• Range Finding and Rubric Validation

IN: Process for involving educators

• Recruitment of educators varies on the process
• Nominations from superintendents
• Standard setting
• Volunteers with independent school or district confirmation
• Content development processes
• Volunteers with third-party screening, training and oversight
• Scoring

IN: Challenges

• Communication
• Difficult for IN to communicate directly with educator population
• Timing
• Many efforts overlap with challenging times in school calendars or

summer breaks

• Establishing protocols
• Reliance on criteria and revisiting this after each cycle to continue

to refine the criteria

IN: Benefits

• Professional Development
• Internalizing quality processes
• Development of assessment
• Scoring of items based on a rubric
• Assessment literacy
• Alignment to instructional practices
• Educators are content experts
• Highlight dimensions of standards as they are used in daily

instruction

IN: Lessons learned

• Recruitment
• Be proactive in recruiting—it always takes longer than you think!
• Communicate with local organizations
• Who participated, what they did, how they can apply this to their

daily work

• Be intentional in procurement efforts
• Clearly define where it is expected for educators to be involved so

that the schedule can support these efforts

GA: Why engage educators?

• Georgia has a longstanding history of involving educators

in various aspects of assessment development.

• Educator involvement is not a legislative mandate, but is

considered best practice.

• Assessment Peer Review guidance indicates that test items should

be:

• reviewed by an external committee for alignment to academic content

standards, intended levels of cognitive complexity, intended levels of difficulty, construct-irrelevant variance, and consistency with item specifications; and

• reviewed with field test data to evaluate the quality of items and select

items for operational use.

GA: Process for involving educators

• Nominations are solicited from districts each year.
• System Test Coordinators, Curriculum Coordinators, Special

Education Directors

• All nominees are asked to submit an application with

additional background information.

• Invitees are selected to provide a balanced representation

based on:

• Gender
• Ethnicity
• Region
• Experience
• Special populations
• Educators are involved in the development of all state

assessments, including formative assessments for our youngest learners.

GA: Process for involving educators

Georgia Milestones Georgia Alternate Assessment Georgia Kindergarten Inventory of Developing Skills Keenville (Game-Based Assessment for Grades 1 and 2)

Blueprint Development Assessment Specifications Development Committee Development Committee ALD Development Blueprint Development Task Review Playtesting New Item Review Extended Standards Review Piloting Dashboard Review Rangefinding ALD Development Item Matching Workshops Calibration Study Data Review New Item Review Inter-rater reliability study Standard Setting Data Review Standard Setting

GA: Challenges

• Communication
• District communication to teachers
• Recruitment
• Fewer nominations for some programs
• Declines/cancellations
• Balanced representation
• Timing
• Available meeting space
• Conflicting obligations for teachers and other Department staff
• Compressed field test scoring window for constructed-response
• Differences of opinion from one committee to the next

GA: Benefits

• Buy-in from educators
• “I felt we were contributing to a process that allows teachers to feel assured

that these assessments are valid and fair to all of our students.”

• “It was a great comfort to me to know that so many steps are taken and that

educators have a role in that process.”

• “The people who create our tests really do care about the children and try to

create the most fair and reliable questions for all students.”

• Professional development for educators
• “The process was incredible. It's probably one of the most beneficial

experiences of my teaching career.”

• “It was interesting to collaborate with others from around the state. We don't
• ften get that opportunity.”
• Ensure alignment and expectations
• “It was refreshing to learn how much attention was afforded to ensuring that

the questions met the expectations of student learning in accordance to the GSE.”

• “This was a great way to ensure that the test items are valid.”

GA: Lessons learned

• Frequent communication
• Districts
• Educators
• Be more transparent
• All teachers would benefit from a better understanding
• f the assessment development process

Discussion

• Requirements for participation

○

How does this influence the activities in which teachers are engaged? How has the state selected teacher participation in some but not other aspects of the program?

○

What other opportunities exist within your programs where there could be value in involving teachers?

• Participation as professional development

○

Instructional impact - how do we measure this, is it important? ○ If valuable, how can this be broadly disseminated to encouarge participation?

• Common challenges - how do we overcome them?