STATE ASSESSMENTS: HOW THEY ARE CREATED OFFICE OF EDUCATIONAL - - PowerPoint PPT Presentation

STATE ASSESSMENTS: HOW THEY ARE CREATED

OFFICE OF EDUCATIONAL ASSESSMENT AND ACCOUNTABILITY (OEAA)

PRESENTERS

Linda Howley (Test Development Manager) Scott Koenig (Social Studies Assessment Consultant) Julie Murphy (K-5 Assessment Specialist) Kyle Ward (Mathematics Assessment Coordinator)

TAKEAWAYS

• Improved understanding about how

M-STEP assessments are developed

• Gain assessment ideas and

structures to take back to your districts to help you develop your local assessments

• Opportunity and/or share

information to apply and be a part

• f our future development

committees

THE LIFE CYCLE OF A M-STEP ITEM

THIS IS THE MICHIGAN ITEM BANK SYSTEM (IBS)

THE LIFE CYCLE

ITEM WRITER

Standard Assignment Item Idea and Options Content Lead Feedback

ITEM VIEW – CONTENT REVIEW

CAC Comments

ITEM VIEW – BIAS REVIEW

BSC Comments

ITEM DATA REVIEW

DATA Committee Comments

Item with Statistics Committee Comments On Statistics

ITEM BANK SYSTEM – ACCEPTED ITEM

ITEM BANK SYSTEM – ACCEPTED ITEM

ITEM WRITING BASICS

ITEMS SHOULD

• assess a content standard and/or benchmark.
• Include accurate content information
• be grade level appropriate including: appropriate cognitive level, reading levels, and thinking skills
• be equitable based on gender, ethnicity, race, social class, disability, geography, etc.
• be fair, clear, concise, and free of cueing
• avoid emotionally charged topics (natural disasters, illness, divorce, loss of jobs) unless standard

and/or benchmark specifies it

• adhere to the MDE approved style guidelines
• contain graphics that are clear, relevant, accurate and necessary
• be reviewed and approved by content review committees and bias/sensitivity committees

before field testing

• require a professional editor to review items for verification before field testing

MULTIPLE CHOICE (MC) AND TECHNOLOGY ENHANCED (TE) ITEMS SHOULD

• propose a single problem, although the

solution may require more than one step

• avoid repeated words from the stem in the

answer choices

• restrict negatives “not” or “none”
• have distractors that are in a logical order,

avoid clues, are comparable in length, complexity, and grammatical form

• make certain a distractor is not a possible

answer

MULTIPLE CHOICE (MC) AND TECHNOLOGY ENHANCED (TE) ITEMS

CONSTRUCTED RESPONSE (CR) ITEMS SHOULD

• clearly state how student is to respond
• allow for a variety of acceptable answers
• allow for enough time and space to respond
• conform to item development specifications

and be scaffolded if necessary

• be measured according to a scoring rubric

that is defensible

CONSTRUCTED RESPONSE (CR) ITEM

CONSTRUCTED RESPONSE (CR) ITEM

Carefully read the passage. Then complete the task that follows. My Trip to the City Aquarium Most of the class was excited to find out we were going on a field trip to an aquarium. I was not. I had never been to an aquarium, so I didn’t know what to expect. But something about the idea of spending the day looking at giant tanks filled with slimy creatures and fish did not excite me. I knew I had to go; plus, my parents had always told me to learn about new things. I agreed with them. I decided I would try to be positive. I liked the idea of learning something new. We boarded the bus and headed downtown, where the aquarium was located. When the bus pulled up to the building, I thought we were just stopping for a minute. Then I saw the sign that read “City Aquarium.” I was amazed at the size of the

• building. It was massive with giant white pillars and wide stone steps. I wondered why they put fish in such a big building. It

looked more like a state capitol building than an aquarium. . . . . . The passage tells about a student taking a trip to an aquarium. Write an essay analyzing how the student changes his or her opinion about the aquarium trip. Use evidence from the passage to support your essay.

CONSTRUCTED RESPONSE (CR) ITEM

RESOURCES: RUBRIC FOR CONSTRUCTED RESPONSE (CR) ITEM

RESOURCES: EXAMPLES OF TEXT DEPENDENT ANALYSIS (TDA) ITEMS

You can find examples of constructed responses on the M-STEP Website under Content Specific Information http://www.michigan.gov/mde/0,4615,7-140-22709_70117-350540--,00.html

COMMITTEE REVIEWS

EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

REVISED EXAMPLE POOR EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

EXAMPLE A EXAMPLE B

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

POOR EXAMPLE REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: POOR EXAMPLE VS. REVISED EXAMPLE

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD

6.SP .A. Develop understanding of statistical variability. 6.RP .B.3 Use ratio and reasoning to solve real-world and mathematical problems.

Answer: 6.RP.B.3 Use ratio and reasoning to solve real-world and mathematical problems.

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD

5U3.1.06 Identify the role that key individuals played in leading the colonists to revolution, including George Washington, Thomas Jefferson, Benjamin Franklin, Patrick Henry, Samuel Adams, John Adams, and Thomas Paine. 4G2.0.01 Describe ways in which the United States can be divided into different regions (e.g., political regions, economic regions, landform regions, vegetation regions). 5U3.1.02 Describe the causes and effects of events such as the Stamp Act, Boston Tea Party, the Intolerable Acts, and the Boston Massacre.

THE COMMITTEE REVIEW GAME: MATCH THE STANDARD

Answer: 5U3.1.06 Identify the role that key

individuals played in leading the colonists to revolution, including George Washington, Thomas Jefferson, Benjamin Franklin, Patrick Henry, Samuel Adams, John Adams, and Thomas Paine.

BLUEPRINTS, TEST MAPS, AND RENDERING

TEST DEVELOPMENT

• Guided by a set of test specifications
• Universal design describes the use of test formats that allow tests to be taken

without adaptation by as broad a range of individuals as possible

• Summative forms in Michigan
• Online fixed form (K-2 Benchmarks in ELA and Math, MI-Access)
• Online Computer Adaptive (M-STEP)
• Paper/pencil (M-STEP, MI-Access)
• Accommodated forms (M-STEP, MI-Access)

TEST DEVELOPMENT

• All items align to Michigan’s content standards
• Item selection follows Psychometric guidelines
• p-values between .2 and .9
• item total correlations > .2
• Items selected must cover a wide variety of questions
• Avoid items rejected by CAC, BSC (should be marked

DNU)

• Items must not be previously been released

TEST BLUEPRINTS

• Original test blueprint approved by MDEs Test Development and

Psychometric unit

• Use original, approved blueprint for valid score reporting and interpretation
• Content Lead creates modified blueprint each testing cycle
• Ensure variety of standards assessed per domain while maintaining points

per domain consistency year-to-year

• Pre-selected anchor (equating) items from prior year
• Blueprint dictates # of operational items and # of field test items
• Length of test vs testing time – big consideration
• Content Lead selects item type (CR, MC, TE) and mode (Online/P-P)

BLUEPRINTS AND FORMS – THE BEGINNING

BLUEPRINT DETAILS

BLUEPRINT LAYOUT

IBS MAGIC

• Once blueprint is completed, Content Lead selects ‘IBS generate’ to build the test

forms containing specific items

• IBS designed to select items following pre-determined formula
• Content Leads review each item on each form to:
• Prevent clueing
• Replace an item that is too similar to another on the form
• Check that answer keys are comparable on each form
• Content Lead has discretion to replace any and all items
• Upon Content Lead approval, test map submitted for Psychometric review and

approval

IBS GENERATES TEST MAP

A TEST FORM

TEST ITEM/FORM REVIEW

• After Psychometricians approve test map, it is exported to DRC
• DRC renders each item according to MDE-approved style guide, in the online test

delivery system and delivers to OEAA

• OEAA Rendering Lead and Content Lead review each item in the way they will

appear to students, comparing each rendered item to the item in IBS

• If changes are needed, the item goes back to DRC for re-rendering
• Once all items in a program, grade, and content are approved, DRC creates forms

in the delivery engine

• Content Lead reviews forms for accuracy, using the IBS test map
• Examples of rendered items are in Sample Item Sets

THE IMPORTANCE OF MICHIGAN EDUCATORS

WHY we need local experts?

• Local Leaders are our

Experts

• Locals assist “buy in” to

the importance of assessment

• Sound practice for

assessment development

• Fair equitable assessments

Assessments developed by Michigan educators for Michigan classrooms.

DISCUSSION TIME

Using your elbow partner:

• 1. à Discuss one topic you were already aware of and one “A-HA” moment.
• 2. à What is one question you have?
• 3. à What is one “take home” you gained?

CONTACT INFORMATION

Office of Educational Assessment and Accountability Phone: 1-877-560-8378 Website: www.michigan.gov/oeaa Email: Mde-oeaa@Michigan.gov