What Do We Get From CR and TE Items? Findings from the study The - PowerPoint PPT Presentation

What Do We Get From CR and TE Items? Findings from the study The Utility of Online Mathematics Constructed-Response Items: Maintaining Important Mathematics in State Assessments and Providing Appropriate Access to Students Phoebe C. Winter, Scott W. Wood, Susan M. Lottridge, Thomas B. Hughes, & Teresa E. Walker Presentation at the National Conference on Student Assessment, June 20, 2013

This work was supported by a Grant for Enhanced Assessment Instruments awarded by the U.S. Department of Education to the North Carolina Department of Public Instruction as lead state, award S368A100003. The opinions in this document are those of the authors and do not necessarily reflect those of the North Carolina Department of Public Instruction, the U.S. Department of Education, or the participating project states. For more information, including sample items and the full report, visit http://www.pacificmetrics.com/omap/

Overview • Background • Overall results – Reliability – Information functions • Item set results – TE vs. MC • Information functions – CR vs. MC • Information functions • Student responses

Purpose • To investigate the feasibility and utility of using mathematics assessments consisting solely of constructed-response (CR) and technology-enhanced (TE) items to measure mathematics knowledge and skills.

Design 4 states • Grade 7 and Algebra I • – Subset of CCSS in mathematics Two core test forms per content area • – Cores designed to measure the same KS – Form 1 consisted of CR/TE items – Form 2 consisted of KS-matched MC items – Same number of points for each core form – Additional items added to include all item formats and balance response time across forms Tests delivered online • All items machine scored • Forms randomly assigned within content area •

Grade 7 Forms N Items on T est Core Form Items CR/TE MC T otal Form 1 CR/TE 13 5 18 Form 2 MC 5 16 21

Algebra I Forms N Items on T est Core Form Items CR/TE MC Total Form 1 CR/TE 14 5 19 Form 2 MC 4 17 21

CR/TE Item Types Graphing items allow the student to draw lines or points (or both) on a grid. Drag-and-Drop items allow the student to match images to their correct counterpart or sort and order components of a response. Constrained Response items have an input tool that pops up when the student clicks in the answer box. The student can either click on the tool buttons or use the matching keys on the keyboard to enter the response. Table Entry items have students use the constrained response tool to enter responses in a table. Free Text Entry items allow the student to use the entire keyboard to respond; restrictions on what the student can enter are removed. Mixed items use both constrained response and free text entry response in different parts of the item.

Research Sample: Schools and Students by State Grade 7 Algebra I Schools Students Schools Students State A 14 1,395 12 746 (31%) (24%) (20%) (22%) State B 13 2,026 18 904 (29%) (34%) (31%) (27%) State C 10 1,134 15 604 (22%) (19%) (25%) (18%) State D 8 1,361 14 1,151 (18%) (23%) (24%) (34%) T otal 45 5,916 59 3,405

Test Information Functions, Grade 7 Form1_Info 10 9 8 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Test Information Functions -- Grade 7 Red = MC Form, Blue = CR/TE Form Blue X = Form 1 Red + = Form 2

Comparison of Item Information by Set, Grade 7 Theta Value KS CR/TE –2 –1 0 1 2 Assessed Type 7SP8b TE      7RP2ac TE    7SP6* ET      7RP3* ET    7EE4a ET     7NS3 ET    7G5* ET      7EE4a* ET    7NS3* ET     7G5 ET      7RP3* ET      7NS3 ET     7G5 ET    

Test Information Functions, Algebra I Form1_Info 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Test Information Functions -- Algebra Blue X = Form 1 Red + = Form 2 Red = MC Form, Blue = CR/TE Form

Test Information Functions, Algebra I with Anomalous Item Set Removed Form1_Info 7 6 5 4 3 2 1 0 -3 -2 -1 0 1 2 3 Theta Red = MC Form, Blue = CR/TE Form Test Information Functions -- Algebra (Item 17,19 from Form 2, Item 16 from Form 1 Removed) Blue X = Form 1 Red + = Form 2

Comparison of Item Information by Set, Algebra I Theta Value KS CR/TE –2 –1 0 1 2 Assessed Type SID6 GR   FIF4 GR    AREI1 MX    AREI6 TE    FIF4 DD      FLE5 FT     AAPR1 ET      SID6 FT   ACED ET     AREI1 DD    AAPR1 ET    AREI6 ET      FLE5* FT    

Technology Enhanced Items

Algebra I, Set B, TE KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value 0 pt 31% 1 pt 4% 2 pt 65% b -parameter -0.64 point biserial 0.50

Algebra I, Set B, MC1 KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 2% B 3% C 92% D 2% b -parameter –2.17 point biserial 0.20

Algebra I, Set B, MC 2 KS Statement: Identify graphs given a verbal description of a relationship between quantities. Statistics Aspect Value A 70% B 9% C 6% D 15% b -parameter -0.61 point biserial 0.25

Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Algebra I Set B Item Information Functions -- Algebra -- Item 4 vs. Item 3 & 5 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

Algebra I, Set J, TE KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution. Statistics Aspect Value 0 pt 31% 1 pt 69% b -parameter -0.70 point biserial 0.49

Algebra I, Set J, MC KS Statement: Understand that solving an equation is a logical process that maintains equality while moving toward a solution. Statistics Aspect Value A 49% B 8% C 24% D 19% b -parameter 0.43 point biserial 0.37

Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Algebra I Set J Item Information Functions -- Algebra -- Item 14 vs. Item 18 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

Constructed Response Items

Grade 7, Set B, CR KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx. Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b -parameter 0.39 point biserial 0.57

Grade 7, Set B, MC #1 KS Statement 1: Find a matching value given a proportional relationship and a given value. Statistics Aspect Value A 11% B 4% C 10% D 75% b -parameter –0.89 point biserial 0.37

Grade 7, Set B, MC #2 KS Statement 2: Represent a proportional relation with an equation of type y = kx. Statistics Aspect Value A 83% B 7% C 6% D 5% b -parameter –1.07 point biserial 0.39

Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Grade 7 Set B Item Information Functions -- Grade 7 -- Item 3 vs. Item 2 & 3 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

Grade 7, Set B, CR KS 1: Find a matching value given a proportional relationship and a given value. KS 2: Represent a proportional relation with an equation of type y = kx. Exemplary Response DVDs Cost 10 $160.00 15 $240 20 $320 n 16n Statistics Aspect Value 0 pt 35% 1 pt 48% 2 pt 18% b -parameter 0.39 point biserial 0.57

Additional Information from CR Item Students use proportional reasoning to find the values, but • do not find the unit rate needed for the expression. Students determine the unit rate for the correct expression, • but do not correctly calculate values. Students do not write an accurate algebraic expression. •

Grade 7, Set G, CR KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Exemplary Response: x = 40 Statistics Aspect Value 0 pt 58% 1 pt 42% b -parameter 0.31 point biserial 0.50

Grade 7, Set G, MC KS Statement: Use facts about supplementary, complementary, vertical, and adjacent angles to find an unknown angle in a single step. Statistics Aspect Value A 10% B 13% C 65% D 13% b -parameter 0.41 point biserial 0.28

Information 2 1 0 -3 -2 -1 0 1 2 3 Theta Item Information Functions for Grade 7 Set G Item Information Functions -- Grade 7 -- Item 10 vs. Item 13 Red = MC Form, Blue = CR/TE Form Blue = Form 1 Red = Form 2

Algebra I, Set G, CR KS Statement: Add and subtract polynomials, combining like terms as needed. Statistics Aspect Value Exemplary 0 pt 67% Response: 1 pt 33% 3 x – 2 y b -parameter 0.62 point biserial 0.51

Algebra I, Set G, MC KS Statement: Add and subtract polynomials, combining like terms as needed. Statistics Aspect Value A 16% B 48% C 14% D 22% b -parameter 0.67 point biserial 0.28