and Diagnostic Testing to the High Schools Lessons Learned from - PowerPoint PPT Presentation

Transitioning College Placement and Diagnostic Testing to the High Schools – Lessons Learned from P.E.R.T. Matthew Schultz, PhD McCann Associates

College Placement Testing in Florida • Statutorily required placement testing • Acceptable instruments defined in rule – SAT, ACT, Accuplacer, and PERT • Students who do not meet the scores placed in developmental education • Required developmental education coursework must be completed before students may enroll in entry-level college credit courses 2

From Accuplacer to PERT American Diploma Project

Faculty Involvement • Over 100 faculty from five school districts, two non-public colleges, nine universities and 24 colleges • Identified Florida’s Postsecondary Readiness Competencies • Prioritized the readiness competencies in terms of determining a student’s readiness for college -level coursework • Helped develop the PERT blueprints based on the prioritized list of readiness competencies • Reviewed test items to ensure they are aligned to the competencies and to ensure accuracy and clarity • Took the test “as a student” to ensure the test is functioning as expected 4

Postsecondary Readiness Competencies • PERT is aligned to Florida’s Postsecondary Readiness Competencies (PRCs) – PRCs in reading, writing and mathematics – Prioritized list of competencies for success in initial college credit courses (living document) – True assessment of what students need to know before enrolling in introductory courses – Follow up diagnostics that identify which competencies are not met for the purposes of remediation 5

Blueprint Development • Series of meetings with stake holders from department and colleges • PRC’s for each content area reviewed and prioritized by SME’s • Critical PRC’s identified and weighted • Blueprints developed to reflect most critical PRC’s for success

Test Design Process • PRC’s and Blueprints used to drive content development • College Faculty developed content exemplars • McCann developed item banks of content aligned specifically to PRC’s • College Faculty reviewed all content and approved alignment • DOE, Florida Colleges and McCann collaborated on pilot to test content

Test Development Process • Pilot included all colleges and most campuses • 900+ items administered to samples of student test takers • Online platform used to deliver content • Psychometric review of item performance • Item calibrations (3 parameter IRT) completed and CAT developed • PERT launched operationally Fall 2010 – Includes ongoing field testing of content

Diagnostic Competencies • Diagnostic exams are available for students not meeting the cut-scores – Lower and upper level in reading, writing and mathematics – Based on statewide common course descriptions – Prioritized list of competencies for success in given course – No passing/failing score, just results by competency 9

PERT Components • PERT System Components – Placement Tests (3 sub-tests) • Identifies appropriate course placement (developmental education through college credit) – Reading – Writing – Mathematics – Diagnostic Tests • Lower and upper level for each subject • Based on separate competencies for exiting remedial courses 10

PERT Test Design • 30 questions per placement subtest – 25 operational items will be the basis of the student’s placement score – 5 field test items to continually enhance the operational test bank • Each test item is aligned with a PRC per the test blueprint • All items have been approved and aligned by Florida faculty 11

PERT Test Formats • PERT Placement is delivered using a CAT engine – Draws on item banks of 200+ items per content area – Item banks constantly being expanded • PERT Diagnostics are linear – Content delivered collaboratively between Florida College Faculty and McCann

PERT Interim Cut Score Setting Data • Pilot data • Data reflecting current student enrollment – first course, first year of enrollment • Data reflecting CPT/Accuplacer scores Target • Establish PERT cut scores/ranges that result in course placement comparable to current enrollment numbers – Two ranges for developmental education – lower and upper – One score for entry level college credit – Additional score for MAC 1105 College Algebra 16

Cut Scores and Expected Placement Math Placement Level 3-Year Average Percent of Cut Score Expected Percent at Students Range Placement Advanced Math 0.5% NA - College Algebra or Equiv 6.1% 123-150 7.1% Intermediate Algebra 13.9% 113-122 14.1% Upper Level Dev Ed 40.6% 96-112 41.9% Lower Level Dev Ed 38.9% 50-95 36.9% Reading Placement Level 3-Year Average Percent Cut Score Expected Percent at of Students Range Placement College Ready (ENC 1101) 40.5% 104-150 42.0% Upper Level Dev Ed 46.3% 84-103 45.1% Lower Level Dev Ed 13.2% 50-83 12.9% Writing Placement Level 3-Year Average Percent Cut Score Expected Percent at of Students Range Placement College Ready (ENC 1101) 54.9% 99-150 55.3% Upper Level Dev Ed 20.4% 90-98 21.5% Lower Level Dev Ed 24.7% 50-89 23.2%

PERT in the Colleges Population Math Reading Writing College N 385,978 329,066 325,758 Mean 97.9 103.5 101.7

PERT in the High Schools Population Math Reading Writing High School N 207,687 178,258 163,392 Mean 103.2 97.7 99.9

Test Taker Counts 10,000 6,000 9,000 1,000 2,000 3,000 4,000 5,000 7,000 8,000 0 50 53 56 Sample Distribution 59 62 65 68 PERT Writing Score Distribution 71 74 77 80 83 86 89 92 PERT Writing Scores 95 98 101 104 107 110 113 116 119 122 125 128 131 134 137 140 143 146 149

Contrasting Colleges and High Schools Population Math Reading Writing High School Mean 103.2 97.7 99.9 College Mean 97.9 103.5 101.7

Future Directions with PERT • Standard Setting • Legislation

Contact Information Scott Kramer Matthew Schultz Director Psychometric Services Vice President of Strategic Accounts McCann Associates McCann Associates skramer@mccanntesting.com mschultz@mccanntesting.com