Multiple Measures Assessment of Student Readiness for College - - PowerPoint PPT Presentation

Multiple Measures Assessment of Student Readiness for College Courses

Elisabeth Barnett, CCRC and NCREST Jennifer Kim, NCREST

NACEP October 2017

Agenda

• The college-readiness assessment landscape and

the emerging use of multiple measures assessment

• Middle and early colleges’ current assessment

practices

• Discussion and Q&A

Organization of CAPR

CCRC CCRC MDRC

Descriptive Study of Developmental Education Evaluation of The New Mathways Project (RCT in TX) Evaluation of New Assessment Practices (RCT in NY) Supplemental Studies

Research on an Alternative Placement Strategy using Multiple Measures

• 5 year study; seven colleges; 10,000 students
• Alternative placement system
• Random assignment: half of students placed using

algorithm

• Analysis of student outcomes and implementation.

4

Slides available at: bit.ly/capr_ashe16 5

RAPS – Partner Sites

A – CAPR/CCRC/MDRC B – Cayuga CC C – Jefferson CC D – Niagara County CC E – Onondaga CC F – Rockland CC G – Schenectady County CC H – Westchester CC

Predictive analytics used to develop an algorithm

Use data from previous cohorts

Develop formula to predict student performance Use formula to place entering cohort of students

6

Why Use Multiple Measures Assessment?

7

Community college 8-year graduation rates (Attewell,

Lavin, Domina, and Levey, 2006)

28% 43% 0% 10% 20% 30% 40% 50% Students Needing Remediation Students Not Needing Remediation

8

Under-placement and Over-placement (severe)

Placement According to Exam Developmental College Level Student Ability Developmental



Over-placed

(English – 5%) (Math – 6%)

College Level

Under-placed

(English – 29%) (Math – 18%)



9

COLLEGE 2: ENGLISH COLLEGE 2: MATH

10

9.9% 2.7% 12.0% 14.5% 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 GPA only Test only GPA and test Full model 3.8% 1.0% 4.8% 7.5% 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 GPA only Test only GPA and test Full model

Slides available at: bit.ly/capr_ashe16 11

Model R-Squared Statistics English

0.02 0.04 0.06 0.08 0.1 0.12 College 1 College 2 College 3 College 4 College 5 College 6 College 7

R-Squared Statistics – Graphical Representation

GPA ACCUPLACER GPA + ACCUPLACER Full Model

Slides available at: bit.ly/capr_ashe16 12

Model R-Squared Statistics Math

0.05 0.1 0.15 0.2 0.25 College 1 College 2 College3 College 4 College 5 College 6 College 7

R-Squared Statistics – Graphical Representation

GPA ACCUPLACER GPA + ACCUPLACER Full Model

Conclusions so far

• Better assessment systems are needed.
• HS GPA is the best predictor of success in

college math and English.

13

Multiple Measures Assessment

14

Why Use Multiple Measures

• Existing placement tests are not good

predictors of success in college courses.

• More information improves most predictions.
• Different measures may be needed to best

place specific student groups.

15

Current assessment practices in the US

(preliminary; CAPR 2017)

% using….. Math English Standardized tests 88% 87% High school performance 40% 37% Planned course of study 29% 18% Other indicators of motivation 11% 14% No assessment done 6% 6%

16

Possible Measures

Type Examples Placement test Accuplacer ALEKS High school GPA, course grades, test scores From transcript Self-report From SAT, SAT, SB, etc. Non-cognitive assessments GRIT Questionnaire SuccessNavigator or Engage Career inventory, computer skills Kuder Career Assessment Home grown computer skills test Writing examples Faculty-assessed portfolio Home-grown writing assessment Individual advisement

17

Non-cognitive assessments

Development of non-cognitive skills promotes students’ ability to think cogently about information, manage their time, get along with peers and instructors, persist through difficulties, and navigate the landscape

• f college…(Conley, 2010).

Non-cognitive assessments may be of particular value for:

• Nontraditional (older) students.
• Students without a high school record.
• Students close to the cut-off on a test.

18

Multiple Measures Options (Barnett and Reddy, 2017)

MEASURES SYSTEMS OR APPROACHES PLACEMENTS Administered by college: 1. Traditional or alternative placement tests 2. Non-cognitive assessments 3. Computer skills or career inventory 4. Writing assessments 5. Questionnaire items Obtained from elsewhere: 1. High school GPA 2. Other HS transcript information (courses taken, course grades) 3. Standardized test results (e.g., ACT, SAT, Smarter Balanced)

• Waiver system
• Decision bands
• Placement formula

(algorithm)

• Decision rules
• Directed self-placement
• Placement into

traditional courses

• Placement into

alternative coursework

• Placement into

support services

19

But what about….

• Students placed via multiple measures will not be successful.
• Our test is different/better/more awesome.
• High school GPA is only predictive for recent graduates.
• High schools grade differently.
• It’s too hard to get or use transcripts/it’s not worth it.

20

Davidson County, NC, 2013-15 Ivy Tech, IN, 2014-15

Students placed via multiple measures will likely be successful.

21

59% 48% 76% 65%

0% 20% 40% 60% 80% 100%

English Math

Comparison HS Data

Rule s use d fo r E ng lish and Math: HSGPA >=2.6 and c o mple tio n o f fo ur ye ars o f mathe matic s inc luding o ne ye ar b e yo nd Alg e b ra 2 in HS

Rule s use d fo r E ng lish and Math: HSGPA >=2.6 F ROM HE T T S, 2016

57% 59% 57% 64% 68% 64%

0% 20% 40% 60% 80% 100%

English Math Reading

Accuplacer HS Data

NC ENGLISH NC MATH

Our test is different/better/more awesome.

From Bostian (2016), North Carolina Waves GPA Wand, Students Magically College Ready adapted from research of Belfield & Crosta, 2012 – see also Table 1)

HS GPA is a better predictor than test results for long time (Hetts, 2016)

MMAP (in preparation): correlations b/w predictor and success (C or better) in transfer-level course by # of semesters since HS

For the most part, college grades stay parallel with feeder high school grades. (Bostian, 2016)

Sources of HS transcript data Self-report research

• The students bring a

transcript.

• The high school sends.
• Obtained from state data files.
• Self report.

Note: Consider using the 11th grade GPA.

• UC admissions uses self-report but

verifies after admission. In 2008, at 9 campuses, 60,000 students. No campus had >5 discrepancies b/w reported grades and student transcripts (Hetts, 2016)

• College Board: Shawn & Matten,

2009: “Students are quite accurate in reporting their HSGPA”, r = .73.

• ACT research often uses self-reported

GPA and generally find it to highly correlated with students actual GPA: ACT, 2013: r = .84.

25

Examples of alternative assessment systems

• Indiana
• California
• North Carolina
• New York
• Minnesota
• Wisconsin

26

IVY TECH, INDIANA – WAIVERS (CCCSE, 2016)

I vy Tech Community College (I N) has been using a multiple measures

placement policy for degree-seeking students since 2003.

• Students may submit any of the following documents: ACT, SAT, or

PSAT scores for tests taken within the past four years

• The college added the acceptance of high school GPA to the policy for

students entering the college in fall 2014.

• Students who do not have the above documentation are required to

take the college’s custom ACCUPLACER diagnostic assessment.

27

IVY TECH PASS RATES

28

CALIFORNIA

MMAP Project (Hetts, 2016)

• Collaborative effort of CCCCO, Common Assessment Initiative (CAI), Cal-

PASS Plus, RP Group and ~ 45 CCC pilot colleges*

• Identify, analyze, & validate multiple measures data
• For English, Mathematics, ESL and Reading
• Focus on predictive validity (success in course)
• Key variables included HSGPA, last course in discipline, course grade
• r level, AP course-taking, CST scores, etc.

bit.ly/MMAP2015 and http://bit.ly/MMAPRules

29

Example– CA Math Placement

Level Direct Matriculants (from HS) Non-Direct Matriculants Calculus I Passed Precalculus or Trigonometry (or better) 11th-grade GPA ≥ 3.6 11th-grade GPA ≥ 3.2 and Precalculus C (or better) 12th-grade GPA ≥ 3.1 and took Calculus 12th-grade GPA ≥ 3.5 Precalculus Passed Algebra II (or better) 11th-grade GPA ≥ 3.4 11th-grade GPA ≥ 2.6 and took Calculus 12th-grade GPA ≥ 3.3 12th-grade GPA ≥ 3 and Algebra II California Standards Test ≥ 340 12th-grade GPA ≥ 3 and Calculus C (or better) Trigonometry Passed Algebra II (or better) 11th-grade GPA ≥ 3.4 11th-grade GPA ≥ 3 and Precalculus C+ (or better) 11th-grade GPA ≥ 3 and Algebra II B (or better) 12th-grade GPA ≥ 3.3 12th-grade GPA ≥ 2.8 and Precalculus C (or better) College Algebra ( ) 11th-grade GPA ≥ 3.2 11th-grade GPA ≥ 2.9 and Precalculus C (or better) 12th-grade GPA ≥ 3.2 12th-grade GPA ≥ 3.0 and (

30

NORTH CAROLINA (Bostian, 2016)

The North Carolina Community College System has adopted a multiple measure placement hierarchy.

• The 58 colleges began implementing in 2013.
• All colleges required to implement by fall 2016.

CRITERIA 1. Unweighted HS GPA 2.6 + 4 college prep math courses 2. ACT/SAT at national benchmark scores 3. NCDAP placement test

31

32

Relevance for dual/concurrent enrollment

• There may be opportunities to improve access

– Can more students enroll in DE? – Can analyses be done of your students’ high school GPA and DE

performance?

– Can students be prepared to meet new criteria for inclusion?

Good time for a conversation at the college…...

Maps and Tools from Other Settings

34

Middle and Early College Partnerships (NCREST) Outcomes and Multiple Assessments

NCREST

• National Center for

Restructuring Education, Schools and Teaching

• Focused on school

change, mainly in high schools

• Our team – college/high

school partnership programs, especially Early Colleges

36

Related projects:

• MCNC – Middle College National Consortium

schools

• MEMCA – Michigan Early Middle College

Association schools

• MEMPHIS – District Early College initiative
• NEW YORK – Smart Scholars initiative
• BARD – High School Early College model
• SECEP – STEM Early College Expansion

Partnership (I3 Grant: Michigan, Connecticut)

Middle and Early College Design Principles

• Rigorous college-focused academic program
• Comprehensive student support
• Robust high school – college partnerships
• Culture of continuous improvement

37

MCNC Middle and Early College High Schools Data Project 2015-16

38

• 6 – California
• 4 – Michigan
• 4 – New York

14 – Three States

• 1 – Colorado
• 1 – Connecticut
• 1 – Illinois
• 1 – Maryland
• 1 – Tennessee
• 1 – Texas

6 – Six States

MCNC Students Served

39

African American 26% Hispanic 46% Caucasian 15% Asian 7% Other 6% Male 42% Female 58%

69% Free/Reduced Lunch

Criteria used by Middle and Early College High Schools for Readiness for College Course-taking

40

32% 68% 47% 16% 16% 0% 20% 40% 60% 80% 100% High school GPA College placement exams High school counselor, teacher recommendation SAT or ACT High school state exam

College Coursework Outcomes: Access and Success

41

Access to College Courses

42

54% 79% 85% 92% 94% 95% 85% 86% 87% 95% 0% 20% 40% 60% 80% 100% 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 MCNC Graduating Class (12th Graders)

Percent of 12th Graders Who Enrolled in College Courses

Credit Completion in College Courses

43

25.4 30.8 27.0 31.5 31.7 31.8 33.7 33.8 37.6 32.8 0.0 10.0 20.0 30.0 40.0 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 College Course-Taking Students, MCNC Graduating Class

Average College Credits Earned (cumulative)

Success in College Courses

44

87% 86% 87% 89% 90% 90% 0% 20% 40% 60% 80% 100% 2010 2011 2012 2013 2014 2015 MCNC Graduating Class (12th Graders)

Percent of course grades resulting in C grade or higher

(includes A, B, C, D, F grades in calculation)

Percentage of 2014-15 MCNC Students Taking College Courses

45

100% 95% 92% 80% 44% 0% 20% 40% 60% 80% 100% 13th graders 12th graders 11th graders 10th graders 9th graders

College Coursework Summary by 2014-15 Grade Cohort

2015-16 Grade Cohort Number

• f Students

Taking College Courses Average GPA Average Credits Earned Percentage of Courses Passed

(C grade or higher)

9th graders 597 3.08 5.4 94% 10th graders 1,293 3.10 11.9 94% 11th graders 1,297 2.88 21.1 91% 12th graders 1,379 2.90 32.7 90% 13th graders 220 2.68 40.6 84% Total 4,786 2.96 20.9 91%

46

College Course Grade Distribution: ENG & MATH All MCNC 2014-15 Students

Subject Course Enrollments A B C D F W Other

ENGLISH

3777

(87% A-C)

42.3% 30.7% 13.9% 3.7% 6.2% 2.7% .6% MATH

2520

(79% A-C)

34.7% 25.0% 19.0% 6.5% 8.8% 5.4% .6%

47

Assessment Practices 2 Middle and Early College High School – College Partnerships

48

• 1. Washtenaw Community College & Washtenaw

Technical Middle College High School - Assessments

49

• High school course grades (As and Bs at first eligibility point)
• Academic cut scores on Accuplacer
• Soft Skills credential (successful completion of high school 6

modules by end of term; high school teachers must sign off that student has mastery and proficiency in these skills in action

Soft Skills  College Readiness and Success

• Similar to college course, College 101/Prep… but starting 9th grade
• In addition to academic skills, students must learn and practice a

number of "soft" skills needed for transition to and success in college-level work.

– Time management and organization – Goal setting – Decision-making – Conflict resolution – Persistence and seeking help

50

Soft Skills Assessment Grades

• Grade ‘R’ = Recommended for college coursework. This student has

demonstrated effective self-management skills and academic proficiency in this subject area.

• Grade ‘S’ = Soft skill credentialed. This student has demonstrated

effective self-management skills, but has not yet demonstrated academic proficiency in this subject area.

• Grade ‘N’ = Needs improvement. This student needs to improve in
• ne or more soft skill area before being ready to self-manage in a

college class.

51

• 2. Buffalo State University & Buffalo Middle Early

College High School - Assessments

• Interview with entering students – assess whether kids are committed

to full summer and college coursework program

• MATH: Accuplacer (2-year partner), SAT scores (4-year partner)
• ENGLISH: 65 score or better on state high school exam
• Calculated approach on how college classes are introduced  Arts or

Music college course in early years.

• Preparing students from the start of high school  Advisory (FOCUS)

52

9th Grade Advisory (FOCUS) Objectives and Topics

53

9th Grade Advisory (FOCUS) – Parent Engagement

54

School Principal on Early Preparation

• Support support support….The more you can introduce

and talk about ‘college’ to the kids, how possible it is, just starting with freshman class and talking about their grades and being realistic with them about college work. So that later, our kids are mentally prepared.

• Guidance, seminar teachers are all saying the same

messages, and mentioning this multiple times from multiple teachers. Even principal knows and so the kids take it seriously.

55

Assessments for College Readiness vs. Assessment + Early Prep/Support for College Readiness

• How do we help students be successful in high school courses early
• n… to increase college access and success?
• How do we equip students, academically and socially, with the

confidence ‘to do’ and be successful in college?

– Middle Early College High Schools  Comprehensive supports –

academic and social

56

Sample MCNC Schools: Starter and Final Year College Courses

School Course names – Grade 10 Course names – Grade 12

School A

• THEA 1001: INTRO THEATER ARTS
• THEA 1101: INTRO TO ACTING
• CORC 1130: MUSIC LANG HIST & CULT
• SPEC 1103: COMMUNICAT TECHNIQ
• CHEM 1050: GENERAL CHEM 1A
• ENGL 2301: INTRO CREATIVE WRITING
• MATH 1011: PRECALC or MATH 1201: CALCULUS
• PSYC 2810: ABNORMAL PSYCHOLOGY

School B

• CPT101: Intro Comp or MUS105: Music App
• PSY201: Gen Psych or SOC-101: Intro Soc
• SPA101: Elementary Spanish I
• COL03: College Skills
• ENG101: English Composition I
• ENG102: English Composition II
• BIO101: Bio Science I
• MAT110: College Algebra or MAT120: Statistics

School C

• FA 205: DRAWING
• FA 305: PAINTING
• History or Political Science course
• EN 110: COLLEGE COMPOSITION
• EN 114: PUBLIC SPEAKING
• PS 100: GENERAL PSYC or SO 100: INTRO TO SOC
• GS 111: COLLEGE SUCCESS SKILL
• MT 112: SURVEY OF MATH

School D

• COM 115: Public Speaking
• HPR 104: Health Career Opt & Readiness
• CIS 118: Intro PC Applications
• Criminal Justice or Health Careers course
• ENG 121: English Composition I
• ENG 122: English Composition II
• HPR 178: Medical Terminology
• PSY 101: General Psychology I
• ACC 101: Fundamentals of Accounting
• MAT 135: Intro to Statistics

57

MCNC Students: The best part of my middle early college experience has been…

• Being able to take a oral communications class at the college I
• attend. It's what interested me in my current career path and I

would not know that without my dual enrollment experience.

• The best part of my Middle Early College experience would have to

be that my love for learning came back. I am forever grateful that I was given the teachers, counselors and support staff at the school. Coming to an early college gave me hope that I would actually be able to make it in a college environment.

• Not having to worry about a crippling college debt. I know that I will

have one, but because of this school it will be less of a debt. I am also way more prepared for university life thanks to this program.

58

Community College Research Center \ Institute on Education and the Economy \ Teachers College \ Columbia University 525 West 120th Street, Box 174 New York, NY 10027 \ E-mail: ccrc@columbia.edu \ Telephone: 212.678.3091

Contact Us Visit us online:

Elisabeth Barnett: Barnett@tc.columbia.edu Jennifer Kim

jek51@tc.columbia.edu

ccrc.tc.columbia.edu www.mdrc.org To download presentations, reports, and briefs, and sign-up for news announcements. We’re also on Facebook and Twitter.

59