Developing Reading Proficiency Using Accuplacer as a Tool for - - PowerPoint PPT Presentation

Developing Reading Proficiency Using Accuplacer as a Tool for Research and Assessment

Jody Worley Director of Institutional Research and Assessment Margaret E. Lee Dean of Student Services, Metro Campus Tulsa Community College

League for Innovation in the Community College Innovations Conference 2004 San Francisco, CA March 3, 2004

New statewide requirement

• In 1994: OSRHE required all higher

education institutions to require reading competency for incoming student admission

• The regents allowed for institutional

discretion to implement the requirement

Assessing college reading Assessing college reading competency competency

• Can our students read?
• How do we know?

Prevailing Assumption: Prevailing Assumption:

• Student success does not depend solely on

taking developmental courses. THEREFORE:

• Developmental courses should be optional.

Tulsa Community College’s approach: Tulsa Community College’s approach:

• Mandatory assessment of reading skill at

college entry

• Mandatory advisement of developmental
• ptions for students below college level
• Optional enrollment in developmental

reading courses

Time passed, things changed: Time passed, things changed:

• 1999: new student information system

changed TCC’s enrollment procedure

• Increasing sense of institutional accountability
• Increasing concern for student success

Links Between Assessment of Learning Outcomes, Planning and Budgeting TCC Mission TCC Strategic Vision Assessment Budgeting Planning Accountability Governing Boards Accrediting Agencies

• Entry Level
• General Education
• Discipline/Program Outcomes
• Support Areas
• Student Satisfaction

Linkage between the Institution’s Mission Statement and Use of Assessment Results

TCC’s Strategic Vision Goal Statements: Focused statements for student outcomes assessment efforts. Intended Outcomes / Objectives Statements of what is currently being assessed - descriptions of what we intend for students to . . .

• 1. Know (cognitive)
• 2. Think (affective)
• 3. Do (behavioral)

Means of Assessment What are the criteria for meeting the

• bjectives?

How will we know if the

• bjectives

have been met? Results Document results from the various assessment measures. TCC’s Mission

Entry Level Assessment Committee’s Goals: Entry Level Assessment Committee’s Goals:

• Effectively apply the OSRHE reading

competency requirement

• Allow maximum student autonomy
• Encourage reading skill development in

ways that improve both student attainment and performance

Guiding Principles Guiding Principles

• Identify and “stick to” your mission.
• State your assumptions a priori
• Listen to the data, but rely on your expertise

and experience.

• Be open and willing to change directions

midstream, or at least consider alternatives.

New Hypothesis: New Hypothesis:

• Student success is directly connected to

taking developmental courses. THEREFORE:

• Developmental courses should be mandatory

and should precede college level work.

Proposal: Proposal:

• Restrict enrollment options for students who

need reading development

• Test restrictions for effectiveness through

performance data

• Evaluate the effectiveness of the

developmental program

Assessment Tool: Assessment Tool:

The College Board’s Accuplacer Computerized Placement Test Successful placement program in mathematics:

• Validation studies over 7 years
• Placement program for:

– College level mathematics – Developmental mathematics

Assessment Tool: Assessment Tool:

The College Board’s Accuplacer CPT Reading Comprehension

• College-Level:

> 80

• Underprepared:

66 – 79

(1 developmental Reading Course)

• Seriously Deficient:

< 66

(2 developmental Reading Courses)

Initial Faculty Recommendations: Initial Faculty Recommendations:

• Restrict enrollment in certain courses to

students with demonstrated college level reading competency.

• Require reading skill development BEFORE

enrollment in college courses

• Require qualifying cut score for Reading II
• Measure improvement in Reading I and II
• Consider mandatory development

The Study The Study

• Select the cohort
• Classify entering students by their developmental

path

• Analyze academic attainment and performance

after 3 years

Preliminary Results: Attainment Preliminary Results: Attainment

• Students who develop

their skills earn more hours than those who need to develop but don’t.

• Development makes

a difference for student persistence.

3555 613 248 N =

College Ready No Development Developed Skills

Total Earned Hours as of Summer 1998

20 18 16 14 12 10 8 6

Preliminary Results: Performance Preliminary Results: Performance

3555 613 248 N =

College Ready No Dev. Developed Skills

Performance (qpts_2/ehrs_2) as of Summer 1998

2.6 2.4 2.2 2.0 1.8 1.6 1.4 1.2

• Students who develop

their skills outperform those who need to develop but don’t.

• Development makes a

difference for student performance.

What we learned: What we learned:

• Development matters!
• Student success is directly related to taking

developmental courses. THEREFORE:

• Development should be mandatory.

Conclusion #1: Conclusion #1:

• Developmental courses improve student success.

THEREFORE:

• Developmental courses should be mandatory.

Recommendation #1: Recommendation #1:

• Block enrollment for reading proficiency as originally

planned in preliminary recommendations.

New Question: New Question:

• Should development be complete before

enrolling in restricted courses?

• Faculty recommendation: YES

Developmental options:

102 146 613 3555 500 1000 1500 2000 2500 3000 3500 4000 D e v . O n l y C

• n

c u r r e n t N

• D

e v . C

• l

l e g e L e v e l

• Group 1: only

developmental courses

• Group 2: developmental

and college level courses

• Group 3: no

developmental courses

• Group 4: entered at

college reading level

Total N = 4416

Assessment Tool: Assessment Tool:

The College Board’s Accuplacer CPT Reading Comprehension

• College-Level:

> 80

• Underprepared:

66 – 79

(1 developmental Reading Course)

• Seriously Deficient:

< 66

(2 developmental Reading Courses)

Results: Attainment Results: Attainment

• Students who developed

skills while taking college level courses earned more hours than students who developed skills prior to taking any college level courses.

3555 613 146 102 N =

C

• l

l e g e R e a d y N

• D

e v e l

• p

m e n t D e v C

• n

c u r r e n t D e v O n l y

Total Earned Hours as of Summer 1998

30 25 20 15 10 5

Results: Performance Results: Performance

Reading development with college level coursework:

• Outperform students

who take only developmental courses.

• Outperform students

who do not develop their skills.

3555 613 146 102 N =

C

• l

l e g e R e a d y N

• D

e v e l

• p

m e n t D e v . C

• n

c u r r e n t D e v . O n l y

Performance as of Summer 1998

2.6 2.4 2.2 2.0 1.8 1.6 1.4 1.2 1.0

New question: New question:

Why do developmental students who take college courses outperform students who only take developmental courses? Hypothesis: The disparity in performance derives from differences in student skill level.

Attainment and Deficiency Level Attainment and Deficiency Level

3555 75 71 60 42 N =

C

• l

l e g e R e a d y S D & r e m c

• n

c u r r e n t U p & r e m c

• n

c u r r e n t S D & r e m i n d e v

• n

l y U p & r e m . d e v

• n

l y

Total Hours as of Summer 1998

30 20 10

• Seriously deficient students

attained more hours than underprepared students when they took college level courses with developmental courses.

• Seriously deficient students

who developed skills concurrently also attained more hours than college ready students.

Performance and Deficiency Level Performance and Deficiency Level

• Contrary to our expectations, Level of

deficiency did not affect performance among students who developed their skills.

• As expected, underprepared students
• utperformed students with serious

deficiencies among students did not develop their reading skills.

Conclusion #2: Conclusion #2:

• Differences in student success are NOT attributable

SOLELY to taking developmental courses.

• Enrollment in college level courses contributes to

student success, IF development is taking place.

Recommendation #2: Recommendation #2:

• Permit concurrent enrollment in college level

courses, even for seriously deficient students.

Conclusion #3: Conclusion #3: Recommendation #3: Recommendation #3:

• In addition to taking developmental courses,

student success is affected by other factors we have not yet identified.

• Examine student experience within the

developmental reading program to identify other key factors related to student success.

• Pre- and post-test for reading skill in all

developmental reading courses.

Unanswered question: Unanswered question:

• Development improves student success….

BUT:

• Why do college-ready students still outperform

students who take one developmental course?

Follow-up study: Follow-up study:

How much improvement in reading skill can students expect from participating in developmental reading courses?

Pre- Post-Testing on Placement Tests Pre- Post-Testing on Placement Tests

Assumption: If developmental courses are designed to remove reading deficiencies, post- test scores on placement test should be higher than pre-test scores (hopefully “college-level”).

Reading I Results: 2002-2003 Reading I Results: 2002-2003

N Avg. Pre-test Score Avg. Post-test Score Sig. Nelson-Denny Comprehension 291 33.11 (GE = 9.3) 37.07 (GE = 9.7) YES Nelson-Denny Total 292 64.94 (GE = 9.1) 71.30 (GE = 9.6) YES CPT Reading Comprehension 340 50.69 56.02 YES

Reading II Results: 2002-2003 Reading II Results: 2002-2003

N Avg. Pre-test Score Avg. Post-test Score Sig. Nelson-Denny Comprehension 352 38.65 (GE = 10.1) 44.03 (GE = 11.5) YES Nelson-Denny Total 351 77.19 (GE = 10.1) 85.19 (GE = 11.3) YES CPT Reading Comprehension 479 66.51 65.54 NO

What are these results telling us? What are these results telling us?

• These results suggest that while the raw scores improve
• n the post-test, the deficiencies that were identified on

the pre-test might still be present. However, We know from the previous study that successful completion of developmental courses correlates with success in college-level courses.

Conclusions Conclusions

• Something from the developmental course

experience contributes to college success, but the tests do not measure whatever that is.

• Although the placement tests do not appear

to adequately measure improvement, they are effective for identifying students who need to develop skills and ensuring that they get the help they need.

Still to be explored: Still to be explored:

• Why do college ready students outperform

students who meet development requirements?

• What college level courses do students

take concurrent with developmental courses?

• Why do seriously deficient students attain

more credit hours than underprepared students?

• How can we appropriately place students in
• ur developmental reading program?

Action taken: Action taken:

• Entry level assessment committee shared information

with developmental reading discipline review

• Faculty innovation grant
• Federal grant proposals

Decisions Based on Evidence Decisions Based on Evidence

• Enrollment Practices based on Student

Performance

• Ongoing collaboration between the Office
• f Institutional Research and the Entry-

Level Assessment Committee to study placement in mathematics and writing

Contact Information: Contact Information: Jody Worley

jworley@tulsacc.edu 918 - 595 - 7925

Director of Research & Assessment Tulsa Community College 6111 E. Skelly Drive Tulsa, OK 74135

End of presentation End of presentation