Predictive Analytics: Practical insights into Goals, Means, & - - PDF document

Predictive Analytics: Goals, Means, & Managing Expectations

Tony Scinta Nevada State College

BIG Goals

Practical insights into the development of an analytics platform Key questions to ask:

Of your faculty About your students About your institution About your philosophy

Nevada middle tier of higher education

• circa 1999

Nevada middle tier of higher education

• circa 2016

Watch entire season

• f Breaking Bad in
• ne weekend

100% Commuter Inadequate Academic Preparation First- generation/No n-cognitive challenges Work

• bligations/Poo

r finances

Institutional Challenges

Gateways

to

Completion

Best practices in teaching & learning . . .

Enforcement of prerequisites . . .

Assessment techniques. . . Institutional Policy . . .

Academic support services . . .

Course scheduling . . .

Fail Often . . . Fail Early . . .

1st semester GPA below 2.0

Fail Often . . .

4-Year Rate

Graduate Don't Graduate

100 Percent

Fail Early . . . Fail Often . . .

5-Year Rate

Graduate Don't Graduate

Fail Early . . .

100 Percent

Fail Often . . .

6-Year Rate

Graduate Don't Graduate

Fail Early . . .

100 Percent

Silver Lining

Year-to-Year Retention

Used Advising Used Tutoring 16% Higher 19% Higher

HS GPA < 3.0

Year-to-Year Retention

Used Advising 25% Higher 19% Higher

GOAL: Help students before it is too late MEANS: Early identification of at-risk students Assistance from academic support services

Institutional Challenges

BIG QUESTIONS

What should I ask before proceeding with an analytics effort?

First Question: What is your goal? Second Question: What predicts that goal? Third Question: How can we know if students

are on track to reach the goal? Gender Ethnicity Distance

Transfer GPA HS GPA Credits Taken 1st T erm Academic Year Attended Orientation

Major

Remedial Math

Semester

Cumulative GPA

Academic Level

Credits Passed

1st Generation

Enrollment status Class Add Date

Pell Eligible

Pass Ratio

Instruction Mode

Expected Family Contribution

72

Predictive Model

Predicts the probability that a student will earn a grade

• f C or better in the course

GREAT, RIGHT?

Usability Pitfalls

• 1. Student feedback was inadequate
• SOLUTION: Added new dashboards

Usability Pitfalls

• 1. Student feedback was inadequate
• SOLUTION: Added new dashboards
• 2. Faculty wanted more data and they wanted it to be

more accessible

SOLUTIONS:

• More data on student cards (e.g., time since last log in)
• Emails to faculty
• Ability to view grades as raw points or percentages

Modeling Pitfalls

• 1. Bad predictions

Modeling Pitfalls

• 1. BAD

PREDICTIONS!

Modeling Pitfalls

• 1. Bad predictions
• Anomalous predictions
• Too lenient

Low Risk 76-100% Low Risk 76-100% Low Risk 0-50% Low Risk 0-50% Low Risk 0-50% Low Risk 51-75% Low Risk 51-75% Low Risk 51-75%

Type II Error?

3207 752 356 500 1000 1500 2000 2500 3000 3500 Green Yellow Red

Number of Students

Modeling Pitfalls

• 1. Bad predictions
• Anomalous predictions
• Too lenient
• 2. Faculty vs. Gradebook

Modeling Pitfalls

SOLUTIONS

Low Risk 90-100 Low Risk 90-100 Low Risk 90-100 Low Risk 75-89.9% Low Risk 75-89.9% Low Risk 75-89.9% Low Risk 0-74.9% Low Risk 0-74.9% Low Risk 0-74.9%

Modeling Pitfalls

SOLUTIONS

Characteristics/History Grades

Philosophical Pitfalls

“Non-cognitive” Concerns

No solution yet

Structural Pitfalls

Structural Pitfalls Small Class Size Structural Pitfalls InsufficientAdvisors

Structural Pitfalls

SOLUTIONS

New Comprehensive Dashboard

Take Home Lessons

• 1. Analytics platforms are not easy to do right
• Clarify roles BEFOREHAND
• Modeling is not magic
• Manage expectations BEFOREHAND
• Choose the right parameters

2. One size does not fit all

• Our experience – faculty need it less in small courses
• Advising may be critical
• 3. Even done right, there are concerns
• Belonging/efficacy should be accounted for
• Advising may be critical
• 4. If it works, it is worth the effort