Is Academic Development Ready for Academic Analytics? Using - - PowerPoint PPT Presentation

1

Brad Wuetherick Executive Director, Learning and Teaching

Is Academic Development Ready for Academic Analytics? Using Academic Analytics to Improve Teaching and Learning

1

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Tansi … Tawow Kwe … Pjila’si I would like to acknowledge that we are gathered on the traditional and ancestral territory of the Cherokee and Creek peoples.

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Dalhousie University

Halifax, NS, Canada Medical-Doctoral Research (member of U15) 19,000+ students (~15,000 undergrad) 1000+ faculty 13 faculties -- 180+ degree programs

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

• On a scale of 1-10, where are you:

1 – I know nothing ………………………………………………… 10 – I am an analytics about analytics expert

• On a scale of 1-10, where is your

Centre:

1 – We are not involved ………………………………………… 10 – We lead analytics in analytics

• n my campus

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

A plethora of data about learners + Tools to analyse, cluster, model and predict Deeper personalized information about learners

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

How are Academic Analytics (Learning Analytics) being used?

• Improve administrative data for strategic

enrolment management

• Provide personalized support, inform holistic

advising and early alerts initiatives

• Improve quality of communication between

learners, teachers, and advisors.

• Guide and inform course and program design
• Improve quality and accuracy of student

assessment & program evaluation

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Case Example #1:

John N. Gardner Institute Gateway Courses Initiative

https://www.jngi.org/

Looked at the first year American History gateway course across 32 US universities The average D/F/W/I rate for students was 25.5%

• A. Koch, "Many Thousands Failed"

https://www.historians.org/publications-and-directories/perspectives-on-history/may-2017/many- thousands-failed-a-wakeup-call-to-history-educators

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

When institutional demographic data is added, does the conversation change?

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Faculty Perceptions of Academic Analytics

• several studies report faculty skepticism and

uncertainty about using such data to inform changes to teaching, learning, and curriculum practices (Andrade, 2011; Dykoff, 2011; Parry, 2012)

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Faculty Perceptions of Academic Analytics

• 1. Faculty skepticism about the motivations behind

the initiative

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Why Academic Analytics?

• Increased focus on retention and student success

(Campbell, DeBlois, and Oblinger, 2007).

• what motivates institutions?
• Focus on desire for understanding, developing

and sustaining a high quality education to help students towards their individual goals and ambitions

• Focus on practical realities that retention and

student success impacts - rankings, reputation, recruitment, and revenues

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Faculty Perceptions of Academic Analytics

• 1. Faculty skepticism about the motivations behind

the initiative

• 2. Concerns about Ethics and Privacy

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Ethics and the Privacy of Data

• Knowledge of student risk factors can result

in bias (even if unintentional) from advisors, instructors, etc.

• Profiling can be discriminatory and prejudicial
• Students have a right to keep personal

information private – and a right to be given appropriate notice about the use of their data for institutional purposes

• BUT … Institutions have an ethical

responsibility to act in the best interest of students based on the data they gather

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

‘Balanced’ Approach to Academic Analytics Students’ rights to privacy Institution’s responsibility to act Needs of the Learner Needs of the Institution*

Ethics and Academic Analytics

Vendor Interests*

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Faculty Perceptions of Academic Analytics

• 1. Faculty skepticism about the motivations behind

the initiative

• 2. Concerns about Ethics and Privacy
• 3. Data Literacy

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

What is Data Literacy?

Data literacy is the ability to collect, manage, evaluate, and apply data; in a critical manner.

• Data collection; Data management; Data analysis;

Data visualization; Data policy; Data dissemination; Effective and ethical use of data

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Faculty Perceptions of Academic Analytics

• 1. Faculty skepticism about the motivations behind

the initiative

• 2. Concerns about Ethics and Privacy
• 3. Data Literacy
• 4. Resistance to Change

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics at Dalhousie

Remembering that the vast majority of Dalhousie’s students are successful, we explored:

• Retention patterns: Who is leaving?
• Retention Analysis: Who is at the highest risk of

leaving?

• Retention Analysis: Why do students leave?

What are the common characteristics of students who leave?

• Predictive Modelling: What have we learned to

help us support potentially ‘at-risk’ incoming students?

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model A - Academic

Preparation

• 1. Incoming grades (particularly Math and English)
• 2. Writing Fluency/Organization
• 3. International baccalaureate (IB) and/or advanced

placement (AP) vs traditional high school stream

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model B – Pre-Entry Non-

Academic Factors

• 1. Province/Country of origin
• 2. Rural/Urban
• 3. Gender
• 4. Age
• 5. Socio-Economic Background
• 6. Family Educational Background
• 7. Ethnicity

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model C – Post-Entry

Academic and Non-Academic

• 1. Faculty enrolled
• 2. High risk (DFW) courses
• 3. Low SRI courses
• 4. Not ‘First Choice’ program
• 5. Residence
• 6. Loans/Bursaries
• 7. Awards/Scholarships
• 8. Varsity Athletics

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model D - Academic

Performance/Behaviours

• 1. Attendance
• 2. Early Midterm Grades (need to make this

systematic)

• 3. Fall-term GPA
• 4. Credit Hours Completed
• 5. Switch to part-time in Winter term

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model E – Surveys (NSSE,

CUSC, CSI)

1. High scores on Levels of Academic Challenge (LAC), Supportive Campus Environment (SCE), Active and Collaborative Learning (ACL), and Student-Faculty Interactions (SFI) 2. Campus engagement – positive relationships with other

students; helping other students with academic work; participation in co-curricular; living on or close to campus

3. Academic engagement – tutoring other students; receiving

prompt feedback; belief they were gaining work-related knowledge and skills

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• What Factors Matter: Model F – Learning Activity

Data (Analytics from LMS and other Educational Technologies)

• 1. System-Level Data – course grades, comparative, etc.
• 2. Individual-Level Data – individual assessment

performance, response to individual items, etc.

• 3. Transaction-level Data – number of times logging on,

time on task, ‘click rate’, use of hints/help systems, etc.

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Case Example #2

• Course design consultation in STEM Gateway course –

new to faculty member, historically high D/F/W rate (~30%)

• Been taught as a traditional 3 hr lecture/3 hr lab
• Students without 85% in HS math are more likely to

struggle (3x more likely to receive D/F/W)

• All students previously rated SFI and ACL (on NSSE) lower

than peers (a known risk factor for retention)

• Students on Pell Grants were more likely to report that they

did not believe they are gaining work-related knowledge and skills (a known risk factor for retention)

• Students in past five years who lived in residence were less

likely to receive D/F/W (a protective factor for retention)

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Case Example #2

• 2 months before the term starts, you meet with the faculty

member

• Institutional research office was able to send the following

update: 62% of this year’s cohort has below 85% in HS math 33% of the students are in residence 68% of the students are on Pell Grants

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics – Required Steps

DATA ACCESS

THE POLICY FRAMEWORK AROUND ETHICAL ACCESS AND USE OF ANALYTICS DATA

DATA VERIFICATION

ENSURING APPROPRIATE DATA STANDARDS ARE IN PLACE ACROSS INSTITUTIONAL DATA SETS

DATA INTEGRATION

BRINGING TOGETHER DISPARATE INSTITUTIONAL DATA INTO COMMON DATA SET

DATA ANALYSIS

ANALYZING AND INTERPRETING DATA APPROPRIATELY AND ETHICALLY

DATA SUPPORT

PROVIDING THE RIGHT SUPPORT FRAMEWORK FOR THE EFFECTIVE USE OF DATA

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

• Understanding the data about our students is
• nly useful if we use that information to make

better decisions about how we design learning experiences, support students, and support faculty/instructors/academic leaders

• Who should have access to the data?
• Under what conditions?

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Using Academic Analytics to Improve Teaching and Learning

INPUTS (I) + ENVIRONMENT(E) = STUDENT OUTCOMES (O) (Astin, 1993)

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Improving Student Success

• Cannot rely on the ‘I’, but rather must focus on the ‘E’, to

improve the ‘O’

• Bias (especially unconscious bias), which may come from

profiling students, can have an unintended negative consequence

• The interventions – administrative and educational – that

most benefit students at-risk, are also highly beneficial for ALL students

• Educational experiences (undertaken by programs – at the

curriculum level, or faculty/instructors – at the course level) are more likely to have an impact on student success and retention than other administrative or student life interventions

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

Institutional Research Office Lead for Academic Analytics

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

Institutional Research Office Student Affairs (Registrar, Advising, Student Life, etc) Lead for Analytics

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

Institutional Research Office Student Affairs (Registrar, Advising, Student Life, etc) Centre for Learning and Teaching Lead for Analytics

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Academic Analytics

Institutional Research Office Student Affairs (Registrar, Advising, Student Life, etc) Centre for Learning and Teaching Academic Units Faculty and Instructors Students Lead for Analytics

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Providing the Appropriate Supports

• For academic and administrative leaders
• For faculty
• For advisors and other student affairs

professionals

• For the students themselves

__________________________________

• For educational developers (and other expected

support roles)

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Are the supporters ready to support?

• Are Academic Developers ready to support the

analysis, interpretation, and decision-making that come from academic analytics?

• Barriers?
• Challenges?

June 3, 2014 | presented by Jane Smith

PRESENTATION TITLE

Questions?

Brad Wuetherick Executive Director, Learning and Teaching Centre for Learning and Teaching brad.wuetherick@dal.ca 902-494-6646