Institutional research and decision support Marin Clarkberg Associate - - PDF document

9/11/2019 1

Institutional research and decision support

Marin Clarkberg Associate Vice Provost for Institutional Research & Planning Cornell University

Institutional Research

• Research about the institution itself
• Institutional researchers collect, analyze, report, and store data

about their institution’s students, faculty, staff, curriculum, course

• fferings, and learning outcomes
• Using data effectively to help make better decisions in universities
• Supports excellence in delivering the mission

1 2

9/11/2019 2

“Institutional Effectiveness” and “Quality Assurance”

• Nearly all U.S. universities have an office of institutional research
• Some other countries have IE or QA and not IR

Institutional Research at Cornell

The mission of Institutional Research & Planning (IRP) is to provide

• fficial, accurate, and unbiased information and analysis about the

university in support of institutional planning, decision‐making, and reporting obligations.

3 4

9/11/2019 3

Institutional Research at Cornell

The mission of Institutional Research & Planning (IRP) is to provide

• fficial, accurate, and unbiased information and analysis about the

university in support of institutional planning, decision‐making, and reporting obligations. IRP supports “data‐informed decision making.”

Support excellence in delivering the mission

“Cornell’s mission is to discover, preserve and disseminate knowledge, to educate the next generation of global citizens, and to promote a culture of broad inquiry throughout and beyond the Cornell community.”

Learning. Discovery. Engagement.

5 6

9/11/2019 4

Learning.

What leads to students’ academic success?

Learning.

What leads to students’ academic difficulties?

• Lack of academic preparation
• Lack of maturity, e.g. time management
• Biting off more than one can chew
• Too many extra curriculars
• Taking too many classes

7 8

9/11/2019 5

The Data: What predicts academic success?

• Tens of thousands of student records
• Outcomes (that is, grades) in thousands of courses
• We know a lot about the students (including things from their

application to Cornell)

• We know some about other things a student is doing
• Where they live
• Some extracurriculars
• Other classes that they are taking

Can we use what we know build data‐driven advising?

Advice to undergraduates

Give yourself the opportunity to do well:

• Don’t exceed 18 (or maybe 21?) credits in a semester
• Don’t take all the hardest courses all at once

That’s old‐fashioned human advice… what do the data say?

9 10

9/11/2019 6

B 12 18 A C D

My theory My theory

This line says that every increase in course load of four credit hours is associated with an increase in GPA of a full letter grade! B 12 18 A C D

11 12

9/11/2019 7

B A C D 12 18 25

Spurious relationships

Observed in the data Reason for the relationship Ice cream sales and death by drowning Both increase in hot summers Shoe size and reading ability in children Older children have bigger feet and read better than younger children Radiation therapy is associated with death Cancer causes both the need for radiation and death Course overloads are associated with better grades Only the best students attempt to take a course overload See also https://www.tylervigen.com/spurious‐correlations

13 14

9/11/2019 8

The problem with being “data‐driven”

• Universities should not operate “like Amazon.” The student

experience is complicated. Life advice is complicated.

• It is difficult to isolate the effect of any given facet of the student

experience (like course load… or a specific program at the university)

• “Correlation is not causation”
• Statistical relationships may overlook qualitative differences. For

example, are credit hours taught really comparable constructs when they are taught in physics or history or art?

Okay, then why use data at all?

• Anecdotal evidence is not enough. Life experiences, pet theories,

and gut feelings can be biased.

• Facts are an important part of building a shared understanding

among multiple parties who don’t always agree.

• Universities hone rhetorical skills, but decisions should not be based
• n force of argumentation, personal reputation or stature, or political

connections.

15 16

9/11/2019 9

Data‐informed decisions take place in a cycle

Commitment to unbiased, impartial inquiry

Questions Assemble data Analysis Insight

Data‐informed decisions take place in a cycle

Commitment to unbiased, impartial inquiry

Questions Assemble data Analysis Insight Communication: Sharing analysis to support insight

17 18

9/11/2019 10

Enrollment from 1868‐present

Only to 1937 shown below… impossible to see this information in a meaningful way in a table

http://irp.dpb.cornell.edu/

19 20

9/11/2019 11

Law School Vet School

21 22

9/11/2019 12

How should data be displayed?

Table Graph Data are expressed as text: words and numbers Data are expressed graphically, as a picture Best choice when the display will be used to look up individual values or the quantitative values must be precise. Works best when the message you wish to communicate resides in the shape of the data: patterns, trends, exceptions Accessible by screen readers Can be difficult for the visually impaired Sometimes, the choice is driven by the “client”…

Sometimes you want specific numbers…

23 24

9/11/2019 13

Maybe both

Numbers and graphs can support each other

Wow?

• Co‐Authorship on

Pharmacology, Toxicology and Pharmaceutics Articles

25 26

9/11/2019 14

Is staff headcount increasing? Headcount has increased… a bit.

27 28

9/11/2019 15

Unnecessary use of a third dimension Misleading use of a third dimension

29 30

9/11/2019 16

This just in.. Good analysis and visualization…

• Clarifies patterns rather than distorts
• Is designed with a purpose, to communicate particular findings
• Facilitates making comparisons of various data elements
• Change over time (such as in a line chart)
• Differences across schools or colleges or programs (maybe stacked bar chart)
• Helps to tell a story
• Guides the viewer to think about patterns in data rather than

thinking about graphic design frills or fancy software

31 32

9/11/2019 17

Data‐informed decisions take place in a cycle

Commitment to unbiased, impartial inquiry

Questions Assemble data Analysis Insight Dialogue, exploration

Importance of impartiality

The mission of Institutional Research & Planning (IRP) is to provide official, accurate, and unbiased information and analysis about the university in support of institutional planning, decision‐making, and reporting obligations.

• Facts are an important part of building a shared understanding

among multiple parties who don’t always agree.

• Universities hone rhetorical skills, but decisions should not be based
• n force of argumentation, personal reputation or stature, or political

connections.

33 34

9/11/2019 18

Decisions are made by people…

… but those decisions should be informed decisions

• Data are not licensed to drive
• Data do not speak for themselves
• Data relationships need to be interrogated; “theory” is important
• Data and analysis must be trusted and nonpartisan
• Data and analysis must be effectively communicated—in a

clear and nonpartisan manner—to be useful

Supporting decision‐making with IR

• Skilled analysts and communicators with a deep

understanding of higher education

• A location in the institution that is separate from the

functional offices that are generating the data and accountable for outcomes

• A commitment to discovery with impartiality

35 36

9/11/2019 19

Thanks!

Marin Clarkberg Associate Vice Provost for Institutional Research & Planning clarkberg@cornell.edu irp.dpb.cornell.edu

37