NSSE: More Than Your Average Lake Monster John Ryan Director, - - PowerPoint PPT Presentation

NSSE: More Than Your Average Lake Monster

John Ryan Director, Office of Institutional Research The University of Vermont

1/25/13

Presentation Overview

§ An Introduction: NSSE & Student Engagement § NSSE Uses and Select Results at UVM § Questions & Discussion § Contact Information

An Introduction: NSSE and the Concept of Student Engagement

What is Student Engagement?

§ What students do -- time and energy devoted to educationally purposeful activities § What institutions do -- using effective educational practices to induce students to do the right things § Educationally effective institutions channel student energy toward the right activities

Time on task (Tyler, 1930s) Quality of effort (Pace, 1960-70s) Student involvement (Astin, 1984) Social, academic integration (Tinto, 1987, 1993) Good practices in undergraduate education

(Chickering & Gamson, 1987)

College impact (Pascarella, 1985) Student engagement (Kuh, 1991, 2005)

Foundations of Student Engagement

Seven Principles of Good Practice in Undergraduate Education

(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)

ü Student-faculty contact ü Active learning ü Prompt feedback ü Time on task ü High expectations ü Experiences with diversity ü Cooperation among students

How College Affects Students

(Pascarella & Terenzini, 2005)

Because individual effort and

involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student

• engagement. – p. 62

Student Success in College: Creating Conditions that Matter

(Kuh, Kinzie, Schuh, Whitt, and Associates, 2005)

Illustrates actual institutional policies, programs, and practices that promote student success. § Comprehensive report of in-depth study

• f successful educational practices

across institutional types § Practical guidance on how to implement effective practice in a variety of contexts § Co-sponsored by leading higher education groups (CFAT, Pew Forum on Undergraduate Learning)

NSSE Survey Content

Student Behaviors in College Institutional Actions And Requirements Student Reactions to College Student Background Information

Student Learning & Development

NSSE Indicators of Effective Educational Practice

Level of Academic Challenge Active & Collaborative Learning Enriching Educational Experiences Student – Faculty Interaction Supportive Campus Environment

Survey Administration

§ Administered to random samples of first-year & senior students across U.S. § Administration occurs in the spring academic term § Paper & Web-based survey § Flexible to accommodate consortium questions § Multiple follow-ups to increase response rates

How Valid is Self-Reported Data?

Research Confirms Self-Reported Data Validity When Following Conditions Exist…

§ Requested information is known to respondents § Questions are clear and unambiguous § Respondents take questions seriously and thoughtfully § Answering does not threaten, embarrass, or violate privacy or compel a socially desirable response

The NSSE survey instrument was designed to meet these conditions…but there is ongoing debate

For more details, see: NSSE Psychometric Portfolio at http://nsse.iub.edu/html/NSSE_Psychometric_Portfolio.cfm

NSSE Uses and Select Results at UVM

NSSE 2011 Institution Response Rates

§ UVM’s response rate = 33%

§ Slightly more female, ALANA similar compared to UVM student population; weighted for comparison reports; participated 2005, 2008, 2011…2014 next

Average Institutional Response Rates

§ 23% for Carnegie Class comparison institutions § 28% for NSSE 2011 Peer Group institutions § 26% for Writing Consortium institutions

NSSE 2011 Select Results

§ Many strategies for analyzing the results

§ Comparisons with other universities/groups § “Change” over time at UVM (and relative to other schools) § First year v. senior respondents § Differences among colleges within UVM § Compare similar majors across universities § Predictive modeling for outcomes (learning, development, satisfaction, retention, academic performance) § Cohort tracking (2008 first year respondents and as seniors in 2011)

NSSE 2011 UVM Results

§ Thinking about your overall experience at this institution, how would you rate the quality of relationships with faculty and administrative personnel and offices (FY)?

§ What percentage of our first year students would say “quite a bit” or “very much” about the extent to which UVM provides the support they need to thrive socially?

Survey says?

Answer: 55% in 2008 and 48% in 2011

§ Thinking about your overall experience at this institution, to what extent does the college provide the support you need to thrive socially (FY)?

Selected Peer Comparison with UVM Results

§ In thinking about your undergraduate program as a whole, including your major, have you done a culminating senior experience (e.g., senior comprehensive exam, capstone course, thesis or project)?

NSSE 2011 UVM Findings

What percentage of our students (in comparison to selected peers) participated in community service or volunteer work?

Class UVM Carnegie Peers First-Year 29% 39% Seniors 64% 58%

Survey Says?

§ What percentage of our FY and SR students reported typically spending more than 15 hours per week studying/preparing for class?

NSSE 2011 UVM Findings

What percentage of UVM students spent more than 15 hours per week studying/preparing for class?

Class UVM Carnegie Peers First-Year 55% 41% Seniors 42% 42%

Using NSSE at UVM: Student Success and Satisfaction

"SENIORS" 2008 NSSE+Engagement+Factors Impact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+over Academic(Challenge Active(Collaborative(Learning Student(Faculty(Interaction Enriching(Educational(Experiences Supportive(Campus(Environment SAT(Math SAT(Verbal Female Ethnicity( Hispanic Rank Major(Category Business Transfer Campus(residence Mother's(education Father's(education 2008 NSSE+Engagement+Factors "FIRST+YEARS" Impact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+over Academic(Challenge Active(Collaborative(Learning Student(Faculty(Interaction Enriching(Educational(Experiences Supportive(Campus(Environment SAT(Critical(Reading SAT(Critical(Writing SAT(Math Female Ethnicity( Rank Major(Category Professional,+Social+Science,+Other Transfer Campus(residence Mother's(education Attended+college+or+completed+degree Father's(education Attended+but+did+not+complete+college Attended+but+did+not+complete+college

Student Success and Satisfaction

"SENIORS" 2008 UVM.Engagement.Factors Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.over Inclusion)and)Support Active)and)Enriching)Learning Higher8Order)Critical)Thinking Openness)to)New)Ideas SAT)Math SAT)Verbal Female Ethnicity) Hispanic Rank Major)Category Arts.and.Humanities,.Business,.Professional Transfer Campus)residence Mother's)education Attended.but.did.not.complete.college Father's)education Completed.associate.degree Completed.a.bachelor's.degree 2008 UVM.Engagement.Factors "FIRST.YEARS" Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.over Inclusion)and)Support Active)and)Enriching)Learning Higher8Order)Critical)Thinking Openness)to)New)Ideas SAT)Critical)Reading SAT)Critical)Writing SAT)Math Female Ethnicity) Caucasian,.Hispanic Rank Major)Category Business,.Professional Business,.Professional,.Social.Science,.Other Arts.and.Humanities,.Business,.Professional Transfer Campus)residence Mother's)education Attended.college.or.completed.degree Attended.but.did.not.complete.college Father's)education Completed.a.bachelor's.degree

Student Persistence

Likelihood of Persistence Fall 2011-Fall 2012 ì ë Grade-Point Supportive Average Campus Environment

Student Persistence

...and the predictors for the predictors… GPA Supportive Environment

• SAT Score
• Level of Academic
• Active Learning

Challenge

• Female
• Student-Faculty

Interaction

• Not transfer student
• Fewer hrs./wk. working off

campus

Best Practice Benchmarking

Class Mean a S ig b Effect S ize c Mean a S ig b Effect S ize c Mean a S ig b Effect S ize c

First-Year 63.0 .00 61.5 * .07 63.0 .00 Senior 58.9 *

• .10

57.4

• .02

59.1 **

• .11

Mean a

62.9 57.1

a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.

Supportive Campus Environment (SCE)

Mean Comparis

• ns

University of Vermont compared with:

UVM Writing Cons

• rtium

Carnegie Clas s NSSE 2011 Peer Group

• Class

Mean a S ig b Effect S ize c Mean a S ig b Effect S ize c Mean a S ig b Effect S ize c

First-Year 53.8 * .09 53.3 *** .12 54.5 .03 Senior 57.1 .02 56.7 .05 57.5

• .01

Level of Academic Challenge (LAC)

Mean Comparis

• ns

University of Vermont compared with:

UVM Writing Cons

• rtium

Carnegie Clas s NSSE 2011 Peer Group

Mean a

54.9 57.4

a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.

Other examples of NSSE at UVM

§ “Strategic Value” § College Key Performance Indicators § Board of Trustees Dashboard Indicators § College Portrait/VSA § Student learning outcomes (writing, diversity, …) § Accreditation evidence

Examples from Other Universities

§ NSSE results are being used across all sectors and types of institutions. § Discovering and sharing ways student engagement results are being used is one of NSSE’s most important activities. § The following slides illustrate how NSSE data can inform educational policy and practice and provide examples of how specific institutions have used their NSSE results in productive ways.

Areas of Effective Educational Practice Areas for Institutional Improvement

Internal Campus Uses

§ Gauge status of campus priorities § Examine changes in student engagement between first and senior years § Assess campus progress

• ver time

§ Encourage dialogue about good practice § Link with other data to test hypotheses, evaluate programs § Improve curricula, instruction, services

Institutional Improvement

Learning Communities 1ST Year and Senior Experience Academic Affairs Learning Assessment Faculty Development Academic Advising Peer Comparison Student Affairs Institutional Research Enrollment Management

External Campus Uses

§ Assess status vis-à-vis peers, competitors § Identify, develop, market distinctive competencies § Encourage collaboration in consortia (e.g., statewide NSSE conference) § Provide evidence of accountability for good processes (while awaiting improvement in

• utcomes)

Public Accountability

Fund Raising Governing Boards Prospective Students Alumni State Policy Makers Performance Indicators Focus on Right Things Accrediting Bodies Media Parents

Level of Academic Challenge

UNIVERISTY OF MICHIGAN

§ Finding: An item of discussion at a meeting of the Provost’s Faculty Advisory Committee raised the question of students’ writing abilities and how much writing students have to do as U of M undergraduates. § Action: NSSE and FSSE results pertaining to academic challenge, i.e. those that focus on the amount of effort involved in class preparation and how much the environment emphasizes studying and academic work, were prepared and shared with faculty.

Active and Collaborative Learning

WASHINGTON STATE UNIVERSITY

§ Finding: Results indicated the campus was not meeting student expectations for collaborative learning, student-faculty interaction, and enriching educational experiences. § Action: Freshman Focus learning communities were created to provide all incoming freshmen the

• pportunity to engage in an

extensive living-learning community system.

Student – Faculty Interaction

CALIFORNIA STATE UNIVERSITY - FRESNO

§ Finding: NSSE results showed that student-faculty interaction was lower than expected. § Action: Student success task force identified ways to improve student

• success. Participated in Building

Engagement and Attainment for Minority Students (BEAMS) program to develop Mentoring Institute. Now 200+ faculty members, staff and student mentors have been trained.

Enriching Educational Experiences

SIMONS ROCK COLLEGE OF BARD

§ Finding: NSSE results for out-of- class engagement were not as strong as results on pure academic measures. § Action: Student affairs staff members used NSSE results to guide discussions and develop new student activities and community engagement efforts. Have added a new professional position in Student Activities.

Supportive Campus Environment

TOWSON UNIVERSITY

§ Finding: Wished to examine and enhance experience of female students. § Action: Women’s Center staff developed a narrative about the collegiate experience of women

• n their campus, strengthened

programs that offer leadership

• pportunities and self-

empowerment content and process.

Sharing NSSE Results Campus Wide

CLAYTON STATE UNIVERSITY:

Discusses NSSE results at faculty counsel, presidential retreats, student success forums, and in various standing committees.

§ UNIVERSITY OF COLORADO AT COLORADO SPRINGS:

Institutional Research Office publishes a series of one- to two- page research briefs on their NSSE Web site, for faculty and staff members to access. Each brief is dedicated to a specific aspect of NSSE, such as “Diversity” or the “Academic Experience.”

Improving Civic Engagement

JAMES MADISON UNIVERSITY

§ Finding: First-year student involvement in service-learning programs was lower than desired. § Action: Increased the number and quality of class presentations about service- learning; increased the number of contacts with new faculty members regarding the importance of emphasizing service-learning.

Improving General Education

WORCESTER POLYTECHNIC UNIVERSITY

§ Finding: General education courses provide foundation essential to numerous learning

• utcomes. A faculty-appointed

committee defined 5 objectives to enhance core education. § Action: Have begun developing a new first-year general education curriculum featuring interdisciplinary, inquiry-based seminars, better integration of the disciplines, and broader, more engaging introductions to major areas of study.

Questions & Discussion

Contact Information

John F. Ryan, Ph.D. jfryan@uvm.edu 656-4418 UVM Office of Institutional Research (OIR) www.uvm.edu/~isis/

.