Adrift or Engaged? Exploring a Multi-Engagement Model of the - - PowerPoint PPT Presentation

Adrift or Engaged?

Exploring a Multi-Engagement Model of the Student Experience Using SERU Data

BEAR Seminar – Graduate School of Education – UC Berkeley – March 21 2017 Gregg Thomson (CSHE/Saint Mary’s College) - John Douglass (CSHE)

Center for Studies in Higher Education – Goldman School of Public Policy – UC Berkeley SERU RUA Project Colleagues: Igor Chirikov (CSHE/HSE-Moscow) Ron Huesman (UMN), Tongshan Chang (UCOP), and Steve Brint (UC Riverside)

SERU - Turning to the Student Side of the Research University’s Mission

Teaching and Learning Research/Know ledge Production Public/ Community Service

Seeking a Holistic Understanding of the Student Experience

SERU - Turning to the Student Side of the Research University’s Mission

Teaching and Learning Research/Know ledge Production Public/ Community Service

Seeking a Holistic Understanding of the Student Experience

UG Survey – 2002 Graduate Student Survey – 2014 Census – Online - Customized

SERU-AAU Consortium Members

University of California – Berkeley+ all the UC UG Campuses Rutgers University University of Florida University of Michigan University of Minnesota University of Oregon University of Pittsburgh University of Texas University of Southern California University of North Carolina University of Virginia Texas A&M University Purdue University University of Iowa University of Washington Michigan State University University of Toronto University of Connecticut** University of Delaware

SERU-AAU Consortium Members

University of California – Berkeley+ all the UC UG Campuses Rutgers University University of Florida University of Michigan University of Minnesota University of Oregon University of Pittsburgh University of Texas University of Southern California University of North Carolina University of Virginia Texas A&M University Purdue University University of Iowa University of Washington Michigan State University University of Toronto University of Connecticut** University of Delaware

SERU-I Consortium Members

Universidade Estadual de Campinas (Unicamp) Hunan University Nanjing University Xi’an Jiaotong University Amsterdam University College (AUC) University of Cape Town* University of Bristol* University of Oxford* Higher School of Economics – Moscow Osaka University Tongji University Hiroshima University Lancaster University Utrecht University Lund University Universität Bremen Georg-August-Universität Göttingen Humboldt-Universität zu Berlin University of Freiburg National University of Singapore

SERU-AAU Consortium Members

University of California – Berkeley+ all the UC UG Campuses Rutgers University University of Florida University of Michigan University of Minnesota University of Oregon University of Pittsburgh University of Texas University of Southern California University of North Carolina University of Virginia Texas A&M University Purdue University University of Iowa University of Washington Michigan State University University of Toronto University of Connecticut** University of Delaware

SERU-I Consortium Members

Universidade Estadual de Campinas (Unicamp) Hunan University Nanjing University Xi’an Jiaotong University Amsterdam University College (AUC) University of Cape Town* University of Bristol* University of Oxford* Higher School of Economics – Moscow Osaka University Tongji University Hiroshima University Lancaster University Utrecht University Lund University Universität Bremen Georg-August-Universität Göttingen Humboldt-Universität zu Berlin University of Freiburg National University of Singapore

SERU Undergraduate Survey Instrument

Core Module (36 questions)

Academic and research engagement, time use, learning outcomes, personal development, plans and aspirations, satisfaction and background characteristics

Module 1

(11 questions)

Academic Experience and Globalization Module 2

(23 questions)

Community and Civic Engagement Module 3

(11 questions)

Student Life and Development

Module 4

(12 questions)

Uses of Technology Module 5

(6 questions)

International Students

Campus Wildcard: a set of questions specific to the campus

Uses of SERU Data

Internal

• Catalyst for Campus

Discussion on Strengths and Weaknesses

• Program Review
• Campus

Studies/Departmental Use

• Common Source of Info –

what does SERU tell us?

External

• Campus accreditation
• National Accountability

Regimes – in US VSA

• Media and public

relations

Policy

• Research and Policy

Analysis:

• RUA Project
• International Students
• Research Engagement
• Global Skills . . .
• In Turn:
• Leads to New HEI’s Policy
• Informs the SERU Survey

SERU DATA

SERU - Research University Advantage Project (RUA)

UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:
• the curriculum is not rigorous.
• students do not spend enough time in class and studying.
• students do not learn enough and most are drifting through their academic careers – at all types
• f HE institutions.

“Its official,” stated a one news article, “college student learn next to nothing.”

UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:
• the curriculum is not rigorous.
• students do not spend enough time in class and studying.
• students do not learn enough and most are drifting through their academic careers – at all types
• f HE institutions.

“It’s official,” stated one news article, “college student learn next to nothing.”

Is this True?

UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:
• the curriculum is not rigorous.
• students do not spend enough time in class and studying.
• students do not learn enough and most are drifting through their academic careers – at all types
• f HE institutions.

“It’s official,” stated a one news article, “college student learn next to nothing.”

Is this True?

• Perception (reality?) that Research Universities are not concerned enough about

the quality of Undergraduate Education – a la Boyer Report

UG Education - The Narrow View

• Academically Adrift (2011) induced critique of American higher education:
• the curriculum is not rigorous.
• students do not spend enough time in class and studying.
• students do not learn enough and are drifting through their academic careers – at all types of

HE institutions.

“It’s official,” stated a one news article, “college student learn next to nothing.”

Is this True?

• Perception (reality?) that Research Universities are not concerned enough about

the quality of Undergraduate Education – a la Boyer Report

• The primary [only?] function of higher education is the production of a skilled labor

force and promise of future personal income.

UG Education - Our Broader SERU View

• Undergraduates

at Research Intensive Universities have opportunities for multiple forms of engagement and learning that are perhaps unique in their range and depth.

• The purpose of higher education should be

to produce graduates equipped for both careers and democratic citizenship.

UG Student Experience

What Are Employers Looking For?

Template for Recent Graduates’ Resume from Yale University

Curricular Engagement Study Abroad Research Engagement Extra-Curricular Engagement Civic Engagement

Students and the Multiversity Metaphor

The modern comprehensive university is like a

• city. It has many subcultures, experiences, and

responsibilities that reflect its growing role in society.

UG Student Experience

Students and the Multiversity Metaphor

The modern comprehensive university is like a

• city. It has many subcultures, experiences, and

responsibilities that reflect its growing role in society. “Some get lost in the city,” Clark Kerr once explained, “while

• thers,

both faculty and students, find communities within the larger institution.” The “city,” continued Kerr, “is more like the totality of civilization.”

UG Student Experience

Students and the Multiversity Metaphor

The modern comprehensive university is like a

• city. It has many subcultures, experiences, and

responsibilities that reflect its growing role in society. “Some get lost in the city,” Clark Kerr once explained, “while

• thers,

both faculty and students, find communities within the larger institution.” The “city,” continued Kerr, “is more like the totality of civilization.”

• Are our AAU universities helping students

navigate and take advantage

• f

the

• pportunities

for multiple forms

• f

Engagement and learning?

• Are they innovating in an era of reduce public

investment and often rising student to faculty ratios?

UG Student Experience

Our SERU RUA Conceptual Lens

The Four RUA Spheres of Multi-Engagement

UG Student Experience

Curricular Engagement Research Engagement Public/Community Service Extracurricular Activities

Seeking a Holistic Understanding of the Student Experience

Popular Sense of the Average?

UG Student Experience

Curricular Engagement

Research Engagement

Public/Com munity Service

Extracurricular Activities

Closer to AAU Reality Average?

UG Student Experience

Curricular Engagement Research Engagement Public/Community Service Extracurricular Activities

Four RUA Spheres – Most Students are Multi-Engaged

UG Student Experience

Curricular Engagement Research Engagement Public/Community Service Extracurricular Activities

Most Students are Multi-Engaged – Disciplines Matter

UG Student Experience

Curricular Engagement Research Engagement

Public/Comm unity Service

Extracurricular Activities

The Average for a Third Year Engineering Major?

RUA Research Questions

• Do students appear to be engaged or adrift

when various modes of engagement (curricular, research, extracurricular, and civic) are examined?

UG Student Experience

Curricular Engagement Research Engagement Public/Communit y Service Extracurricular Activities

RUA Research Questions

• Do students appear to be engaged or adrift

when various modes of engagement (curricular, research, extracurricular, and civic) are examined?

• To what extent are various modes of

engagement mutually exclusive, independent, or correlated?

UG Student Experience

Curricular Engagement Research Engagement Public/Communit y Service Extracurricular Activities

RUA Research Questions

• Do students appear to be engaged or adrift

when various modes of engagement (curricular, research, extracurricular, and civic) are examined?

• To what extent are various modes of

engagement mutually exclusive, independent, or correlated?

• How do the patterns of engagement vary by

student background, GPA, level of study, field of study?

UG Student Experience

Curricular Engagement Research Engagement Public/Communit y Service Extracurricular Activities

RUA Research Questions

• Do students appear to be engaged or adrift

when various modes of engagement (curricular, research, extracurricular, and civic) are examined?

• To what extent are various modes of

engagement mutually exclusive, independent, or correlated?

• How do the patterns of engagement vary by

student background, GPA, level of study, field of study?

• How various forms of engagement affect

student learning outcomes, satisfaction, career plans, and job outcomes?

UG Student Experience

Curricular Engagement Research Engagement Public/Communit y Service Extracurricular Activities

RUA Data Sources

SERU/UCUES 2014 Data – Total of 130,125 Responses

UC-Berkeley 9785 University of Minnesota 8332 UC-Davis 10456 Rutgers University 7186 UC-Irvine 10193 University of Pittsburgh 5733 UC-Los Angeles 6807 University of Southern California 1960 UC-Merced 1947 Texas A&M University 11806 UC-San Diego 8064 University of Virginia 4763 UC-Santa Barbara 6822 Indiana University 3205 UC-Santa Cruz 6089 Purdue University 2944 UC-Riverside 6160 University of Iowa 4814 University of Michigan 4930 University of Washington 8129

RUA Data Sources

• SERU/UCUES 2014 Data – Total of 130,125 Responses
• RUA Data source: SERU-2014 Data (UCUES data is not included)
• 11 AAU SERU Campuses with >63,000 Student Responses

UC-Berkeley 9785 University of Minnesota 8332 UC-Davis 10456 Rutgers University 7186 UC-Irvine 10193 University of Pittsburgh 5733 UC-Los Angeles 6807 University of Southern California 1960 UC-Merced 1947 Texas A&M University 11806 UC-San Diego 8064 University of Virginia 4763 UC-Santa Barbara 6822 Indiana University 3205 UC-Santa Cruz 6089 Purdue University 2944 UC-Riverside 6160 University of Iowa 4814 University of Michigan 4930 University of Washington 8129

Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables. How to measure them?

Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables. How to measure them?

Equal weights (Stuart and Thomson, 2013; NSSE) Pros: Easy to interpret, Easy to Calculate Cons: Not all forms of engagement are equal in terms of efforts, time, etc.

Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables. How to measure them?

Weights are determined based on experts opinion Pros: Inequality between forms of engagement is taken into account Cons: How to

• rganize experts

nomination? How to resolve disagreement? Equal weights (Stuart and Thomson, 2013; NSSE) Pros: Easy to interpret, Easy to Calculate Cons: Not all forms of engagement are equal in terms of efforts, time, etc.

Construction of Indices

Curricular, research, civic, and extracurricular engagement are latent variables. How to measure them?

Weights are determined based on Factor Analysis (PCA, CatPCA) Pros: The most accurate method Cons: Harder to interpret, Multiple solutions, Data- sensitive Weights are determined based on experts opinion Pros: Inequality between forms of engagement is taken into account Cons: How to

• rganize experts

nomination? How to resolve disagreement? Equal weights (Stuart and Thomson, 2013; NSSE) Pros: Easy to interpret, Easy to Calculate Cons: Not all forms of engagement are equal in terms of efforts, time, etc.

Construction of Indices

During this year, how often have you- Contributed to a class discussion

• Never
• Rarely
• Occasionally
• Somewhat Often
• Often
• Very often

How frequently have you - Communicated with a faculty member by e-mail or in person:

• Never
• Rarely
• Occasionally
• Somewhat Often
• Often
• Very often

a* + b*

Equal distance? Categorical Principal Component Analysis (CatPCA)

Major Initial SERU RUA Findings

Correlations Between Indices

Curricular Engagement Index Research Engagement Index Civic Engagement Index Extracurricular Engagement Index

Curricular Engagement Index

Pearson Correlation

1 .212 .296 .259

Control variables: Class level & University & Field of study

1 .168 .271 .232

Control variables: Gender & Social class & Race

1 .206 .321 .259

Control variables: GPA

1 .192 .310 .259

• Sig. (2-tailed)

.000 .000 .000

Research Engagement Index

Pearson Correlation

.212 1 .181 .167

Control variables: Class level & University & Field of study

.168 1 .123 .119

Control variables: Gender & Social class & Race

.206 1 .202 .220

Control variables: GPA

.192 1 .183 .214

• Sig. (2-tailed)

.000 .000 .000

Civic Engagement Index

Pearson Correlation

.296 .181 1 .333

Control variables: Class level & University & Field of study

.271 .123 1 .285

Control variables: Gender & Social class & Race

.321 .202 1 .345

Control variables: GPA

.310 .183 1 .334

• Sig. (2-tailed)

.000 .000 .000

Extracurricular Engagement Index

Pearson Correlation

.259 .167 .333 1

Control variables: Class level & University & Field of study

.232 .119 .285 1

Control variables: Gender & Social class & Race

.259 .220 .345 1

Control variables: GPA

.259 .214 .334 1

• Sig. (2-tailed)

.000 .000 .000

0.69%

• f students are completely

disengaged. They belong to the lowest quartile in all 4 areas (curricular, research, civic, extracurricular engagement)

Are Students Engaged or Adrift?

Are Students Engaged or Adrift?

8%

• f students have below

median engagement in all 4 areas (they belong to the lowest 2 quartiles)

Which means according to this initial analysis …

Are Students Engaged or Adrift?

92%

• f students have above

average level of engagement in at least one area (curricular, research, civic or extracurricular)

But a more narrow analysis looks at only the top quartile - those most engaged in one or more of the Four Spheres of Engagement …

Are Students Engaged or Adrift?

67%

• f students have high level of

engagement (upper 25%) in at least one area (curricular, research, civic or extracurricular)

Are Students Engaged or Adrift?

67%

• What are the Patterns of

Engagement?

• How about Learning Outcomes?

Curricular, Research and Extracurricular Engagement – Above Average Curricular Engagement 12% 14% 24% 14% 12% 12% 12% Extracurricular Engagement Research Engagement

Multi-Engagement: Above Average Engagement in Two or More Areas Curricular Engagement Civic Engagement Research Engagement Extracurricular Engagement

73%

• f students at research universities

are multi-engaged

Multi-Engagement: High Engagement in Two or More Areas

Curricular Engagement Civic Engagement Research Engagement Extracurricular Engagement

37%

• f students at research universities

have high (upper 25%) engagement in two or more areas

NEW RESEARCH

• 2012 SERU/UCUES DATA - Includes 9 UG UC Campuses
• Has extensive co-curricular and civic engagement module
• Expanded framework for the four types of engagement
• Defines highly engaged in absolute terms rather than

distribution of scores across quartiles

NEW RESEARCH USING 2012 SERU

Includes UC data Has extensive co-curricular and civic engagement module Expanded framework for the four types of engagement Defines highly engaged in absolute terms rather than distribution of scores across quartiles

CURRICULAR ENGAGEMENT

Curricular Engagement 1 Time spent studying: 26 hours/week or more Curricular Engagement 2 Active engagement with instructor and in classroom: Average of “Often” or higher on five-item scale The two measures overlap very slightly.

RESEARCH ENGAGEMENT

Index of research activity– two or more of the following Research seminar (often) Assist faculty research for course credit Assist faculty research for pay Assist faculty research as volunteer

CIVIC ENGAGEMENT

Two factor analysis of reasons for community service: Belief in the cause, change conditions Opportunity for self-development Civic Engagement 1 and Civic Engagement 2 defined by positive factor score AND either 10 hours/week

• r total of more than 50 hours service

Two measures overlap substantially

C0-CURRICULAR ENGAGEMENT

Co-Curricular Engagement 1 Officer or leader in at least one the following: Student government, Fraternity/sorority, campus political group, advocacy group, service group Co-Curricular Engagement 2 Officer or leader in at least one the following: media, performing, recreational group or sports team OR at least 10 hours/week in student club or organization The two measures have little overlap

Research and Co-Curricular Engagement Research engagement 10% 45% 45% Co-Curricular engagement

Civic and Co-Curricular Engagement Civic engagement 12% 45% 43% Co-Curricular engagement

Curricular and Research Engagement Curricular engagement 16% 31% 53% Research engagement

Curricular and Co-Curricula Engagement Curricular engagement 11% 34% 55% Co-Curricular engagement

Civic and Research Engagement Civic engagement 13% 44% 43% Research engagement

Curricular and Civic Engagement Curricular engagement 13% 32% 55% Civic engagement

HOW MANY STUDENTS ARE HIGHLY ENGAGED?

No Engagement

52

One Form of Engagement

33

Two Forms of Engagement

12

Three Forms of Engagement

3

Four Forms of Engagement

0.3

ENGAGEMENT BY YEAR IN SCHOOL

Frosh Soph Junior Senior Curricular1 11 12 11 9 Curricular2 11 13 15 17 Research

7

10 20 26 Civic1 9 11 14 16 Civic2 6 10 11 12 Co-Curricular1 6 10 12 13 Co-Curricular2

4 7

10 11 No Engagement 63 51 44 39 Any Engagement 37 49 56 61 Multiple Engagement

8

14 21 24

GPA, GENDER, SENSE OF BELONGING, AND SLEEP

GPA Female Belong Sleep <7 hrs Curricular1 3.22 60 74 60 Curricular2 3.28 56 70 52 Research 3.29 59 70 50 Civic1 3.22 71 72 58 Civic2 3.21 71 70 56 Co-Curricular1 3.17 55 76 50 Co-Curricular2 3.22

52

71 48 No Engagement 3.10 59 59 47 Any Engagement 3.22 60 70 52 Multiple Engagement 3.29 60 73 55

ENGAGEMENT BY FIELD OF STUDY (%)

Curr1 Curr2 Reas Civic1 Civic2 Co-Curr1 Co-Curr2 Arts 10

23

9 8 5 8

12

English 7

25

8 11 7 9

10

Psychology 8 12

26 15

12 9 7 Economics 9 12 10 8 6

14

8 Political Scie 7

22

12 14 8

18

8 Sociology 8 16 14

17

11 10 7 Biology 13 9

26 16 16

7 7 Business 7 16 10 11 8

13

8 Phys Science

14

13

26

6 6 6 6 Engineering

17

10 16 7 4 10 8 Mathematics 12 11 11 8 6 9

10

Top-5 Areas with Most Significant Self-Assessed Learning Gains for Each Area of Engagement

Curricular Engagement Research Engagement Civic Engagement Extra-Curricular Engagement

Ability to prepare and make a presentation Research skills Ability to appreciate cultural and global diversity Leadership skills Research skills Library research skills Analytical and critical thinking skills Understanding your field of study Analytical and critical thinking skills Understanding your field

• f study

Research skills Ability to understand international perspectives Understanding your field

• f study

Ability to prepare and make a presentation Ability to appreciate the fine arts Ability to appreciate and understand racial and ethnic diversity Library research skills Ability to read and comprehend academic material Ability to be clear and effective when writing Interpersonal (social) skills

Upper-division students, self-reported learning gains, 2 or more steps increase

Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Curricular Engagement ! Curricular Engagement 2 Curricular Engagement 2 (continued) Research Engagement

Ability to prepare and make a presentation Ability to be clear and effective when writing Research skills Quantitative skills Analytical and critical thinking skills Ability to prepare and make a presentation Research skills Ability to prepare and make a presentation Ability to read and comprehend academic material Ability to read and comprehend academic material Research skills Leadership skills Understanding your field of study Understanding your field of study Ability to read and comprehend academic material Understanding your field of study

Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Civic Engagement 1 Civic Engagement 1 (continued) Civic Engagement 2

Leadership skills Leadership skills Research skills Research skills Ability to prepare and make a presentation Interpersonal (social) skills Ability to prepare and make a presentation Ability to appreciate and understand racial and ethnic diversity Self-awareness and understanding Understanding the importance of personal social responsibility Understanding the importance of personal social responsibility Ability to appreciate and understand racial and ethnic diversity

Areas Where Self-Assessed Learning Gains Significantly Exceeded Gains for Non-Engaged – Upper-Division Students

Co-Curricular Engagement 1 Co-Curricular Engagement 2

Leadership skills Leadership skills Interpersonal (social) skills Self-awareness and understanding Ability to prepare and make a presentation Self-awareness and understanding

Impact of Changing Just One Criterion for Engaged

26 per week or more 21 per week or more No Engagement 52 38 One Form of Engagement 33 41 Two Forms of Engagement 12 17 Three Forms of Engagement 3 4 Four Forms of Engagement 0.3 1

Conclusions

• New analysis expands our four forms
• f engagement framework
• Results illustrate the multiple

pathways to engagement at research universities

• The number of students who will be

identified as “engaged” is highly sensitive to the methodology and criteria for engagement used

PLANNED RUA RESEARCH AGENDA

• Explore Further Disciplinary and Subpopulations
• More Fully Explore Engagement with Academic

Performance and Graduation

• Link SERU AAU Data with Job and Income Outcomes
• Seek Selected Comparative Data from Other Non-

Research Intensive Colleges and Universities

• Utilize the SERU RUA Analysis for Diagnostic

Purposes and to Further Improve the UG Experience

We Seek Your Ideas and Input!

Thank you!