Alumni Survey: Three conceptualizations to alumni research Alberto - - PowerPoint PPT Presentation

1

Alumni Survey: Three conceptualizations to alumni research

Alberto F. Cabrera

Professor & WISCAPE Sr. Researcher E-mail: cabrera@education.wisc.edu

David J. Weerts

Project Director& WISCAPE Research Associate E-mail: dweerts@education.wisc.edu

&

Bradford J. Zulick

Doctoral Student & Graduate Research Assistant E-mail: bzulick@wisc.edu

University of Wisconsin- Madison

Métodos de análisis de la inserción laboral de los universitarios Universidad de León 9- 11 de Junio de 2003

2

Topics

1. Approaches to alumni assessment

– Conceptualizations – Policy questions – Methodological considerations

• 2. Common

methodological issues

• 3. Conclusions &

recommendations

3

4

Growth of Alumni Surveys in the United States

Alumni Surveys 1979 – present

70% 15% 15%

Outcome Engagement/Competency Giving

5

• I. Outcomes Approach

1) Oldest Approach – Used since 1937 (Pace, 1979) 2) Assumption – Institutional quality and effectiveness can be appraised by what alumni have accomplished 3) Policy Questions 1) How satisfied are graduates with their employment? 2) How satisfied are the graduates with the degree granting institution? 3) To what extent are graduates fully participating in civic activities?

6

Why stress job outcomes?

Most perspective students rate getting a good job as important in selecting an institution

7

Top 6 reasons when selecting a college Top 6 reasons when selecting a college

(281,064 students at 421 baccalaureate colleges & universities)

• 1. Academic Reputation (56%)
• 2. Graduates getting good jobs (51%)
• 3. Size of the institution (35%)
• 4. Financial Assistance (33%)
• 5. Admission to top graduate schools (30%)
• 6. School’s social activities (28%)

Source: HERI Fall 2001 Survey

8

Outcomes Approach

4) Forces – Accreditation – State Mandates – Social Justice Issues (Shape of the River) 5) Measures – Job Satisfaction – Satisfaction with the Institution – Income – Occupational Status – Participation in civic activities/organizations

9

10

Civic Engagement

11

• II. Engagement & Competencies Approach
• Assumption

– Student engagement in college and attainment of competencies while attending college are the best predictors of alumni success.

• General advice

– Align assessment efforts with what research shows matters in a student’s development, learning and attainment of competencies (e.g. Pascarella, 2002; Kuh 2001; Roberson, Carnes & Vice, 2002).

12

Policy Questions

• 1. What are the competencies (i.e., outcomes,

abilities & values) that college education should foster most?

• 2. To what extent are alumni engaged with

faculty/staff/peers while attending college?

• 3. To what extent colleges and universities engage

students on those learning activities most prone to produce critical competencies?

• 4. To what extent graduates apply those

competencies in the job or in graduate school?

13

Measures of Engagement

• Active &

collaborative learning

• Interactions with

faculty & staff

• Educational

experiences

• Campus climate
• Enriching educational

experiences

• Classroom practices

14

Measures of competencies

ACT College Outcomes Survey

• Gains in problem solving

& group skills

• Critical thinking
• Practical Competence
• Cognitive Complexity
• Knowledge Acquisition &

Application

• Preparation for career

15

The Extent to Which 35 Abilities and Skills Were Enhanced by the Undergraduate Program at Albany (N=1775 Alumni)

Average Response (Scale of 1 to 5) Exercise personal responsibility Maintain openness to new ideas Think analytically & logically Function independently Learn how to learn Exercise self-discipline Acquire new skills & knowledge Adapt to social situations Listen effectively Understand myself Exercise problem solving skills Write effectively Possess clear goals Speak effectively Carry out research Relate to people of different races Know literature of recognized merit 70 68 66 66 64 63 59 50 50 49 47 43 43 43 41 41

Percent responding 4 or 5(high)

• n a 5 point scale

100

8 8 8 9 17 11 10 17 15 17 16 17 18 23 26 27 29 Percent responding 1 or 2(low)

• n a 5 point scale

3.9 3.9 3.8 3.8 3.8 3.8 53 3.7 3.5 3.5 3.4 3.4 3.4 3.3 3.3 3.2 3.2 3.2

20 40 60 80 100 80 60 40 20

Source: Volkwein (2002)

16

The Extent to Which 35 Abilities and Skills Were Enhanced by the Undergraduate Program at Albany (N=1775 Alumni)

Average Response (Scale of 1 to 5) Evaluate and choose among alternatives Understand cultural differences Be effective team member Produce scholarship Analyze literature critically Understand scientific findings Cope with conflict Appreciate artistic expression Use statistics Understand math concepts Take moral & ethical action Employ scientific methods Lead and supervise others Solve math problems Analyze scientific data Use foreign language skills Utilize computing skills 20 40 60 80 100 11 34 27 33 35 36 28 36 37 35 41 37 42 38 44 45 58 61 39 39 38 37 37 37 36 34 34 32 32 31 31 30 30 29 27 19 Place problems in historical perspective 3.2 3.1 3.1 3.0 3.0 3.0 3.1 2.9 2.9 3.0 2.8 2.9 2.8 2.9 2.7 2.7 2.4 2.2

Percent responding 4 or 5(high)

• n a 5 point scale

Percent responding 1 or 2(low)

• n a 5 point scale

20 40 60 80 100

Source: Volkwein (2002)

17

Approaches in identifying competencies

• Focus groups with experts
• Content Validation approach

1. Define a universe of competencies. Do a comprehensive job analyses of those jobs graduates are most likely to hold. Single

• ut those skills, knowledge and abilities each task involves.

Then, identify those competencies that can be taught in college from those that are learned on the job. Pay particular attention to those contexts where those competencies are most likely to be effective. 2. Develop and adopt assessment measures. Make certain that your assessment tool reflects the competencies that enable graduates succeed on the job under a variety of relevant contexts 3. Validate assessment measures with experts. Pilot test the instrument with graduates, job holders, employers and other experts.

18

Competencies & Skills for Business Program

Honesty & trust, ethics & social responsibility Integrity & character Coaching others, developing others, motivating

• thers

Coaching & facilitating Approachability, composure, interpersonal savvy, negotiating, managing peer relationships, conflict management Interpersonal skills Written & oral communication, presentation skills, informing others Communication Collecting information, analyzing root causes, developing alternatives, making effective decisions Problem solving & decision making Planning, organizing, time management, processes management, delegating, directing others Implementation Vision & strategic agility, action oriented, costumer focus, building teamwork & political savvy Leadership influence

Skill group Competency dimension

Source: Roberson, M. T., Carners, L. W. & Vice, J. P. (2002). Defining and measuring student competencies: A content validation approach for Business program outcome assessment. Delta Phi Epsilon Journal, XLIV (1), p. 17.

19

Forces pushing for adoption of engagement & competency approach

• 1. Private funding organizations
• 2. Accreditation
• 3. State regulations

20

Good teaching practices as an alternative to rankings

(Chronicle, October 22, 1999) “Unless we develop measures of quality where

colleges can provide evidence of their contribution to student, then this whole system [of ranking colleges] turns on resources and reputation, and reinforces the elitism of higher education Russell Edgerton. Pew Charitable Fund

21

How regional & professional accreditors are changing

• Middle States Association of Colleges & Schools:

Middle States Association of Colleges & Schools: – Revising standards to focus on student learning and other outcomes such as the quality of student services. – 1996 Middle States: Framework for Outcomes Assessment:

• “The ultimate goal of outcomes assessment is the improvement
• f teaching and learning”
• New England Association of Schools & Colleges:

New England Association of Schools & Colleges: – Developing better ways for colleges to measure student learning.

• North Central Association of Colleges & Schools

North Central Association of Colleges & Schools: – Drafted revisions in eligibility requirements to require evidence of student achievement and institutional effectiveness. Changes must still be voted on by the group. Revisions of standards to begin this fall.

Based on The Chronicle of Higher Education (July 7, 00) A29

22

How regional & professional How regional & professional accreditors accreditors are are changing (continued) changing (continued)

• Southern Association of Colleges & Schools:

– Revising standards to make them less prescriptive and more focused on measuring results, such as student achievement.

• Western Association of Schools & Colleges,

Commission for Community & Junior Colleges

– Revised standards in 2001 similar to North Central’s

• Commission for Senior Colleges & Universities

– Approved revised standards simplifying the compliance process and stress evidence of student learning.

Based on The Chronicle of Higher Education (July 7, 00) A29

23

ABET: Engineering Criteria 2002

Engineering programs must demonstrate that their graduates have:

• Ability to apply knowledge of

mathematics, science & engineering

• Ability to design & conduct

experiments, as well as to analyze and interpret data

• Ability to design a system,

component, or process to meet desired needs

• Ability to function on multi-

disciplinary teams

• Ability to identify, formulate, and

solve engineering problems

• Ability to communicate effectively
• The broad education necessary

to understand the impact of engineering solutions in a global & societal context

• A recognition of the need for,

and ability to engage in life-long learning

• Knowledge of contemporary

issues

• An ability to use the techniques,

skills, and modern engineering tools necessary for engineering practice

24

Use of indicators in performance funding among 11 Use of indicators in performance funding among 11 states. states.

8 44 25 3

15 42 23 16 4

20

5 10 15 20 25 30 35 40 45 50

Inputs & Reputation Process Outputs Outcomes Combined 2-year 4-year

Fuente: Burke & Serban (1998). Performance funding for public higher education: Fad or trend?

25

Methodological concerns

• Can be expensive & time consuming

– Content validation approach can be prohibited for all majors

• Gathering information from alumni

regarding collegiate experiences & current employment experiences & attainments can introduce ‘halo’ errors into results.

26

• III. Alumni Giving Approach

1) Assumption 1) Giving is a function of positive experiences with the institution and ability to contribute 2) Policy Question 1) What is the ability and inclination of alumni to support their alma mater through philanthropy, service, and advocacy?

27

Declining state support is driving alumni assessment towards alumni giving

28

29

Alumni Giving

Increasing reliance on alumni support in the USA

Office of the Chancellor, University of Wisconsin – Madison

30

General advice:

Concentrate efforts in identifying alumni with greatest capacity and willingness to give

31

Concentrate efforts on 10% of the people who contribute 90% of gift dollars

Capacity Rating (10 is highest) 5 5 10 These constitute your primary “prospect list” Require long-term work: need to become engaged Great volunteers and lower level gifts You have better things to do with your time Inclination Rating (10 is highest) 10

32

Measures of Alumni Giving

• Age
• Family income
• Career and educational history
• Current job duties and responsibilities
• Board memberships
• Honors achievements, publications, or

creative works

• Leisure, activities, and hobbies
• Spouse’s career and educational

history, memberships, and activities

• Ages and education of children and

grandchildren

• Quality of Educational Experiences
• Extent to which institution prepared

alumnus for career

• Degree to which faculty members

exerted a positive influence

• Extent to which alumni maintain

contact with faculty and former classmates

• Current impressions with the institution
• History of involvement with the

institution

• Willingness to use influence on behalf of

the alma mater

• Willingness to consider specific

assignments of gifts Capacity Willingness

Based on: Leslie & Ramey, (1988); Brittingham & Pezzulo, (1990); Bodigan & Dehne, (1997); Taylor & Martin, (1995).

33

UW Law School Alumni Giving Survey

34

Common Methodological Issues

• May yield lower

response rates

• Instrument not

accessible to entire population

• Highly efficient

Internet Based

• Low response rate
• Self selecting responses

affect validity of results

• Low Expense
• Highest reliability of

instrument Mail

• Higher cost
• Interviewer bias may

affect validity

• High response rate
• Most likely to be

completes in full Telephone

Cons Pros Type of survey

35

Future directions for alumni surveys

• Avoid capturing in a single survey data from

college experiences and post-college outcomes

– Cause & effect problems – Measurement artifacts (‘Halo effect’ & ‘All Positive or All Negative Answers’).

• Make surveying alumni part of your enrollment

management system.

– Incorporate all major campus units in the conceptualization, design, analyses & application of results. – Approach middle school students as if they were future contributing alumni – Collect information at points in time for critical developmental stages

36

What to What to assess assess & & when when to do to do it? it?

37

A longitudinal approach to assessing alumni success

College Experiences

Engagement Inside & outside The classroom Social Integration Classroom Experiences Climate & Diversity

College Outcomes College Decision

Persistence Transfer Stop-out

Competencies

Satisfaction & Commitment Degree Completion Employment & Income Job Satisfaction Civic Engagement Engagement With alma nater Giving & Advocacy for Alma mater Preparation for College Family Encouragement & Involvement K-16 Communication & Engagement Aspirations & Plans Awareness of College Characteristics, Admission Standards, & Costs

38

What to What to measure measure & & when when to do to do it? it?

39

Graduation/ Post College Collegiate Experiences Preparation for College

Alumni Survey Grad school Degrees earned Occupation/Income Job/Career/ Satisfaction Adequacy of college Preparation Leadership, service Awards, recognition Alumni giving Student survey Advising Faculty & peer interactions Classroom & learning experiences Family & work responsibilities Faculty survey Student survey Orientation & advisement Remediation Faculty & peer interactions Classroom & learning experiences Family & work responsibilities Faculty survey Student Survey Family education Knowledge of careers Family involvement Information about universities Plans & intentions Transfer plans Intended major Inquire process Student Survey Family education Occupation Encouragement Family involvement Saving for college Aspirations Plans & intentions Campus information & transcripts Time to degree Major & changes Transfer Stop-out, return,drop Campus information & transcripts Enrollment information Financial aid data Campus profile Academic performance Campus information & transcripts Enrollment information Financial aid data Campus profile Academic performance School information & transcript Preparation for college School district profile School information & transcript Preparation for college School district profile Outcomes Junior Year Freshman Year High School Application & readiness (11th-12th) Middle School Aspirations & readiness (7th –10th)