Establishing Legitimacy among Project STEP-UP STEM Intervention - - PDF document

SLIDE 1

1

CASEY GEORGE-JACKSON, Ph.D. BLANCA RINCON University of Illinois at Urbana -Champaign ASQ Advancing the STEM Agenda University of Wisconsin -Stout JULY 19-20,2011

Establishing Legitimacy among STEM Intervention Programs: The Need for Evaluation

Project STEP-UP

 STEM Trends In Enrollment & Persistence for

Underrepresented Populations (STEP-UP)

 Examines factors that impact the entrance into,

persistence in, and degree attainment in the STEM fields at large, public, research universities

 By gender  By race/ethnicity  By socioeconomic status  By STEM field

2

STEM Intervention Programs (SIPs)

 Examine the design, implementation, and impact of

STEM intervention programs on underrepresented undergraduate students

 Serve women, students of color, low-income, and first-

generation students

 Sample programs: tutoring, mentoring, financial aid,

research experiences, first-year seminars, living-learning communities

 Questions of Interest:

 Has the program been formally evaluated (internally or

externally)?

 If so, what was the focus and results of the evaluation?

3

Objectives of STEM Intervention Programs

 Recruitment and/or Retention  Encourage and prepare students for graduate

education and/or careers in STEM

 Aid in the transition to college  Increase awareness of STEM majors and careers  Assist in transforming composition of the STEM

workforce

 Create opportunities for access and success  Increase representation and success of select

populations in STEM

4

Evaluation

 The systematic review of a program or policy,

which uses various methodological approaches, to determine its merit, quality, worth, or value.

 Evaluations are used to

 Respond to the need for accountability, including efforts

“to improve and better programs and society” (Alkin and Christie, 2004, p. 12).

 Inform decisions and changes  Determine best practices  Apply for funding  Determine and demonstrate merit or Worth

5

Legitimacy Theory

 Found in Organizational Theory  Describes how an organization (SIP) gains acceptance

due to their relationship with mainstream norms and values

 SIPs may be influenced to align their missions and goals with certain

values, but also to demonstrate their value

 As an institution becomes legitimate, it sustains the flow

• f resources from the environment to the organization

 SIPs use evaluations to demonstrate the value of their services, gain

support, and secure funding based on the demonstration of desired

• utcomes

 Legitimacy increases resources and support over time;

sustains services to students

6

SLIDE 2

2

Legitimation Process for SIPs

SIP gains Legitimacy & is viewed as legitimate by stakeholders SIP gains access to Resources (finances & human resources) SIP uses Evaluation to demonstrates the value of services and desired outcomes SIP provides recruitment and retention Services to students

7

Data & Methods

 Data

 Collected in 2009-2010  10 large, public, research

universities

 SIP Directors & Administrators  137 invited; 55 interviewed

 Qualitative Analysis

 Semi-structured interviews with

program administrators

 Coded for common themes and

issues

8

Summary of Evaluation Efforts

 Structure of evaluations:

 42% —Formal internal

evaluation

 18% — Informal internal

evaluation

 18% — Formal external

evaluation

 15% — Combination of

internal and external

 6% — No response

 Focus of Evaluations

 Students’ experiences,

including issues of climate

 Student outcomes, including

recruitment and retention in STEM

 Extent SIP’s mission is being

met  Evaluation methods and

techniques

 Pre- and Post- Tests  Focus Groups  Exit Interviews  Observation  Students’ Self-Evaluation  Comparison Groups

9

Internal vs. External Evaluations

 Internal Evaluations

 Conducted own data collection and assessment  Formal (e.g., surveys) and informal (e.g., anecdotal

information) approaches

 Some staff members trained and/or experienced in

conducting evaluations  External Evaluations

 Hired outside evaluators; paid from program’s budget  Partnered with evaluators on campus (e.g., graduate

students in education or evaluation programs)

10

Results: Evaluation As A Requirement

 Meet requirements established by funder (e.g. NSF)  Provide evidence of specific outcomes in order for

funding to be renewed

 “Good stewards of that money”  “College wants to make sure they’re getting their money’s

worth”  Criticisms:

 Funds for evaluation should go towards serving students  Expectations may not be clearly articulated  “Vague pressure to evaluate our programs”

11

Results: Use of Evaluation Results

 Make decisions about and inform changes to SIP

 “For us to figure out what we’re doing right and wrong”

 Develop new programs and services

 “Where our students fit in and where they’re lacking”

 Report to others, including campus diversity offices  Bottom line is the “numbers” (students entering and

succeeding in STEM)

 Share with funders who want to assess the impact of

a particular program and its services

 Aids in establishing legitimacy  Enables program to secure recurring funding & new resources

12

SLIDE 3

3

Results: Evaluation Expertise

 Partner with local experts

 Faculty or graduate students in education departments and/or

evaluation programs  Experts speak the language of evaluation

 Demonstrates level of expertise

 Lack of staff expertise

 Need for training and/or purposeful hiring  “I’m not a statistician. I don’t know how to design a questionnaire.

I don’t know how to do that … I’m a community organizer.”

 Importance of evaluation still recognized

13

Results: Resource Constraints

 A significant roadblock to performing evaluations

 Funding  (Qualified) Staff  Knowledge of Evaluation  Time

 Can result in incomplete evaluation efforts

 Data gathered but not analyzed

 Can lead to difficult decisions

 Using funds to conduct evaluations or provide services

14

Limitations

 Limited generalizability

 10 large, public, research universities  Four-year, doctoral-granting universities  Predominately White Institutions (PWIs)

 Participant recruitment based on publicly available

information of SIPs on each university’s website

 Response rate based on self-selection  Analysis based on opinions and perceptions of

directors and administrators

 May not reflect opinions or perceptions of funders or other

stakeholders

15

Recommendations

 Conduct evaluations and assessments

 To demonstrate and report value and worth of SIP to

• thers

 To secure additional funding and support  To inform decisions, changes, development of new

programs/services

 Online Resources:

 National Center for Women in Information Technology http://www.ncwit.org/resources.assessment.html  Pell Institute Evaluation Toolkit http://toolkit.pellinstitute.org/

16

Recommendations (con’t)

 Seek out strategic partnerships

 Hire graduate student and/or faculty evaluators from

local campus

 Budget evaluation activities in requests for funding  Provide training opportunities to current staff members  Hire new staff members with evaluation experience  Collaborate with similar SIPs or student service programs

to combine evaluation efforts and resources

17

Conclusions

 View evaluations as an important and necessary way

to gain legitimacy, garner support, and secure funding

 Program design improves based on use of evaluation

results, so more students benefit and/or receive services they need

 Legitimized SIPs are better situated and able to serve

students, recruit and retain students in STEM fields, and contribute to students’ educational success

18

SLIDE 4

4

Questions & Discussion

Contact Information Project STEP-UP stem@education.illinois.edu http://stepup.education.illinois.edu/ http://twitter.com/ProjectStepUP Facebook: step-up project

This material is based upon work supported by the National Science Foundation under Grant

• No. 0856309. Any opinions, findings, and conclusions or recommendations expressed in

this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

19