EPIC Evaluation: Measuring Community Change Cynthia Matthias, - - PowerPoint PPT Presentation

EPIC Evaluation:

Measuring Community Change

Cynthia Matthias, Program Evaluator, College of Education and Human Development,

University of Minnesota

Mike Greco, Director, Resilient Communities Project, University of Minnesota

Overview

Ripple Effects Mapping (REM): what it is, why it’s useful in this context Resilient Communities Project (RCP) evaluation findings Using evaluation findings to inform the evolution of RCP

Why did RCP do this?

We wanted to answer some questions about the program:

What project-specific and community-wide changes result from RCP partnerships? What’s working well from partner perspectives? What could we do to improve the partnership experience for communities? Is the program making good on its promises to partner communities?

What goals is the program achieving? Are these the goals we want to achieve?

Evaluation Approach & Rationale

Interrogating our program model using:

• Bottom-up approach: partners tell us what benefits/outcomes they realize
• Reflection: are these the outcomes we want to see?

Developmental evaluation approach—incorporate ongoing feedback into program processes in real time

• Observe, propose, test, repeat
• Not formative or summative evaluation
• What are the standards for “measuring merit, value, and worth”?

Methods: Surfacing Community Outcomes

Method: Ripple Effects Mapping--reflecting on work, collecting data

• Participatory group process with 6-14 people in a room
• Pair-share using a short interview, report back to the group
• Facilitators frantically record, move comments around, organize
• “But for…” principle

Approach: Appreciative Inquiry

• Pair-share interview questions focus on outcomes
• Used seven community capitals as a heuristic device
• Did some projects make leaps forward
• Also asked participants to air grievances

Overall Data Collection Process

REM Sessions (worked with REM expert from UMN Extension)

• One session for each partner community (4 total, 6-8 attendees each)
• Communities chosen based on how much time had passed

Post-Session Follow-Ups

• Emailed interview questions to people who couldn’t attend
• Follow-up interviews to get more depth

Generate a map during the REM session

Note: We used XMind software, but could do low-tech version with Post-Its + flipcharts

Making Sense of Outcomes for RCP

Recontextualizing REM data for RCP program improvement

• Read REM responses, thought about categories of
• utcomes
• Generated (as a team) new themes for organizing outcomes
• Used an iterative process to name outcomes
• Mapped outcomes backward to program activities
• Generated an “outcome map” to visualize how RCP works

“Pros” of the Process

Group discussion surfaces lots of information Structure helps participants think about outcomes systematically Participants connected ideas and events they hadn’t before Sessions produced a tangible reminder of the partnership ⇢ validation of all the work partner communities did Sessions give partner communities a chance to “air grievances” Helped RCP recognize consistent challenges across communities Generated metadata about projects

“Cons” of the Process

It’s hard to get a bunch of busy people in the room at the same time The projects represented are the ones you hear about Time is short, and revelations are incomplete Staff dynamics have a strong impact on success of these sessions

What We Learned

Impacted Municipal Programs, Policies, & Processes

Adopting new policies Undertaking new programs and initiatives Incorporating new design ideas Hiring new staff

Strengthened Relationships

Establishing ongoing relationships between city/county staff and University of Minnesota Strengthening interagency relationships Improving communication with residents and businesses

Changed Staff Attitudes and Perceptions

Working across silos New perspectives and ideas bring new energy and motivation Shifting focus to devote time to neglected issues

Reframed Issues

Rethinking problems and strategies Refocusing community goals Validating existing community initiatives Building momentum for ongoing projects

Created Space to Consider New Ideas

Floating unpopular or “radical” ideas Raising awareness of an issue Creating opportunities to engage with the public

Saved Money/Was Cost-Effective

Monetary savings from process efficiencies and cost-beneficial solutions Laying groundwork to engage professionals more effectively/efficiently Providing professional-level assistance more cheaply (sometimes at the

expense of consultants)

Ultimately, a good return on investment

Challenges

Workload associated with RCP projects Projects that lead to “less useful” results Projects assigned to unwilling/unmotivated staff Staff turnover resulting in lack of continuity/follow-through Information overload

What We Learned

Validated many of the “benefits” claimed for the EPIC model

(. . .and highlighted contingencies that can intervene)

A lot of impactful program work happens before students begin work on projects (the pre-work can make or break a partnership) Many outcomes involve changes in how staff see, think about, and do things Some staff found the demands of the partnership overwhelming Some partners were unable to fully capitalize on the partnership

• nce the formal relationship ended (free kittens vs. free beer)

VS.

Building the Plane While Flying It:

Responding to What We Learned

Potential Program Changes

Scale and length of partnerships (“deep immersion”) Pre-partnership preparation

• Community involvement (listening session? charette? advisory committee?)
• "Resiliency” assessment
• Asset mapping
• Project project lead orientation/training (job descriptions?)

Non-course based assistance

Potential Program Changes

Program efficiencies (survey example) Post-partnership processing

• Distill and aggregate findings
• Focused follow-up work on selected projects
• Prioritize projects for further action (longitudinal implementation plan)

Assessment and Evaluation Changes

• Document feedback from project leads on the heels of presentations

○ Reaction to presentation and key takeaways ○ Next steps ○ Resources to assist (poster, project brief, community presentation…)

• Redesigned and streamlined end-of-semester surveys for students,

faculty, and staff (what, so what, now what)

• Two-year follow-up with partners using REM/interviews
• Two-year retrospective survey to students and faculty to assess outcomes
• Regularly incorporate feedback to inform program operation/strategic plan

Resources

Ripple Effects Mapping Emery, M., Higgins, L., Chazdon, S., & Hansen, D. (2015). Using Ripple Effect Mapping to Evaluate Program Impact: Choosing or Combining the Methods That Work Best for You. Journal of Extension, 53(2), n2. Kollock, D. H., Flage, L., Chazdon, S., Paine, N., & Higgins, L. (2012). Ripple effect mapping: A" radiant" way to capture program impacts. Journal of Extension, 50(5), 1-5. Chazdon, S., Emery, M., Hansen, D., Higgins, L., & Sero, R. (2017). A Field Guide to Ripple Effects Mapping. https://conservancy.umn.edu/handle/11299/190639

EPIC-N Evaluation and Assessment Resources

Elise Amel, Faculty Director, Sustainable Communities Partnership,

University of St. Thomas

Marshall Curry, EPIC-N Program Associate, University of Oregon

. www.rcp.umn.edu @RCPumn rcp@umn.edu www.epicn.org @epicn.org @EPICNtweet

Questions?