Network 17 th July 2019 Agenda Welcome & introductions - - PowerPoint PPT Presentation
Regional Impact Network 17 th July 2019 Agenda Welcome & introductions Embedding evaluation into practice - training Break Embedding evaluation into practice action planning Creative ways to become a learning organisation London
Welcome & introductions Embedding evaluation into practice - training Break Embedding evaluation into practice – action planning Creative ways to become a learning organisation – London Bubble Theatre What’s coming up? What do we want to see in the future? Free space/networking
evaluation into your organisation
Today we’ll cover
Evaluation culture: the ideas, customs and behaviours of your organization and those in it pertaining to why and how you gather and use evaluation data. Evaluation ecosystem: the different evaluation roles, often carried out by different people, that interact with each other (or not) to enable evaluation to happen effectively (or not)
Engagement
http://www.camman-evaluation.com/blog/2015/11/25/better- program-evaluation-through-frontline-staff-engagement
Before starting an evaluation, be able to answer these questions:
Practitioners are often hesitate to take part in an evaluation because they’re worried about high stakes.
What’s at stake?
Planner: Setting evaluative goals and priorities, identifying and selecting appropriate evaluation methodologies. Participant/data contributor: Participating in surveys, interviews, and focus groups; generating program data; facilitating access to other stakeholder groups for the purpose of data collection (e.g., program recipients). User/ analyst: Being an intended user of the evaluation findings, reviewing deliverables and recommendations and making decisions about what steps to take based on this information. Implementer: Carrying out recommendations and any changes in operations or service delivery as a result of evaluation findings.
Common roles in evaluation
http://www.camman-evaluation.com/blog/2015/11/25/better- program-evaluation-through-frontline-staff-engagement
A broken evaluation ecosystem
Planner Participant/ data contributor User/ analyst Implementer
feasible or relevant
not of high quality
informed by good evidence
changes to service delivery
A balanced evaluation ecosystem
culture of continuous learning and improvement
Planner Participant/ data contributor User/ analyst Implementer
Write down…. In your organisation
Your evaluation ecosystem
How do these people talk to each other?
evaluation methodology?
the implementer?
How balanced is your evaluation ecosystem?
culture of continuous learning and improvement
Planner Participant/ data contributor User/ analyst Implementer
Are there communication links in your evaluation ecosystem that could be improved? Which links require improvement? Are there individuals who feel like they contribute more than they gain from evaluation? Who are they? Who feels like they gain the most from evaluation?
Description: What happened? Feelings: What were you thinking and feeling? Evaluation: What was good and bad about the experience? Analysis: What sense can you make
situation? Conclusion: What else could have you done? Action Plan: If it arose again, what would you do?
Gibbs Cycle of Reflection (1988)
What forms the basis of your organization’s reflective practice?
Are actions that come out of reflective practice
Engagement data Quality data Feedback data
Using evaluation results to form the basis of reflective practice
Description: What happened? Feelings: What were you thinking and feeling? Evaluation: What was good and bad about the experience? Analysis: What sense can you make
situation? Conclusion: What else could have you done? Action Plan: If it arose again, what would you do?
You can also apply this cycle to your evaluation approach
Description: What happened? Feelings: What were you thinking and feeling? Evaluation: What was good and bad about the experience? Analysis: What sense can you make out of the situation? Conclusion: What else could have you done? Action Plan: What would you do the same and/or differently next time?
Reflect on…. Your experience gathering data from young people Young people’s reactions to being asked for data Your experience analysing evaluation data Your experience reading evaluation results Other’s responses to your evaluation data
Steps of a closed feedback loop
https://feedbacklabs.org/feedback-loops-steps/ Closing the loop with young people shows them you value their feedback and will make them more forthcoming with it. Also close the feedback loop with practitioners – what is changing for the better as a result of the evaluation? How could the evaluation process be changed for the better and lead to better evaluation?
Getting feedback from your team about evaluation
Ways to close the feedback loop
1. Agree your evaluation plan 2. Tell everyone about your evaluation plan 3. Get buy-in by involving people 4. Demonstrate the value of evaluation 5. Define roles and responsibilities 6. Recruit and train people 7. Think about incentives 8. Make data collection and use as easy as possible 9. Set up systems and processes for learning 10. Tolerate mistakes and failure
— Mistakes in data collection stick with it! You will get better — Data that shows no change or negative change have a plan for dealing with unexpected results, and change for the better
call it a Learning Report rather then an Evaluation or an Impact Report
for some small, but quick wins at the start and then build from that as your team gets more comfortable with collecting and using evidence
discuss it in line management sessions
Managing risk
http://www.evaluationsupportscotland.org.uk/media/uploads/resources/final_making _it_stick.pdf
Making it Stick – Evaluation Support Scotland
Resources for embedding evaluation into your
the organization
Action Plan
How would you currently describe your evaluation culture? How would you like to describe your eval How you’ll get there: strengthen your evaluation ecosystem What do they already do well? What do they need to improve? What can be done about it? Timeline Planner Participant/ data contributor User/ analyst Implementer of learning
Action Plan
Your ecosystem:
What could be improved?
What can be done about it?
different to the thing we say we are going to do?
Learning About Culture Aim 1: Build a stronger evidence base for cultural learning Aim 2: Improve the use of evidence in cultural learning
UEL comparison group study 2015/16. Statistically significant improvements in:
We don’t make a statistically significant impact on:
Dr J. Barnes - Arts and Health Study 2015.
friendliness… the development of these psychological dispositions appears closely related to increased capacity to learn, speak out, communicate and listen.’ School reporting 2016/2017:
behaviour.
Next meetings