Are we making a difference? Lessons from evaluating civil - - PowerPoint PPT Presentation

Are we making a difference?

Lessons from evaluating civil society-led action on food

• Dr Hannah Pitt
• Sustainable Places Research Institute, Cardiff University
• Social scientist
• Food sustainability, gardening & growing
• Research in partnership with civil society organisations

What is evaluation?

Monitoring = What are we doing? (outputs) Evaluation = How have we done? (outcomes + process)

Why evaluate?

Aims :

• to present evidence of the programme impacts;
• to reveal and document how impacts are achieved;
• to support and facilitate sharing of learning; and
• to advise on research and evidence elements of programme

development.

Process

Report Collect data Agree plan Awarded Contract Tender & selection Project funded

Approach

Theory of change / logic model

= how you expect outcomes to occur over the short, medium and longer term http://bhfood.org.uk/wp-content/uploads/2018/11/Outcomes- and-impact-framework.pdf

e.g. Food Growing Schools London

”the ambition to inspire and equip every school in London to grow their own food”

Outcomes & Indicators

Outcome Indicators Collection Method

Through involvement in food growing activities at school, children and young people will have gained life skills, knowledge and confidence in food growing. I 1. Head teachers report an increase in positive behaviour, attention or attainment: 40% of respondents Annually School survey I 2. Number of pupils physically involved in food growing: 10% increase over baseline Annually School survey I 3. Number of pupils reporting increased knowledge, skills and confidence in food growing: 20% increase

• ver baseline

Annually Pupil survey

Data collection

• 1. What change are we trying to demonstrate and understand?
• 2. What will indicate or measure change? numbers & stories
• 3. Who can help us understand change?

e.g. questionnaires/surveys, interviews, focus groups

Case Studies

Food Growing Schools London report: https://www.foodgrowingschools.org/resources/files/F GSL_InterimReport_2016_v2.pdf Academic publication on Food Growing Schools London: Pitt, H. , Jones, M. and Weitkamp, E. (2018) Every city a food growing city? What food growing schools London reveals about city strategies for food system

• sustainability. Sustainability, 10 (8). p. 2924. ISSN 2071-

1050 Available from: http://eprints.uwe.ac.uk/38053

Food For Life

https://www.foodforlife.org.uk/

Food For Life

Do pupils consume more fruit and vegetables in schools engaged with FFL than pupils in schools not engaged with FFL? Day in the Life Questionnaire : Health Educ Res. 2002 Apr;17(2):211-20 Development and validation of the Day in the Life Questionnaire (DILQ) as a measure of fruit and vegetable questionnaire for 7-9 year olds. Edmunds LD1, Ziebland S.

All Food For Life evaluation reports available at: https://www.foodforlife.org.uk/abou t-us/our-impact/evaluation-reports Academic papers:

What is the value of investing in Food For Life?

Social Return on Investment

• Aims to capture the full the value of investment beyond

market prices

• Quantifies the relative importance people place on change

they experience

• Measured from the perspective of those affected

SROI = value of benefits

Value of investments

Outcome Indicator Financial Proxy Data source Curriculum development linked to local issues Value of staff time spent in curriculum development work Cost per head of half day local authority area- based training session, based upon attendance of 10 trainees. Plus teacher cover supply costs Cost half day in- school training session £30 per head. http://www.aqa.or g.uk/professional- development/in- school-training.

What is the value of investing in Food For Life?

Report on SROI Results: https://www.foodforlife.org.uk/~ /media/files/evaluation%20repo rts/4foodforlifelcssroifullreportv 04.pdf

Academic publications on Food For Life:

• Gray, S., Orme, J., Pitt, H., Jones, M. (2017) Food for Life: evaluation of the impact of the Hospital Food Programme in England using a case study

approach Journal of the Royal Society of Medicine Open 8 (10)1-9 DOI: 10.1177/2054270417712703

• Jones, M., Pitt, H., Oxford, L., Bray, I., Kimberlee, R., Orme, J. (2017) Association between Food for Life, a Whole Setting Healthy and Sustainable Food

Programme, and Primary School Children’s Consumption of Fruit and Vegetables: A Cross-Sectional Study in England. International Journal of Environmental Research and Public Health,14, 639. ISSN 1660-4601

• Gray, S., Jones, M., Means, R., Orme, J., Pitt, H., and Salmon, D. (2017) Inter-sectoral Transfer of the Food for Life Settings Framework in England. Health

Promotion International, pp.1-10. ISSN 0957-4824

• Gray, S., Means, R., Orme, J., Pitt, H., Jones, M. and Salmon, D. (2015) Improving hospital food: Evaluating the impact of the UK Food for Life

Partnership.European Journal of Public Health, 25 (Suppl3). p. 380. ISSN 1101-1262

• Weitkamp, E., Jones, M., Salmon, D., Kimberlee, R. and Orme, J. (2013) Creating a learning environment to promote food sustainability issues in primary

schools? Staff perceptions of implementing the food for life partnership programme.Sustainability, 5 (3). pp. 1128-1140. ISSN 2071-1050

• Orme, J., Jones, M., Salmon, D., Weitkamp, E. and Kimberlee, R. (2013) A process evaluation of student participation in a whole school food

programme.Health Education, 113 (3). ISSN 0965-4283

• Kimberlee, R., Jones, M., Orme, J. and Salmon, D. (2013) Whole school food programmes and the kitchen environment.British Food Journal, 115 (5). pp.

756-768. ISSN 0007-070X

• Jones, M., Dailami, N., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Engaging secondary school students in food-related citizenship:

Achievements and challenges of a multi-component programme.Education Sciences, 2 (2). pp. 77-90. ISSN 2227-7102

• Jones, M., Dailami, N., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Food sustainability education as a route to healthier eating: Evaluation
• f a multi-component school programme in English primary schools.Health Education Research, 27 (3). pp. 448-458. ISSN 0268-1153
• Jones, M., Weitkamp, E., Kimberlee, R., Salmon, D. and Orme, J. (2012) Realizing a holistic approach to food through school gardens and growing

activities.Children, Youth and Environments, 22 (1). pp. 75-98. ISSN 1546-2250

Growing Together

Does involvement in gardening improve mental health? Evaluation Report available: Jones, M. and Pitt,

• H. (2016) Growing Together: A mixed methods

evaluation of a community gardening project delivered by a social housing association. Year 1. Project Report. UWE Bristol, Bristol. Available from: http://eprints.uwe.ac.uk/28428

Warwick Edinburgh Mental Wellbeing Scale

Guidance on the scale: https://warwick.ac.uk/fac/sci/med/research/platform/wemwbs/

Food Power

“Food Power aims to strengthen the ability of local communities to reduce food poverty, through solutions developed in partnership and with the support of peers from across the UK. Our goal is to transform the way that people experiencing food poverty access support so they can create long-term, sustainable lives that are free from hunger.” https://www.sustainweb.org/foodpower/

Food Power

Are local food poverty alliances making a difference? Year 1 Evaluation Report available: https://www.sustainweb.org/foodpower/publi cations/

Collective impact

• Collective impact is an approach to understanding change

related to complex problems.

• Considers organisations working together rather than in

isolation.

• A structured way of understanding and measuring

collaboration and its impacts.

• Research identified 5 conditions of success.

dfdvdv

30

Evaluating Collective Impact

How:

• Promotes continuous learning
• Seeks to understand progress (performance, outcomes) and how & why progress

(process)

• Uses understanding of collective impact processes to devise a framework for

measurement and evaluation.

• Evolves over time
• Early years - focus on what needs to happen & establishing 5 core conditions
• Middle years – increasing focus on impacts, how well is it working
• Later years – focus on goals and outcomes, what difference did it make

Guide available: https://www.collectiveimpactforum.org/resources/guide- evaluating-collective-impact

Food Power Collective Impact Tool

FIVE FOUNDATIONS Why this matters: These are the underpinning foundations of working

collectively, and are likely to be the focus of activity in the earlier stages of forming a food poverty alliance. Meeting these conditions means a greater chance of achieving collective impact. Reflecting on your foundations is also a way to health check your alliance, something you might review periodically.

Is this relevant to us? Are you in the early stages of building an alliance?

Do you want to check the 'health' of your collaboration? Do you want to identify areas to improve the effectiveness of your alliance?

Figure from 'Guide to Evaluating Collective Impact' Preskill et al 2013 https://www.fsg.org/publications/guide-evaluating-collective-impact

TOTAL SCORE FOR THIS SECTION 14 AVERAGE SCORE FOR THIS SECTION 2.33

Foundation Goal Indicators Score Example of evidence

• 1. Common Agenda

Diverse voices have developed a common agenda Alliance includes key local actors Alliance members have a common understanding of the problem Alliance has articulated a shared definition of food insecurity and their vision for tackling it 2 Alliance members have consensus on their ultimate goal Alliance members have publicly committed to collective action Alliance members have committed to solving the problem together Members look to the Alliance for strategic guidance

• 2. Backbone function

Sufficient resources are available for core infrastructure – finance, knowledge & skills, staff Coordinator(s) have sufficient capacity for the life of the Action Plan 2

• 3. Mutually Reinforcing Activities

The Alliance has a collective plan of action Food Poverty Action Plan agreed by alliance members Alliance members coordinate their activities to align with the plan Alliance members have sought to address duplication of efforts 4 Alliance members have sought to

Challenges

• Demonstrating impact on complex problems
• Attribution and not over claiming
• Balancing robust evidence with practical approaches
• Being driven by what’s measurable
• What if we’re not making a difference?

Challenges – For organisations

• Resources and capacity
• Working with academics – timescales, priorities, jargon
• Skills and expertise
• Selecting the right tool
• Is it robust?

Challenges – for researchers

• Flawed logic models
• Managing expectations
• Bringing bad news
• Working with organisations - timescales, priorities, jargon
• Demanding people’s time
• Is it research?

Some solutions

• Plan it early
• Resource it
• Pick the right people
• Build a relationship
• Capacity building e.g. Toolkits
• Remember its simple!