Developmental Evaluations March 2020 This presentation is made - - PowerPoint PPT Presentation

1

Evaluation Café: A Study of Three Developmental Evaluations

This presentation is made possible by the support of the American People through the United States Agency for International Development (USAID.) The contents of this presentation are the sole responsibility of DEPA-MERL Consortium and do not necessarily reflect the views of USAID or the United States Government.

March 2020

2

Who we are

3

Before we begin,

• How familiar are you with Developmental Evaluation (DE)?
• How familiar are you with Outcome Harvesting?
• How many are interested in international evaluation work?

4

• Introduction to DEPA-MERL
• Overview of research questions & methods
• Key lessons learned
• Audience questions

Our agenda today,

5

• WDI working with USAID since 2015
• MERLIN Program:

– Innovate on traditional approaches to monitoring, evaluation, research and learning (MERL)

• DEPA-MERL is a consortium under

MERLIN that implements developmental evaluations in the USAID context

Introduction to DEPA-MERL

6

• Supports the continuous

adaptation of development interventions in complex environments

• Developmental Evaluators are

“embedded” within the program – Use a variety of M&E approaches

What is developmental evaluation?

7

How we applied DE

• Three DE pilots were conducted

–

Family Care First in Cambodia

–

Sustained Uptake DE, the US Global Development Lab (Washington, DC)

–

Bureau for Food Security (Washington, DC)

• Each DE pilot included:

–

1 full-time, external Developmental Evaluator

–

A “DE Administrator” from DEPA-MERL

8

Our research questions

INSERT DE RQ QUESTIONS TABLE FROM IP AAP REPORT

Research question Methods Data sources

1: How does DE capture, promote, and enable the utilization of emergent learnings in support of ongoing programming in a complex system, in the USAID context? Outcome harvesting (qualitative)

• Monthly reflection interviews with two Developmental

Evaluators (n=35)

• Substantiation interviews with keys stakeholders at

endline of two DEs (n=26)

• Document reviews, as required

2: What are the barriers and enablers to implementation of DE in the USAID context? Semi- structured interviews (qualitative)

• Monthly reflection interviews with two Developmental

Evaluators (n=35)

• Substantiation interviews with keys stakeholders at

endline of two DEs (n=26) 3: What is the perceived value of conducting a DE, especially versus a traditional evaluation approach? Survey (quantitative and qualitative)

• Value of Developmental Evaluation Survey with DE

stakeholders at endline (n=30)

9

Outcome harvesting approach

10

What was the most surprising finding from our research on DE?

Ben White

11

What is the biggest difference between DE in theory and DE in practice?

12

If you were approached by new DE implementer, what advice would you give them about how to successfully launch and oversee a DE?

https://bit.ly/32kJGOA

13

What is the most controversial idea or topic in DE right now?

Diego PH

14

Resources!

https://bit.ly/2JXCUru https://bit.ly/2VkTbgO

15

Questions?

16

USAID-MERLIN (DEPA-MERL) Shannon Griswold, sgriswold@usaid.gov WDI WDI-performancemeasurement@umich.edu

Contact us!

WDI Website https://bit.ly/2NtuVU7

17

Annex

18

What is developmental evaluation?

• Development Evaluation (DE) is an approach to evaluation that supports the

continuous adaptation of development interventions.

• DE provides evaluative thinking and timely feedback to inform ongoing adaptation as

needs, findings, and insights emerge in complex dynamic situations.

• The DE helps facilitate the process from findings to action in a collaborative process

with the DE stakeholders.

Realized Outcomes Intended Outcomes Emergent Outcomes Unrealized Outcomes Implemented Outcomes

Source: Henry Mintzberg, Sumatra Ghoshal, and James B. Quinn, The Strategy Process, Prentice Hall, 1998.

What outcomes can you expect from DE?

20

Traditional Evaluation Developmental Evaluation Purpose Supports improvement, summative test and accountability Supports development of innovation and adaptation in dynamic environments Standards Methodological competence and commitment to rigor, independence, credibility with external authorities Methodological flexibility and adaptability; systems thinking, creative and critical thinking balanced; high tolerance for ambiguity; able to facilitate rigorous evidence-based perspectives Options Traditional research and disciplinary standards of quality dominate options Utilization focused: options are chosen in service to developmental use Evaluation Results Detailed formal reports; validated best practices, generalizable across time and space. Rapid, real time feedback; diverse, user- friendly forms of feedback. Evaluation aims to nurture learning.

How is DE different?

• The DE evaluator works

collaboratively with implementing teams to conceptualize, design, and test new approaches in a long-term,

• ngoing process of adaptation,

intentional change, and development.

• The DE evaluator thinks and

engages evaluatively; questions assumptions; applies evaluation logic; uses appropriate methods; and stays empirically grounded— that is, rigorously gathers, interprets, and reports data.

21

What is a developmental evaluator?

22

Analysis Categories for the Harvested Outcomes (RQ1)

Factor Percent of all enablers* Percent of all barriers* Skills of the Developmental Evaluator 17% 8% Data collection and sharing 16% 10% Data utilization 12% 8% Integration of the Developmental Evaluator 11% 10% Leadership (of program being evaluated) 11% 15% Stakeholder relationships 9% 11% DE readiness 8% 8% USAID dynamics 7% 14% Funding dynamics 2% 6% Local and international dynamics 2% 4% * Percentages do not total 100% because only 10 of the 13 factors that were coded across DE pilots for are shown.

23

Barriers and Enablers (RQ2)

24

Value of Developmental Evaluation Survey (RQ3) (1/2)

Takeaway: In all areas except DE cost and time savings, survey respondents said the DEs were better than traditional evaluation (n=29)

25

Value of Developmental Evaluation Survey (RQ3) (2/2)

Takeaway: The overwhelming majority of respondents reported positive interactions with their Developmental Evaluators.

26

Description of codes (1/2)

27

Description of codes (2/2)

28

The DEPA-MERL DE pilots