Practical Methods and Approaches to Evaluating Your ADR Program - - PowerPoint PPT Presentation

▶
practical methods and approaches to evaluating your adr
SMART_READER_LITE
LIVE PREVIEW

Practical Methods and Approaches to Evaluating Your ADR Program - - PowerPoint PPT Presentation

Mar 07, 2024 716 likes •896 views

How Are We Doing and Where Do We Go From Here? Practical Methods and Approaches to Evaluating Your ADR Program Russell Saltz & Zachary Miller Center for Alternative Dispute Resolution Annual Conference June 27, 2014 Presentation Overview

slide-1
SLIDE 1

How Are We Doing and Where Do We Go From Here? Practical Methods and Approaches to Evaluating Your ADR Program Russell Saltz & Zachary Miller Center for Alternative Dispute Resolution Annual Conference June 27, 2014

slide-2
SLIDE 2

Presentation Overview

  • Introductions
  • Evaluation & ADR
  • Evaluation Design Exercise
  • Q & A Discussion

2

slide-3
SLIDE 3

Getting to Know Us

  • Russell Saltz, Senior Research Associate

M.P.A. Public and Economic Policy London School of Economics and Political Science

  • Zachary Miller, Senior Research Analyst

M.P.A Nonprofit Management George Washington University

3

slide-4
SLIDE 4

Getting to Know You

  • Name, affiliation
  • Where does your work or program fall on the ADR

continuum?

  • What do you hope to take-away from today’s

session?

  • Where on the conflict management mechanisms

continuum do you fall?

4

Conflict Management Mechanisms

slide-5
SLIDE 5

Evaluation Overview

  • What does evaluation mean?

– Oxford Dictionary: to form an idea of the amount, number, or value of; assess [something] – AEA: To assess the strengths and weaknesses of programs, policies, personnel, products, and

  • rganizations to improve their effectiveness

– Rossi: Use of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are deigned to inform action to improve conditions

5

slide-6
SLIDE 6

6

Program Evaluation Policy Analysis Performance Analysis / Management

Evaluation Overview

slide-7
SLIDE 7

Deciding Whether to Evaluate

  • When should you evaluate a program?
  • Dimension 1: Utility
  • Will the evaluation be useful? For whom? Is the evaluation

mandated?

  • Dimension 2: Significance
  • Is there an important issue with your ADR program? Is your

ADR program or are there components of your ADR program that are a strategic priority?

  • Dimension 3: Feasibility
  • Is your ADR program evaluable and ready to be evaluated?
  • Dimension 4: Accuracy
  • Can your ADR program be evaluated well?

7

slide-8
SLIDE 8

Deciding Whether to Evaluate

  • What Questions Can an ADR Evaluation Address?

– Formative- Programmatic elements

  • What worked well?
  • What did not work well?
  • What needs to improve or change?

– Summative- Outcomes

  • How many people served?
  • How many cases resolved?
  • How much did it cost?
  • Value of time and money saved?

8

slide-9
SLIDE 9

Deciding Whether to Evaluate

  • Decisions:

– What are the evaluation questions? – Identify and review the data sources (adequate & available). – Determine resource requirements (financial & human/professional). – Timing (enough for a good study?) – Institutional support? – What is the program’s status? (proposed vs. new; steady-state vs. startup; temporary vs. permanent/ongoing)

9

slide-10
SLIDE 10

Deciding Whether to Evaluate

  • What type of evaluation is appropriate?

10

Type of Evaluation Recognizable Features

Implementation Analysis Fieldwork, observation, surveys, administrative data Performance Analysis Program outcome data and performance measures Cost-Benefit/Cost-effectiveness Analysis/Return on Investment Cost analysis Quasi-experimental, Net Impact, and Outcome Analysis Multivariate statistical modeling; comparison group Experimental Net Impact Analysis Random assignment; control group

slide-11
SLIDE 11

Examples

  • ADR at the World Bank

– Outcomes examined- direct impact: cost; time; client satisfaction; jobs lost, retained, or created; avoidance

  • f bankruptcy; avoid a negative public image

– Outcomes examined – indirect impact: increase effectiveness of courts; improve perceptions of the quality of the legal system and increase trust in the fair resolution of conflicts; influence on investors’ perceptions; improvements in business relationship quality

11

slide-12
SLIDE 12

Logic Model - A Helpful Tool

  • What is a Logic Model?

– A tool to help evaluate the effectiveness of a program – Visual representation of inputs and how they translate to outputs and ultimately outcomes – Identifies the “What”, “If” and “Then”

12

slide-13
SLIDE 13
  • Ex. Logic Model for an ADR Program

13

slide-14
SLIDE 14

Evaluation Design Exercise

Please fill out the provided Evaluation Design Worksheet.

14

slide-15
SLIDE 15

Evaluation Design Exercise

1. Involve intended users (key informant interviews/surveys) (participatory) 2. Clarify program design (goals, priorities, structure, population target groups, client flow) 3. Explore program reality (data systems, structure, implementation status) 4. Assess plausibility of the program (readiness, demo, test, etc.) 5. Reach agreement on any needed changes to program design or implementation 6. Reach agreement on evaluation questions, focus, use

  • f findings.

15

slide-16
SLIDE 16

16

Contact Information

Russell Saltz Senior Research Associate rsaltz@impaqint.com Office: (443) 718-4342 Zachary Miller Senior Research Analyst zmiller@impaqint.com Office: (443) 283-6231