Pro Bono Design & Management Accelerator 1 Decem ecember er - - PowerPoint PPT Presentation

Pro Bono Design & Management Accelerator

Decem ecember er 12, 2 2018

1

2

Session 3

Impact Evaluation & Data Tracking

3

Coach introductions

4

Renée J. Schomp Senior Staff Attorney, Pro Bono Consulting Peter James Senior Manager of Research & Evaluation

Logistics - Nuts and bolts

• Thank you to DREDF & Ed Roberts Campus!
• Restrooms
• Water
• Lunch
• Snacks

5

Mindfulness moment

6

Pro bono accelerator

• bjectives
• 1. Shared pro bono language
• 2. Inspiration from peers
• 3. Role of pro bono in larger civil justice

movement

• 4. Lens of equity & inclusion
• 5. Support on concrete action steps towards
• rganizational change & pro bono design

7

Pro bono accelerator roadmap

• 1. October 10: Volunteerism Overview
• 2. November 14: Recruitment, Cultivation, & Training
• 3. December 12: Impact Evaluation & Data Tracking
• 4. January 9: Placement, Supervision, & Technical Assistance
• 5. February 13: Capstone Project Presentations &

Organizational Change Planning

8

Today’s agenda ...

• 1. Morning: the theory and the methods
• a. Evaluation frameworks
• b. Quantitative methods
• c. Qualitative methods
• 2. Afternoon: the realities
• a. Doing evaluation in practice
• b. Working with funders
• 3. Capstone work time

9

Grounding pro bono programs in a larger civil justice movement

10

Who benefits from evaluation and data tracking?

Ground rules

• Beach ball conversations
• One diva, one mic
• Make space, take space
• Be here now
• Confidentiality

11

Introduction to Impact Evaluation and Data Tracking

12

A simple example

13

Insights from data

14

Understanding -- usually a bit late, but sometimes really late!

Why evaluate le legal a l aid?

15

Why evaluate pro b ro bon

• no p
• prog

rogra rams?

16

Evaluation Frameworks

17

• Evaluation frameworks help ensure that your

evaluation is answering relevant questions

Defining your problem

18

• Define and understand the problem that your
• rg/program is trying to solve

Theory of change

19

• Explains how your org/program/new initiative

solves problem and achieves goals

• Think about comparison and counterfactuals:

how does it bring about a change that otherwise would not happen? We believe that more people will file successfully with the help of a pro bono attorney (than without) because the applications are complex and daunting for most people without legal training.

Logic models

20

• A step-by-step diagram showing how your

program will achieve its results in the real world

Inputs Outputs Outcomes Impact Staff time Pro bono services Completed applications Improved status

Focusing evaluation

21

• For detailed evaluation, you can typically only

focus on parts of the theory or logic model

• Choose and define your focus area

Focus area: process evaluation

• Sometimes we are most interested in evaluating

a specific process (rather than full program)

• In this context, the process measures become

the outcomes/impacts of interest

Focus area: service evaluation

• We are often interested in one or more phases of

service provision

Focus area: training evaluation

• The Kirkpatrick Model: 4 Levels
• Useful for defining outcomes and impacts

Creating questions

25

• Ultimately, all evaluation activity needs to be

grounded in a question (or set of questions) What proportion of clinic participants successfully file?

Today’s exercise: Part 1

26

Introduction

• Problem + goal
• Theory & logic model
• Evaluation question

The Evaluation Toolbox:

• verview of mixed methods

27

Introduction to mixed methods

28

• Actively choose your method(s)
• Consider using multiple methods
• Tailor the methods to question (esp whether

you are interested in causation)

Quantitative methods

29

• For things that you can count or measure
• Use to either explore or to confirm theories
• Examples for legal aid/pro bono:
• What proportion of clients in our pro bono

program achieve a successful outcome?

• Did our new retention strategy result in

increased retention of pro bono volunteers?

Selecting a measure for your question

30

• Two broad types of quantitative measure that

may be relevant to your evaluation question

• Categorical variables
• e.g. win/lose
• Counts/totals and

proportions

• Numerical variables
• e.g. # cases handled
• Descriptive statistics

(mean, median etc)

Finding a data source

31

• Internal administrative data - case management

system and data tracking tools

• External administrative data - matching to

records held by courts, firms etc

• Survey data - data provided by clients/pro

bonos

• Think about:
• Appropriateness
• Completeness (often a limitation of surveys)
• Accuracy

Observational frameworks

32 Caseload Caseload

Post-intervention

• utcomes unknown

Post-intervention

• utcomes known

Limitation of observational frameworks

33 Caseload (- help)

Counterfactual unknown

33 Caseload

Post-intervention

• utcomes known

Causal frameworks

34 Caseload (- help) Caseload

Outcomes without intervention Outcomes with intervention

Pragmatic approaches

35

• Causal framework is gold standard but

experiments often require professional assistance

• Observational data is still valuable:
• Understand what is happening
• Develop new questions and theories
• You can also try to approximate causal methods

by using comparisons:

• Comparing groups
• Change over time

Example of observational study

36

Question How effective are limited scope services in the housing context? Methods Comparison of case outcomes between groups:

• tenants with no representation
• tenants with limited scope representation
• tenants with full scope representation.
• Jessica Steinberg (2011), In Pursuit of Justice

Learning from In P In Pursuit o

• f J

Justice?

37

Outcome measure Unrepresented Limited scope Full scope Possession 14% 18% 55% Mean days to move 47 54 97

• Insight: in this jurisdiction, lawyer may be needed

at each step

• Return to theory: what have we learned about

the problem we are trying to solve?

• Return to logic model: how might we adapt our

program in light of new information?

Demo of data analysis

38

https://docs.google.com/spreadsheets/d/1RUobI78KlCXhq5_qCaV3KgRcydv6zUvXYgJrR_52Aj0/edit?usp=sharing

Tech tips: new skills, new tools

39

Quick brainstorm

40

What is on

• ne way that you could use an

quantitative m method to evaluate your pro bono program?

Qualitative methods

41

• To understand the nuance - expectations,

experiences, reasons, perceptions

• Good for why and how questions
• Inductive: generating ideas and theories
• Examples for legal aid/pro bono:
• Individual interviews with clients to explore

how comfortable they felt working with their pro bono attorney

• Focus groups with pro bono attorneys to

explore experiences of new mentorship model

Qualitative methods detail

42

Formats

• Semi-structured interview
• Topic guides: sequence of prompts/questions
• Individual interviews: individualized
• Focus groups: collective
• Take notes or use a recorder

Samples:

• Principle of saturation
• Professional rule of thumb = 25 participants
• But think about subgroups

Qualitative methods: example

43

• Sarah Sternberg Greene (2016), Race, Class and

Access to Civil Justice

• What explains inaction in response to legal

problems? Why does this differ by race?

• 97 interviews with public housing residents in

Cambridge, MA

• Interview explored a range of related topics

that might influence decision-making on using legal aid (e.g. past experience with justice system and other social institutions)

Key findings

44

• Negative experiences/perceptions spillover from

criminal justice system

• View that your treatment in justice system is

dependent on money

• Prior experience w/ public institutions -

“ashamed, inadequate, degraded, and confused”

• Desire for self-sufficiency
• Racial differences in levels of trust and in level of

comfort seeking help from formal systems

Quotation from study

45

“More money, more justice. I mean it. More money, more justice. It is true. The more money you have for an attorney, whether you are a big case or not, the more

• justice. If you have more money, they have more time to

do the paperwork, investigate, that kind of thing. Oh I can get an attorney, let me tell you. No problem at all. But it won't be one of the good ones.”

Listening exercise

46

https://www.legalaidsmc.org/videos/

Quick brainstorm

47

What is on

• ne way that you could use an

qualitative m method to evaluate your pro bono program?

Observational methods

48

• Specifically motivated observation
• Often used in design research (recall Hagan)
• Best for understanding (inter)actions - often

phenomena without a formal record

• Consider the impact of the observer
• Examples for legal aid/pro bono:
• Observing client flow at workshop
• Observing client interviews
• Observing court operations & hearings

Observational methods: example

49

• Barbara Bezdek (1992), Silence in Court
• Classic study of Baltimore rent court
• Observing the functioning of an institution

and its exclusionary dynamics

Quick brainstorm

50

What is on

• ne way that you could use an
• b
• bservation
• nal m

method

• d to evaluate your pro

bono program?

Exercise: Part 2

51

Methods section

• What method(s), why?
• How are method(s)

tailored to question?

Implementing impact evaluation & data tracking in legal aid

52

Looking ahead

53

Defining scope

54

Monitoring Systems Evaluation Projects

Timelines

55

Preparation Data collection Analysis

Management & resources

56

Logistics: process changes

57

Partnerships

58

Previewing results

59

Example: ILRC NAC evaluation

60 https://newamericanscampaign.org/about/our-impact/

?

Exercise: Part 3

61

Evaluation Plan

• Scope
• Timings
• Logistics
• Results preview

Collaborating with funders: feedback & insight session

62

Summary of exercise

63

• What is the goal of this program?
• How is the program designed to achieve this goal?
• What key evaluation questions have been identified?
• What methodology will be used to answer these

questions?

• What kind of results would indicate success?

Sequence of events

64

• 1. Participant shares ideas with partner for feedback

and development

• 2. Switch partners; repeat exercise using new

information

• 3. Group discussion to explore learning from exercise

...to our thought p partners!

65

Breaktime!

66

Individual capstone activity

• Outline the elements of internal analysis for

your capstone issue(s) that you plan to research.

• Deadline reminder: January 4, 2019: Submit
• ne-page description of the internal analysis to

Renée and Lea via email

67

Today, we...

• 1. Introduced evaluation frameworks, such as

logic models and theory of change

• 2. Learned some basics about quantitative and

qualitative models

• 3. Started creating an evaluation and data tracking

plan for your pro bono program

• 4. Road-tested your evaluation plan with Funders
• 5. Started work on the Internal Analysis for your

capstone project

68

One-word closeout

69

Thank you!

• Complete evaluation surveys

70