Checklists for Improving Evaluation Practice June 10, 2015 - - PDF document

▶
checklists for improving evaluation practice
SMART_READER_LITE
LIVE PREVIEW

Checklists for Improving Evaluation Practice June 10, 2015 - - PDF document

Oct 06, 2023 118 likes •534 views

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 Checklists for Improving Evaluation Practice June 10, 2015 Introductions Lori Wingate Emma Perk Goldie MacDonald This material is based upon work supported by the

slide-1
SLIDE 1

1

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Checklists for Improving Evaluation Practice

June 10, 2015

This material is based upon work supported by the National Science Foundation under grant number 1204683. Any
  • pinions, findings, and conclusions or recommendations expressed in this material are those of the presenters and do not
necessarily reflect the views of NSF.

Introductions

Lori Wingate Goldie MacDonald Emma Perk

slide-2
SLIDE 2

2

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Behind the Scenes

Janet Pinhorn Mike Lesiecki

Jason

Slides

www.evalu-ate.org/webinars/2015-june/

Webinar Materials

Recording Handout

slide-3
SLIDE 3

3

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

  • 1. Checklist Form & Function

  • 2. Using Checklists to Improve

Evaluation Practice

Checklists help us remember what we need to do

slide-4
SLIDE 4

4

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Have you read this book?

We need a different strategy for overcoming failure…

Photo courtesy of the MacArthur Foundation www.macfound.org
slide-5
SLIDE 5

5

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

slide-6
SLIDE 6

6

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

“

…It is a checklist.

It is a checklist.

Checklists Save Lives

bit.ly/WHO-check

slide-7
SLIDE 7

7

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Lori Wingate

Checklist Form and Function

Memory Aid Task Completion Decision Making Evaluation

Laundry list

Procedural checklist

 

Diagnostic checklist

  

Criteria

  • f merit

checklist

   

Checklist Form & Function

based on Scriven’s (2007) ”The Logic and Methodology of Checklists” bit.ly/check-logic
slide-8
SLIDE 8

8

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Laundry List

A nonordered list of items or tasks, sometimes grouped into categories

Checklist Form & Function

slide-9
SLIDE 9

9

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org bit.ly/fema-check bit.ly/fema-check

Is there anything on this list you wouldn’t have thought of on your own? Get ready to use your markers…

slide-10
SLIDE 10

10

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org bit.ly/fema-check

Procedural Checklist

A list of tasks or issues that must be addressed in a particular order

Checklist Form & Function

slide-11
SLIDE 11

11

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

bit.ly/WHO-check

slide-12
SLIDE 12

12

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Diagnostic Checklist

A series of questions or considerations that lead the user to a descriptive conclusion about the status of something

Checklist Form & Function

[example – PTSD checklist? Auto diagnostic?]

bit.ly/think-ebola

slide-13
SLIDE 13

13

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

[example – PTSD checklist? Auto diagnostic?]

bit.ly/think-ebola

Criteria of Merit Checklist

A list of characteristics that define quality for a particular product or phenomena to aid the user in reaching an evaluative judgment

Checklist Form & Function

slide-14
SLIDE 14

14

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

This is a checklist to be utilized by authors, report developers, and

  • ther interested

parties to aid in the development and maximization of readability of written documents. This is a checklist to be utilized by authors, report developers, and

  • ther interested

parties to aid in the development and maximization of readability of written documents.

(Will help (Will help you you avoid writing avoid writing terribl terrible sentences e sentences like this one!) like this one!)

slide-15
SLIDE 15

15

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

This is a checklist to be utilized by authors, report developers, and

  • ther interested

parties to aid in the development and maximization of readability of written documents. This is a checklist to be utilized by authors, report developers, and

  • ther interested

parties to aid in the development and maximization of readability of written documents.

slide-16
SLIDE 16

16

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

1-page checklist is supported by 5 pages of explanation

Guide Task Completion Guide Decision Making Reach Evaluative Conclusions Aid Memory

CHECKLIS CHECKLISTS TS

MAIN FUNCTIONS MAIN BENEFITS

Distill Complex Content Transfer Knowledge Quality Control

Checklist Form & Function

slide-17
SLIDE 17

17

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Checklists Distill Complex Content

Checklist Form & Function

Photo from NSF Flickr stream
slide-18
SLIDE 18

18

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org bit.ly/cg-checklists

Photo from NSF Flickr stream

52 pages 15 pages

(Six 1-3 page checklists)

52 pages 15 pages

(Six 1-3 page checklists)

bit.ly/cg-checklists

Photo from NSF Flickr stream
slide-19
SLIDE 19

19

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Checklists Support Quality Control

Checklist Form & Function

slide-20
SLIDE 20

20

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

EvaluATE’s Webinar Checklist

bit.ly/webinar-check

Checklists Aid in Knowledge Transfer

Checklist Form & Function

slide-21
SLIDE 21

21

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org Michael Scriven Michael Quinn Patton Daniel Stufflebeam

 Key Evaluation Checklist  Utilization-Focused Evaluation Checklist  Qualitative Evaluation Checklist  Evaluation Plans and Operations Checklist  CIPP Evaluation Model Checklist  Evaluation Design Checklist  Evaluation Contracts Checklist

Daniel Stufflebeam

 Evaluation Plans and Operations Checklist  CIPP Evaluation Model Checklist  Evaluation Design Checklist  Evaluation Contracts Checklist  Evaluation Plans and Operations Checklist Daniel Stufflebeam (2000) “Guidelines for Developing Evaluation Checklists: The Checklists Development Checklist” www.wmich.edu/evalctr/checklists

“ “

Students began pressing me to explain what exactly I was doing in designing project evaluations. Answering their questions proved difficult. In retrospect, I was developing and exercising a kind of personal art of evaluation design rather than laying out and following any particular systematic approach. When the students persisted in pressuring me to give them an evaluation planning protocol, I decided to try to respond. Thus, I developed my first evaluation checklist.
slide-22
SLIDE 22

22

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

www.wmich.edu/evaluation/checklists

Discussion

Goldie MacDonald

slide-23
SLIDE 23

23

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Questions

Type them Type them in in the chat box now the chat box now

Lori Wingate Goldie MacDonald Emma Perk

slide-24
SLIDE 24

24

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

#evalchecklists EvaluATE WMUEvaluATE @EvaluATE_WMU EvaluATE Group

Join us on Social Media Kalamazoo WE’RE HIRING! WE’RE HIRING!

Assistant Director of The Evaluation Center

bit.ly/wmu-ec-job

slide-25
SLIDE 25

25

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Lori Wingate

Using Checklists to Improve Evaluation Practice

www.wmich.edu/evaluation/checklists

Categories

 Evaluation Capacity Building  Evaluation Design and Management  Evaluation Models  Evaluation Values and Criteria  Metaevaluation  Checklist Development

slide-26
SLIDE 26

26

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

by Daniel Stufflebeam

bit.ly/eval-contracts

Evaluation Contracts Checklist

Evaluation Contracts Checklist

by Daniel Stufflebeam

bit.ly/eval-contracts

Evaluation Contracts Checklist

by Daniel Stufflebeam

bit.ly/eval-contracts

slide-27
SLIDE 27

27

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Evaluation Contracts Checklist

by Daniel Stufflebeam

bit.ly/eval-contracts

Evaluation Contracts Checklist

by Daniel Stufflebeam

bit.ly/eval-contracts

slide-28
SLIDE 28

28

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Evaluation Budget Checklist

by Jerry Horn bit.ly/eval-budget

Photo courtesy of www.seniorliving.org

 Basic Considerations  Travel  Supplies and Materials  Communications  Copying Printing  Equipment  Consultants  Subcontracts  In-kind or Cost-sharing  Indirect/Overhead

6 pages of checkpoints across 10 categories

Evaluation Budget Checklist

by Jerry Horn bit.ly/eval-budget

Photo courtesy of www.seniorliving.org
slide-29
SLIDE 29

29

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

667 pages 19 pages

Utilization-Focused Evaluation Checklist

bit.ly/ufe-check Utilization-Focused Evaluation Checklist by Michael Quinn Patton

Utilization-Focused Evaluation Checklist Step 12. Simulate use of findings

 Fabricate findings based on the proposed design and measures of implementation and outcomes  Guide primary intended users in interpreting the potential (fabricated) findings.  Interpret the simulation experience to determine if any design changes, revisions, or additions to the data collection would likely increase utility.  As a final step before data collection, have primary intended users make an explicit decision to proceed with the evaluation given likely costs and expected uses.

slide-30
SLIDE 30

30

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Photo from CDC Global Health Flickr stream

58 pages 8 pages

Photo from CDC Global Health Flickr stream

bit.ly/cdc-eval

slide-31
SLIDE 31

31

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

58 pages 8 pages

Photo from CDC Global Health Flickr stream

bit.ly/cdc-eval Framework for Program Evaluation in Public Health: A Checklist of Steps and Standards

Step 2. Describe the Program

 Need  Expected effects  Activities  Resources  Stage of development  Context  Logic Model

slide-32
SLIDE 32

32

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org Framework for Program Evaluation in Public Health: A Checklist of Steps and Standards

Step 3. Focus the Evaluation Design

 Purpose  Users  Uses  Questions  Methods  ____________ What is the missing checkpoint? Agreements Agreements

slide-33
SLIDE 33

33

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

slide-34
SLIDE 34

34

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Criteria for Selection

  • f High-Performing

Indicators

by Goldie MacDonald bit.ly/indicator-eval bit.ly/indicator-eval

Criteria for Selection

  • f High-Performing

Indicators

by Goldie MacDonald

slide-35
SLIDE 35

35

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Checklist for

Criteria for Selection

  • f High-Performing

Indicators

by Goldie MacDonald bit.ly/indicator-eval

 Nondirectional Language The indicator is written as neutral, not defined as positive or negative in advance of data collection A) Retention rate among first generation college students B) Increased retention among first generation college students Indicator Selection Criterion Which indicator meets this criterion?

slide-36
SLIDE 36

36

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

 Nondirectional Language The indicator is written as neutral, not defined as positive or negative in advance of data collection A) Percentage of participants who attended 75% or more of program events B) Mean number of events attended by participants Indicator Selection Criterion Which indicator meets this criterion?

Discussion

Goldie MacDonald

slide-37
SLIDE 37

37

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

slide-38
SLIDE 38

38

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org bit.ly/indicator-eval

Questions

Type them Type them in in the chat box now the chat box now

Lori Wingate Goldie MacDonald Emma Perk

slide-39
SLIDE 39

39

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

EvaluATE’s Blog: Read and Be Read

New today! Corey Smith on ATE program recruitment and retention strategies

Next Webinar

August 19 | 1-2:30 p.m. EDT

Register at www.evalu-ate.org/webinars

slide-40
SLIDE 40

40

EvaluATE Webinar: Checklists for Improving Evaluation Practice 6/10/2015 evalu‐ate.org

Thank You!