Quick Reference Guides Evaluators Cant Live Without AEA 2019 - - PowerPoint PPT Presentation

▶
quick reference guides
SMART_READER_LITE
LIVE PREVIEW

Quick Reference Guides Evaluators Cant Live Without AEA 2019 - - PowerPoint PPT Presentation

Nov 12, 2023 482 likes •1.04k views

Quick Reference Guides Evaluators Cant Live Without AEA 2019 Introduction Kelly Robertson Lyssa Wilson Becho This material is based upon work supported by the National Science Foundation under grant no. 1600992 and 1841783. The content

slide-1
SLIDE 1

Quick Reference Guides

Evaluators Can’t Live Without

AEA 2019

slide-2
SLIDE 2

Kelly Robertson Lyssa Wilson Becho

Introduction

This material is based upon work supported by the National Science Foundation under grant no. 1600992 and 1841783. The content reflects the views of the authors and not necessarily those of NSF.
slide-3
SLIDE 3

AGENDA

Quick-reference-what? How we learned about QRGs in use Top quick reference guides Top guides you likely missed

1 2 3 4

Full list of evaluation QRGs

5

slide-4
SLIDE 4

Practical reference or resource materials

Worksheet Infographic Checklist Cheat sheet Table or page in a book

What is a Quick Reference Guide?

slide-5
SLIDE 5

Why do Quick Reference guides matter ?

Solve problems in action Learn new tools Open access Increase evaluation quality

slide-6
SLIDE 6

2

What did we do ? What did we do ?

slide-7
SLIDE 7

2

What did we do ? What did we do ?

Do you use quick reference guides? How? Which ones? (n = 79)

slide-8
SLIDE 8

2

What did we do ? What did we do ?

Do you use quick reference guides? How? Which ones? (n = 79) How useful are these quick reference guides (from list)? (n = 45)

slide-9
SLIDE 9

What did we learn ?

78%

(n = 45)

slide-10
SLIDE 10

What did we learn ?

53%

(n = 45)

to teach others to guide work to enhance knowledge to help clients

51% 29% 27%

slide-11
SLIDE 11

Top Guides

Overall by Categories

slide-12
SLIDE 12

Top Guides

Evaluation Planning Evaluation Design Data Collection Data Analysis Evaluation Reporting

slide-13
SLIDE 13

Evaluation Flash Cards

Michael Patton (2017)

slide-14
SLIDE 14

Evaluation Flash Cards

Purpose

Introduce core concepts of evaluation in an easily accessible and retrievable format.

Intended Use

Information refresher

Topics Covered

  • Evaluative Thinking
  • Evaluation Questions
  • Logic Models
  • Theory of Change
  • Evaluation vs Research
  • Dosage
  • Disaggregation
  • Changing Denominators,

Changing Rates

  • SMART Goals
  • Distinguishing Outcomes From

Indicators

  • Performance Targets
  • Qualitative Evaluation
  • Triangulation Through Mixed

Methods

  • Important and Rigorous Claims of

Effectiveness

  • Accountability Evaluation
  • Formative Evaluation
  • Summative Evaluation
  • Development Evaluation
  • The IT Questions
  • Fidelity or Adaptation
  • High-Quality Lessons Learned
  • Evaluation Quality Standards
  • Complete Evaluation Reporting
  • Unitization-Focused Evaluation
  • Distinguish Different Kinds of

Evidence

Michael Patton (2017)

slide-15
SLIDE 15

About Concept Example Review Bottom Line

Evaluation Flash Cards

Michael Patton (2017)

slide-16
SLIDE 16

Evaluation Data Matrix Template

EvaluATE (2017)

slide-17
SLIDE 17

Evaluation Data Matrix Template

EvaluATE (2017) Purpose

Organize evaluation indicators, collection source, analysis, and interpretation.

Intended Use

Provide structure for evaluation design

Topics Covered

  • Indicators
  • Data sources and methods
  • Responsible party
  • Timing
  • Analysis plan
  • Interpretation
slide-18
SLIDE 18

Evaluation Data Matrix Template

EvaluATE (2017) Outline for matrix structure Explanation of each column

slide-19
SLIDE 19

Evaluation Data Matrix Template

EvaluATE (2017) Example

slide-20
SLIDE 20

Likert-Type Scale Response Anchors

Vegias (2006)

slide-21
SLIDE 21

Likert-Type Scale Response Anchors

Vegias (2006) Purpose

Examples of likert scale labels for a variety of constructs.

Intended Use

Inspiration during survey item development

Topics Covered

  • Acceptability
  • Appropriateness
  • Importance
  • Agreement
  • Knowledge
  • Beliefs
  • Priority
  • Concern
  • Priority
  • Level of Problem
  • Affect
  • Level of Consideration
  • Level of Support
  • Agreement
  • Desirability
  • Participation
  • Frequency
  • Frequency of Use
slide-22
SLIDE 22

Measurement construct Example anchor labels

Likert-Type Scale Response Anchors

Vegias (2006)

slide-23
SLIDE 23

You’re Invited To A Data Party

Hutchinson (2016)

slide-24
SLIDE 24

You’re Invited To A Data Party

Hutchinson (2016) Purpose

Summarize the Who, What, and When of a Data Party

Intended Use

Plan a data interpretation session

Topics Covered

  • Who should be at your Data Party
  • What is a Data Party
  • When you should hold it
  • Why you should hold a Data Party
  • Where to hold it
  • How to ask reflective questions
slide-25
SLIDE 25

You’re Invited To A Data Party

Hutchinson (2016) Answer common questions Example of reflective questions to ask

slide-26
SLIDE 26

Data Visualization Checklist

Evergreen & Emery (2016)

slide-27
SLIDE 27

Data Visualization Checklist

Evergreen & Emery (2016) Purpose

Guide for the development of high impact data visualizations.

Intended Use

Rating graphs and charts

Topics Covered

  • Text
  • Arrangement
  • Color
  • Lines
  • Overall
slide-28
SLIDE 28

Data Visualization Checklist

Evergreen & Emery (2016) Explanation of aspects Data visualization guidelines with explanations Ratings

slide-29
SLIDE 29

Data Visualization Checklist

Evergreen & Emery (2016) Visual comparison

slide-30
SLIDE 30

Top Guides

You Likely Missed

slide-31
SLIDE 31

Guides you Likel y Missed

Evaluation Planning Evaluation Design Data Collection Data Analysis Evaluation Reporting

slide-32
SLIDE 32

Checklist of key considerations for development of program logic models

MacDonald (2018)

slide-33
SLIDE 33

Checklist of key considerations for development of program logic models

MacDonald (2018) Purpose

Support the development of logic models.

Intended Use

Provide guidance or refresh knowledge on creating logic models

Topics Covered

  • Scope
  • Basic components
  • Underlying logic of program
  • Key items to include
  • Accompanying narrative
slide-34
SLIDE 34

Checklist of key considerations for development of program logic models

MacDonald (2018) Steps for development Practical guidance

slide-35
SLIDE 35

Get to know your Evaluation Stakeholders

Hutchison

slide-36
SLIDE 36

Get to know your Evaluation Stakeholders

Hutchison Purpose

Provide ideas of questions to ask in meeting with evaluation stakeholders.

Intended Use

Facilitate conversation with stakeholders

Topics Covered

  • Questions for evaluation

stakeholders

slide-37
SLIDE 37

Get to know your Evaluation Stakeholders

Hutchison Example questions

slide-38
SLIDE 38

Collecting Race and Ethnicity Data from Students and Staff Using the New Categories

IPEDS

slide-39
SLIDE 39

Collecting Race and Ethnicity Data from Students and Staff Using the New Categories

IPEDS Purpose

Guide for the development of questions to collect demographic data.

Intended Use

For programs and institutions that need to align data collection with NCES

Topics Covered

  • Guidelines for questions
  • Reporting missing or messy

data to NCES

slide-40
SLIDE 40

Collecting Race and Ethnicity Data from Students and Staff Using the New Categories

IPEDS Separate questions for race and ethnicity How not to ask questions

slide-41
SLIDE 41

Choosing the Correct Statistical Test in SAS, Stata, SPSS, and R

Leeper

slide-42
SLIDE 42

Choosing the Correct Statistical Test in SAS, Stata, SPSS, and R

Leeper Purpose

Increase appropriateness of statistical analyses.

Intended Use

Identifying and coding analysis

Topics Covered

  • Type of inferential test
  • Code for conducting test in

SAS, Stata, SPSS, and R

slide-43
SLIDE 43

Choosing the Correct Statistical Test in SAS, Stata, SPSS, and R

Leeper Identifies correct test for data Provides example code

slide-44
SLIDE 44

recommendations that rock

Hutchinson (2015)

slide-45
SLIDE 45

recommendations that rock

Hutchinson (2015) Purpose

Support the development of evaluation recommendations.

Intended Use

Writing evaluation reports that include recommendations

Topics Covered

  • Criteria for high quality

recommendations

  • Guidance for organization
  • Practical strategies
slide-46
SLIDE 46

recommendations that rock

Hutchinson (2015) Criteria for quality recommendations Practical strategies for writing Organizational tips

slide-47
SLIDE 47

Quick Reference GUIDES

slide-48
SLIDE 48

Quick Reference GUIDES

slide-49
SLIDE 49

Quick Reference GUIDES

slide-50
SLIDE 50

Quick Reference GUIDES

slide-51
SLIDE 51

WHERE TO FIND MORE?

BetterEvaluation Racial Equity Tools USAID Performance Monitoring and Evaluation TIPS Community Tool Box Evaluation Checklist Project EvaluATE

slide-52
SLIDE 52

Questions?

slide-53
SLIDE 53

Thank You

Kelly Robertson & Lyssa Wilson Becho