FOOD FOR PEACE DATA QUALITY ASSESSMENT Pamela Velez Vega, - - PowerPoint PPT Presentation

▶
food for peace data quality assessment
SMART_READER_LITE
LIVE PREVIEW

FOOD FOR PEACE DATA QUALITY ASSESSMENT Pamela Velez Vega, - - PowerPoint PPT Presentation

Dec 25, 2022 303 likes •736 views

FOOD FOR PEACE DATA QUALITY ASSESSMENT Pamela Velez Vega, Monitoring & Evaluation Advisor, FANTA Project Dan Houston, Monitoring and Evaluation Specialist, USAID/Southern Africa/FFP March 2 nd , 2016 Food and Nutrition Technical Assistance

slide-1
SLIDE 1

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

FOOD FOR PEACE DATA QUALITY ASSESSMENT

Pamela Velez‐Vega, Monitoring & Evaluation Advisor, FANTA Project Dan Houston, Monitoring and Evaluation Specialist, USAID/Southern Africa/FFP March 2nd, 2016

Food and Nutrition Technical Assistance III Project (FANTA) FHI 360 1825 Connecticut Ave., NW Washington, DC 20009 Tel: 202‐884‐8000 Fax: 202‐884‐8432 Email: fantamail@fhi360.org Website: www.fantaproject.org

slide-2
SLIDE 2

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 2

Session Objectives

Participants will:

  • 1. Identify four key data quality assessment (DQA)

requirements from the FFP Monitoring and Evaluation Policy document

  • 2. Assess an indicator using five data quality

standards

  • 3. Review an illustrative DQA process and

potential pitfalls to avoid

slide-3
SLIDE 3

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 3

FFP Data Quality Assessment (DQA) Definition

A systematic and periodic review of the data quality of indicators that FFP development projects report annually.

slide-4
SLIDE 4

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 4

Purpose

To improve data quality with the ultimate goal of improving accountability and decision making.

Bad Data Good Data

slide-5
SLIDE 5

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 5

Purpose

A DQA is designed to:

  • 1. Verify the quality of data
  • 2. Assess the system that produces that data
  • 3. Develop action plans to improve both
slide-6
SLIDE 6

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 6

DQA Requirements for FFP Development Projects

slide-7
SLIDE 7

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 7

Requirement Number 1

DQA on annual monitoring indicator data Preferably on project‐ specific indicators from non‐survey data collection methods

Your Your Data Data

slide-8
SLIDE 8

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 8

Requirement Number 1 (cont.)

Universe of annual monitoring indicator data

FFP annual monitoring indicators Project‐specific annual monitoring indicator data collected through beneficiary‐ based surveys Project‐specific annual monitoring indicators collected through routine monitoring Focus of DQA

slide-9
SLIDE 9

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 9

Requirement Number 2

Provide description of plans for DQA on an annual basis

Life of Award M&E Plan PREP PREP PREP PREP Year 1 Year 2 Year 3 Year 4 Year 5

slide-10
SLIDE 10

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 10

What must the description of the plan for DQA include?

slide-11
SLIDE 11

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 11

Requirement Number 3

Description of plan for DQA should include:

  • Indicators to be assessed and justification

for selection

  • Timeframe: timing and duration
  • Methodology
  • DQA staff roles and qualifications
slide-12
SLIDE 12

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 12

Requirement Number 4

Select a sample of indicators for DQA annually Purposive sample based on:

  • Importance of indicator to ToC
  • Identified and perceived data quality risks

associated with indicator

  • Timing and availability of staff
  • Frequency and timing of data collection
  • Other factors
slide-13
SLIDE 13

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 13

Selecting Indicators for DQA

Categorize indicators:

  • Similar data flows
  • Output vs. Outcome
slide-14
SLIDE 14

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 14

Data Flows

REPORTING LEVELS

M&E Unit Intermediate aggregation levels (e.g., districts, regions) Service delivery points

Source: Adapted from Measure Evaluation Photo: Jessica Scranton, FANTA/FHI 360

slide-15
SLIDE 15

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 15

A DQA is designed to:

  • 1. Verify the quality of data
  • 2. Assess the system that produces that data
  • 3. Develop action plans to improve both
slide-16
SLIDE 16

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 16

Data Quality

slide-17
SLIDE 17

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Data Quality Standards

17

Validity Reliability Precision Integrity

Timeliness

slide-18
SLIDE 18

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 18

A DQA is designed to:

  • 1. Verify the quality of data
  • 2. Assess the system that produces that data
  • 3. Develop action plans to improve both
slide-19
SLIDE 19

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 19

Data‐Management and Reporting System

slide-20
SLIDE 20

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 20

REPORTING LEVELS

Quality Data

Source: Adapted from Measure Evaluation

M&E Unit Intermediate aggregation levels (e.g., districts, regions) Service delivery points

Data‐management and reporting system

slide-21
SLIDE 21

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 21

Functional Components of Data‐Management System Needed to Ensure Data Quality

I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines

  • III. Data collection tools and reporting forms
  • IV. Processes of data verification, aggregation, processing,

management, storage, and safeguarding

  • V. Data use and dissemination
  • VI. Links with national reporting systems (where relevant)

Source: Adapted from Measure Evaluation

slide-22
SLIDE 22

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 22 REPORTING LEVELS

Quality Data M&E Unit Intermedi aggregation (e.g., distr regions ate levels icts, ) Service delivery points

Data‐management and reporting system

Quality Standards Validity, Reliability, Precision, Integrity, Timeliness

Functional Components of Data‐ Management System Needed to Ensure Data Quality I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

Source: Adapted from Measure Evaluation

slide-23
SLIDE 23

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

DQA Requirements for FFP Development Projects Data Quality Standards

23

Validity Reliability Precision Integrity

Timeliness

slide-24
SLIDE 24

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Case Example

You are the DQA team leader for a FFP development food assistance project. You are verifying the quality of the data for the following indicator: Number of kilograms (kg) produced as a result of participation in project’s technology transfer

  • Tilapia
  • Maize

24 Photos: Jessica Scranton, FANTA

slide-25
SLIDE 25

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Number of kg of tilapia/maize produced as a result

  • f participation in project’s technology transfer

25

Are we measuring

Validity

what we believe we are measuring?

slide-26
SLIDE 26

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 26

Validity

Key Functional Components of a Data‐Management System that Impact Validity

I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

slide-27
SLIDE 27

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Number of kg of tilapia/maize produced as a result

  • f participation in project’s technology transfer

Reliability

Do data reflect stable and consistent definitions and data collection processes and analysis methods over time?

FOOD FOR PEACE DATA QUALITY ASSESSMENTS 27

slide-28
SLIDE 28

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 28

Reliability

Key Functional Components of a Data‐Management System that Impact Reliability

I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

slide-29
SLIDE 29

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Number of kg of tilapia/maize produced as a result

  • f participation in project’s technology transfer

Precision

Do data have a sufficient level of detail to permit management decision making and/or comply with reporting requirements? E.g. level of disaggregation, avoid over

  • r underreporting.

FOOD FOR PEACE DATA QUALITY ASSESSMENTS 29

slide-30
SLIDE 30

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 30

Precision

Select Key Functional Components of a Data‐ Management System that Impact Precision

I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

slide-31
SLIDE 31

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 31

Number of kg of tilapia/maize produced as a result

  • f participation in project’s technology transfer

Integrity

Do the data collected, analyzed, and reported have established mechanisms in place to reduce manipulation or simple errors in transcription?

slide-32
SLIDE 32

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 32

Integrity

Select Key functional Components of a Data‐Management System that Impact Integrity

I. M&E structures, functions and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

slide-33
SLIDE 33

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 33

Number of kg of tilapia/maize produced as a result

  • f participation in project’s technology transfer

Timeliness

Are data available at a useful frequency? Are data current and timely enough to influence management decision making?

slide-34
SLIDE 34

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 34

Select Key Functional Components of a Data‐Management System that Impact Timeliness

I. M&E structures, functions, and capabilities II. Indicator definitions and reporting guidelines III. Data collection tools and reporting forms IV. Processes of data verification, aggregation, processing, management, storage, and safeguarding V. Data use and dissemination VI. Links with national reporting systems (where relevant)

Timeliness

slide-35
SLIDE 35

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Illustrative DQA Process

  • Step 1. Develop an overall approach and schedule
  • Step 2. Identify the indicators and sites to be included

in the review

  • Step 3. Identify the DQA team
  • Step 4. Develop a budget and logistics plan
  • Step 5. Develop and pilot DQA tools or instruments
  • Step 6. Train DQA reviewers

35

slide-36
SLIDE 36

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

  • Step 7. Conduct the DQA
  • Step 8. Prepare DQA draft report
  • Step 9. Report review
  • Step 10. Follow up on Actions
  • Step 11. Submit DQA report to FFP through FFPMIS

as part of the Annual Results Report

FOOD FOR PEACE DATA QUALITY ASSESSMENTS 36

Illustrative DQA Process

slide-37
SLIDE 37

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS 37

Sample DQA Checklist

slide-38
SLIDE 38

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Potential Pitfalls to Avoid

  • 1. Data security issues
  • 2. Lack of assigned budget and personnel for

annual DQA

  • 3. Lack of data traceability standards
  • 4. Filing system inconsistencies
  • 5. Incomplete requirements in collection forms
  • 6. Insufficient training and knowledge refreshers

38

slide-39
SLIDE 39

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Potential Pitfalls to Avoid

  • 7. Lack of standardized processes/tools/indicators

definitions

  • 8. Inconsistent data‐collection methodology among

prime and sub‐awardees (for projects run by consortium)

  • 9. Insufficient standards to verify and cross‐check

data 10.Insufficient follow up on actions

39

slide-40
SLIDE 40

FOOD FOR PEACE DATA QUALITY ASSESSMENTS FOOD FOR PEACE DATA QUALITY ASSESSMENTS

Resources

  • USAID. Draft. USAID’s Office of Food for Peace Policy and Guidance for

Monitoring, Evaluation, and Reporting for Development Food Assistance Projects, Section 3.2: Data Quality Assurance, Management, and Safeguard.

  • USAID. 2012. ADS Chapter 203: Assessing and Learning. Available at:

http://www.usaid.gov/sites/default/files/documents/1870/203.pdf

  • Measure Evaluation. Data Quality Assurance Tools. Available at:

http://www.cpc.unc.edu/measure/resources/tools/monitoring‐ evaluation‐systems/data‐quality‐assurance‐tools

  • USAID. 2010. Performance Monitoring & Evaluation TIPS: Conducting Data

Quality Assessments. Available at: http://pdf.usaid.gov/pdf_docs/Pnadw118.pdf

  • USAID. 2009. Performance Monitoring & Evaluation TIPS: Data Quality
  • Standards. Available at: http://pdf.usaid.gov/pdf_docs/Pnadw112.pdf

40

slide-41
SLIDE 41

FOOD FOR PEACE DATA QUALITY ASSESSMENTS

This presentation is made possible by the generous support of the American people through the support of the Office of Health, Infectious Diseases and Nutrition, Bureau for Global Health, U.S. Agency for International Development (USAID) and the Office of Food for Peace, Bureau for Democracy, Conflict and Humanitarian Assistance, under terms of Cooperative Agreement No. AID‐OAA‐A‐ 12‐00005, through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360. The contents are the responsibility of FHI 360 and do not necessarily reflect the views of USAID or the United States Government.

41