Surveillance system evaluation Potjaman Siriarayapon, MD, FETP, - - PowerPoint PPT Presentation

Surveillance system evaluation

Potjaman Siriarayapon, MD, FETP, DrPH

Why do we evaluate surveillance Why do we evaluate surveillance system? system?

Number of acute diarrhea cases in 2002 compare with median 5 years (1997-2001), Akat-Umnuay District, Sakonnakorn

97 180 96 80 122 161 48 23 61 45 70 35 78 14 18 18 17 66

50 100 150 200 Jan Feb Mar Apr May June July Aug Sep Oct Mov Dec

Source : Akat-Umnuay Health Office, Sakonnakorn Median 5 yrs 2002

Evaluation

Systematic investigation of the merit (quality), worth (cost-effectiveness), or significant (importance) of an object

Questions to be answered? Questions to be answered?

• What will be evaluated?
• What aspects of the program will be

considered?

• What standard must be reached for

considered successful program?

Questions to be answered? Questions to be answered?

• What evidence will be used to indicate how the

program has performed?

• What conclusion regarding program

performance are justified?

• How will the lessons learned from the inquiry be

used to improve public health effectiveness?

Program evaluation

Input Process Output

Man Money Management (Context) Detail step

• f working

Outcome Effect Impact

Framework for program evaluation in public health

Engage Stakeholders Describe the program Focus the evaluation design Gather credible evidence Justify conclusions Ensure use and share lessons learned

Purpose of evaluating public health surveillance system

To ensure that problems of public health importance are being monitored efficiently efficiently and effectively effectively

Engage the stakeholders in the evaluation

• Those involved in program operations
• Those served or effected by the program
• Primary user of the evaluation

Describe the surveillance system to be evaluated

• Describe the public health importance of the

disease (or health-related event) under surveillance

• Describe the purpose and operation of the

system

• Describe the resources used to operate the

system

• A. Public health importance
• Disease frequency (number of cases/death,

incidence, prevalence, mortality rate)

• Severity (case-fatality ratio, hospitalization rate,

disability rate)

• Preventability
• Public interest

• B. Purpose and operation of the

surveillance system (1)

• List the purpose and objectives of the system
• Describe the planned uses of the data from the

system

• Describe the disease under surveillance,

including case definition

• Cite any legal authority for the data collection

• B. Purpose and operation of the

surveillance system (2)

• Where in the organization the system resides
• Level of integration with other systems
• Flow chart
• Component of the system

– Population under surveillance – Period of time of data collection – Collected what data and how – …

• C. Resources used to operate the

surveillance system

• Funding source
• Personnel
• Other resources

– Travel – Training – Supplies – Computer and other equipment – Related services (e.g., mail, telephone, internet, laboratory support, hardware & software maintenance)

Focus the evaluation design

• Determining the specific purpose of the

evaluation

• Identifying stakeholders
• Considering what will be done with the result
• Determining standards for assessing the

performance of the system

Gathering credible evidence

• Usefulness
• System attribute

– Simplicity

• Sensitivity

– Flexibility

• Predictive value positive

– Data quality

• Representativeness

– Acceptability

• Timeliness

– Stability

About public health surveillance

• Surveillance for action!!!
• How?

– Early detection of abnormal event – Estimated magnitude of morbidity & mortality – Detect trend – Permit assessment of preventive & control measures – Stimulate research intended to lead to prevention and control

Usefulness Usefulness

• Describing the “actions taken”

as a result of the data from the surveillance system

• Should begin with review
• bjectives of the system and

assess whether the utilization

• f the system meet the
• bjectives

Sensitivity Sensitivity

Can be considered at 2 levels

• Proportion of cases detect by the surveillance

system

• Ability to detect outbreak,

Comparing data from surveillance system with the real situation

Surveillance system

Case Non-case Non-case Case

A B C D

Real situation

Sensitivity = A A+C

Sensitivity of suspected avian influenza reported in Nakonnayok province by district, 2005

80.0% 50.0% 76.9% 76.9% 0.0% 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% เมือง บานนา องครักษ มศว ปากพลี โรงพยาบาล รอยละ

Predictive value positive (PVP) Predictive value positive (PVP)

• Proportion of reported cases of Disease A

that really be the Disease A

• PVP provide different perspectives to tell

how well the system is functioned

Comparing data from surveillance system with the real situation

Surveillance system

Case Non-case Non-case Case

A B C D

Real situation

PVP = A A+B

Predictive value positive of suspected avian influenza cases

Suspected AI cases reported to central level Met case definition

87

Not met case definition

8

Total

95

PVP of suspected Avian influenza cases

= 87/95 = 91.58%

Representativeness Representativeness

• Represent disease occurrence over time

and distribution

• These characteristic generally related to

time, place, person

Attack rate of severe diarrhea cases by sub-district, Akat Umnuay District (per 100,000 population)

Timeliness Timeliness

• Reflect speed between steps in a public

health surveillance system

• Should be evaluated in terms of availability
• f information to control the disease

Simplicity Simplicity

• Refer to both its structure and ease of
• peration
• Surveillance system should be as

simple as possible while still meeting their objectives

Flexibility Flexibility

A flexible public health surveillance system can adapt to changing information needs on operating conditions with little additional time, personnel or allocated funds.

Data quality Data quality

Reflect completeness and validity

• f the data from public health

surveillance

Acceptability Acceptability

Willingness of persons and

• rganizations to participate in the

surveillance system

Stability Stability

Refer to reliability (ability to collect, manage, and provide data properly without failure) and availability (ability to be

• perational when needed) of the public

health surveillance system

Example of different surveillance Example of different surveillance evaluation evaluation

• Main surveillance system
• HIV/AIDS surveillance system
• EWARS

Thank you for your attention Thank you for your attention