Assessing EARS Ability to Locally Detect the 2009 H1N1 Pandemic Ron - - PowerPoint PPT Presentation

Assessing EARS’ Ability to Locally Detect the 2009 H1N1 Pandemic

Ron Fricker, Katie Hagen, Krista Hanni, Susan Barnes, and Kristy Michie January 9, 2011

Clinic 1 Clinic 2 Clinic 3 Clinic 4 Clinic 5 Clinic 6

Early Aberration Reporting System

• EARS designed as a drop-in surveillance system
• But also used by local public health departments for

routine health surveillance

– Including Monterey County Health Department (MCHD)

2

MCHD’s “Dose Report”

3

Research Goal

• Assess how well the Early Aberration

Reporting System (EARS) can detect known outbreaks (seasonal ILI and H1N1)

– Including two EARS variants implemented by the Monterey County Health Department – Compared to an alternative CUSUM-based methodology

4

Deriving Daily Syndrome Counts

• County clinics and hospital ERs send chief

complaint data to MCHD on daily basis

– Weekdays less holidays – Examples:

5

• Text matching searches for terms in the data to derive

symptoms

– E.g., existence of word “flu” results in classifying an individual with the flu symptom

• Symptoms then used to determine whether to classify

an individual with a syndrome

• MCHD has used three definitions for ILI syndrome:

Deriving Daily Syndrome Counts

6

Detection Algorithms

Base Case Expanded Restricted

9,093 total ILI 5,154 added 291 deleted 13,956 total ILI (53% ↑) 51 added 8,410 deleted 734 total ILI (92% ↓)

ILI Definition Counts

Out of 153,696 total records…

7

Daily Count Time Series

8

Early Aberration Reporting System

• EARS’ detection algorithms:
• Often referred to as CUSUMs, but not true
• In SPC parlance, C1 and C2 are Shewhart

variants

9 1 1 1

( ) ( ) ( ) ( ) Y t Y t C t s t  

3 2 3

( ) ( ) ( ) ( ) Y t Y t C t s t  

 

2 3 2

( ) max 0, ( ) 1

t i t

C t C i

 

 



• Sample statistics calculated from

previous 7 days’ data

• Signal when C1 > 3
• Sample statistics calculated from

7 days’ of data prior to 2 day lag

• Signal when C2 > 3
• Signal when C3 > 2

• Adaptive regression: regress a sliding baseline of
• bservations on time relative to current observation

– I.e. regress on

• Calculate standardized residuals from one day ahead

forecast, , where and

• CUSUM:

where a signal is generated if S(t)>h

CUSUM on Adaptive Regression Forecast Errors

10

( 1),..., ( ) Y t Y t n   ,...,1 n

1

ˆ ˆ ˆ ( ) ( ) ( 1)

j

R t Y t n             

ˆ ( ) ( ) /

Y

Z t R t  

 

( ) max 0, ( 1) ( ) S t S t Z t k    

• We looked at the performance of three

CUSUMs based on choices of k and h:

– Smaller k: Can detect smaller increases in mean – Larger h: Fewer false positive signals (i.e., larger ATFS) but slower to signal

Three CUSUMs Evaluated

11

Determining the Outbreak Periods

12

Baseline ILI Definition Results

13

Expanded ILI Definition Results

14

Restricted ILI Definition Results

15

• Metrics:

– Sensitivity: # outbreak days with signal / # outbreak days – Specificity: # non-outbreak days without signal / # non-outbreak days – Average delay:

• average time to signal from start of outbreak period
• average time to signal from earliest signal
• Results:

Quantifying Performance

16

1

d

2

d

Conclusions: ILI Definitions

• With EARS algorithms:

– Baseline definition did not perform well – Expanded definition performed even worse – Restricted definition gave best performance

• With CUSUM algorithms

– Choice of ILI definition seems less critical

• Hard to conclude CUSUM performed better under one

definition than the others

• Suggests CUSUM is robust – a good characteristic since

connection of definition to arbitrary outbreak type tenuous

• More syndrome definition research warranted

– “Low hanging fruit” to improve performance

17

Conclusions: Algorithms

• Consider incorporating (real) CUSUM

detection algorithms into EARS

– SPC literature shows that CUSUM methods can detect smaller mean increases than Shewhart methods (such as C1 and C2)

• For routine surveillance applications, at

least incorporate methods that use more historical data

18