Forecasting based on surveillance data Sebastian Meyer Institute of - - PowerPoint PPT Presentation

▶

Jul 07, 2023 202 likes •408 views

Forecasting based on surveillance data Sebastian Meyer Institute of Medical Informatics, Biometry, and Epidemiology Friedrich-Alexander-Universitt Erlangen-Nrnberg, Erlangen, Germany GEOMED 2019, Glasgow, 27 August 2019 Based on joint work

SLIDE 1

Forecasting based on surveillance data

Sebastian Meyer Institute of Medical Informatics, Biometry, and Epidemiology Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany GEOMED 2019, Glasgow, 27 August 2019 Based on joint work with Leonhard Held (University of Zurich) and Junyi Lu (FAU Erlangen-Nürnberg)

SLIDE 2

Epidemics are hard to predict

World Health Organization (2014)

Forecasting disease outbreaks is still in its infancy, however, unlike weather forecasting, where substantial progress has been made in recent years.

Meanwhile . . .

Epidemic Prediction Initiative in the USA (https://predict.cdc.gov/): online

platform to collect real-time forecasts from multiple research groups

Integration of social contact patterns (Meyer & Held, 2017), human

mobility data (Pei, Kandula, Yang, & Shaman, 2018), and internet data (Osthus, Daughton, & Priedhorsky, 2019)

Adoption of forecast assessment techniques from weather forecasting

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 1

SLIDE 3

“Forecasts should be probabilistic” (Gneiting & Katzfuss, 2014)

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 2

SLIDE 4

Proper scoring rules S(F,y)

Quantify discrepancy between forecast F and observation y
“Proper”: forecasting with true distribution is optimal
Most scoring rules are easy to compute:
Squared error score: SES(F,y) = (y − µF)2
Logarithmic score: LS(F,y) = −logf(y)
Dawid-Sebastiani score: DSS(F,y) = log(σ 2

F)+ (y−µF )2

σ 2

Scoring rules summarize two complementary measures of forecast quality:
Sharpness: width of prediction intervals (property of F)
Calibration: statistical consistency of forecast F and observation y

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 3

SLIDE 5

Histogram of F(y) = PIT (probability integral transform) values

PIT Density

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0

verestimation

underdispersed forecasts

verdispersed

forecasts

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 4

SLIDE 6

Case study I: Weekly ILI counts in Switzerland, 2000–2016

10 100 1 000 10 000 100 000 2001 2003 2005 2007 2009 2011 2013 2015 2017

Time (weekly) ILI counts

Compute one-week-ahead forecasts in the test period (from December 2012)
Compare average scores between different models

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 5

SLIDE 7

Useful statistical models to forecast epidemic spread

Scope: well-documented open-source R implementations
We compare five different models:
forecast::auto.arima() for log-counts → ARMA(2,2)
glarma::glarma() → NegBin-ARMA(4,4)
surveillance::hhh4(): “endemic-epidemic” NegBin model (lag 1)
Kernel conditional density estimation (kcde) by Ray et al. (2017)
prophet::prophet() for log-counts: harmonic regression with changepoints
Naive historical reference forecast: log-normals by calendar week

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 6

SLIDE 8

Performance of 213 one-week-ahead forecasts

Method RMSE DSS LS runtime [s]

arima

2287 13.78 7.73 0.51

glarma

2450 13.59 7.71 1.49

hhh4

1769 13.58 7.71 0.02

kcde

1963 13.79 7.80 1128

prophet

5614 15.00 8.03 3.01 naive 5010 14.90 8.06 0.00

Runtimes vary considerably (time for single refit and forecast)
The two autoregressive NegBin models score best
Non-dynamic methods: prophet does not outperform naive forecasts

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 7

SLIDE 9

PIT histogram for hhh4-based one-week-ahead forecasts

PIT Density

0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.5 1.0 1.5 2.0

Some evidence of miscalibration

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 8

SLIDE 10

hhh4-based one-week-ahead forecasts

10 100 1000 10000 100000

surveillance::hhh4() ILI counts

1% 25% 50% 75% 99%

Score

15 30 DSS (mean: 13.58) LS (mean: 7.71) 2013 2014 2015 2016

Relatively sharp forecasts → penalty in wiggly off-season 2016
Off-season counts tend to be lower than predicted

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 9

SLIDE 11

Case study II: Weekly ILI activity in the USA, 1998–2018

2 4 6 8 2001 2005 2009 2013 2017

wILI (%)

Inspired by CDC’s FluSight competition (https://predict.cdc.gov/)
Forecast ILI proportion 1 to 4 weeks ahead, plus peak week & proportion

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 10

SLIDE 12

Seasonal epidemic curves

14/15 15/16 16/17 17/18

2 4 6 8 10 20 30 40

Season week wILI (%)

(Intermediate) peak at the end of the year (dashed line)
Test seasons with late peak (15/16) and high intensity (17/18)

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 11

SLIDE 13

Forecasting machinery

Gaussian models of logit-transformed proportions:

[S]ARIMA, Prophet, naive historical

Kernel conditional density estimation (KCDE)
hhh4 not applicable for proportions

→ Idea: “Endemic-epidemic” beta regression (Beta(p)), via betareg:

Xt|Ft−1 ∼ Beta(µt,φt) logit(µt) = νt +

∑

k=1

βk logit(Xt−k) νt = α(ν) +β (ν)T

z(ν)

log(φt) = α(φ) +β (φ)T z(φ)

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 12

SLIDE 14

Overall performance of short-term forecasts (all horizons)

Method DSS LS max(LS) runtime [min] #par ARIMA(5,1,0) –1.81 –0.02 5.24 6.2 16 SARIMA(1,0,0)(1,1,0)[52] –1.69 0.04 4.92 110.4 3 Beta(1) –2.02 –0.11 5.59 2.9 19 Beta(4) –2.07 –0.12 4.34 2.6 20 KCDE –2.29 –0.12 4.08 266.6 28 Prophet –0.75 0.48 5.04 11.8 50 Naive –1.13 0.42 5.29 0.1 106

Runtimes vary considerably (total time for [re]fitting and forecasting)
Higher order lags improve Beta forecasts
Worst case prediction is less worse with KCDE than with Beta(4)
Non-dynamic methods: prophet does not outperform naive forecasts

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 13

SLIDE 15

Relative performance wrt Beta(4), by season and horizon

3 weeks ahead 4 weeks ahead 1 week ahead 2 weeks ahead

ARIMA KCDE ARIMA KCDE

−1.0 −0.5 0.0 0.5 1.0 −1.0 −0.5 0.0 0.5 1.0

Log score difference (positive favours Beta(4)) Season

2014/2015 2015/2016 2016/2017 2017/2018

No model consistently outperforms another, and rankings vary by season
KCDE tends to produce better 3- and 4-week-ahead forecasts

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 14

SLIDE 16

Overall performance of peak forecasts

Method Timing (LS) Intensity (LS) ARIMA(5,1,0) 1.44 1.59 SARIMA(1,0,0)(1,1,0)[52] 1.78 1.57 Beta(1) 1.99 1.46 Beta(4) 1.47 1.51 KCDE 1.43 1.41 Prophet 1.44 1.68 Naive 1.46 1.46 Equal bin (uniform) 3.50 3.30

KCDE has best peak forecasts overall
Naive historical forecasts are not that bad either

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 15

SLIDE 17

Relative performance wrt equal-bin forecast, by season

2014/2015 2015/2016 2016/2017 2017/2018 Peak Intensity Peak Timing

ARIMA SARIMA Beta(1) Beta(4) KCDE Prophet Naive ARIMA SARIMA Beta(1) Beta(4) KCDE Prophet Naive ARIMA SARIMA Beta(1) Beta(4) KCDE Prophet Naive ARIMA SARIMA Beta(1) Beta(4) KCDE Prophet Naive

−4 −2 2 4 −4 −2 2 4

Model LS diff (positive favours model)

2014/2015 2015/2016 2016/2017 2017/2018 10 20 30 40 10 20 30 40 10 20 30 40 10 20 30 40 2 4 6

Season week wILI (%)

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 16

SLIDE 18

Discussion

Endemic-epidemic approach useful for short-term forecasts:

fast, performant, and easy to implement

Peak prediction is hard: no model outperformed naive historical forecasts in all

seasons (KCDE did the best job)

Any missing competitive forecasting method with a well-documented

implementation in open-source software?

Ensemble forecasts (Reich et al., 2019)
Underreporting and reporting delays
Multivariate forecasting by region or age group

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 17

SLIDE 19

References

Gneiting, T., & Katzfuss, M. (2014). Probabilistic

forecasting. Annual Review of Statistics and Its

Application, 1(1), 125–151. https: //doi.org/10.1146/annurev-statistics-062713-085831 Held, L., & Meyer, S. (2019). Forecasting based on surveillance data. In L. Held, N. Hens, P . D. O’Neill, &

J. Wallinga (Eds.), Handbook of infectious disease

data analysis. Chapman & Hall/CRC. Meyer, S., & Held, L. (2017). Incorporating social contact data in spatio-temporal models for infectious disease spread. Biostatistics, 18(2), 338–351. https://doi.org/10.1093/biostatistics/kxw051 Osthus, D., Daughton, A. R., & Priedhorsky, R. (2019). Even a good influenza forecasting model can benefit from internet-based nowcasts, but those benefits are

limited. PLOS Computational Biology, 15(2), 1–19.

https://doi.org/10.1371/journal.pcbi.1006599 Pei, S., Kandula, S., Yang, W., & Shaman, J. (2018). Forecasting the spatial transmission of influenza in the United States. Proceedings of the National Academy

f Sciences of the United States of America, 115(11),

2752–2757. https://doi.org/10.1073/pnas.1708856115 Ray, E. L., Sakrejda, K., Lauer, S. A., Johansson, M. A., & Reich, N. G. (2017). Infectious disease prediction with kernel conditional density estimation. Statistics in Medicine, 36(30), 4908–4929. https://doi.org/10.1002/sim.7488 Reich, N. G., Brooks, L. C., Fox, S. J., Kandula, S., McGowan, C. J., Moore, E., . . . Shaman, J. (2019). A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the united states. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.1812594116

Data and reproduction code for case study I: https://HIDDA.github.io/forecasting/

Sebastian Meyer | FAU Erlangen-Nürnberg | Forecasting based on surveillance data GEOMED 2019, Glasgow 18