The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt.
Cécile Viboud, Kaiyuan Sun, Robert Gaffey, Marco Ajelli, Laura Fumanelli, Stefano Merler, Qian Zhang, Gerardo Chowell, Lone Simonsen, Alessandro Vespignani, RAPIDD Ebola Forecasting Challenge group
Author Information
Cécile Viboud: Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA. Electronic address: viboudc@mail.nih.gov.
Kaiyuan Sun: Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA.
Robert Gaffey: Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA.
Marco Ajelli: Bruno Kessler Foundation, Trento, Italy.
Laura Fumanelli: Bruno Kessler Foundation, Trento, Italy.
Stefano Merler: Bruno Kessler Foundation, Trento, Italy.
Qian Zhang: Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA.
Gerardo Chowell: Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA; School of Public Health, Georgia State University, Atlanta, GA, USA.
Lone Simonsen: Division of International Epidemiology and Population Studies, Fogarty International Center, National Institutes of Health, Bethesda, MD, USA; Department of Global Health, George Washington University, Washington DC, USA.
Alessandro Vespignani: Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA, USA; Institute for Quantitative Social Sciences at Harvard University, Cambridge, MA, USA; Institute for Scientific Interchange Foundation, Turin, Italy.
Infectious disease forecasting is gaining traction in the public health community; however, limited systematic comparisons of model performance exist. Here we present the results of a synthetic forecasting challenge inspired by the West African Ebola crisis in 2014-2015 and involving 16 international academic teams and US government agencies, and compare the predictive performance of 8 independent modeling approaches. Challenge participants were invited to predict 140 epidemiological targets across 5 different time points of 4 synthetic Ebola outbreaks, each involving different levels of interventions and "fog of war" in outbreak data made available for predictions. Prediction targets included 1-4 week-ahead case incidences, outbreak size, peak timing, and several natural history parameters. With respect to weekly case incidence targets, ensemble predictions based on a Bayesian average of the 8 participating models outperformed any individual model and did substantially better than a null auto-regressive model. There was no relationship between model complexity and prediction accuracy; however, the top performing models for short-term weekly incidence were reactive models with few parameters, fitted to a short and recent part of the outbreak. Individual model outputs and ensemble predictions improved with data accuracy and availability; by the second time point, just before the peak of the epidemic, estimates of final size were within 20% of the target. The 4th challenge scenario - mirroring an uncontrolled Ebola outbreak with substantial data reporting noise - was poorly predicted by all modeling teams. Overall, this synthetic forecasting challenge provided a deep understanding of model performance under controlled data and epidemiological conditions. We recommend such "peace time" forecasting challenges as key elements to improve coordination and inspire collaboration between modeling groups ahead of the next pandemic threat, and to assess model forecasting accuracy for a variety of known and hypothetical pathogens.
