Calibration of individual-based models to epidemiological data: a systematic review

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of the calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs in HIV, tuberculosis, and malaria epidemiology that were published between 2013 and 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies and GOF measures. We also recorded the reporting of model validation. The PubMed search identified 653 candidate articles, of which 84 met the criteria for review. Of the included articles, 40 (48%) combined a quantitative measure of GOF with an algorithmic parameter-search strategy, which was an optimisation algorithm (14 articles) or a sampling algorithm (26 articles). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 articles, the parameter-search strategy could either not be identified (32 articles) or was described as an informal, non-reproducible method (12 articles). Of these 44 articles, the majority (25 articles) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Thirty-two (38%) articles reported on model validation. Less than half of the articles reviewed used algorithmic parameter-search strategies. Only one-third of articles quantified the uncertainty around calibrated parameter values and could incorporate this uncertainty into their model projections. There was no consensus on which algorithmic calibration method to use. Only 38% of the articles performed model validation. The adoption of better-documented algorithmic calibration and validation methods could improve both the reproducibility and the quality of inference in model-based epidemiology.