Can ensemble machine learning improve the accuracy of severe maternal morbidity screening in a perinatal database

Background Severe maternal morbidity (SMM) is an important maternal health indicator, but existing tools to identify SMM have substantial limitations. Our objective was to retrospectively identify true SMM status using ensemble machine learning in a hospital database and to compare machine learning algorithm performance with existing tools for SMM identification. Methods We screened all deliveries occurring at Magee-Womens Hospital, Pittsburgh, PA (2010-2011 and 2013-2017) using the CDC list of diagnoses and procedures for SMM, intensive care unit (ICU) admission, and/or prolonged postpartum length of stay (PPLOS). We performed detailed medical record review to confirm case status. We trained ensemble machine learning (SuperLearner) algorithms, which "stack" predictions from multiple algorithms to obtain optimal predictions, on 171 SMM cases and 506 non-cases from 2010-2011, then evaluated the performance of these algorithms on 160 SMM cases and 337 non-cases from 2013-2017. Results Some SuperLearner algorithms performed better than existing screening criteria in terms of positive predictive value (0.77 versus 0.64, respectively) and balanced accuracy (0.99 versus 0.86, respectively). However, they did not perform as well as the screening criteria in terms of true-positive detection rate (0.008 versus 0.32, respectively) and performed similarly in terms of negative predictive value. The most important predictor variables were ICU admission and PPLOS. Conclusions Ensemble machine learning did not globally improve the ascertainment of true SMM cases. Our results suggest that accurate identification of SMM likely will remain a challenge in the absence of a universal definition of SMM or national obstetric surveillance systems.