Ranking Hospitals Based on Preventable Hospital Death Rates: A Systematic Review With Implications for Both Direct Measurement and Indirect Measurement Through Standardized Mortality Rates

Objectives There is interest in monitoring avoidable or preventable deaths measured directly or indirectly through standardized mortality rates (SMRs). We reviewed studies that use implicit case note reviews to estimate the range of preventable death rates observed, the measurement characteristics of those estimates, and the measurement procedures used to generate them. We comment on the implications for monitoring SMRs and illustrate a way to calculate the number of reviews needed to establish a reliable estimate of preventability of one death or the hospital preventable death rate. Design Systematic review of the literature supplemented by re-analysis of authors previously published and un-published data and measurement design calculations. Data source Searches in PubMed, MEDLINE (OvidSP) and Web of Knowledge in June 2012, updated December 2017. Eligibility criteria Studies of hospital-wide admissions from general and acute medical wards where preventable deaths rates are provided or can be estimate and which can provide inter- observer variations. Results Twenty-four studies were included from 1983-2017. Recent larger studies suggest consistently low rates of preventable deaths (3.0-6.5% since 2012). Reliability of a single review for distinguishing between individual cases with regard to the preventability of death had a Kappa rate of 0.27-0.50 for deaths and 0.24-0.76 for adverse events. A Kappa of 0.35 would require an average of 8-17 reviews of a single case to be precise enough to have confidence about high stakes decisions to change care procedures or impose sanctions within a hospital as a result. No study estimated the variation in preventable deaths across hospitals, although we were able to re-analyse one study to obtain an estimate. Based on this estimate, 200-300 total case-note reviews per hospital could be required to reliably distinguish between hospitals. The studies display considerable heterogeneity: 13/24 studies defined preventable with a threshold of ≥4 in a six-category Likert scale; 11/24 involved a two-stage screening process with nurses at the first stage and physicians at the second. Fifteen studies provided expert clinical review support for reviewer disagreements, advice, or quality control. A ‘generalist/internist’ was the modal physician specialty for reviewers and they received 1-3 days of generic tools orientation and case-note review practice. Methods did not consider the influence of human or environmental factors. Conclusions The literature provides limited information about the measurement characteristics of preventable deaths that suggests substantial numbers of reviews may be needed to create reliable estimates of preventable deaths at the individual or hospital level. Any operational program would require population specific estimates of reliability. Preventable death rates are low, which is likely to make it difficult to use SMRs based on all deaths to validly profile hospitals. The literature provides little information to guide improvements in the measurement procedures. Systematic review registration The systematic review was conceived prior to PROSPERO, and so has not been registered.