ROC plot and AUC with binary classifiers: pragmatic analysis of cognitive screening instruments

Receiver operating characteristic (ROC) plots are a performance graphing method showing the relative trade-off between test benefits (true positive rate) and costs (false positive rate) with the area under the curve (AUC) giving a scalar value of test performance. It has been suggested that ROC and AUC may be potentially misleading when examining binary predictors rather than continuous scales. The purpose of this study was to examine ROC plots and AUC values for two binary classifiers of cognitive status (applause sign, attended with sign), a cognitive screening instrument producing categorical data (Codex), and a continuous scale screening test (Mini-Addenbrooke's Cognitive Examination), the latter two also analysed with single fixed threshold tests. For each of these plots, AUC was calculated using different methods. The findings indicate that if categorical or continuous measures are dichotomised then the calculated AUC may be an underestimate, thus affecting screening or diagnostic test accuracy which in the context of clinical practice may prove to be misleading.