Bayesian random effects for interrater and test–retest reliability with nested clinical observations

Abstract Objective The assessment of inter- and intrarater reliability usually involves more than one level of nesting structures in the collected data, where repeated observations are made by multiple raters. Most approaches, however, are not designed to accommodate both inter- and intrarater reliability jointly, not to mention further difficulties arising when modeling with dichotomous responses. The multiple sources of dependence because of nesting structures and the existence of covariates can result in complexity in inference. Study Design and Setting We first establish the equivalence between correlation and kappa under common positive correlation models for multiple raters and then apply a Bayesian generalized linear mixed-effects model to interpret simultaneously both types of reproducibility through different annotations of similarity. In addition to marginal correlations, the correlated random effects among raters are adopted to infer similarity between raters, whereas the correlation for random time effects may contribute to test–retest reliability. Results This model accounts for individual covariates and random effects because of subjects, raters, and time, and it covers a wide variety of data structures and types. An application of endodontic radiographic examinations is illustrated. Conclusion This Bayesian hierarchical correlation model offers a wide applicability, flexibility, and feasibility in modeling inter- and intrarater reliability together.