Data quality and reliability metrics for event-related potentials (ERPs): The utility of subject-level reliability.

Abstract Event-related brain potentials (ERPs) represent direct measures of neural activity that are leveraged to understand cognitive, affective, sensory, and motor processes. Every ERP researcher encounters the obstacle of determining whether measurements are precise or psychometrically reliable enough for an intended purpose. In this primer, we review three types of measurements metrics: data quality, group-level internal consistency, and subject-level internal consistency. Data quality estimates characterize the precision of ERP scores but provide no inherent information about whether scores are precise enough for examining individual differences. Group-level internal consistency characterizes the ratio of between-person differences to the precision of those scores, and provides a single internal consistency estimate for an entire group of participants that risks masking low internal consistency for some individuals. Subject-level internal consistency considers the precision of an ERP score for a person relative to between-person differences for a group, and an estimate is yielded for each individual. We apply each metric to published error-related negativity (ERN) and reward positivity (RewP) data and demonstrate how failing to consider data quality and internal consistency can undermine statistical inferences. We conclude with general comments on how these estimates may be used to improve measurement quality and methodological transparency. Subject-level internal consistency computation is implemented within the ERP Reliability Analysis (ERA) Toolbox.