Categorical perception of p-values

Author(s): Rao, V. N. Vimal; Bye, Jeffrey K.; Varma, Sashank | Abstract: Traditional statistics instruction emphasizes a .05 significance level for hypothesis tests. Here, we investigate the consequences of this training for researchers’ mental representations of probabilities –– whether .05 becomes a boundary, i.e., a discontinuity of the mental number line, and alters their perception of differences between p-values. Graduate students (n = 25) with statistical training viewed pairs of p-values and judged whether they were ‘similar’ or ‘different’. After controlling for covariates, participants were more likely and faster to judge p-values as ‘different’ when they crossed the .05 boundary (e.g., .047 vs. .052) compared to when they did not (e.g., .027 vs. .032). This categorical perception effect suggests that traditional statistical instruction creates a psychologically real divide between so-called statistically significant and non-significant p-values. Such a distortion is undesirable given modern approaches to statistical reasoning that de-emphasize dichotomizing p-values.