Semiparametric Bayesian analysis of transformation linear mixed models

Abstract In classical linear mixed models (LMMs), it is commonly assumed that the random effects and within-individual errors independently follow a Gaussian distribution. However, in some applications, this assumption may be inappropriate. To this end, this paper proposes a novel LMM by assuming that the random effects follow an unknown distribution, and the within-individual errors associated with the transformed responses are Gaussian. A semiparametric Bayesian approach is developed to make Bayesian inference on the novel LMM by using the truncated centered Dirichlet Process prior to approximate the unknown distribution of the random effects and using Bayesian P-splines to approximate the transformation function, and combining the Gibbs sampler and the Metropolis–Hastings algorithm. A Bayesian local influence analysis method is developed to assess the effect of minor perturbations. Simulation studies and an example are used to illustrate the proposed methodologies.