A flexible multivariate random effects proportional odds model with application to adverse effects during radiation therapy

Radiation therapy in patients with head and neck cancer has a toxic effect on mucosa, the soft tissue in and around the mouth. Hence mucositis is a serious common side effect and is a condition characterized by pain and inflammation of the surface of the mucosa. Although the mucosa recovers during breaks of and following the radiotherapy course, the recovery will depend on the type of tissue involved and on its location. We present a novel flexible multivariate random effects proportional odds model that takes account of the longitudinal course of oral mucositis at different mouth sites and of the radiation dosage (in terms of cumulative dose). The model is an extension of the proportional odds model that is used for ordinal response variables. Our model includes the ordinal multivariate response of the mucositis score by site, random intercepts for individuals, and a nonlinear function of cumulative radiation dose. The model allows to test whether sensitivity differs by mouth sites after having adjusted for site-specific cumulative radiation dose. The model also allows to check whether and how the (nonlinear) effect of site-specific dose differs by site. We fit the model to longitudinal patient data from a prospective observation and find that after adjusting for cumulative dose, upper, lower lips, and mouth floor are associated with the lowest mucositis scores and hard and soft palate are associated with the highest mucositis scores. This implies the possibility that tissues at different mouth sites differ in their sensitivity to the toxic effect of radiation.