Predicting Esophageal Fistula Risks Using a Multimodal Self-attention Network.

Radiotherapy plays a vital role in treating patients with esophageal cancer (EC), whereas potential complications such as esophageal fistula (EF) can be devastating and even life-threatening. Therefore, predicting EF risks prior to radiotherapies for EC patients is crucial for their clinical treatment and quality of life. We propose a novel method of combining thoracic Computerized Tomography (CT) scans and clinical tabular data to improve the prediction of EF risks in EC patients. The multimodal network includes encoders to extract salient features from images and clinical data, respectively. In addition, we devise a self-attention module, named VisText, to uncover the complex relationships and correlations among different features. The associated multimodal features are integrated with clinical features by aggregation to further enhance prediction accuracy. Experimental results indicate that our method classifies EF status for EC patients with an accuracy of 0.8366, F1 score of 0.7337, specificity of 0.9312 and AUC of 0.9119, outperforming other methods in comparison.