A multi-scale strategy for deep semantic segmentation with convolutional neural networks

Abstract A novel multi-scale scheme is proposed to improve the performance of deep semantic segmentation based on Convolutional Neural Networks(CNNs). The fundamental idea is to combine the information from different intermediate layers by introducing new multi-scale loss(m-loss) function. We also show that it can be calculated by three different modules. The advantage of m-loss functions is that the loss of all layers could be optimized in one-shot without additional modifications of the training algorithm. The proposed strategy is also applied to improve the performance of Unet and FCN, and the structures of multi-scale loss functions are presented as well. Numerical validations are performed on two datasets, including the benchmark Pascal VOC 2012 dataset and the PICC dataset from medical treatment. It is illustrated that our multi-scale approach yields faster learning convergence rate and better accuracy.