Multi-scale hierarchical recurrent neural networks for hyperspectral image classification

Abstract This paper presents a novel hyperspectral image (HSI) classification framework by exploiting multi-scale spectral-spatial features via hierarchical recurrent neural networks. The neighborhood information plays an important role in the image classification process. Convolutional neural networks (CNNs) have been shown to be effective in learning the local features of HSI. However, CNNs do not consider the spatial dependency of non-adjacent image patches. Recurrent neural networks (RNNs) can effectively establish the relationship of non-adjacent image patches, but it can only be applied to single-dimensional (1D) sequence. In this paper, we propose multi-scale hierarchical recurrent neural networks (MHRNNs) to learn the spatial dependency of non-adjacent image patches in the two-dimension (2D) spatial domain. First, to better represent the objects with different scales, we generate multi-scale 3D image patches of central pixel and surrounding pixels. Then, 3D CNNs extract the local spectral-spatial feature from each 3D image patch, respectively. Finally, multi-scale 1D sequences in eight directions are constructed on the 3D local feature domain, and MHRNNs are proposed to capture the spatial dependency of local spectral-spatial features at different scales. The proposed method not only considers the local spectral-spatial features of the HSI, but also captures the spatial dependency of non-adjacent image patches at different scales. Experiments are performed on three real HSI datasets. The results demonstrate the superiority of the proposed method over several state-of-the-art methods in both visual appearance and classification accuracy.