A Novel Multitemporal Deep Fusion Network (MDFN) for Short-Term Multitemporal HR Images Classification

High-resolution (HR) satellite images, due to the technical constraints on spectral and spatial resolutions, usually contain only several broad spectral bands but with a very high spatial resolution. This provides rich spatial details of the objects on the Earth surface, while their spectral discrimination is relatively low. Recently, the increase of the satellite revisit times made it possible to acquire more frequent data coverage for finer classification. In this article, we proposed a novel multitemporal deep fusion network (MDFN) for short-term multitemporal HR images classification. Specifically, a two-branch structure of MDFN is designed, which includes a long short-term memory (LSTM) and a convolutional neural network (CNN). The LSTM branch is mainly used to learn the joint expression of different temporal-spectral features. For the CNN branch, the three-dimensional (3-D) convolution is firstly applied along the temporal and spectral dimensions to jointly learn the temporal-spatial and spectral-spatial information, respectively, and then the 2-D convolution is performed along the spatial dimension to further extract the spatial context information. Finally, features generated from the two different branches are fused to obtain the discriminative high-level semantic information for classification. Experimental results carried on two real multitemporal HR remote sensing datasets demonstrate that the proposed MDFN provides better classification performance over the state-of-the-art methods, and it also shows the potentiality to use short-term multitemporal HR images for more accurate land use/land cover mapping.