A Deep Learning Architecture for Heterogeneous and Irregularly Sampled Remote Sensing Time Series

Remote sensing present some new challenges for deep learning, because (also to compensate the scarce detail level) multimodal, multisource and multitemporal data should be jointly exploited. For example, time series of optical multispectral/hyperspectral or synthetic aperture radar (SAR) data probe different properties of the observed scene, based on their different wavelength, acquisition geometry, etc., and with possible data gaps.To address this task, we propose a new deep learning architecture that exploits a sequence of deep convolutional neural networks (CNN) and a recurrent neural network (RNN). In the proposed architecture, all the data (with their spectral, spatial and temporal information) are used jointly and optimally in the sense that no imputation is enforced, but the internal weights providing the best classification results are estimated from the data themselves (hence the proposed name ODIN - Optimal Data Imputation Network).We have tested the proposed architecture, using Sentinel SAR and multispectral image series, on land cover and crop classification, an important remote sensing application. The obtained results are very promising, with an error rate below 1%, and show good spatial consistency without loss of spatial resolution.