A Unified Multimodal Deep Learning Framework For Remote Sensing Imagery Classification

In this paper, we present a unified deep learning framework for multimodal remote sensing image classification, U-MDL for short. U-MDL attempts to develop a general network architecture that consists of two subnetworks for feature extraction and feature fusion, respectively, with a focus on "which", "when", and "how" to fuse. For this purpose, we detail several common but effective fusion modules in the networks, e.g., early fusion, middle fusion, late fusion, and encoder-decoder fusion. These modules can be generalized well into our U-MDL framework. More significantly, we also emphasize to investigate a special case of multi-modality learning (MML), that is, cross-modality learning (CML) which widely exists in real applications. Moreover, extensive experiments are conducted to demonstrate the superiority and effectiveness of the proposed U-MDL framework in the remote sensing image classification task. The codes and datasets are available at: https://github.com/danfenghong/IEEE_TGRS_MDL-RS for the sake of reproducibility.