Panchromatic Side Sparsity Model-Based Deep Unfolding Network for Pansharpening

Deep learning (DL) recently receives state-of-the-art results in pansharpening. However, most of the existing pansharpening neural networks are purely data-driven, without taking account of the characteristics of the pansharpening task. To address this issue, we propose a novel deep unfolding network for pansharpening that combines the insight of the variational optimization (VO) model and the capability of deep neural network (DNN). We first develop a panchromatic side sparsity (PASS) prior-based VO model for pansharpend image reconstruction, which is formulated as the $\ell _{1}-\ell _{1}$ minimization. In particular, the PASS prior is defined using the transform sparsity, which can alleviate the influences of the irregular outliers between the multispectral (MS) and panchromatic (PAN) images. The iterations of half-quadratic splitting algorithm for solving the $\ell _{1}-\ell _{1}$ minimization are then deeply unfolded into a DNN, referred as PASS-Net. To capture the nonlinear relationship between MS and PAN images, the linear transforms used in PASS prior are extended into the subnetworks in PASS-Net. Moreover, a pair of learnable downsampling and upsampling modules are designed to realize the downsampling and upsampling operations, which can improve the flexibility. The experimental results on different satellite datasets confirm that PASS-Net is superior to some representational traditional methods and state-of-the-art DL-based methods.