Multistage Dual-Attention Guided Fusion Network for Hyperspectral Pansharpening

Abstract

Deep learning, especially the convolutional neural network, has been widely applied to solve the hyperspectral pansharpening problem. However, most do not explore the intraimage characteristics and the interimage correlation concurrently due to the limited representation ability of the networks, which may lead to insufficient fusion of valuable information encoded in the high-resolution panchromatic images (HR-PANs) and low-resolution hyperspectral images (LR-HSIs). To cope with this problem, we develop a hyperspectral pansharpening method called multistage dual-attention guided fusion network (MDA-Net) to fully extract the important information and accurately fuse them. It employs a three-stream structure, which enables the network to incorporate the intrinsic characteristics of each input and correlation among them simultaneously. In order to combine as much information as possible, we merge the features extracted from three streams in multiple stages, where a dual-attention guided fusion block (DAFB) with spectral and spatial attention mechanisms is utilized to fuse the features efficiently. It identifies the useful components in both spatial and spectral domains, which are beneficial to improving the fusion accuracy. Moreover, we design a multiscale residual dense block (MRDB) to extract dense and hierarchical features, which improves the representation power of the network. Experiments are conducted on both real and simulated datasets. The evaluation results validate the superiority of the MDA-Net.