Hyperspectral Image Super-Resolution Network Based on Cross-Scale Nonlocal Attention

Abstract

Hyperspectral image (HSI) super-resolution generally means the fusion of low-spatial resolution hyperspectral image (LRHSI) and high-spatial resolution multi-spectral (MSI)/panchromatic (PAN) image (HRMPI) to get high-spatial resolution HSI (HRHSI). Existing fusion methods have not sufficiently considered the huge spectral and spatial resolution difference between the LRHSI and HRMPI. In addition, most DL-based methods that adopt the CNN structure are limited by its local feature learning, and it is difficult to exploit the global dependency of image features. To fully adapt to the huge modality difference between LRHSI and HRMPI and release the limitation of local feature learning, we design the cross spectral-scale and shift-window based cross spatial-scale non-local attention networks (CSSNet) to effectively fuse the LRHSI and HRMPI. These two networks could explicitly learn the spectral and spatial correlations between two input images. These correlations are then used to reconstruct the HRHSI feature, which makes the obtained HRHSI feature to maintain the spectral and spatial feature consistency with the input images. Finally, a 'feature aggregation module' is designed to aggregate the image features from these two networks and output the fused HRHSI. Extensive experimental results on both HM-fusion (fusion with MSI) and HP-fusion (fusion with PAN image) tasks demonstrate CSSNet's state-of-the-art (SOTA) performance compared to other fusion methods. (The codes could be available at https://github.com/rs-lsl/CSSNet).