SS-INR: Spatial-Spectral Implicit Neural Representation Network for Hyperspectral and Multispectral Image Fusion

Abstract

Due to the limitation of imaging equipment, it is difficult to acquire hyperspectral images with high spatial resolution directly. Existing approaches improve the resolution of HSIs by fusing multispectral image (MSI) and hyperspectral image (HSI). However, most of them are only feed-forward. They only learn low- to high-resolution feature mappings without considering the ill-posedness of super-resolution tasks, leading to a large solution space of mapping functions and making it difficult to learn a complete mapping function. Moreover, there is a large resolution difference between HSI and MSI, and some up-sampling operations are inevitably employed in the network. Nevertheless, traditional upsampling methods only represent pixel points in a discrete way, failing to adequately restore the continuous spatial and spectral information. To this end, this paper proposes a spatial-spectral implicit neural representation network for hyperspectral and multispectral image fusion (SS-INR). Inspired by the success of implicit neural representation(INR) in continuum reconstruction, we design spatial-INR and spectral-INR for spatial and spectral resolution reconstruction, respectively. SS-INR contains two processes: forward fusion (FF) and back-projection fusion(BPF). In the FF process, the input HSI is first spatially upsampled with Spatial-INR to overcome spatial resolution differences while performing initial fusion with MSI. In the BPF process, we explore the spatial and spectral degradation processes and use them as prior knowledge for error correction. Extensive experiments on five public hyperspectral datasets demonstrate the effectiveness of SS-INR, and SS-INR achieves competitive results compared with existing state-of-the-art fusion methods. The source code for SS-INR will be released at https://github.com/wxy11-27/SS-INR.