Two‐branch global spatial–spectral fusion transformer network for hyperspectral image classification

Abstract

Abstract Transformer has achieved outstanding performance in hyperspectral image classification (HSIC) thanks to its effectiveness in modelling the long‐term dependence relation. However, most of the existing algorithms combine convolution with transformer and use convolution for spatial–spectral information fusion, which cannot adequately learn the spatial–spectral fusion features of hyperspectral images (HSIs). To mine the rich spatial and spectral features, a two‐branch global spatial–spectral fusion transformer (GSSFT) model is designed in this paper, in which a spatial–spectral information fusion (SSIF) module is designed to fuse features of spectral and spatial branches. For the spatial branch, the local multiscale swin transformer (LMST) module is devised to obtain local–global spatial information of the samples and the background filtering (BF) module is constructed to weaken the weights of irrelevant pixels. The information learned from the spatial branch and the spectral branch is effectively fused to get final classification results. Extensive experiments are conducted on three HSI datasets, and the results of experiments show that the designed GSSFT method performs well compared with the traditional convolutional neural network and transformer‐based methods.