CGTF: Convolution-Guided Transformer for Infrared and Visible Image Fusion

Abstract

Deep learning has been successfully applied to infrared and visible image fusion due to its powerful ability of feature representation. Existing most deep learning based infrared and visible image fusion methods mainly utilize pure convolution model or pure transformer model, which leads to that the fused image cannot preserve long-range dependencies (global context) and local features simultaneously. To this end, we propose a convolution-guided transformer framework for infrared and visible image fusion (CGTF), which aims to combine the local features of convolutional network and the long-range dependency features of transformer to produce satisfactory fused image. In CGTF, the local features are calculated by convolution feature extraction module, and then the local features are used to guide the transformer feature extraction module to capture the long-range dependencies of the image, which can overcome not only the lack of long-range dependencies that exists in convolutional fusion methods, but also the deficiency of local feature that exists in transformer models. Moreover, the convolution-guided transformer fusion framework can consider the inherent relationship of local feature and long-range dependencies due to the alternate use of convolution feature extraction module and transformer module. In addition, to strengthen local feature propagation, we employ dense connections among convolution feature extraction modules. Ablation experiments demonstrate the effectiveness of convolution-guided transformer fusion framework and loss function. We employ two datasets to compare our method with other nine methods, which includes three traditional methods, five deep learning based methods and one transformer based method. Qualitative and quantitative experiments demonstrate the advantages of our method.