MAFusion: Multiscale Attention Network for Infrared and Visible Image Fusion

Abstract

The infrared and visible image fusion aims to generate one image with rich information by integrating thermal regions from the infrared image and texture details from the visible image, which is beneficial to facilitate the capacity of video surveillance and object detection in complex environments. Although there is great progress in image fusion algorithms, artifacts and inconsistencies are still challenging tasks. To alleviate these problems, a multi-scale attention network for infrared and visible image fusion (MAFusion) is proposed. The network consists of encoder, fusion strategy, and decoder. Specifically, the encoder is adopted to extract multi-scale features by feeding the source images. An attention-based model is then designed as the fusion strategy to integrate different features in the infrared and visible images. The attention-based model can highlight the thermal targets in the infrared image and maintain details in the visible image, so as to avoid the generation of artifacts. The decoder is based on multi-scale skip connection to incorporate low-level details with high-level semantics at different scales. The vital features of infrared and visible images can be fully preserved by the multi-scale skip connection network to restrict the introduction of inconsistencies. Furthermore, we develop a feature-preserving loss function to train the proposed network. Experimental results demonstrate that the proposed network delivers advantages and effectiveness compared with the state-of-the-art fusion methods in qualitative and quantitative assessments. Besides, we apply the fused image generated by MAFusion to crowd counting, which can effectively improve the crowd counting performance in low illumination conditions.