MEFNET: Multi-expert fusion network for RGB-Thermal semantic segmentation

Abstract

Semantic segmentation using RGB and thermal images is crucial in a variety of applications, including autonomous driving and video surveillance. However, the validity of information differs between modalities, which is typically addressed by weighting image features using complex and inefficient networks. To address this issue, we propose a Multi-Expert Fusion Network (MEFNet) that decouples the three-dimensional attention matrix of image features into a two-dimensional modal weight matrix and a channel attention vector. This approach focuses more on modal and channel differences while excluding interferences from other factors. Specifically, MEFNet multiplies RGB and thermal features by their respective modal weights, and then uses channel attention to select important feature channels. Comprehensive experiments demonstrate that MEFNet is competitive with state-of-the-art methods, achieving 62.6% mIoU on the IR SEG dataset.