Dual-Branch Multimodal Fusion Network for Driver Facial Emotion Recognition

Abstract

In the transition to fully automated driving, the interaction between drivers and vehicles is crucial as drivers’ emotions directly influence their behavior, thereby impacting traffic safety. Currently, relying solely on a backbone based on a convolutional neural network (CNN) to extract single RGB modal facial features makes it difficult to capture enough semantic information. To address this issue, this paper proposes a Dual-branch Multimodal Fusion Network (DMFNet). DMFNet extracts semantic features from visible–infrared (RGB-IR) image pairs effectively capturing complementary information between two modalities and achieving a more accurate understanding of the drivers’ emotional state at a global level. However, the accuracy of facial recognition is significantly affected by variations in the drivers’ head posture and light environment. Thus, we further propose a U-Shape Reconstruction Network (URNet) to focus on enhancing and reconstructing the detailed features of RGB modes. Additionally, we design a Detail Enhancement Block (DEB) embedded in a U-shaped reconstruction network for high-frequency filtering. Compared with the original driver emotion recognition model, our method improved the accuracy by 18.77% on the DEFE++ dataset, proving the superiority of the proposed method.