Heterogeneous Convolutional Neural Networks for Emotion Recognition Combined with Multimodal Factorised Bilinear Pooling and Mobile Application Recommendation

Abstract

The field of emotion recognition has garnered considerable interest due to its diverse applications in mental health, personalised advertising and enhancing user experiences. This research paper introduces a unique and innovative method for emotion recognition by integrating heterogeneous convolutional neural networks (CNNs) with multimodal factorised bilinear pooling. Furthermore, the paper also incorporates the integration of mobile application recommendations as part of the overall approach. The proposed method leverages the power of CNNs to extract high-level features from different modalities, including facial expressions, speech signals and physiological signals. By using heterogeneous CNNs, each modality is processed independently to capture modality-specific emotional cues effectively. To fuse the extracted features, multimodal factorised bilinear pooling is employed, which captures the complex interactions between different modalities while reducing the computational complexity. This pooling technique efficiently combines the modality-specific features, resulting in a compact and discriminative representation of the emotional state. In addition to emotion recognition, this paper also introduces the integration of mobile app recommendations. By leveraging the recognised emotion, the system recommends relevant mobile applications that are tailored to the user’s emotional state. This integration enhances user experience and facilitates emotion regulation through the utilisation of appropriate mobile apps. Experimental evaluations are conducted on benchmark emotion recognition datasets, including the DEAP and MAHNOB_HCI datasets. The findings of the study highlight the effectiveness of the proposed methodology in terms of accuracy and robustness, surpassing existing approaches in the field. Additionally, the integration of the mobile app recommendation system showcases encouraging outcomes by offering personalised recommendations tailored to the user’s emotional state.