Multimodal Approach: Emotion Recognition from Audio and Video Modality

Abstract

Emotion recognition is a domain of artificial intelligence that recognizes human emotions similar to the cognitive capabilities of Affective computing. It is an interdisciplinary field of computer science that is emerging as a new area for research. Recognizing emotions from different input sources using computational intelligence is an advanced application of artificial intelligence and has benefitted numerous applications in human-computer interaction, healthcare, and psychology. The proposed work extensively studies a multimodal approach to emotion recognition that takes input from audio and video modalities. The study critically analyses the performance and improvement of emotion recognition deep learning models and fusion techniques of audio-video input modalities. The proposed approach demonstrates the effectiveness of the multimodal approach and the challenges of multimodal fusion while building a model on RAVDESS, and IEMOCAP dataset. In essence, the research strives to contribute to the growing body of knowledge in the field of audio-based emotion recognition by providing a comprehensive analysis of feature extraction techniques and deep learning models. By understanding the strengths and limitations of each approach, researchers and practitioners can make informed decisions when developing systems that accurately capture and interpret emotions from audio and video modalities. Conclusively, the advancement has the potential to enhance human-computer interaction, improve mental health assessment, and enrich various other domains where emotion plays a pivotal role.