Interactive Multimodal Attention Network for Emotion Recognition in Conversation

Abstract

In this letter, we propose a novel Interactive Multimodal Attention Network (IMAN) for emotion recognition in conversations. IMAN introduces a cross-modal attention fusion module to capture cross-modal interactions of multimodal information, and employs a conversational modeling module to explore the context information and speaker dependency of the whole conversation. Concretely, the cross-modal attention fusion module captures the cross-modal interactions and complementary information among the pre-extracted unimodal features from textual, visual, acoustic modalities based on the cross-modal attention block. Afterward, the updated features from each modality are fused to concentrate more on the informative modality and achieve a refined feature for each constituent utterance. The conversational modeling module defines three different gated recurrent units (GRUs) with respect to the context information, the speaker dependency, and the emotional state of utterances. In this way, we exploit the speaker dependency and contextual information to obtain the emotional state of utterances for emotion classification. Empirical evaluations on the multimodal benchmark IEMOCAP dataset demonstrate that our IMAN achieves competitive performance compared to the state-of-the-art approaches.