Multiple Contrastive Learning for Multimodal Sentiment Analysis

Abstract

Multimodal sentiment analysis has received extensive attention with the explosion of multimodal data. For multimodal data, representations should have disparate distributions in the feature space under different labels. The paired multi-modal image-text posts should be closer than unpaired. We propose Multimodal fine-grained interaction with the Multiple Contrastive Learning (M ² CL) model for image-text multi-modal sentiment detection. Specifically, we first obtain the reinforced global representation of one modality with the assistance of fine-grained information from another via the Multimodal Interaction Component. Then, we introduce the Multiple Contrastive Learning Component, including Supervised Contrastive Learning (SCL) and Dual Multimodal Contrastive Learning (DMCL). SCL accomplishes pushing the posts with the same sentiment closer and pulling the instances of different sentiments apart within each modality. DMCL pushes the paired image-text features together and pulls the unpaired apart with multiple stages. Extensive experiments conducted on three datasets confirm the effectiveness of our approach.