Emotion Recognition in Tweets Using Multi-Modal Data

Abstract

People on an average tend to express their emotions and opinions farmore freely in social media community than anywhere else. Understanding human emotional states is necessary to improve Human computer interaction (HCI). Emotion classification systems use unsupervised, supervised or hybrid learning techniques to classify emotions expressed in text, images etc. In this paper, we propose a fuzzy-rule based unsupervised learning technique for textual data and a late fusion model formed by fusion of objects and places based Convolutional Neural Network (CNN) classifiers for images to identify the emotions associated with social media posts. The proposed fuzzy system integrates Word Sense Disambiguation and Natural language processing using a novel rule base to identify emotions with their intensity. The images are annotated and its relevance with respect to the related tweet is checked by calculating the cosine similarity between them.