Rumor Detection Method Based on Attention and Multi-Modal Hybrid Fusion

Abstract

Content on social media is characterized by high structural complexity.Many rumors are mixed with real information, and real pictures are tagged with fabricated description. So it is difficult to detect rumors effectively based on single modal methods.In order to solve the problem, a method is proposed based on hybrid fusion of the attention mechanism and Dempster's rule of combination.The method adds three kinds of modal feature vectors, including text, vision and user.Then the attention mechanism is utilized to give more weight to words and visual neurons that contribute more to rumor detection, making bidirectional matching between words and vision.The attention mechanism is added to early fusion and late fusion to achieve the automatic weighting of the features and decisions.The hybrid fusion of early fusion and late fusion is implemented by using Dempster's rules of combination.The experimental results show that the proposed method displays an accuracy of 97.44% on the Chinese data set of Weibo and 92.35% on the data set of Twitter.The accuracy and F1-score of the method are both better than those of the base-line methods and advanced multimodal methods.