Confidence-Aware Multimodal Learning for Trustworthy Fake News Detection

Abstract

With the rapid growth of social media, fake news has been widespread. Recent studies have focused on using multimodal information to improve the accuracy of fake news detection. However, there are two issues that have been ignored. The first is that existing methods have not seriously considered the effectiveness of multimodal semantic information. Blindly using all the semantic information of modalities for multimodal fusion would introduce noise or irrelevant information to the results. The second is that existing multimodal fake news detection models have not taken into account the reliability of their output. Although post hoc confidence calibration methods can be directly applied to existing models to improve the trustworthiness of the detection results, they lack theoretical guarantees for their performance. Regarding these, we develop a confidence-aware multimodal learning framework for trustworthy fake news detection with the following two main technical contributions: (a) we propose a new fusion module that effectively utilizes the multimodal information, and (b) we introduce a new post hoc confidence calibration method with theoretical guarantees. In the experiments, we apply the proposed framework to three public data sets. The results demonstrate that our framework outperforms existing fake news detection models. Moreover, our post hoc calibration method performs better than the state-of-the-art ones. In addition, we conduct a comprehensive discussion to show the advantages and properties of our proposed framework and calibration method. History: This paper has been accepted by Kaushik Dutta for the Special Issue on the Responsible AI and Data Science for Social Good. Funding: Financial support from the National Natural Science Foundation of China [Grants 72121001, 72101066, and 72131005] as well as the Heilongjiang Natural Science Excellent Youth Fund [Grant YQ2022G004], is gratefully acknowledged. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2024.0655 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2024.0655 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .