Cross-modal Hypergraph Optimisation Learning for Multimodal Sentiment Analysis

Abstract

Sentiment expressions are multimodal,and more accurate emotions can be derived through multiple modalities such as verbal,audio,and visual.Studying the interactions among modalities can effectively improve the accuracy of multimodal sentiment analysis.Previous studies have used graph models to capture rich interactions across modalities and time to obtain highly expressive and fine-grained sequence representations,but there is a greater need to tap into the expression of higher-order information in multimodal data,which can only be achieved on a one-to-one basis in graph neural networks,which restricts the utilisation of the interactions of higher-order information.This paper explores the application of hypergraph neural networks in multimodal sentiment analysis,where the hypergraph structure can connect two or more nodes to make full use of intra-and inter-modal higher order information and to achieve the interaction of higher-order information between data.Furthermore,this paper proposes a hypergraph adaptive module to optimise the structure of the original hypergraph,where the hypergraph adaptive network is designed to detect potential hidden information by means of point-edge cross-attention,hyperedge sampling and event node sampling to discover potential implicit connections and prune redundant hyperedges as well as irrelevant event nodes to update and optimise the hypergraph structure,the updated hypergraph structure represents the higher-order correlations of the data more accurately and completely than the initial structure.Extensive experiments on two publicly available datasets show that the proposed framework improves 1% to 6% in several performance metrics over other state-of-the-art algorithms on the CMU-MOSI and CMU-MOSEI datasets.