Relation Extraction with Knowledge-Enhanced Prompt-Tuning on Multimodal Knowledge Graph

Abstract

Recently, Multimodal Knowledge Graphs (MKGs) with visual and textual factual knowledge have been widely used in tasks such as knowledge question answering, recommender systems, and entity disambiguation. Since most of the current MKGs still have defects, a multimodal knowledge graph completion technology is proposed, and multimodal relation extraction (MRE) is one of the basic processes. However, visual objects with high object classification scores are usually selected in previous tasks, which may result in the addition of noise from objects that are either irrelevant or redundant, which can adversely affect multimodal relationship extraction. For this reason, in this paper, we propose a Relation Extraction with Knowledge-enhanced Prompt-tuning modal on multimodal knowledge graph (REKP) to address these issues. Specifically, we inject potential knowledge from relational labels into the prompt construction of answer words and optimize their representation with structured constraints. A Transformer architecture with cross-modal attention is then used to fuse the visual and textual representations. We conduct extensive experiments to verify that our REKP model can achieve SOTA performance on the MNRE dataset with multimodal relational extraction.