Prompt Learning for Multi-modal COVID-19 Diagnosis

Abstract

The outbreak of COVID-19 pandemic has spread rapidly and severely affected all aspects of human lives. Recent researches has shown artificial intelligence and deep learning based approaches have achieved successful results in detecting diseases. How to accurately and quickly detect COVID-19 has always been the core topic of research. In this paper, we propose a novel approach based on prompt learning for COVID-19 diagnosis. Different from the traditional "pre-training, fine-tuning" paradigm, we propose the prompt-based method that redefine the COVID-19 diagnosis as a masked predict task. Specifically, we adopt an attention mechanism to learn the multi-modal representation of medical image and text, and manually construct a cloze prompt template and a label word set. Selecting the label word corresponding to the maximum probability by pre-training language model. Finally, mapping the prediction results to the disease categories. Experimental results show that our proposed method obtains obvious improvement of 1.2% in terms of Mi-F1 score compared with the state-of-the-art methods.