Distillation-Based Hashing Transformer for Cross-Modal Vessel Image Retrieval

Abstract

Cross-modal image retrieval has attracted much attention in remote sensing(RS) data analysis these years, however, the retrieval of target images such as surface vessels receives little interest. Considering the complex geometric features of vessel images and the modality gap, the widely used joint feature learning based on CNNs tends to have low precision. In this letter, a distillation-based hashing transformer(DBHT) is proposed to solve the above problems. Specifically, we adopt vision transformer(ViT) as the feature extractor for target images and a hash token is designed and attended to ViT for hashing generation. To avoid the precision attenuation caused by the uncontrollability in common feature space construction, we design a two-step feature learning strategy and build a well-performed unimodal hashing retrieval framework firstly, and then transfer the hashing knowledge to another modality. Two distillation strategies, as well as cross-modal weighted triplet loss, are designed to supervise the above process and ensure complete knowledge transfer. Cross-modal weight transfer is also adopted to bridge the modality gap. Extensive experiments on two bimodal vessel image datasets show that the proposed DBHT is superior to several cross-modal hashing baselines in cross-modal vessel image retrieval tasks.