A Transformer-Based Three-Branch Siamese Network For Hyperspectral Object Tracking

Abstract

Hyperspectral videos can provide more information for the object tracking task. Due to the limited training samples, most current hyperspectral trackers do not fully use hyperspectral information to improve the tracking performance. To solve this problem, we propose a Transformer-based three-branch Siamese network (TrTSN) for hyperspectral object tracking. First, we construct a three-branch structure based on the Siamese network to obtain the semantic information of hyperspectral data fully. Second, we design a Transformer-based fusion module (TFM) and use two TFMs to adaptively combine the information obtained by different branches to obtain more robust features. Finally, the two sets of classification response and regression response generated by two fusion features are corresponding merged to improve the tracking network's ability to predict the object's position. Experimental results show that the TrTSN tracker is superior to the state-of-the-art trackers, demonstrating the effectiveness of this method.