VOTA: A Heterogeneous Multicore Visual Object Tracking Accelerator Using Correlation Filters

Abstract

A correlation filter (CF) is a lighter and faster solution for visual object tracking (VOT) compared with a deep neural network (DNN). However, the CF-based trackers introduce diverse computational kernels and require FP16 computations. To address these challenges, we present a VOT accelerator (VOTA), a domain-specific accelerator for CF-based VOT that meets real-time performance at high efficiency, while providing flexibility and programmability. The VOTA encompasses a Winograd convolution core (WINO), a fast Fourier transformation (FFT) core (FFT), and a vector core (VEC) for diverse kernels, integrated in a high-bandwidth star-ring topology. The VOTA’s frame-based instruction set and execution enable a 537 GFLOPS performance, adapt to variances of CF trackers, and reduce the code size. An instruction-chaining mechanism permits inter-core pipelining and improves the hardware utilization up to 84.2%. A 10.2-mm 2 28-nm FP16 system on chip (SoC) prototype incorporating the VOTA, an RISC-V host CPU, and other supportive peripherals is taped out and measured to achieve 2.45 TFLOPS/W at 0.72 V. Running the oriented particle CF (OPCF), a CF tracker enhanced by adaptive boosting and particle filtering, the SoC chip achieves 1157 frames/s ( $640\times480$ frame size) at 0.9 V and 175 MHz, consuming 296 mW.