Multi-Modal Feature Fusion Network for Ghost Imaging Object Detection

Abstract

Ghost imaging is a new imaging technology with the advantage of anti-interference and environmental adaptability, but it's far from the practical application in the field of object detection for ghost data. The challenge is that ghost data only contains the depth information with the low resolution and visibility, in the lack of textural features. In this paper, we propose a multi-modal feature fusion network which adapts recent convolutional neural networks (CNNs) based detectors. We also generate a synthetic ghost imaging dataset. To fully exploit the complete characteristics of ghost data, we encode the original data into new feature maps. Our architecture is divided into two streams, one for ghost data and one for encoded maps, which utilizes multi-modal features by mid-level fusion. We obtained an average 4.2% improvement over ghost data baseline and also achieved competitive accuracy on the NYUD2 dataset. Our research is a relatively novel field with significant application value and potential demands.