Image Radar Point Cloud Segmentation with Segment Anything Model

Abstract

Radar point clouds are a rich source of information for various applications. However, clustering or classification of radar point clouds is challenging due to their sparsity, noise, and ambiguity. In this paper, we propose a novel approach that leverages the Segment Anything Model (SAM), a segmentation model introduced by Meta AI that can produce high-quality segment masks from 2D images, to predict 3D masks in image-radar point clouds. We extend SAM to handle 3D points indirectly, by associating point clouds with time-synchronized and calibrated image data, we first get masks from 2D images using SAM and then project the masks onto 3D points. Based on the masks, we can cluster radar point cloud and predict the remaining parameters of the objects by involving other radar point clouds attributes. This is also a convenient method to label radar point clouds for radar-only neural network development without supervision. Our approach is experimented on a self-made dataset and the results demonstrate reasonable qualitative accuracy without any further fine-tuning or training of SAM.