PPF-Net: Efficient Multimodal 3D Object Detection with Pillar-Point Fusion

Abstract

Detecting objects in 3D space using LiDAR is crucial for robotics and autonomous vehicles, but the sparsity of LiDAR-generated point clouds limits performance. Camera images, rich in semantic information, can effectively compensate for this limitation. We propose a simpler yet effective multimodal fusion framework to enhance 3D object detection without complex network designs. We introduce a cross-modal GT-Paste data augmentation method to address challenges like 2D object acquisition and occlusions from added objects. To better integrate image features with sparse point clouds, we propose Pillar-Point Fusion (PPF), which projects non-empty pillars onto image feature maps and uses an attention mechanism to map semantic features from pillars to their constituent points, fusing them with the points’ geometric features. Additionally, we design the BD-IoU loss function, which measures 3D bounding box similarity, and a joint regression loss combining BD-IoU and Smooth L1, effectively guiding model training. Our framework achieves consistent improvements across KITTI benchmarks. On the validation set, PFF (PV-RCNN) achieves at least 1.84% AP improvement in Cyclist detection performance across all difficulty levels compared to other multimodal SOTA methods. On the test set, PPF-Net excels in pedestrian detection for moderate and hard difficulty levels and achieves the best results in low-beam LiDAR scenarios.