Unsupervised deep depth completion with heterogeneous LiDAR and RGB-D camera depth information

Abstract

In this work, a depth-only completion method designed to enhance perception in light-deprived environments. We achieve this through LidarDepthNet, a novel end-to-end unsupervised learning framework that fuses heterogeneous depth information captured by two distinct depth sensors: LiDAR and RGB-D cameras. This represents the first unsupervised LiDAR-depth fusion framework for depth completion, demonstrating scalability to diverse real-world subterranean and enclosed environments. To facilitate unsupervised learning, we leverage relative rigid motion transfer (RRMT) to synthesize co-visible depth maps from temporally adjacent frames. This allows us to construct a temporal depth consistency loss, constraining the fused depth to adhere to realistic metric scale. Furthermore, we introduce measurement confidence into the heterogeneous depth fusion model, further refining the fused depth and promoting synergistic complementation between the two depth modalities. Extensive evaluation on both real-world and synthetic datasets, notably a newly proposed LiDAR-depth fusion dataset, LidarDepthSet, demonstrates the significant advantages of our method compared to existing state-of-the-art approaches.