HFF-Net: An Efficient Hierarchical Feature Fusion Network for High-Quality Depth Completion

Abstract

Depth completion aims to achieve high-quality dense depth prediction from a pair of synchronized sparse depth map and RGB image, and it plays an important role in many intelligent applications, including urban mapping, scene understanding, autonomous driving, and augmented reality. Although the existing convolutional neural network (CNN)-based deep learning architectures have obtained state-of-the-art depth completion results, depth ambiguities in large areas with extremely sparse depth measurements remain a challenge. To address this problem, an efficient hierarchical feature fusion network (HFF-Net) is proposed for producing complete and accurate depth completion results. The key components of HFF-Net are the hierarchical depth completion architecture for predicting a robust initial depth map, and the multi-level spatial propagation network (MLSPN) for progressively refining the predicted initial depth map in a coarse-to-fine manner to generate a high-quality depth completion result. Firstly, the hierarchical feature extraction subnetwork is adopted to extract multi-scale feature maps. Secondly, the hierarchical depth completion architecture that incorporates a hierarchical feature fusion module and a progressive depth rectification module is utilized to generate an accurate and reliable initial depth map. Finally, the MLSPN-based depth map refinement subnetwork is adopted, which progressively refines the initial depth map utilizing multi-level affinity weights to achieve a state-of-the-art depth completion result. Extensive experiments were undertaken on two widely used public datasets, i.e., the KITTI depth completion and NYUv2 datasets, to validate the performance of HFF-Net. The comprehensive experimental results indicate that HFF-Net produces robust depth completion results on both datasets.