SG-NeRF: Semantic-guided Point-based Neural Radiance Fields

Abstract

Neural Radiance Fields (NeRF) can successfully reconstruct room-scale scenes and achieve photo-realistic novel view synthesis results with densely captured input images. However, capturing hundreds of high-quality images in a single room is extremely laborious. We tackle this problem by greatly reducing the number of images input to NeRF while maintaining high-quality rendering results in a room-scale scene. In this paper, we propose semantic-guided point-based NeRF (SG-NeRF), which is capable of reconstructing the radiance field of a room-scale scene with tens of images. To this end, we leverage sparse 3D point clouds with neural features to be the geometry constraints of NeRF optimization and semantic prediction of both 2D images and 3D point clouds to guide the neighboring neural points searching at the ray marching procedure. With the semantic guidance, the sampled query points are capable of searching for neighboring neural points, which are structurally related to the query points accurately in a large area since of the unevenly distributed sparse point clouds. Extensive experimental results demonstrate that our approach outperforms previous state-of-the-art methods.