SCFusion: Real-time Incremental Scene Reconstruction with Semantic Completion

Abstract

Real-time scene reconstruction from depth data inevitably suffers from\nocclusion, thus leading to incomplete 3D models. Partial reconstructions, in\nturn, limit the performance of algorithms that leverage them for applications\nin the context of, e.g., augmented reality, robotic navigation, and 3D mapping.\nMost methods address this issue by predicting the missing geometry as an\noffline optimization, thus being incompatible with real-time applications. We\npropose a framework that ameliorates this issue by performing scene\nreconstruction and semantic scene completion jointly in an incremental and\nreal-time manner, based on an input sequence of depth maps. Our framework\nrelies on a novel neural architecture designed to process occupancy maps and\nleverages voxel states to accurately and efficiently fuse semantic completion\nwith the 3D global model. We evaluate the proposed approach quantitatively and\nqualitatively, demonstrating that our method can obtain accurate 3D semantic\nscene completion in real-time.\n