Open-World 3D instance segmentation

Abstract

Existing methods for 3D instance segmentation often operate under the closed-world assumption, where only seen categories are segmented at inference, limiting their adaptability to real-world scenarios. In this work, we pioneer open-world 3D indoor instance segmentation, enabling the model to distinguish between known and unknown classes while incrementally learning the semantic category of unknown objects. Our approach employs an auto-labeling scheme during training to generate pseudo-labels, enhancing separation between known and unknown categories. We refine pseudo-label quality at inference by adjusting unknown class probabilities based on objectness score distributions. Furthermore, we introduce curated open-world splits, reflecting realistic scenarios, to evaluate our method comprehensively. Moreover, we propose an exemplar-free approach that integrates continual learning and unknown class identification, leveraging self-distillation. By utilizing pseudo-labels from previous tasks, our method improves unknown predictions during training and mitigates catastrophic forgetting. This unified approach outperforms traditional methods, showcasing superior performance in continual learning and unknown class retrieval. Extensive experiments on various ScanNet200 dataset splits validate the efficacy of our proposed approach, demonstrating its potential for real-world applications.