Annotation quality versus quantity for object detection and instance segmentation

Abstract

Deep learning-based computer vision models are typically data-hungry, resulting in the rise of dataset sizes. The consensus for computer vision datasets is that larger datasets lead to better model performance. However, the quality of the datasets is often not considered. Annotating datasets for fully supervised object detection and instance segmentation tasks requires a significant investment in time, effort, and cost. In practice, due to the large sample sizes needed, this often leads to inaccuracies in the annotation process. This research aims to understand and quantify the impact of annotation quality and quantity on the performance of object detection and instance segmentation models. Specifically, the research aims to investigate how introducing additional data with varying levels of annotation quality affects mean average precision (mAP) performance. To investigate the relationship between annotation quality and quantity, subsets of the COCO and ADE20K datasets are used. For each of the datasets, three types of annotation ,uncertainty are added to the annotations, which are localization uncertainty, incorrect class labels, and missing annotations. Mask-RCNN, YOLACT, and Mask2Former models are trained on a variety of sample sizes for varying levels of annotation uncertainties. The results indicate there is utility in adding additional data of lesser annotation quality. The extent of the benefits of the additional data is directly related to how degraded the annotations’ are. Furthermore, the results show that all three annotation uncertainties negatively affect mAP performance, with incorrect class labels degrading mAP performance the most, followed by missing annotations and lastly localization uncertainty.