Visual Scene Understanding through Semantic Segmentation

Abstract

The problem of visual scene understanding entails recognizing the semantic constituents of a scene and the complex interactions that occur between them. Development of algorithms for semantic segmentation, which requires the simultaneous segmentation of an image into regions and the classification of these regions into semantic categories, is at the heart of this problem. This dissertation presents methods that provide improvements to the state of the art in semantic segmentation of images and investigates the use of the obtained semantic segmentation output for related image retrieval and classification tasks. We present a method for non-parametric semantic segmentation of images which can effectively work on image datasets with a large number of categories. The method exploits query time feature channel relevance and also introduces the semantic label descriptor for improving the semantic segmentation output by retrieving images which share semantically similar spatial layouts. We further demonstrate how to associate accurate confidences with the resulting semantic segmentation through the use of the strangeness measure. We show how this measure can be applied for confidence ranking of unlabeled images and associate high uncertainty scores with images containing unfamiliar semantic categories. We then demonstrate the use of semantic segmentation output for additional tasks such as scene categorization, learning related semantic concepts and content based image retrieval.