Image annotation with semi-supervised clustering

Abstract

Methods developed for image annotation usually make use of region clustering algorithms. Visual codebooks are generated from the region clusters of low level features. These codebooks are then, matched with the words of the text document related to the image, in various ways. In this paper, we supervise the clustering process by using three types of side information. The first one is the topic probability information obtained from the text document associated with the image. The second is the orientation and the third one is the color information around each interest point. The side information provides a set of constraints in a semi-supervised k-means region clustering algorithm. Consequently, in clustering of regions not only low level features, but also this extra information is used. Experimental results show that image annotation with semi-supervision of side information is more successful compared to the one that uses low level features alone. Moreover, a speedup is obtained in the modified k-means algorithm because of the constraints. The proposed algorithm is implemented in a high performance parallel computation environment.