Augmenting Shape with Appearance in Vehicle Category Recognition

Abstract

Shape is an important cue for generic object recognition but can be insufficient without other cues such as object appearance. We explore a number of ways in which the geometric aspects of an object can be augmented with its appearance. The main idea is to construct a dense correspondence between the interior regions of two shapes based on a shape-based correspondence so that the intensity and gradient distributions can be compared, e.g., using a mutual information paradigm. Three methods for regional alignment are suggested and compared here, based on: (i) propagation of correspondences from the silhouette to parallel curves in the interior, (ii) intersection of line segments anchored on corresponding points on the contour, and (iii) correspondence of shape skeletons. These methods have been implemented and applied to vehicle category recognition from aerial videos under known viewing and illumination conditions. We have constructed a photo-realistic synthetic video database to explore the performance of these methods under controlled conditions. We have also tested these algorithms on real video collected for this purpose from a balloon. Our findings indicate that (i) augmenting shape with appearance significantly increases recognition rate, and (ii) the region correspondence induced by the shape skeleton yields the highest performance.