Object recognition using tactile and image information

Abstract

Camera provides rich information about objects and therefore becomes the mainstream sensors in robots. However, it often fails when the objects are not visual-distinguished. As a complementary, tactile sensors in the robotic fingertips can be used to capture multiple object properties such as texture, roughness, spatial features, compliance or friction and therefore becomes a very important sense modality for intelligent robot. Nevertheless, how to effective fuse both modality is still a challenging problem. In this paper, we developed tactile-image fusion framework for object recognition task. The multivariate times series model is used to represent the tactile sequence and the covariance descriptor is used to characterize the image. We also develop a practical dataset which includes 18 household object for verification and the experimental results shows that the performance of tactile-image is obviously better than using single modality.