RGB-D object recognition with multimodal deep convolutional neural networks

Abstract

Object recognition from RGB-D images has become a hot topic and gained a significant popularity in recent years due to its numerous applications. In this paper, we propose a novel multimodal deep convolutional neural networks architecture for RGB-D object recognition which composed of three streams with two different types of deep CNNs, where each stream can separately learn from each modality. Finally, we propose a combined architecture of joint network of these three streams to classify the objects. Compared to RGB data, RGB-D images provide additional depth information that can be represented as depth colorization methods or surface normals. Our goal is to exploit both colorization and surface normals information to encode depth images. We show that by utilizing both colorization and surface normals of depth images combined with RGB significantly can improves the classification accuracy. We evaluate our model on one of the most challenging RGB-D object dataset and achieves comparable performance to state-of-the-art methods.