CNN+RNN Depth and Skeleton based Dynamic Hand Gesture Recognition

Abstract

Human activity and gesture recognition is an important component of rapidly\ngrowing domain of ambient intelligence, in particular in assisting living and\nsmart homes. In this paper, we propose to combine the power of two deep\nlearning techniques, the convolutional neural networks (CNN) and the recurrent\nneural networks (RNN), for automated hand gesture recognition using both depth\nand skeleton data. Each of these types of data can be used separately to train\nneural networks to recognize hand gestures. While RNN were reported previously\nto perform well in recognition of sequences of movement for each skeleton joint\ngiven the skeleton information only, this study aims at utilizing depth data\nand apply CNN to extract important spatial information from the depth images.\nTogether, the tandem CNN+RNN is capable of recognizing a sequence of gestures\nmore accurately. As well, various types of fusion are studied to combine both\nthe skeleton and depth information in order to extract temporal-spatial\ninformation. An overall accuracy of 85.46% is achieved on the dynamic hand\ngesture-14/28 dataset.\n