Micro-Expression Recognition Using the Spatiotemporal Feature extraction and Deep-Learning Methods

Abstract

Objective: The Micro-Expression (ME), which automatically reveals genuine human emotions, has gained significant attention. Recognizing the ME is crucial for many real-time applications. However, there are significant challenges to overcome. For instance, the number of ME frames are limited due to their short duration, and the subtle facial movements can be hard to detect due to their low intensity. These challenges need to be addressed to improve ME recognition. Materials and Methods: We propose a novel method for the ME recognition in real-time. In this method, first, the apex frame is spotted using the rotated local binary pattern from six planes (RLBPS) and correlation coefficient (CC). Next, three hand-crafted methods such as the multi-color rotated local binary pattern from six planes (MRLBPS), the histograms of directed gradients from six planes (HDGS), and the histogram of image gradient direction from six planes (HIGDS) extract the features from the apex frame and its surrounding frames. Finally, the stacks of features as matrixes are fed into a three-dimensional convolutional neural network (3D-CNN), and the output is the maximum recognition rate by voting three results. Results: The proposed method has shown promising results when compared to most state-of-the-art methods. According to the results, an average precision of 99% has been obtained using our proposed method. Conclusion: The combination of the RLBPS and the CC creates a strong method for spotting the apex frame. Also, feeding the stacks of spatiotemporal features into the 3D-ResNet increases the ME recognition rate in real-time.