TSNN:Three‐StreamCombining2Dand3DConvolutional Neural Network for Micro‐Expression Recognition

Abstract

Facial micro‐expression recognition is a natural mechanism of facial behavior with subtle muscle movements and short duration, which is widely considered to be hard to recognize. In this paper, we propose the temporal sampling deformation (TSD) to normalize the temporal lengths and conserve time domain information for micro‐expression sequences. A three‐stream combining 2D and 3D convolutional neural network (TSNN) is also proposed to capture the features of micro‐expressions and classify the expressions as well. The proposed network has two variants TSNN‐IF and TSNN‐LF, which can automatically learn spatial and temporal features at the same time. Single domain experiments and cross‐domain experiments are also performed in the three benchmark datasets (chinese academy of sciences micro‐expression II (CASME II), spontaneous micro‐expression database (SMIC), and spontaneous micro‐facial movement dataset (SAMM)) to verify the effectiveness and validity of the proposed framework. Comprehensive results and ablation studies show that the proposed method can achieve comparable or even better results compared with other state‐of‐the‐art methods for micro‐expression recognition. © 2020 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.