Multi-Granularity Feature Fusion Network for Dynamic Sequential Facial Expression Recognition

Abstract

Dynamic sequential facial expression images can effectively reflect the essence of dynamic process of facial expression, and it is easy to eliminate the influence of various interference factors existing in static facial expression images. However, most previous studies only recognize static facial expression images, ignoring the facial expressions change is a dynamic and continuous process. To address these issues, this work proposes a network based on multi-granularity feature fusion for dynamic sequential facial expression recognition. The proposed network firstly extracts 68 facial landmarks of each frame in the sequence, and then calculates two types of offset distances of different frame' s facial landmarks in the sequence. One is the offset distances of landmarks between each frame and the first frame. Another is the offset distances of landmarks between two adjacent frames. These distances are served as the coarse-granularity feature of the facial expression sequence. Meanwhile, the facial features of single image in the sequence are used as the fine-granularity feature of the facial expression sequence. Finally a dual channel neural network model is built to learn muti-granularity features for predicting facial expression. Compared with the experimental results of mainstream facial expression recognition methods on the CK+ and Oulu-CASIA datasets, this method has improved the accuracy by 1.21% and 3.34% respectively.