Facial Expressions and Facial Micro Expressions Recognition Using RGB-D Images and Videos

Abstract

The human face states the inner emotions, thoughts, and physical disorders. These emotions are expressed on the face via facial muscles. Research indicates that facial expressions are the best way to express emotions. Human facial expressions and micro-expressions could be studied in the images and digital video frames. The estimated time through which a facial expression occurs is between 0.5 to 4 seconds and a micro expression between 0.1-0.5. Also, in some references, this value is stated as 1.3, 1.15, and 1.25 seconds. Obviously, for the purpose of recording micro-expressions, obtaining video frames between 30 and 200 fps is essential. Before depth sensors emerged, facial expression recognition (FER) was done only by color images. However, after depth sensors emerged, and due to more data (depth dimension), the recognition rate in FER increased significantly. This was tangible for a decade in this field. Facial expression recognition has applications in human-computer (robot) interaction, 2D and 3D animation, psychology, non-verbal communication or body language, emotion recognition, security issues such as lie detection, etc. Features which are used in this study are Histogram of Oriented Gradient (HOG), Gabor Filter, Speeded Up Robust Features (SURF), Local Phase Quantization (LPQ), Local Binary Pattern (LBP), and Haar Feature. Due to the shortage of RGB-D FER database, and also due to available databases weaknesses, a database including 40 individuals or subjects in a variety of ages and genders with the Kinect V.2 sensor is gathered, which to the extent acceptable has resolved the available databases with similar features weaknesses. On the other hand, it can be said that this is the first depth database for facial micro expression recognition (FMER). This database is named the Iranian Kinect Face Database (IKFDB). Considering that Kinect’s acquired data is divided into color and depth parts, a hybrid feature extraction method for depth data based on pixel distance alterations with a depth sensor is considered. Sections under the titles of age estimation and gender recognition are considered, too. Also, a face detection and extraction algorithm for Depth images is presented. These methods are evaluated and compared with the benchmark databases and the proposed database. Databases used for evaluation are Eurecom Kinect Face DB, VAP RGBD Face DB, VAP RGBD-T Face, JAFFE, IKFDB, Face Grabber DB, Curtin Face, FEEDB, and CASME, which are prepared in two kinds (image and video frames) by different RGB (color), Depth and Thermal sensors. Finally, selected features in the shape of the feature vectors, and for the learning process, are sent to Support Vector Machine (SVM) and Multi-Layer Neural Network (MLNN) classifiers. The results are really satisfactory, and they indicate improvement in classification accuracy in some databases and methods. Also, some of these actions are performed on some of these databases for the first time. Keywords: Facial Expressions Recognition (FER), Facial Micro Expressions Recognition (FMER), Kinect Depth Sensor, Histogram of Oriented Gradient (HOG), Gabor Filter, Speeded Up Robust Features (SURF), Local Phase Quantization (LPQ), Local Binary Pattern (LBP), Haar Feature, Iranian Kinect Face Data Base (IKFDB), Support Vector Machine (SVM), Multi-Layer Neural Network (MLNN)