MaskTheFER: Mask-Aware Facial Expression Recognition using Convolutional Neural Network

Abstract

Facial Expression Recognition (FER) plays a crucial role in understanding people's internal states. While existing FER methods have achieved high accuracy when facial features are fully visible, the widespread use of masks during the COVID19 pandemic has led to a significant loss of facial information. Although a few studies have explored masked FER by masking publicly available datasets, the absence of benchmark datasets on masked facial expressions poses a challenge. In this research, we address this gap by generating and publishing masked versions of two well-known datasets, namely FER2013 and CK+. Our proposed approach focuses on upper facial features in masked images to effectively handle the occlusion caused by masks. Initially, facial landmarks are detected in the masked images, which are then used to crop and align the images, retaining only the region surrounding the eyes. Subsequently, a Convolutional Neural Network (CNN) model based on a modified VGGNet architecture, incorporating fewer convolutional filters and layers, is trained and evaluated on the newly generated MaskedFER2023 and MaskedCK+ datasets. Our method achieves competitive performance – accuracies of 0.6189 and 0.6356 on the MaskedFER2023 and MaskedCK+ datasets, respectively – compared to existing state-of-the-art occlusion-aware and mask-aware FER methods. Additionally, we delve into the impact of masks on the recognition of different emotions. Our experimental results demonstrate that face masks significantly impede the recognition of certain expressions, particularly 'Sad', while other emotions like 'Surprise' exhibit lower sensitivity to masks. Implementation, experimentation and evaluation are publicly available at https://github.com/hasan-rakibul/MaskTheFER.