Remote Facial Emotion Recognition System

Abstract

This paper describes a method for collecting information of user's emotional state based on its facial expressions and linking the data acquired to the user's actions in a time. Methods and Algorithms: At client side, we use reactive programming model implemented by RxJS library to manage data streams. Our client library includes logic to combining events from different sources such as MediaStream APIs, classic DOM event API. We have developed our own neural network architecture that allows us to map the facial expressions of people into a continuous space of arousal/valence. The same architecture for classification of emotions was used. We used inception and residual blocks and batch normalization layers to achieve higher accuracy. In addition, we augment train data by adding pose variations using 3D face reconstruction. Results: We have developed a system that allows us to collect various data such as video stream and events of user interaction with the webpage. We also trained a neural network that shows high accuracy. One part of the system is a client library that does not require the user to install any special software and can be easily integrated into the website. The other part is a server application that enables the site owner to analyze the data collected.