Music Recommendation based on Emotions recognized through Speech and Body Gestures

Abstract

Throughout history, music has consistently remained a widely enjoyed source of entertainment, and technology has swiftly acknowledged its popularity. The emergence of numerous music streaming platforms has provided users with a plethora of choices, emphasizing the need for a system that simplifies the organization and search management of music. A project utilizing human-computer interaction (HCI) techniques to suggest songs by analyzing the user's emotions extracted from speech input can significantly enhance the application's personalization. This approach aims to create a more tailored and emotionally resonant experience for the user, establishing a deeper connection between the music recommendations and the user's emotional state. However, relying solely on emotions from speech may not be very effective as it is subject to variations across people of different genders, race and culture. Speech emotion recognition may also be ineffective for the speech-impaired people. Hence, the objective of this project is to build a music recommendation system capable of discerning emotions from both body gestures and speech, enabling more comprehensive and nuanced song recommendations. In order to make effective recommendations using minimal amount of samples, meta learning models are used in both gesture and speech emotion recognition. This data is in turn used by the recommendation module. We have used the GEMEP Audio-Video dataset for emotion recognition using speech and body gestures. These identified emotions are then correlated with suitable song moods, forming the basis for the generated recommendations. The Emotion module of the model obtained an accuracy of 78.63% for multi-modal emotion recognition. The recommendation module obtained an accuracy of 98.02%.