Sign to Speech Convolutional Neural Network-Based Filipino Sign Language Hand Gesture Recognition System

Abstract

Sign Language Recognition is a breakthrough for helping deaf-mute people and has been studied for many years. Unfortunately, every research has its own limitation and are still unable to be used commercially. In this study, we developed a real-time Filipino sign language hand gesture recognition system based on Convolutional Neural Network. A manually gathered dataset consists of 237 video clips with 20 different gestures. This dataset underwent data cleaning and augmentation using image pre-processing techniques. The Inflated 3D convolutional neural network was used to train the Filipino sign language recognition model. The experiments considered retraining the pretrained model with top layers and all layers. As a result, the model retrained with all layers using imbalanced dataset was shown to be more effective and achieving accuracy up to 95% over the model retrained with top layers to classify different signs or hand gestures. Using the Rapid Application Development model, the Filipino sign language recognition application was developed and assessed its usability by the target users. With different parameters used in the evaluation, the application found to be effective and efficient.