Convolutional Neural Network based Bidirectional Sign Language Translation System

Abstract

A considerable amount of gap in communication exists amongst the speech and hearing-impaired individuals with the other people; which is of paramount importance to be bridged. The aim is to study various methods for effective intercommunication between Sign language and the English language. Initially, a Hardware glove is implemented which has of flex sensors whose accuracy is proved to be not very high. To further improve the accuracy, a model using a convolutional neural network was trained on an existing data-set. Since the data-sets were not versatile and the scope was narrow, a new diversified data-set was created and the model was further improved. The new model has very high accuracy and it can predict almost every alphabet. Various other gestures having facial features and gestures including both the hands were added to our data-set. This model has a huge potential as it can interpret any gesture of various sign languages if provided in the data-set. The user can also add extra gestures in the data-set, making it highly customized. Further, the data is sent to an application which will convert the received text to speech. To reduce the communication gap, the system is made wholly bidirectional i.e. speech can also be converted to the sign language. Initially, the speech is taken at the input and is converted to the text which acts as the input for the next step in which the converted text is directly taken as the input to be converted to the corresponding gesture according to the convenient sign language. Thus, the input speech is translated into a video consisting of a sequence of gestures of the American sign language which can be extended to other languages as well. Bidirectional Sign Language Translating system consists of a software system. It is named as a bidirectional system as it not only converts the sign language to speech via text conversion but also incorporates a system which translates the speech to the prescribed sign language with text conversion as the mediator. The methodology has been explained in the further sections.