BISINDO Sign Language Interpreter System Using YOLOv8 and CNN

Abstract

Many deaf people in Indonesia use sign language as their communication medium, one of which is BISINDO (Bahasa Isyarat Indonesia). However, limited understanding of BISINDO among the general public often becomes a significant barrier to effective communication. To address this, the study aims to develop a BISINDO hand gesture detection system based on computer vision using the YOLOv8 algorithm. The method used is a Convolutional Neural Network (CNN) with the YOLOv8 architecture, trained on a labeled dataset obtained from the Roboflow platform consisting of BISINDO alphabet gestures (A–Z). The system was developed using Python and Streamlit, providing two types of user input: real-time camera feed and manual image upload. Detected gestures are translated into text and converted into speech using the Google Text-to-Speech (gTTS) API. The model was trained over 50 epochs and evaluated using metrics including accuracy, precision, recall, and F1-score. The evaluation results show an accuracy of 89.74%, precision of 89.28%, recall of 96.15%, and F1-score of 92.16%, indicating strong model performance and generalization. Some misclassifications occurred, such as 'R' detected as 'L', and background images mistakenly classified as valid letters. Nonetheless, the system is able to detect and translate BISINDO sign language gestures in semi-realtime with high reliability. This study contributes both practically and theoretically to the field of assistive technologies by providing an accessible, web-based platform for BISINDO recognition, promoting more inclusive communication for the Indonesian deaf community.