MULTIMODAL HUMAN-COMPUTER INTERACTION USING HAND, EYE AND VOICE GESTURES

Abstract

This study presents a multimodal human-computer interaction (HCI) system that integrates hand gestures, eyegestures, and voice commands to control computer actions. Utilizing robust computer vision libraries likeMediaPipe, and SpeechRecognition and PyAutoGUI, the system provides a highly customizable experiencewhere in users can train and map their own gestures or voice commands to particular actions like mouse clicks,volume control, application launching, and screen interaction. Implemented in Python using MediaPipe,PyAutoGUI, and the SpeechRecognition library, the solution supports real-time gesture recognition, actionmapping, and system control. This work demonstrates the potential for inclusive, accessible, and intuitiveinterfaces across platforms.