Multimodal AI-Based Mock Interview System: Integrating Facial Expression Analysis, Speech Emotion Recognition, and NLP for Holistic Candidate Evaluation

Abstract

Traditional interview preparation focuses on technical knowledge while neglecting critical soft skills evaluation including facial expressions, vocal tone, and emotional stability. This research presents a Multimodal AI-Based Mock Interview System (MMIS) that integrates facial expression analysis, speech emotion recognition, and natural language processing to provide comprehensive feedback on interview performance. The system employs convolutional neural networks for emotion detection, MFCC-based speech analysis for vocal assessment, and NLP techniques for answer evaluation using the STAR framework. The architecture includes four modules: facial expression recognition (82% accuracy), speech emotion recognition (91% accuracy), answer evaluation (87% accuracy), and multimodal feedback generation. Testing with 150 candidates shows significant improvements in interview confidence (34.2% improvement, p < 0.001) and communication effectiveness (28.7% improvement, p < 0.001). Candidates using MMIS for three or more sessions achieve 3.2x higher interview success rates compared to traditional methods. The system provides scalable, objective, and accessible interview preparation without requiring professional coaching. This research addresses the research gap in multimodal candidate assessment and contributes to equitable interview readiness technology. Future work includes virtual reality integration and multilingual support.