Multimodal Deep Learning Framework for Analyzing Mobile Video Communication Effectiveness

Abstract

To extend computational communication research in the era of visual media, this study investigates scalable methods for large-scale video content analysis beyond traditional textual data. We built a massive dataset of mobile phone review videos from Bilibili and trained a customized YOLOv11 deep learning model for automatic recognition, extraction, and quantification of core visual elements in video frames, enhancing analytical efficiency. For hand–object interaction detection, we propose Hand-YOLO, a lightweight algorithm combining an HGNetv2 (High Performance GPU Network V2) backbone with a CCFM (Cross-Scale Feature Fusion Module) neck, achieving high accuracy with reduced computational cost—suitable for real-time processing of large datasets. We further integrate advanced natural language processing, employing large language models to analyze video titles and extract emotional intensity features. Correlation analyses link these multimodal variables—visual elements and textual emotion—with indicators of communication effectiveness, including views and user engagement. Statistical results reveal that mobile phone appearance frequency and image brightness are significant positive predictors of communication performance. Conversely, title emotional intensity shows a negative correlation with effectiveness, challenging common assumptions. This study offers empirical insights for optimizing visual communication strategies and presents a replicable, cost-efficient framework for multimodal video analysis. By combining computer vision and NLP in an integrated approach, it advances computational communication methodology and provides a validated paradigm for future large-scale video content research.