Deepfake Detection in Manipulated Images/ Audio/ Videos: A Three-Stage Multi-Modal Deep Learning Framework

Abstract

The proliferation of deepfake content presents a significant threat to digital integrity and necessitates the development of efficient detection techniques. This study aims to establish a three-stage framework utilizing advanced deep learning models for multimedia datasets encompassing audio, video, and image data. The initial stage comprises an XceptionNet-based image deepfake detection model developed by providing its capacity to capture subtle artifacts and inconsistencies through depth-wise separable convolutions. This model, developed using the CelebA dataset, achieved an accuracy of 95.56% for the image data. The second stage, focusing onaudio deepfakes, employs a novel approach combining Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks, selected for their capacity to process both the spatial and temporal aspects of audio data. The hybrid CNN and LSTM achieved an accuracy of 98.5% on the DEEP-VOICE dataset. The third stage, addressing video-based deepfake detection, integrates the XceptionNet and LSTM networks, harnessing the strengths of both spatial and temporal analyses. This integrated approach yields an accuracy of 97.574% across the Forensic++, DFDC, and Celeb-DF datasets. To address class imbalances in the datasets, class weighting is employed, assigning greater weights to the minority class during training, thereby enhancing the robustness of themodel. This framework is used to develop an app for detecting deepfakes across images, audio, and video data. This study underscores the significance of deep learning architectures and comprehensive datasets for accurate deepfake detection across various media forms. By advancing detection methodologies, this research contributes to combating misinformation and safeguarding the authenticity of digital content, thus supporting the preservation of online ecosystems.