De-Obfuscating Emerging Malware Threats Using Reverse Engineering Techniques

Abstract

Abstract: The rapid evolution of malware poses a significant cybersecurity challenge, as attackers increasingly employ sophisticated obfuscation techniques to evade detection. Polymorphic and metamorphic malware utilise different obfuscation techniques such as packing, encryption, and code mutation to evade traditional signature-based detection models. Conventional static and dynamic analysis tools frequently struggle to de-obfuscate these threats. This paper proposes a novel system that integrates reverse engineering techniques with ensemble learning models containing Gradient Boosting Machines, Convolutional Neural Networks, and Gated Recurrent Units, to address these challenges. Reverse engineering enables in-depth structural and behavioural analysis of malicious binaries, exposing hidden payloads and execution patterns. Ensemble learning enhances detection by combining the strengths of multiple algorithms to improve accuracy and adaptability. The proposed system not only identifies obfuscated malware with high precision but also predicts emerging variants, offering resilience against evasion tactics. By uniting explainable reverse engineering with advanced ensemble learning, the system provides scalable, real-time protection against evolving malware threats.