Comparative study of DistilBERT and ELECTRA-Small Models in Spam Email Classification

Abstract

Spam email detection is one of the challenging tasks in cybersecurity due to the variability of spam content. These characteristics make it harder to identify spam, therefore researchers create different spam detection methods. Among these, Natural Language Processing (NLP) and machine learning techniques have shown outstanding results in classifying emails as spam or non-spam. Transformer-based models, such as BERT, have demonstrated pinpoint accuracy in text classification tasks. However, the computational requirements and resources are not practical in resource-limited environments. In order to mitigate this, smaller and more lightweight models, such as the DistilBERT and ELECTRA-Small, have been developed. Both models are renowned for their efficiency and accuracy. This study focuses on the comparison of these models in terms of accuracy, precision, recall, and F1 score. Experimental results revealed that while both models excel in binary classification, notable differences emerge. ELECTRA-small shows exceptional accuracy, precision and faster processing time, while DistilBERT demonstrates superior recall, highlighting its effectiveness in minimizing false negatives.