Multi-objective: hybrid particle swarm optimization with firefly algorithm for feature selection with Leaky ReLU

Abstract

Abstract High-dimensional datasets often pose challenges due to the presence of numerous irrelevant and redundant features, which can compromise the performance of machine learning models. This study proposes a novel optimization algorithm, LR-GPSOFA, designed to improve feature selection by enhancing computational efficiency and classification accuracy. The algorithm integrates Particle Swarm Optimization (PSO) with the Firefly Algorithm and Leaky Rectified Linear Unit (Leaky ReLU), utilizing the K-Nearest Neighbors (KNN) classifier to increase processing speed and ensure accurate classification. The reason behind selecting PSO is that it is well-suited for smaller search spaces, while the Firefly Algorithm (FA) excels in larger search spaces, making their hybridization particularly effective. By combining these strengths, LR-GPSOFA improves adaptability and randomness in particle motion while reducing the tendency for greedy search behavior. The inclusion of Leaky ReLU addresses the “dying ReLU” problem by preserving non-zero gradients, enabling the algorithm to navigate complex, irregular landscapes effectively. A penalty mechanism is incorporated to ensure convergence even when no features are selected, enhancing the robustness of the strategy. The algorithm was evaluated on eight diverse datasets, demonstrating significant improvements in feature selection efficiency and performance. These results highlight the potential of LR-GPSOFA as a powerful tool for feature selection in high-dimensional data, with broad applicability across various domains and real-world challenges.