Multi-Armed Bandit Test Case Prioritization in Continuous Integration Environments

Abstract

Continuous Integration (CI) practices lead the software to be integrated and tested many times a day, usually subject to a test budget. To deal with this scenario, cost-effective test case prioritization techniques are required. COLEMAN is a Multi-Armed Bandit approach that learns from the test case failure-history the best prioritization order to maximize early fault detection. Reported results show that COLEMAN has reached promising results with different test budgets and spends, in the worst case, less than one second to execute. However, COLEMAN has not been evaluated against a search-based approach. Such an approach can generate near-optimal solutions but is not suitable to the CI budget because it takes too long to execute. Considering this fact, this paper analyses the trade-offs of the COLEMAN solutions in comparison with the near-optimal solutions generated by a Genetic Algorithm (GA). We use measures, which better fit with time constraints: Normalized Average Percentage of Faults Detected (NAPFD), Root-Mean-Square-Error (RMSE), and Prioritization Time. We use seven large-scale real-world software systems, and three different test budgets, 10%, 50%, and 80% of the total time required to execute the test set available for a CI cycle. COLEMAN obtains solutions near to the GA solutions in 90% of the cases, but scenarios with high volatility of test cases and a small number of cycles hamper the prioritization.