Resampling Techniques for Learning Under Extreme Verification Latency with Class Imbalance

Abstract

A common, yet rarely addressed, real-world problem in computational intelligence applications is learning from non-stationary streaming data, where the underlying distribution of the data changes over time. This problem, also referred to as concept drift, is made even more challenging if, after initially receiving a small set of labeled data, the streaming data only consists of unlabeled data, requiring the learner to adapt to changing underlying distribution without the benefit of labeled data. This particular scenario is typically referred to as learning in initially labeled nonstationary environment, or as extreme verification latency (EVL), pointing to the fact that the label verification of the test data is indefinitely delayed. In our prior work, we have noted that current EVL algorithms - including the algorithm COMPOSE that we have developed - are largely unable to track changing distributions if the data drawn from those distributions are even mildly imbalanced. In this work, we integrate COMPOSE with 13 different resampling based modified algorithms, and compare accuracy, F1 score, and execution time. The results differed from what we originally expected and provided unique insight on how to choose a data rebalancing approach for different types of drift.