Class-Incremental Learning Based on Big Dataset Pre-Trained Models

Abstract

Deep neural networks have shown excellent performance in the field of pattern classification and are widely used. However, real-world data are often cannot be obtained at once, and the knowledge of old classes will be heavily forgotten when training new classes of data on the network, which is called catastrophic forgetting. Therefore, the incremental learning method to solve this problem came into being. In this paper, we propose a class-incremental learning method based on a big data pre-trained model, which makes full use of the large amount of public knowledge in the pre-trained model’s front network to reduce the forgetting problem of the network in subsequent classification tasks. On the basis of our previous incremental learning method based on PEDCC, we discuss the effects of different pre-trained models, training strategy, training hyperparameters, etc. PEDCC-Loss is used to constrain the cosine distance between the latent feature and the pre-defined class center, and finally the joint prediction is determined by multiple network prediction results. The algorithm in this paper is verified on the CIFAR100, Tiny ImageNet, and FaceScrub datasets with and without partial retention of old samples, and achieves the best results compared to the previous typical class-incremental learning methods. The performance in coarse-grained datasets even exceeds the accuracy of non-incremental learning without pre-trained model. Code is available in https://github.com/byBinWen/Class-Incremental-Learning-Based-on-Big-Dataset-Pre-trained-Models.