A continual learning framework to train robust image recognition models by adversarial training and knowledge distillation

Abstract

Deep learning has been widely adopted in many image recognition tasks with great success. It has now been applied to conducting tasks on vision-based edge devices with resource limitation. To securely deploy image services on such devices, this work develops a framework to investigate the key issues relate the most to training robust and lightweight models. The primary issue is about safety. It is well known that the deep learning method is vulnerable to well-designed attacks. To meet the challenge of continuously evolving attacks, we develop a new defensive approach that integrates continual learning and adversarial training to improve the defensive model's corruption robustness and structure compactness. This approach adopts the structure of a progressive neural model to establish a robust model over time. The second issue is about how to train a lightweight model. Regarding this, we develop another new knowledge distillation approach that integrates both self-distillation and multi-teacher distillation techniques. With such a specially designed student-teacher learning structure and new loss functions, our approach is thus adaptable to different task situations. To verify the proposed framework, we have conducted a series of experiments to evaluate our approach. The results well confirm its functions and effectiveness.