Multi-Scale Feature Distillation for Anomaly Detection

Abstract

Due to the diversity and scarcity of defective samples, anomaly detection is usually taken as one of the main solutions of defect detection in intelligent manufacturing industry. Unsupervised anomaly detection usually faces two main challenges: (1) Without effective knowledge transferring, diversity and scarcity of defect samples led to model's training difficulties, especially for deep CNN model; (2) Large scale variations of defects led to the lack of generalization ability for typical CNN models with single or a few scale represented features. In this work, we propose a simple yet powerful unsupervised anomaly detection method by: i) The teacher and student networks are introduced to train deep CNN-based detection networks. With a powerful deep pre- trained neural network as teacher, and one proposed combined- distance based loss function, a student network with the same architecture is developed to learn teacher's knowledge, and the defective regions are recognized by the larger feature distances compared with the normal regions; ii) Multi-scale feature distillation is developed to against the scale variations. Unlike most knowledge distillation methods that only distill feature output by the last layer, we distill the features on different levels of feature maps output by the intermediate layers, and develop one combined-distance and Gaussian Kernel based receptive field up-sampling mechanism, to generate high resolution anomaly heatmap. Evaluation of the widely used MVTec anomaly detection dataset shows that our method is competitive to the state-of-the-art methods with over 10 times inference speed increase: from 2.8fps@256 × 256 to 33.2fps; The experiments on other two anomaly detection datasets, ShanghaiTech Campus dataset (STC) and Magnetic Tile Defects dataset (MTD), are also given to verify the generalization of the proposed work.