Unsupervised Video Anomaly Detection with Self-Attention Based Feature Aggregating

Abstract

Anomaly detection in surveillance videos is a crucial and challenging task in the intelligent transportation systems. Previous methods utilize a memory module to store prototypical feature embeddings as normal patterns learned from normal training data. However, due to the complexity of real-world scenarios, it is difficult to choose an appropriate size of memory module which can not only memorize normal patterns comprehensively, but also capable of dealing with unseen normal scenarios. To tackle this problem, we learn normal video patterns by constructing and exploring correlations between visual semantics. In the training stage, we act the self-attention map between embeddings as a description of information association between different visual semantics. A self-attention based feature aggregating module is designed to regenerate a feature map through aggregating embeddings with similar information based on the attention map. By decoding the generated feature map instead of the original one to predict the future frame of the input video clip, the model learns to build strong information association between normal visual semantics. Moreover, we observe that abnormal embeddings hardly build strong association with others. Thus, we design a feature-level anomaly criterion referred as prior deviation to increase the difference between attention maps generated by normal and abnormal frames. In the inferring stage, the proposed prior deviation jointly detects anomalies with pixel-level frame prediction error. Experiment results and ablation studies on mainstream benchmarks demonstrate the effectiveness of our design.