Multi-Modal Anomaly Detection by Using Audio and Visual Cues

Abstract

This paper considers the problem of anomaly detection in an outdoor environment where surveillance cameras are usually installed to monitor activities of general public. A novel solution is proposed which combines audio and visual data to automatically detect abnormal activities. The proposed anomaly detection algorithm makes use of both visual and audio features to automatically detect anomalous activities in scenes. Visual features such as optical flow technique combined with particle swam optimization and social force model are used, whereas, acoustic features such as, energy, zero crossing rate, volume, spectral-centroid, spectral spread, spectral roll-off, spectral flux, cross correlation and the mel-frequency cepstral coefficients (MFCCs) are used. An anomaly inference is developed which is based on both visual and audio features. The performance of the proposed algorithm is evaluated by testing it on the publicly available UMN datasets combined with the audio recordings. The proposed algorithm is compared with state-of-the-art techniques and is shown to achieve improved performance in terms of accuracy.