Real-Time Multi-Camera Multi-Person Action Recognition using Pose Estimation

Abstract

Action recognition possesses challenging issues in real-time multi-camera scenario when dealing with multi-person such as occlusion, pose variance and action interaction. In this paper, a real-time pipeline is proposed to address multi-person action recognition in multi-camera setup using joint key-points sequences from detected person. Joints trajectory is the important time-series information to identify actions. 14 key-points from human joints are scaled with relative to the Euclidean distance of neck-to-pelvis to obtain standard size of person, which is invariant to camera distance. Subsequently, 3D histogram correlation is applied to match multi-person identity. An indexed person with a series of action attribute are collected and fed into Long Short-Term Memory (LSTM) recurrent neural network. The proposed pipeline uses spatial-temporal feature of person's joint key-points trajectory for action recognition. Minimal single pass forward time through the LSTM network enables real-time multi-person action recognition in a video sequence. The proposed pipeline achieved up to 13 frames per second with 92% recognition rate with two camera setups.