Two-Branch Stacked Transformer for 2D Skeleton-based Action Recognition

Abstract

Human Action Recognition (HAR) is a challenging computer vision task with various applications, ranging from smart surveillance to human-computer interaction. Recently, the human skeleton, a compact and intuitive data modality, has attracted increasing attention in many studies and has achieved good results in HAR. However, some challenges such as body occlusion and action similarity still need to be addressed. In this paper, to overcome these challenges, we propose a model combining short action-snippets for storing meaningful information about human body transition and a deep network configured by two parallel branches of Transformer for thoroughly learning the temporal correlation of skeletal representations in upper and lower body parts, hence concurrently enabling to handle of partial occlusions of skeleton data and boosting the HAR performance. In experiments, we validate the proposed approach's outperformance compared with the state-of-the-art methods on the JHMDB dataset in terms of both the size (i.e., number of learned parameters) and the accuracy.