A Two-Pathway Convolutional Neural Network with Temporal Pyramid Network for Action Recognition

Abstract

In order to solve the problem of imperfect capture of visual rhythm in action recognition, this paper proposes a novel model which is a combination of a two-pathway network and temporal pyramid networks (TPNs). Specifically, our work involves two aspects, on the one hand, we integrate TPNs into the fast pathway and the slow pathway of SlowFast network to capture multi-level features, and then merge the prediction results of the two pathways in the final recognition stage, which boosts performance of our network by enhancing the semantics extraction at input layer and feature layer. On the other hand, we apply a ConvLSTM module to improve the capability of temporal modeling in TPN, which can further strengthen the capture of features in the long-term dimensions, and the advanced TPN promotes the fusion of temporal and spatial features. Experiments on the Kinetics-400 dataset demonstrate the superiority of our novel architecture combining two-pathway network and advanced TPN in action recognition.