Convolutional Neural Network-Based Video Super-Resolution for Action Recognition

Abstract

For video action recognition, convolutional neural networks (CNNs) especially two-stream CNNs have achieved remarkable progress in the recent years. However, most of the CNNs for action recognition are trained with high-resolution videos and not scale invariant, making it problematic to apply the trained CNNs directly on low-resolution videos. One possible solution to the problem is performing super-resolution (SR) prior to action recognition. In this paper, we investigate the effects of CNN-based video SR on the action recognition accuracy. We adopt a well trained two-stream CNN for action recognition, and analyze the spatial and temporal streams separately. For the spatial stream, we observe that video SR may improve the PSNR but may incur drop in recognition accuracy, this phenomenon is further analyzed in this paper. For the temporal stream, we observe that frame-by-frame SR may produce temporal inconsistency between consecutive video frames, which also incurs drop in recognition accuracy. We then propose a temporal consistency-oriented method for video SR, which indeed improves the recognition accuracy. Finally, we perform proper fusion of the two streams, and achieve a recognition accuracy of 88.95% on the UCF101 dataset when the input video is down-sampled by a factor of 4, compared to 93.49% accuracy on the original-resolution videos.