Unsupervised Temporal Adaptation in Skeleton-Based Human Action Recognition

Abstract

With deep learning approaches, the fundamental assumption of data availability can be severely compromised when a model trained on a source domain is transposed to a target application domain where data are unlabeled, making supervised fine-tuning mostly impossible. To overcome this limitation, the present work introduces an unsupervised temporal-domain adaptation framework for human action recognition from skeleton-based data that combines Contrastive Prototype Learning (CPL) and Temporal Adaptation Modeling (TAM), with the aim of transferring the knowledge learned from a source domain to an unlabeled target domain. The CPL strategy, inspired by recent success in contrastive learning applied to skeleton data, learns a compact temporal representation from the source domain, from which the TAM strategy leverages the capacity for self-training to adapt the representation to a target application domain using pseudo-labels. The research demonstrates that simultaneously solving CPL and TAM effectively enables the training of a generalizable human action recognition model that is adaptive to both domains and overcomes the requirement of a large volume of labeled skeleton data in the target domain. Experiments are conducted on multiple large-scale human action recognition datasets such as NTU RGB+D, PKU MMD, and Northwestern–UCLA to comprehensively evaluate the effectiveness of the proposed method.