Learning Discriminative Feature Representation for Open Set Action Recognition

Abstract

Open set action recognition (OSAR) is a challenging task that requires a classifier to identify actions that do not belong to any of the classes in its training set. Existing methods employ the Evidential Neural Network (ENN) as an open-set classifier, which is trained in a supervised manner on feature representations from known classes to quantify the predictive uncertainty of human actions. In this paper, we propose a novel framework for OSAR that enriches the discriminative representation from a backbone with a reconstructive one to further improve performance. Our approach involves augmenting the input features with their reconstruction obtained from a reconstruction-based model in unsupervised training on known classes. We then use the correspondence between the two features to learn the open-set classifier, forcing it to associate low correspondence both when the feature is from unknown classes as well as when the input feature and its reconstruction variant are inconsistent with each other. Our experimental results on standard OSAR benchmarks demonstrate that our end-to-end trained model significantly outperforms state-of-the-art methods. Our proposed approach shows the effectiveness of combining discriminative and reconstructive representations for OSAR.