Hierarchical Gated Convolutional Networks with Multi-Head Attention for Text Classification

Abstract

Text classification is a fundamental problem in natural language processing. Recently, neural network models have been demonstrated to be capable of achieving remarkable performance in this domain. However, none of existing method can achieve excellent classification accuracy while concerning of computational cost. To solve this problem, we proposed hierarchical gated convolutional networks with multi-head attention which reduces computational cost through its two distinctive characteristics to save considerable model parameters. First, it has a hierarchical structure the same as the hierarchical structure of documents that has word-level and sentence-level, which not only benefits to classification performance but also reduces computational cost significantly by reusing parameters of the model in each sentence. Second, we apply gated convolutional network on both levels that enables our model achieved comparable performance to very deep networks with relatively shallow network depth. To further improve the performance of our model, multi-head attention mechanism is employed to differentiate more or less importance of words or sentences for better construction of document representation. Experiments conducted on the commonly used Yelp reviews datasets demonstrate that the proposed architecture obtains competitive performance against the state-of-the-art methods.