Distributed Hierarchical Sentence Embeddings for Unsupervised Extractive Text Summarization

Abstract

Unsupervised text summarization is a promising approach that avoids human efforts in generating reference summaries, which is particularly important for large-scale datasets. To improve its performance, we propose a hierarchical BERT [1] model that contains both word-level and sentence-level training processes to achieve semantic-rich sentence embeddings. We use the vanilla BERT as the word-level training, and redesign it for the sentence-level training with the new "Sentence Token Prediction" and "Local Shuffle Recovery" training tasks and suitable input format. We first train word-level model to get preliminary sentence embeddings, then we input them into the sentence-level model to further extract higher level and inter-sentence semantic information. After that, we obtain the context sensitive sentence embeddings and utilize them for the KMeans cluster algorithm to finally generate summaries by extracting sentences from the document. To accelerate the training of the BERT model, we adopt the PipeDream [2] model parallelism that distributes the model layers among multiple machines to conduct the training process in parallel. Finally, we show through experimental results that our proposed model outperforms most popular models and achieves a speedup of 2.7 in training time on 4 machines.