Indonesian Abstractive Summarization using Pre-trained Model

Abstract

Automatic text summarization systems are increasingly needed to encounter the information explosion caused by internet growth. Since Indonesian is still considered an under-resourced language, we take advantage of pre-trained language models to perform abstractive summarization. This paper investigates the BERT performance given the Indonesian article by comparing several BERT pre-trained models and evaluated the results based on the ROUGE values. Our experiment shows that an English pre-trained model can produce a good summary given Indonesian text, but it is more effective for using the Indonesian pre-trained model. The default training model only with the abstractive objective is better than using two-stage fine-tuning, where the extractive model must be trained in advance. We also found a lot of meaningless words in the summary words construction. This finding is the result of a preliminary study to improve the Indonesian abstractive summarization model.