Exploring Arabic Pre-Trained Language Models for Arabic Abstractive Text Summarization

Abstract

Abstractive text summarization (ABS) is the task of providing a concise and meaningful summary for a given text. It is accomplished by first understanding the text and then rephrasing it in a shorter form, highlighting the main points of the original text. ABS is useful in applications such as news aggregators, article summarizers, legal-case summary production, business-meeting summarization, and social-media summarization. There are many types of ABS such as headline summaries, highlight summaries and full summaries. Compared to research on ABS in other languages, study of its use for the Arabic language is still limited. Moreover, very few works explore the use of pretrained transformer-based models for Arabic language ABS. This study investigates the effectiveness of pretrained transformer-based models for the downstream task of Arabic ABS. We present several experiments to fine-tune pretrained transformer model AraBArtfor Arabic text summarization. The used datasets are AHS dataset for headline summaries, WikiLingua dataset for highlight summaries, and XL-Sum dataset for full summaries. We report on the performance of the models using the ROUGE, BLEU and BERTScore metrics for ABS evaluation. We compare the results of the models with the state of the art for Arabic ABS. The best performance obtained with our experiments is the AraBART model fine-tuned on the AHS dataset to generate headline summaries, with the following results: ROUGE-1=55, ROUGE-2=40.15, ROUGE-L=54.55, BLEU=56.26, and BERTScore=88.06.