Query-oriented Unsupervised Multi-document Summarization on Big Data

Abstract

Real time document summarization is a critical need nowadays, owing to the large volume of information available for our reading, and our inability to deal with this entirely due to limitations of time and resources. Oftentimes, information is available in multiple sources, offering multiple contexts and viewpoints on a single topic of interest. Automated multi-document summarization (MDS) techniques aim to address this problem. However, current techniques for automated MDS suffer from low precision and accuracy with reference to a given subject matter, when compared to those summaries prepared by humans and takes large time to create the summary when the input given is too huge. In this paper, we propose a hybrid MDS technique combining feature based algorithms and dynamic programming for generating a summary from multiple documents based on user provided query. Further, in real-world scenarios, Web search serves up a large number of URLs to users, and the work of making sense of these with reference to a particular query is left to the user. In this context, an efficient parallelized MDS technique based on Hadoop is also presented, for serving a concise summary of multiple Webpage contents for a given user query in reduced time duration.