Deadline-aware load balancing for MapReduce

Abstract

As cloud computing gains its momentum in big data processing and providing on-line services, there are increasing demands to offer responsive services to users and to improve the effectiveness in server utilization. Most previous work studied the fairness among user requests, the workload balancing among servers, and the support of real-time applications individually. Different from those state-of-the-art work, we focus on the joint considerations of workload balancing and deadline satisfaction in facing user requests for MapReduce. In particular, scheduling algorithms are proposed with a constant approximation bound to balance the server workloads and, at the same time to meet the response time requirements of MapReduce jobs. The proposed scheduling algorithms are then implemented with our proposed resource manager for the open source implementation of Hadoop. We evaluate our design based on performance metrics including balancing server workloads and meeting jobs' response-time requirements. Experimental results show the effectiveness of our design through real testbed implementation.