Energy-Aware Dynamic Resource Allocation on Hadoop YARN Cluster

Abstract

The web big data applications are executed on Hadoop cluster in the cloud datacenter, which requires large amounts of energy. And the energy costs take a considerable fraction of the data center's overall costs. Therefore, the reduction of the energy consumption in the cloud datacenter becomes a critical issue. In this paper, we propose energy-aware dynamic node management technology for online MapReduce jobs by powering on/off nodes in Hadoop cluster to reduce energy consumption while meet user's Service Level Agreements (SLA). Under the dynamic node management policy, the time-varying workload is predicted by extracting the MapReduce job history information continuously. And then, the energy-aware dynamic node management with deadline-driven is used to keep the proper number of nodes for MapReduce tasks based on the average execution time of containers and predictive workloads. Finally, the nodes which have been kept in idle state for threshold duration are turned off to reduce energy costs. We perform extensive simulations on a Yarn Scheduler Load Simulator (SLS) to exploit the energy consumption, the violations on SLA and execution time for each big data application in a period of time. The experimental results demonstrate that our proposed policy to achieve energy savings over comparable four policies with respect to meeting SLA.