Efficient Job Scheduling and Resource Allocation using Load Rebalancing on Big Data

Abstract

In recent days, managing big data has been one of the key challenges for managing data effectively and efficiently. This data is generally utilized in all online media, web-based business, and web applications. To manage and store huge volumes of data sets the Hadoop Distributed File System is quite possibly the most broadly utilized frameworks. With respect to job scheduling, HDFS is additionally testing as it assumes a critical part in upgrading time in huge information. Even though there are many scheduling algorithms in the existing works because they are not very efficient in working with dynamic Hadoop environment that is Hadoop cluster with dynamically available resources due to various issues. For example, there is no time limit for the tasks allocated for the dynamic resource allocation. To deal with such issues, this paper presents efficient scheduling and dynamic resource allocation using load rebalancing techniques that take into account future asset accessibility when limiting job deadline misses. Existing problems can define a job scheduling problem with an optimized scheduling cycle by minimizing iteration, and then dynamically allocating resources using the proposed Load Rebalancing technique. The tasks differ in the existing algorithms and offer algorithms for experiments to prove time and time complexity and their implementation is performed in an open-source Hadoop environment. Experiments have proven that the proposed job scheduling algorithm reduces the quantity of repetitions and improves time productivity by dynamically allocating resources compared to the deadline-aware scheduling algorithm.