Resource Waste-Aware Dynamic Workflow Scheduling in Multicluster

Abstract

In a multicluster environment that allocates resources dynamically, clusters' local tasks also arrive dynamically, making it difficult to schedule and respond to the demands on computing resources. When scheduling a complex scientific workflow, there are problems with available resource uncertainty and dynamics. Here, we show how to predict local tasks' running time in a cluster with accuracy, using a flexible system-prediction method for an environment undergoing dynamic resource changes. This method-Waste-Aware Heterogeneous Earliest Finish Time (WA-HEFT)-formulates a resource waste-perception scheduling strategy, reduces resource competition, and reduces the overall completion time for a complex scientific workflow. In addition, when a significant deviation exists between the actual operation of workflow tasks and the initial scheduling, we rearrange the target workflow tasks by formulating an adaptive rescheduling strategy, so that we can complete the target workflow as soon as possible. Simulation results show that our algorithm is superior to existing algorithms in the makespan.