Scheduling real-time workflow on MapReduce-based cloud

Abstract

As a popular programming model in cloud-based data processing environment, MapReduce and its open source implementation Hadoop, are widely applied both in industry and academic researches. A key challenge in MapReduce-based cloud is the ability to automatically control resource allocations to real-time workflows for achieving their custom-defined deadlines. Current researches on deadline-related MapReduce schedulers only support soft real-time scheduling, where the extension of the deadline is allowed. In this paper, the hard real-time scheduling problem with a strict deadline on MapReduce-based cloud is studied. We propose a SPS scheduler that can guarantee job completion time before the specified deadline for real-time workflows. SPS supports job preemption with low context-switch overhead so that it can make online scheduling decision when workflows randomly arrive in cloud. Experiments on Hadoop show that SPS effectively meets the deadline constraint even if the workflow demands exceed the cluster resources.