Physical Computing Resource Scheduling on Distributed Stream Processing System

Abstract

Recently, distributed stream computing has emerged as a popular model for real-time data stream analytics.This paper focuses on computing resources scheduling of distributed stream processing.To adapt to the fluctuation of data stream, distributed stream processing system must elastically provision the computing resources based on the observed workload.However, there is no system that satisfies all these requirements.Motivated by this, this paper presents a physical resource scheduling algorithm for elastic scheduling computing resources of stream processing task.Base on existing task tier elastic resource scheduling and system-tier resource scheduling, this paper proposes a collaborative algorithm.According to the fluctuant workload, task-tier dynamic resource scheduler config computing resource of stream processing task.Meanwhile, based on the scheduling decision of task-tier scheduler, the system-tier resource scheduler is assign physical computing resources to stream processing tasks.