Hybrid Task Scheduling in Cloud Manufacturing With Sparse-Reward Deep Reinforcement Learning

Abstract

Cloud manufacturing (CMfg) converts the traditional manufacturing system into an Internet-of-things-enabled (IoT-enabled) manufacturing system, where both manufacturing and computational tasks must be scheduled among distributed and heterogeneous resources. Deep reinforcement learning (DRL) has recently become a promising idea for task scheduling in CMfg. However, existing DRL-based methods depend heavily on problem-specific reward engineering and struggle to represent hybrid decision variables. To this end, this paper proposed the sparse-reward deep reinforcement learning (SDRL) method to solve the hybrid task scheduling problem in CMfg. First, the hybrid task scheduling model in CMfg is constructed to minimize the makespan. We reformulate the studied problem as a partially observable Markov decision process (POMDP). Then, the objective hindsight experience replay (objective HER) mechanism is proposed to alleviate the sparse reward issue, through which the scheduling policy can be effectively trained without problem-specific reward engineering. The continuous action space is defined to represent hybrid decision variables, and the implicit action-selection mapping is utilized to alleviate the boundary effect. Numerical experiments validated the effectiveness and superiority of our method compared to eleven popular scheduling algorithms including evolutionary algorithms and DRL. Compared to mainstream DRL scheduling methods, the proposed SDRL outperforms the second-best one at most by $23.6\%$ regarding generalization, and a scheduling solution can be generated in $0.5$ seconds. Note to Practitioners —With the intelligentization of the CMfg platform, hybrid tasks, including manufacturing and computational tasks, need to be scheduled simultaneously. However, this hybrid task scheduling problem is rarely considered by existing works. DRL exhibits many benefits in addressing scheduling problems, but the strong dependency on problem-specific reward engineering limits its application. Additionally, most DRL-based scheduling algorithms are discrete-action DRL, restricting their capacity to effectively represent hybrid decision variables. The studied problem originates from the CMfg platform, but the proposed method holds potential for broader application. The scheduling framework and the POMDP modeling can be applied to similar problems, including hybrid, manufacturing, or computational task scheduling problems. The proposed objective HER serves as a general approach to addressing challenges associated with sparse rewards, which can be extended to diverse combinatorial optimization problems aimed at optimizing an objective. We will open-source our codes to help others to apply the method to other fields.