Federated deep reinforcement learning for dynamic job scheduling in cloud-edge collaborative manufacturing systems

Abstract

The cloud-edge collaborative manufacturing system (CCMS) connects distributed factories to a cloud centre through cloud-edge collaborative communication, introducing both opportunities and challenges to conventional dynamic job scheduling. Enhancing each factory's scheduling performance by sharing general scheduling knowledge among heterogeneous factories under the consideration of data privacy protection remains challenging. To this end, this paper proposes to solve the dynamic job scheduling in the context of CCMS with a novel federated deep reinforcement learning (FDRL) approach. Within each factory, the scheduling objective involves minimising the makespan and energy consumption, accounting for machine warm-up procedures and real-time dynamics. To handle heterogeneous policy structures, we aggregate their hidden parameters through FDRL, with states, actions, and rewards designed to facilitate the aggregation. The two-phase algorithm, comprising iterative local training and global aggregation, trains the scheduling policies. Constraint items are introduced to the loss functions to smooth local training, and the global aggregation considers production scales and obtained objectives. The proposed approach enhances the solution quality and generalisation of each factory's scheduling policy without exposing original production data. Numerical experiments conducted on sixty scheduling instances validate the superiority of the proposed approach compared to twelve dynamic scheduling methods. Compared to independently trained DRL-based approaches, the proposed FDRL-based approach achieves up to an 8.9% reduction in makespan and a 22.3% decrease in energy consumption through knowledge sharing.