Deep Reinforcement Learning Based Cooperative Partial Task Offloading and Resource Allocation for IIoT Applications

Abstract

The Industrial Internet of Things (IIoT) has been regarded as one of the pillars supporting the conceptual paradigm of the Industry 4.0. Compared with traditional cloud computing schemes, edge computing provides an effective solution towards easing congestion in backhaul links and core networks, while meeting real-time, security and reliability demands of compute-intensive and delay-sensitive IIoT applications. Many existing studies only optimize end-edge-cloud cooperative task offloading, and neglect the optimization of the communication and computation resource allocation. In this paper, the cooperative partial task offloading and resource allocation (CPTORA) framework is designed, which jointly considers cooperation among various IIoT devices, local edge computing servers (ECSs), non-local ECSs, and cloud computing servers – for balancing the workload of the ECSs and increasing the resource utilization rate. Then, considering the complex dynamics and unpredictability in IIoT environments, the joint optimization problem is modeled as a constrained Markov decision process. Furthermore, we propose an improved soft actor-critic-based CPTORA (ISAC-CPTORA) algorithm, able to make task offloading and resource allocation decisions for each IIoT device. This algorithm innovatively introduces the idea of distributional reinforcement learning to the soft actor-critic, which can effectively reduce Q-value overestimations or underestimations. Meanwhile, this algorithm employs the prioritized experience replay to enhance its learning efficiency. Extensive laboratory experiments indicate that our CPTORA framework and ISAC-CPTORA algorithm efficiently decrease the total system costs (i.e., latency costs and energy costs), in contrast to various baseline frameworks and algorithms.