QDRL: Queue-Aware Online DRL for Computation Offloading in Industrial Internet of Things

Abstract

Recently, the Industrial Internet of Things (IIoT) has shown great application value in environmental monitoring. However, it suffers from serious bottlenecks in energy and computing capability. To address them, researchers have made lots of effort. Nevertheless, they neglect either the edge–end collaboration or the impact of task queue backlog, resulting in low system revenue. To this end, we design a queue-aware computation offloading method based on DRL (QDRL). Specifically, we represent the long-term system operation as a multistage stochastic mixed-integer optimization problem (M-SMIP), which is further converted into a deterministic problem using Lyapunov optimization. Given that the resource allocation and computation offloading in this deterministic problem are strongly coupled and difficult to solve, we decompose this problem into two subproblems. Subsequently, a reinforcement learning scheme with actor–critic architecture is designed to solve these subproblems. The Actor module is designed based on a deep learning model and quantization strategy for generating computation offloading actions. The mathematical reasoning and learning-based methods are integrated as the Critic module for achieving resource allocation. Extensive simulation results show that the performance of QDRL surpasses four baselines and approaches the approximate optimal algorithm in terms of average task queue length, normalized real computation rate, and computation time.