Cooperative Multi-Agent Deep Reinforcement Learning for Computation Offloading in Digital Twin Satellite Edge Networks

Abstract

With the development of commercial off-the-shelf hardware, low Earth orbit (LEO) satellites are promising to provide flexible edge computing services. In this paper, we investigate a digital twin (DT)-empowered satellite-terrestrial cooperative edge computing network, where computation tasks from terrestrial users can be partially offloaded to the associated base station (BS) edge server, the associated LEO satellite edge server, and an adjacent LEO satellite edge server. We formulate a multi-tier computation offloading optimization problem to minimize the weighted sum of total system delay and satellite energy consumption, where a LEO-layer problem and a DT-layer problem are involved. The LEO-layer problem optimizes the three-tier computation resource allocation and task splitting ratio. From the multi-satellite network perspective, the DT-layer problem optimizes how many resources will be shared between adjacent satellites. We then propose a multi-agent double actors twin delayed deterministic policy gradient (MA-DATD3) algorithm to optimize the LEO-layer problem, and adopt a centralized training and decentralized execution (CTDE) paradigm. The proposed MA-DATD3 algorithm is extended to solve the DT-layer problem in a centralized way, and the resource sharing between adjacent satellites is optimized to maximize the time-averaged reward. Simulation results show that our algorithm achieves a better performance than the MADDPG algorithm, and effectively improves the computation offloading performance while balancing the energy consumption and the total delay.