Multi-Agent Reinforcement Learning for Cooperative Edge Caching in Heterogeneous Networks

Abstract

Heterogeneous networks (HetNets) and mobile edge caching have emerged as promising paradigms for reducing content delivery delay and alleviating network congestion. However, due to the rapid growth of applications and the limited capacity of cache servers, how to make the best caching decision is still a challenging problem in HetNets. Recently, the cooperative caching scheme is proposed as a promising solution. In this paper, we propose a two-phases cooperative edge caching strategy for content caching and content delivery problem in HetNets. The optimization problem is formulated as minimizing the difference between total content delivery delay and cache hit rate in each small cell under the constraint of cache capacity. To solve the above problem, we propose a cooperative edge caching algorithm based on multi-agent deep deterministic policy gradient (MAD-DPG), which enables each small base station (SBS) equipped with an edge server to cooperatively learn the optimal caching strategy. The proposed algorithm has the characteristics of centralized learning and decentralized execution. When the offline training is completed, each SBS can quickly and independently make caching and bandwidth allocation decisions online during the execution phase. Simulation results demonstrate that the proposed MADDPG-based cooperative edge caching algorithm can effectively reduce content delivery delay and improve cache hit rate compared with other baseline schemes.