Multi-Agent Reinforcement Learning for Cooperative Edge Caching in Internet of Vehicles

Abstract

Edge caching has been emerged as a promising solution to alleviate the redundant traffic and the content access latency in the future Internet of Vehicles (IoVs). Several Reinforcement Learning (RL) based edge caching methods have been proposed to improve the cache utilization and reduce the backhaul traffic load. However, they can only obtain the local sub-optimal solution, as they neglect the influence of environment by other agents. In this paper, we investigate the edge caching strategy with consideration of the content delivery and cache replacement by exploiting the distributed Multi-Agent Reinforcement Learning (MARL). We first propose a hierarchical edge caching architecture for IoVs and formulate the corresponding problem with the objective to minimize the long-term cost of content delivery in the system. Then, we extend the Markov Decision Process (MDP) in the single agent RL to the multi-agent system, and propose a distributed MARL based edge caching algorithm to tackle the optimization problem. Finally, extensive simulations are conducted to evaluate the performance of the proposed distributed MARL based edge caching method. The simulation results show that the proposed MARL based edge caching method significantly outperforms other benchmark methods in terms of the total content access cost, edge hit rate and average delay. Especially, our proposed method greatly reduces an average of 32% total content access cost compared with the conventional RL based edge caching methods.