Multi-Agent Deep Reinforcement Learning for Cooperative Edge Caching via Hybrid Communication

Abstract

Though caching on edge servers is widely acknowledged to be essential, it is not trivial to cache content on edge servers adaptively without any prior knowledge of the distribution of content popularity across the users. Several edge caching algorithms have been proposed in the literature based on multi-agent reinforcement learning (MARL) for dynamic control, however, they ignored the non-stationarity and partial-observability issues commonly existing in multi-agent systems. In an MARL-based edge caching application where agents collaborate towards a common goal, communication is essential as their decisions are jointly applied to improve collective intelligence. However, most existing methods proposed to exchange messages between agents have not considered the induced communication overhead, which is critical in practice with real-world multi-agent applications. In this paper, we propose a new MARL framework for edge caching where agents learn to construct, exchange and interpret collective messages for individual benefits, while controlling the complex collaborative task of cache replacement in a communication-efficient manner. With a standard edge caching model, we show that with limited communication and delays introduced, our proposed framework is able to outperform existing rule-based and learning-based caching policy alternatives.