Federated Reinforcement Learning Algorithm with Fair Aggregation for Edge Caching

Abstract

Edge caching is employed to solve the challenge of massive data requests, ensuring the quality of user experience. However, existed edge caching algorithms often overlook issues related to user mobility, and privacy protection, non-identically and independently distributed (non-i.i.d.) characteristics of content requests among base stations. To tackle these challenges, this paper proposes Federated Reinforcement Learning Algorithm with Fair Aggregation for Edge Caching (FFA-PPO) algorithm. This paper primarily focuses on scenario of non-i.i.d. content requests in multi-base-station and multi-mobile-user network. We model this problem as a Markov Decision Process (MDP) problem and propose a federated reinforcement learning method to solve MDP problem. The goal is to minimize the content transmission latency of base stations. FFA-PPO algorithm resolves gradient conflicts by seeking the optimal gradient vector within a local ball centered at the averaged gradient which ensures model’s fairness. In conclusion, simulation results prove that the proposed FFA-PPO algorithm outperforms other baseline algorithms in terms of content transmission latency, model’s fairness.