Neighborhood Graph Filters Based Graph Convolutional Neural Networks for Multi-Agent Deep Reinforcement Learning

Abstract

Multi-agent deep reinforcement learning (MADRL), where a group of agents inside multi-agent systems cooperate to achieve a common goal, has been shown useful in many applications such as collaborative robots, autonomous driving or video games involving teams. In this paper, we propose two multi-agent deep reinforcement learning (MADRL) frameworks for value function factorization built by using Graph Convolutional Neural Networks (GCNN) based on neighborhood graph filters (NGFs). These MADRL frameworks are based on the paradigm of centralized training with decentralized execution (CTDE). In this work, we show that the superior stability of the NGFs as compared to standard graph filters leads also to superior performance for the MADRL algorithms. In the first MADRL framework, the NGF-based GCNN is used to predict the local q-value at each of the agents, while in the other one, the NGF-based GCNN is used to mix the local q-values to generate a global Q-value. We have compared the performance of NGF-based GCNNs over state-of-the-art graph neural networks for value function factorization in the MADRL framework for the StarCraft II and Coalition Structure Generation problems. The results show that the proposed MARL frameworks outperform the existing state-of-art architectures.