Hierarchical Reinforcement Learning in Communication-Mediated Multiagent Coordination

Abstract

Over the past decade, reinforcement learning (RL; e.g., see [9]) has been an active area of AI research in general and of research on agents and multiagent systems (MASs) in particular. In the originalMarkov decision process (MDP; e.g., see [5]) formulation of RL, other agents an agent is coexisting and interacting with are treated as part of its environment. The inability of MDPs to model multiple adaptive agents has explicitly been identified as the main drawback of this approach [4]. As a consequence, interest has grown in extending the RL framework to explicitly take into account other agents as autonomous and self-interested entities (see [8] for an overview). In this paper, we follow this line of research while focussing on communication-mediated multiagent coordination problems. The idea here is that “physical” acting can be preceded by communication to allow for a prediction of actions to come. By assuming that this kind of communication does not manipulate the environment (i.e. hardly affects the states agents find themselves in) and does not have effects w.r.t. utility, we can view the exchanged messages as symbols that “encode” anticipated courses of physical action. This is in accordance with the model of communication we have laid out in [6]. We make two contributions to the solution of communication-mediated multiagent coordination problems: