Hierarchical game-theoretic control for multi-agent autonomous racing

Abstract

We develop a hierarchical control scheme for autonomous racing with realistic safety and fairness rules. The first part constructs a discrete game approximation with simplified dynamics and rules presented as temporal logic specifications. Using the discrete representation, we use a model checking tool to synthesize an optimal strategy in the form of a sequence of target waypoints. We apply the model to several case studies of common racing scenarios, and its resulting strategies are qualitatively verified against those executed by racing experts. This formulation is used as the high-level planner in the hierarchical controller but is solved using Monte Carlo tree search (MCTS) to run in real-time. In the next part, we integrate the high-level planner with a low-level controller to track the target waypoints. Two low-level approaches are considered: a multi-agent reinforcement learning (MARL) trained policy and a linear-quadratic Nash game (LQNG) formulation. As a result, we produce two hierarchical controllers, MCTS-RL and MCTS-LQNG, respectively. The hierarchical agents are tested against three baselines: an end-to-end MARL controller, a MARL controller tracking the optimal racing line, and an LQNG controller tracking the optimal racing line. The controllers compete head-to-head on an oval track and a complex track, and we count the number of wins and a safety score representing the number of rule violations. Our hierarchical controllers outperform their respective baseline methods in terms of wins, but only MCTS-RL is better than its baselines in terms of safety score. The MCTS-LQNG controller has a worse safety score, but this result is due to the simplicity and conservative nature of the fixed trajectory LQNG baseline. Overall, the MCTS-RL controller outperforms all of the other controllers across both metrics and executes maneuvers resembling those seen in real-life racing. In the final part, we extend the hierarchical controllers to team-based racing where they must consider a mixture of competitive and cooperative objectives. The formulations are generalized to consider these challenging objectives while still being required to adhere to the complex rules. We test our controllers against the previously discussed baselines in races where the agents compete in teams of two instead of head-to-head. In addition to counting the number of wins and the safety score, we introduce a third metric to measure the cooperative performance of the controllers. We allocate points based on the finishing position of each agent and aggregate them across the teams, which indicates how the team performed as a whole. The results show our hierarchical agents outperforming their baselines in terms of wins while maintaining similar safety scores. In addition, our controllers also have a higher number of average points per race indicating that they produce greater success as a team. Finally, we observe that the MCTS-RL controller continues to outperform all of the other implemented controllers across all metrics and exhibits tactics performed by expert human drivers. We show that hierarchical planning for game-theoretic reasoning produces competitive behavior even when challenged with complex objectives, rules, and constraints.