A Hierarchical Hybrid Learning Framework for Multi-Agent Trajectory Prediction

Abstract

Accurate trajectory prediction for neighboring agents is crucial for autonomous vehicles navigating complex scenes. Recent deep learning (DL) methods excel in encoding complex interactions but often generate invalid predictions due to difficulties in modeling transient and contingency interactions. This paper proposes a hierarchical hybrid framework that combines DL and reinforcement learning (RL) for multi-agent trajectory prediction, capturing multi-scale interactions that shape future motion. In the DL stage, Transformer-style graph neural network (GNN) is employed to encode heterogeneous interactions at intermediate and global scales, predicting multi-modal intentions as key future positions for agents. In the RL stage, we divide the scene into local scenes based on DL predictions. A Transformer-based Proximal Policy Optimization (PPO) model, incorporated with vehicle kinematics, generates future trajectories in the form of motion planning shaped by microscopic interactions and guided by a multi-objective reward for balanced agent-centric accuracy and scene-wise compatibility. Experimental results on the Argoverse benchmark and driver-in-loop simulations demonstrate that our framework enhances trajectory prediction feasibility and plausibility in interactive scenes.