Modeling Human Driving Behavior Through Generative Adversarial Imitation Learning

Abstract

An open problem in autonomous vehicle safety validation is building reliable\nmodels of human driving behavior in simulation. This work presents an approach\nto learn neural driving policies from real world driving demonstration data. We\nmodel human driving as a sequential decision making problem that is\ncharacterized by non-linearity and stochasticity, and unknown underlying cost\nfunctions. Imitation learning is an approach for generating intelligent\nbehavior when the cost function is unknown or difficult to specify. Building\nupon work in inverse reinforcement learning (IRL), Generative Adversarial\nImitation Learning (GAIL) aims to provide effective imitation even for problems\nwith large or continuous state and action spaces, such as modeling human\ndriving. This article describes the use of GAIL for learning-based driver\nmodeling. Because driver modeling is inherently a multi-agent problem, where\nthe interaction between agents needs to be modeled, this paper describes a\nparameter-sharing extension of GAIL called PS-GAIL to tackle multi-agent driver\nmodeling. In addition, GAIL is domain agnostic, making it difficult to encode\nspecific knowledge relevant to driving in the learning process. This paper\ndescribes Reward Augmented Imitation Learning (RAIL), which modifies the reward\nsignal to provide domain-specific knowledge to the agent. Finally, human\ndemonstrations are dependent upon latent factors that may not be captured by\nGAIL. This paper describes Burn-InfoGAIL, which allows for disentanglement of\nlatent variability in demonstrations. Imitation learning experiments are\nperformed using NGSIM, a real-world highway driving dataset. Experiments show\nthat these modifications to GAIL can successfully model highway driving\nbehavior, accurately replicating human demonstrations and generating realistic,\nemergent behavior in the traffic flow arising from the interaction between\ndriving agents.\n