Reinforcement Learning with Probabilistically Safe Control Barrier Functions for Ramp Merging

Abstract

Prior work has looked at applying reinforcement learning (RL) approaches to autonomous driving scenarios, but the safety of the algorithm is often compromised due to instability or the presence of ill-defined reward functions. With the use of control barrier functions embedded into the RL policy, we arrive at safe policies to optimize the performance of the autonomous driving vehicle through the advantage of a safety layer over the RL methods to ease the design of reward functions. However, control barrier functions need a good approximation of the model of the system. We use probabilistic control barrier functions [4] to account for model uncertainty. Our Safety-Assured Policy Optimization - Ramp Merging (SAPO-RM) algorithm is implemented online in the CARLA [1] Simulator and offline on the US I-80 dataset extracted from the NGSIM Database provided by NHTSA [2]. We further test the algorithm and perform ablation studies of it on the US-101 and exi-D datasets to compare the approaches. The proposed algorithm can also be applied to other driving scenarios by changing the reward and safety constraints.