Modular Deep Reinforcement Learning for Continuous Motion Planning With Temporal Logic

Abstract

This paper investigates the motion planning of autonomous dynamical systems\nmodeled by Markov decision processes (MDP) with unknown transition\nprobabilities over continuous state and action spaces. Linear temporal logic\n(LTL) is used to specify high-level tasks over infinite horizon, which can be\nconverted into a limit deterministic generalized B\\"uchi automaton (LDGBA) with\nseveral accepting sets. The novelty is to design an embedded product MDP\n(EP-MDP) between the LDGBA and the MDP by incorporating a synchronous\ntracking-frontier function to record unvisited accepting sets of the automaton,\nand to facilitate the satisfaction of the accepting conditions. The proposed\nLDGBA-based reward shaping and discounting schemes for the model-free\nreinforcement learning (RL) only depend on the EP-MDP states and can overcome\nthe issues of sparse rewards. Rigorous analysis shows that any RL method that\noptimizes the expected discounted return is guaranteed to find an optimal\npolicy whose traces maximize the satisfaction probability. A modular deep\ndeterministic policy gradient (DDPG) is then developed to generate such\npolicies over continuous state and action spaces. The performance of our\nframework is evaluated via an array of OpenAI gym environments.\n