Goal Space Planning with Reward Shaping

Abstract

Planning and goal-conditioned reinforcement learning aim to create more efficient and scalable methods for complex, long-horizon tasks. These approaches break tasks into manageable subgoals and leverage prior knowledge to guide learning. However, learned models may predict inaccurate next states and have compounding errors over long-horizon predictions. This often makes background planning with learned models worse than model-free alternatives, even though the former uses significantly more memory and computation. Methods that plan in an abstract space, such as Goal-Space Planning, avoid these typical problems of models by background planning with models that are abstract in state and time. This thesis shows how potential-based reward shaping can propagate value and speed up learning with local, subgoal-conditioned models. We demonstrate the effectiveness of this approach in tabular, linear, and deep value-based learners, and study its sensitivity to changes in environment dynamics and the chosen subgoals.