An Enhanced Deep Reinforcement Learning Approach to Motion Planning With Knowledge Transfer and Online Demonstrations

Abstract

Abstract. Deep reinforcement learning is now widely applied in motion planning problems for autonomous systems due to its model-free nature and its ability to solve complex control problems through trial and error. However, the success of deep reinforcement learning depends heavily on the exploration policy and the design of the reward function. This dependence makes it challenging to solve long-range planning problems and requires careful reward function design to avoid the sparse reward problem. In this article, we propose an enhanced deep reinforcement learning framework that learns the high-level planning action policy while considering the low-level control properties and improves training efficiency and navigation optimality. The high-level policy enables the agent to make long-term decisions with a flexible horizon. Using a high-level policy also mitigates the sparse reward problem in long-range planning tasks. By storing only high-level actions and transitions in the experience buffer, the agent can efficiently learn in long-range trajectory planning tasks. The proposed parallel architecture with separate actor and critic neural networks allows for the integration of high-level domain knowledge transfer while maintaining the ability to generate new knowledge tailored to specific problems. Integrating online demonstrations during training using global planning algorithms can significantly enhance the quality of experiences employed during reinforcement learning. Experimental results show that transferring high-level geometry knowledge and applying online error correction through demonstrations can significantly enhance the agent's performance in long-range trajectory planning tasks.