An EnergySaving Snake Locomotion Pattern Learned in a Physically Constrained Environment With Online ModelBased Policy Gradient Method

Abstract

Snake robots, composed of sequentially connected joint actuators, have recently gained increasing attention in the industrial field, like life detection in narrow space. Such robots can navigate the complex environment via the cooperation of multiple motors located on the backbone. However, controlling the robots in a physically constrained environment is challenging, and conventional control strategies can be energyinefficient or even fail to navigate to the destination. This work develops a snake locomotion gait policy for energyefficient control via deep reinforcement learning (DRL). After establishing the environment model, we apply a physics constrained online policy gradient method based on the proximal policy optimization (PPO) objective function of each joint motor parameterized by angular velocity. The DRL agent learns the standard serpenoid curve at each timestep. The policy is updated based on the robot’s observations and estimation of the current states. The robot simulator and task environment are built upon PyBullet. Compared to conventional control strategies, the snake robots controlled by the trained PPO agent can achieve faster movement and a more energyefficient locomotion gait. This work demonstrates that DRL provides an energyefficient solution for robot control.