Locomotion Control Method for Humanoid Robot Based on United Hierarchical Reinforcement Learning

This paper proposes a hierarchical reinforcement learning control method and applies it to the walking control of a humanoid robot. Firstly, a reinforcement learning algorithm called proximal policy optimization(PPO) is combined with central pattern generator(CPG). Thus an united hierarchical reinforcement learning (UHRL) is built which could cooperates with high and low levels control tasks. Secondly, the particle swarm optimization algorithm is used to obtain the initial parameter configuration of CPG. So that the robot can generate basic walking gait at the beginning of the experiment. The particle swarm variance fitness is used as the variation constraint to prevent the optimization process from falling into precocious convergence. Thirdly, the reward function of the high-level controller is designed to help the humanoid robot avoid deviation from the original path.