Similar to their counterparts in nature, the flexible bodies of snake-like robots enhance their movement capability and adaptability in diverse environments. However, this flexibility corresponds to a complex control task involving highly redundant degrees of freedom, where traditional model-based methods usually fail to propel the robots energy-efficiently. In this work, we present a novel approach for designing an energy-efficient slithering gait for a snake-like robot using a model-free reinforcement learning (RL) algorithm. Specifically, we present an RL-based controller for generating locomotion gaits at a wide range of velocities, which is trained using the proximal policy optimization (PPO) algorithm. Meanwhile, a traditional parameterized gait controller is presented and the parameter sets are optimized using the grid search and Bayesian optimization algorithms for the purposes of reasonable comparisons. Based on the analysis of the simulation results, we demonstrate that this RL-based controller exhibits very natural and adaptive movements, which are also substantially more energy-efficient than the gaits generated by the parameterized controller. Videos are shown at blue this https URL this https URL .
@cite_26: Gait optimization is a basic yet challenging problem for both quadrupedal and bipedal robots. Although techniques for automating the process exist, most involve local function optimization procedures that suffer from three key drawbacks. Local optimization techniques are naturally plagued by local optima, make no use of the expensive gait evaluations once a local step is taken, and do not explicitly model noise in gait evaluation. These drawbacks increase the need for a large number of gait evaluations, making optimization slow, data inefficient, and manually intensive. We present a Bayesian approach based on Gaussian process regression that addresses all three drawbacks. It uses a global search strategy based on a posterior model inferred from all of the individual noisy evaluations. We demonstrate the technique on a quadruped robot, using it to optimize two different criteria: speed and smoothness. We show in both cases our technique requires dramatically fewer gait evaluations than state-of-the-art local gradient approaches.
@cite_9: 
@cite_21: As part of the winning team in the Sony legged robot league of the RoboCup 2003 international robotic soccer competition, we used automatic gait optimisation. This was the first such technique seen in the competition and allowed an improvement in both speed and stability over previous gaits. Additionally, the technique was fast enough to run on-site at the competition, optimising the walk for the particular surface used. This paper describes the gait optimisation technique as well as some surprising features of the final gait discovered by the algorithm.
@cite_24: Developing fast gaits for legged robots is a difficult task that requires optimizing parameters in a highly irregular, multidimensional space. In the past, walk optimization for quadruped robots, namely the Sony AIBO robot, was done by handtuning the parameterized gaits. In addition to requiring a lot of time and human expertise, this process produced sub-optimal results. Several recent projects have focused on using machine learning to automate the parameter search. Algorithms utilizing Powell's minimization method and policy gradient reinforcement learning have shown significant improvement over previous walk optimization results. In this paper we present a new algorithm for walk optimization based on an evolutionary approach. Unlike previous methods, our algorithm does not attempt to approximate the gradient of the multidimensional space. This makes it more robust to noise in parameter evaluations and avoids prematurely converging to local optima, a problem encountered by both of the previously suggested algorithms. Our evolutionary algorithm matches the best previous learning method, achieving several different walks of high quality. Furthermore, the best learned walks represent an impressive 20 improvement over our own best hand-tuned walks.
@cite_2: For a robot, the ability to get from one place to another is one of the most basic skills. However, locomotion on legged robots is a challenging multidimensional control problem. This paper presents a machine learning approach to legged locomotion, with all training done on the physical robots. The main contributions are a specification of our fully automated learning environment and a detailed empirical comparison of four different machine learning algorithms for learning quadrupedal locomotion. The resulting learned walk is considerably faster than all previously reported hand-coded walks for the same robot platform.
black Although there are few studies on optimizing the gaits for snake-like robots, studies on optimizing the locomotion gaits of other kinds of robots have been reported for the purpose of energy saving. Initially, researchers adapted techniques for multidimensional function optimization tasks to design efficient gaits, such as evolutionary algorithms @cite_24 or policy gradient search algorithms @cite_21 @cite_2 . However, these algorithms are usually plagued by local optima, which makes the process slow, inefficient, and manually intensive. Afterwards, gait optimization methods based on prior knowledges are further investigated. first optimized the speed and smoothness of the gait with Gaussian process regression on a quadruped robot @cite_26 . used Bayesian optimization approach to regulate those open-loop gait parameters for bipedal robots, which made the robot move faster and robuster @cite_9 . Even though these optimized gaits outperform the hand-coded gaits, they are still very inefficient forms of locomotion compared with the natural movement achieved by animals.