We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at this https URL
@cite_18: 
@cite_4: Successful performance of a sensorimotor task arises from the interaction of descending commands from the brain with the intrinsic properties of the lower levels of the sensorimotor system, including the dynamic mechanical properties of muscle, the natural coordinates of somatosensory receptors, the interneuronal circuitry of the spinal cord, and computational noise in these elements. Engineering models of biological motor control often oversimplify or even ignore these lower levels because they appear to complicate an already difficult problem. We modeled three highly simplified control systems that reflect the essential attributes of the lower levels in three tasks: acquiring a target in the face of random torque-pulse perturbations, optimizing fusimotor gain for the same perturbations, and minimizing postural error versus energy consumption during low- versus high-frequency perturbations. The emergent properties of the lower levels maintained stability in the face of feedback delays, resolved redundancy in over-complete systems, and helped to estimate loads and respond to perturbations. We suggest a general hierarchical approach to modeling sensorimotor systems, which better reflects the real control problem faced by the brain, as a first step toward identifying the actual neurocomputational steps and their anatomical partitioning in the brain.
@cite_9: In this paper we discuss a new perspective on how the central nervous system (CNS) represents and solves some of the most fundamental computational problems of motor control. In particular, we consider the task of transforming a planned limb movement into an adequate set of motor commands. To carry out this task the CNS must solve a complex inverse dynamic problem. This problem involves the transformation from a desired motion to the forces that are needed to drive the limb. The inverse dynamic problem is a hard computational challenge because of the need to coordinate multiple limb segments and because of the continuous changes in the mechanical properties of the limbs and of the environment with which they come in contact. A number of studies of motor learning have provided support for the idea that the CNS creates, updates and exploits internal representations of limb dynamics in order to deal with the complexity of inverse dynamics. Here we discuss how such internal representations are likely to be built by combining the modular primitives in the spinal cord as well as other building blocks found in higher brain structures. Experimental studies on spinalized frogs and rats have led to the conclusion that the premotor circuits within the spinal cord are organized into a set of discrete modules. Each module, when activated, induces a specific force field and the simultaneous activation of multiple modules leads to the vectorial combination of the corresponding fields. We regard these force fields as computational primitives that are used by the CNS for generating a rich grammar of motor behaviours.
@cite_6: Many control problems take place in continuous state-action spaces, e.g., as in manipulator robotics, where the control objective is often defined as finding a desired trajectory that reaches a particular goal state. While reinforcement learning offers a theoretical framework to learn such control policies from scratch, its applicability to higher dimensional continuous state-action spaces remains rather limited to date. Instead of learning from scratch, in this paper we suggest to learn a desired complex control policy by transforming an existing simple canonical control policy. For this purpose, we represent canonical policies in terms of differential equations with well-defined attractor properties. By nonlinearly transforming the canonical attractor dynamics using techniques from nonparametric regression, almost arbitrary new nonlinear policies can be generated without losing the stability properties of the canonical system. We demonstrate our techniques in the context of learning a set of movement skills for a humanoid robot from demonstrations of a human teacher. Policies are acquired rapidly, and, due to the properties of well formulated differential equations, can be re-used and modified on-line under dynamic changes of the environment. The linear parameterization of nonparametric regression moreover lends itself to recognize and classify previously learned movement skills. Evaluations in simulations and on an actual 30 degree-of-freedom humanoid robot exemplify the feasibility and robustness of our approach.
@cite_3: A new architecture for controlling mobile robots is described. Layers of control system are built to let the robot operate at increasing levels of competence. Layers are made up of asynchronous modules that communicate over low-bandwidth channels. Each module is an instance of a fairly simple computational machine. Higher-level layers can subsume the roles of lower levels by suppressing their outputs. However, lower levels continue to function as higher levels are added. The result is a robust and flexible robot control system. The system has been used to control a mobile robot wandering around unconstrained laboratory areas and computer machine rooms. Eventually it is intended to control a robot that wanders the office areas of our laboratory, building maps of its surroundings using an onboard arm to perform simple tasks.
@cite_2: An incinerating toilet device which burns waste in a combustion chamber after which the combustion chamber is cooled. A control system controls the length of the burning and cooling cycles responsive to the temperature of the combustion chamber and the temperature in the exhaust duct from the combustion chamber. The control senses two combustion chamber temperatures, i.e., a first elevated temperature indicative that waste material has been completely burned and a second more elevated temperature indicative of dangerous conditions in the combustion chamber. Responsive to the first elevated temperature, the control deactivates the combustion chamber burning means and responsive to the second elevated temperature, the control renders the important operative portions of the device, and especially the burning and cooling means, inactivatable until the control is replaced. After deactivation of the burning means responsive to the first elevated temperature, the cooling means cools the combustion chamber to a predetermined significantly lower temperature whereupon the control means deactivates the cooling means.
@cite_20: We present a hierarchical framework for approximately optimal control of redundant manipulators. The plant is augmented with a low-level feedback controller, designed to yield input-output behavior that captures the task-relevant aspects of plant dynamics but has reduced dimensionality. This makes it possible to reformulate the optimal control problem in terms of the augmented dynamics, and optimize a high-level feedback controller without running into the curse of dimensionality. The resulting control hierarchy compares favorably to existing methods in robotics. Furthermore, we demonstrate a number of similarities to (nonhierarchical) optimal feedback control. Besides its engineering applications, the new framework addresses a key unresolved problem in the neural control of movement. It has long been hypothesized that coordination involves selective control of task parameters via muscle synergies, but the link between these parameters and the synergies capable of controlling them has remained elusive. Our framework provides this missing link. © 2005 Wiley Periodicals, Inc.
@cite_11: Neural networks along the spinal cord contribute substantially to generating locomotion behaviours in humans and other legged animals. However, the neural circuitry involved in this spinal control remains unclear. We here propose a specific circuitry that emphasizes feedback integration over central pattern generation. The circuitry is based on neurophysiologically plausible muscle-reflex pathways that are organized in 10 spinal modules realizing limb functions essential to legged systems in stance and swing. These modules are combined with a supraspinal control layer that adjusts the desired foot placements and selects the leg that is to transition into swing control during double support. Using physics-based simulation, we test the proposed circuitry in a neuromuscular human model that includes neural transmission delays, musculotendon dynamics and compliant foot–ground contacts. We find that the control network is sufficient to compose steady and transitional 3-D locomotion behaviours including walking and running, acceleration and deceleration, slope and stair negotiation, turning, and deliberate obstacle avoidance. The results suggest feedback integration to be functionally more important than central pattern generation in human locomotion across behaviours. In addition, the proposed control architecture may serve as a guide in the search for the neurophysiological origin and circuitry of spinal control in humans.
The notion that biological motor systems are hierarchical is ancient, dating to the 19th century @cite_18 . In the 20th century, Bernstein promulgated the notion of hierarchical control through motor synergies,'' or stereotyped, multi-joint muscle activation combinations @cite_2 . More recently, the notion of spinal motor primitives has been forwarded by Mussa-Ivaldi and Bizzi @cite_9 and others @cite_4 . Motor primitives have resonated in robotics, especially as Dynamic Movement Primitives @cite_6 , which are low-dimensionality attractor systems that can simplify the learning of robot movements, and hierarchical robot control abstractions date to at least the 1980s @cite_3 . Modern control theorists have also considered abstract hierarchical architectures for manipulation @cite_20 and more bio-mechanical descriptions of locomotion @cite_11 .