Title: Structured Control Nets for Deep Reinforcement Learning,
Abstract: In recent years, Deep Reinforcement Learning has made impressive advances in
solving several important benchmark problems for sequential decision making.
Many control applications use a generic multilayer perceptron (MLP) for
non-vision parts of the policy network. In this work, we propose a new neural
network architecture for the policy network representation that is simple yet
effective. The proposed Structured Control Net (SCN) splits the generic MLP
into two separate sub-modules: a nonlinear control module and a linear control
module. Intuitively, the nonlinear control is for forward-looking and global
control, while the linear control stabilizes the local dynamics around the
residual of global control. We hypothesize that this will bring together the
benefits of both linear and nonlinear policies: improve training sample
efficiency, final episodic reward, and generalization of learned policy, while
requiring a smaller network and being generally applicable to different
training methods. We validated our hypothesis with competitive results on
simulations from OpenAI MuJoCo, Roboschool, Atari, and a custom 2D urban
driving environment, with various ablation and generalization tests, trained
with multiple black-box and policy gradient training methods. The proposed
architecture has the potential to improve upon broader control tasks by
incorporating problem specific priors into the architecture. As a case study,
we demonstrate much improved performance for locomotion tasks by emulating the
biological central pattern generators (CPGs) as the nonlinear part of the
architecture.