Question: Represent the article for finding a single-sentence TLDR

Question: The physical design of a robot and the policy that controls its motion are inherently coupled. However, existing approaches largely ignore this coupling, instead choosing to alternate between separate design and control phases, which requires expert intuition throughout and risks convergence to suboptimal designs. In this work, we propose a method that jointly optimizes over the physical design of a robot and the corresponding control policy in a model-free fashion, without any need for expert supervision. Given an arbitrary robot morphology, our method maintains a distribution over the design parameters and uses reinforcement learning to train a neural network controller. Throughout training, we refine the robot distribution to maximize the expected reward. This in an assignment to the robot parameters and neural network policy that are jointly optimal. We evaluate our approach in the context of legged locomotion, and demonstrate that it discovers novel robot designs and walking gaits for several different morphologies, achieving performance comparable to or better than that of hand-crafted designs. An agent's ability to navigate through and interact with its environment depends not just on its skill at planning and controlling its motion, but also on its physical design. Different physical designs are inherently better suited to different tasks and environments. By making appropriate choices during fabrication, mechanical elements can be designed to improve robustness to non-idealities such as errors in perception, delays in actuation, etc., and indeed, make control problem an easier one to solve. At the same time, robots that take different forms may find completely different control strategies to be optimal to complete the same task. Therefore, the physical and computational design of an agent are inherently coupled, and must ideally be jointly optimized if the robot is to successfully complete a task in a particular environment. Consider the development of a legged robot for locomotion. Variations in physical design will require changes to the joint torques in order to preserve a particular locomotion behavior (e.g., a heavier torso requires greater torque at the ankle), and will likely in completely different walking gaits, even when the morphology is preserved. In fact, some changes to design may render locomotion impossible for the target operating environment (e.g., a robot with long feet may be unable to locomote up an incline). Meanwhile, careful choice of bipedal design enables passive walking BID20 BID9 BID4. It is therefore beneficial to not simply consider the robot's design or gait to be fixed, but to optimize both jointly for the target environment and task. Similar co-design can be beneficial in other settings-for example for the control policy and physical characteristics of digits in robotic grippers for grasping. While a robot's physical design and the corresponding control policy are inherently coupled, most existing methods ignore this coupling, instead choosing to alternate between separate design and control phases. Existing approaches that jointly reason over design and control BID7 BID12 BID46 assume knowledge of an accurate model of the robot dynamics and require expert supervision (e.g., to provide a suitable initial design and guide the optimization process). However, these restrictive assumptions limits their applicability to a handful of specific settings, and often yield solutions heavily influenced by expert intuition. In this work, we seek a general approach-one that can optimize a robot's physical characteristics jointly with controllers of a desired complexity (Fig. 1), that can be applied to general tasks in some DISPLAYFORM0 Figure 1: Our algorithm learns a robot's physical design jointly with the control policy. Here we show the learned designs evolving over time for the Hopper (top left), the Walker2d (top right) and the Ant (bottom), each with the default Roboschool design for comparison. Scale is fixed for each robot. Note that these designs correspond to modes of the distribution over robot designs that our algorithm maintains during training.given environment, and that can explore the joint search space of physical design and computational control in a purely data-driven way, without a model of the robot dynamics and independent of the biases of expert intuition. We develop this approach in the context of determining the physical parameters of an articulated agent-the lengths and thicknesses of each limbs in a given morphologythrough joint training with a neural network for control, with the objective of achieving locomotion. Our method maintains a distribution over these physical parameters, and simultaneously trains the parameters of this distribution with those of a neural network controller, using deep reinforcement learning. In this way, we pursue a design distribution and control policy that are jointly optimal for the given task and environment. Experimental show that starting from random initializations, our approach is able to find novel designs and walking gaits that match or exceed the performance of manually designed agents. To the best of our knowledge, our method is the first to successfully carry out such a joint optimization of design and control in a completely model-free manner. Attention has been paid recently to the problem of jointly optimizing the parameters that specify a robot's design and motion (e.g., gaits) or control. Early work in this area takes an evolutionary approach to optimizing a robot's design and controller, typically parameterized as a neural network, for virtual agents BID45 BID30 BID0 and physical robots BID19 BID1. Particularly relevant to our approach is the work of BID12, who relate design and motion parameters via a set of implicit functions that express robot dynamics, desired trajectories, and actuation limits. These functions encode a manifold that is then linearized to model the relationship between design and motion via the implicit function theorem. The method then solves for the desired parameters in a local fashion via constraint-based optimization. Similarly, BID46 describe an approach that jointly reasons over physical design and motion parameters for robots with articulated degrees of freedom (e.g., legged robots). They formulate the problem in the framework of trajectory optimization by incorporating parameters involved in the design of articulated robots, including dimensions, masses, mass distribution, and moments of inertia, together with the contact force and torque variables typically associated with trajectory optimization. They use weighted actuation cost as the objective, subject to a regularization penalty on parameters. Their method is guaranteed to find a feasible design so long as the problem is initialized within a small neighborhood of a feasible solution. Unlike our approach, their method requires that the user provide an estimate of the robot configuration at each time step, as well as an accurate analytic model of constraint dynamics (e.g., foot contact), which is computationally expensive to enforce. BID8 propose a derivative-free strategy that optimizes over muscle routing and muscle-based control for simulated bipeds to realize a design that incorporates biomechanical constraints. describe an evolutionary method that jointly reasons over design and motion parameters for legged robots, while also tuning the parameters of a robust controller that tracks these motions. Their method is limited to biologically inspired quadrupedal foothold patterns, and does not account for contact dynamics. In contrast, our approach applies to arbitrary morphologies and does not require that we model contact dynamics. While the focus on simultaneous optimization over robot design and motion parameters is relatively new, there is a long history of research focused on the related problem of co-design of physical structure and control BID35 BID34. BID29 jointly optimize the link geometry of a high-speed robot arm along with the parameters of a PD joint controller. These methods often rely upon access to an accurate model of the robot dynamics in order to design the controller. In order to reduce the dependence upon a detailed analytical model, BID32 use on-robot experiments to refine their model as part of the iterative design process. More recently, BID49 allow for some degree of uncertainty in the model by including the sensitivity of the design and control parameters to model uncertainty along with the task-specific optimization objectives. However, existing methods still rely upon access to an analytical model of the robot dynamics and are typically limited to simple (e.g., linear) control designs. Our method assumes only that the controller can be modeled via a convolutional neural network, and can thereby learn complex, highly nonlinear control policies with no a priori knowledge of the robot dynamics. Far more attention has been paid to the individual problem of task-driven optimization of robot motion and control. Given an arbitrary robot design chosen by novice users, BID21 describe an interactive design framework that solves for a realizable walking gait that in stable locomotion. Similarly, BID27 synthesize emergent behaviors (motion) for arbitrary morphologies and tasks from high-level specifications, by jointly optimizing over contact and motion. BID28 build upon this work by training recurrent neural networks to serve as feedback controllers capable of producing complex, stable behaviors for a variety of dynamical systems. Their method interleaves supervised learning with trajectory optimization, and incorporates noise to improve generalizability. Meanwhile, there is a large body of literature that formulates motion synthesis as a trajectory optimization problem. This approach has proven effective at respecting contact constraints (e.g., between the foot and ground), which make controlling dynamic motion particularly challenging BID6 BID33 BID10. These approaches have been shown to generate sophisticated behaviors for complex robot designs (e.g., humanoids) BID48, and for robots of arbitrary morphologies using only a high-level specification of the robot's shape, gait, and task BID50. Related, a number of methods interleave trajectory optimization and supervised learning with neural network regression BID14 BID15 BID26 BID28. Unlike our method, the use of trajectory optimization makes these approaches reliant upon knowledge of the model. A great deal of attention of-late has focused on the problem of learning complex control policies directly from low-level sensory input, without any knowledge of the system dynamics. Methods that have proven particularly effective combine neural networks that learn representations of the highdimensional raw sensor input with deep reinforcement learning BID36 BID24 BID41. While much of the work in this area focuses on low-dimensional, discrete action spaces, several methods have been recently proposed that learn continuous control policies through deep reinforcement learning. These techniques have been applied to control simple, simulated robots BID52 BID53 BID51 BID18, perform robot manipulation BID16 BID19 BID11, and control legged robots BID42 BID31.Black-box optimization BID40 BID37 is an alternative to using reinforcement learning to training the control policy. These approaches have the advantage that they do not involve backpropagating gradients, are insensitive to reward sparsity, and can handle long time horizons. While black-box optimization strategies have traditionally been thought of as illsuited to difficult reinforcement learning problems, BID38 recently showed that they perform similarly to state-of-the-art RL methods on difficult continuous control problems, including those that involve locomotion. Closely related is the policy-gradient method of BID44, who define the policy as a distribution over the parameters of a controller, which they then sample over. This in gradient estimates that are far less noisy than is typical of policy gradient algorithms. These approaches are similar to the way in which we learn the robot design, which we formulate as a Gaussian mixture model over design parameters. Indeed, the two referenced methods yield the same gradient estimate for Gaussian parameter distributions BID38.Meanwhile, much work has focused on the problem of determining robot designs that meet the requirements of a particular task. Given a user demonstration of the desired behaviors, BID5 learn optimum kinematic linkages that are capable of reproducing these motions. BID22 synthesize electromechanical robot designs in a compositional fashion based upon a complete user-specified structural specification of the robot. BID23 build upon this work, allowing the user to specify functional objectives via structured English, which is parsed to a formal specification using linear temporal logic. describes a theory for co-design that includes the ability to select discrete robot parts according to functional constraints, but do not reason over geometry or motion. Related to our approach is recent work that jointly optimizes sensor design and inference algorithms for perception systems. BID3 considers the problem of jointly learning a camera sensor's multiplexing pattern along with reconstruction methods for imaging tasks. They model inference as a neural network together and use stochastic gradient descent to backpropagate the loss to a neural layer representation of the multiplexing pattern. Related, BID39 jointly learn design and inference for beacon-based localization. They encode beacon allocation (spatially and across transmission channels) as a differential neural layer that interfaces with a neural network for inference. Joint optimization then follows from standard techniques for training neural networks. In this section, we begin by describing the standard reinforcement learning framework for training agent policies, and then describe how we extend this to also learn the physical design of the agent. In the standard reinforcement learning setting, an agent interacts with its environment, usually a Markov Decision Process, over a number of discrete timesteps. At each time step t, the agent receives a state s t ∈ S and takes action a t ∈ A according to a policy π: S → A. Then, the agent receives a scalar reward r t and the next state s t+1 from the environment. This process continues until a terminal state is reached. The goal of reinforcement learning is to then find a policy π * that maximizes the expected return E[R t], where R t = ∞ i=0 γ i r t+i and γ ∈ is a discount factor. Policy gradient methods are a class of algorithms often used to optimize reinforcement learning problems, due to their ability to optimize cumulative reward and the ease with which the can be used with neural networks and other nonlinear function approximators. Consequently, they are commonly used for reinforcement learning problems that involve complex, continuous action spaces. Policy gradient methods directly parameterize a stochastic policy π θ (a t |s t) and perform stochastic gradient ascent on the expected return. "Vanilla" policy gradient methods compute an estimate of the gradient ∇ θ E[R t] using a sample-based mean computed over ∇ θ log π θ (a t |s t)R t BID54, which yields an unbiased gradient estimate BID47. While the variance in the ing estimate decreases with the number of samples, sampling is computationally expensive. Another way to reduce variance while maintaining an unbiased estimate is to estimate the gradient by comparing the reward to a "baseline" reward b(s t).These methods are effective but can be unstable, especially when used for deep reinforcement learning. Small changes to the policy parameters may cause large changes in the distribution of visited states. Several methods have been proposed to mitigate these effects. Among them, BID41 introduce Trust Region Policy Optimization (TRPO), which imposes a constraint on the KLdivergence between policies before and after an update. Recently, proposed proximal policy optimization (PPO), a first-order class of methods similar to TRPO. PPO alternates between sampling data through interaction with the environment and optimizing the objectivê DISPLAYFORM0 where r t (θ) = π θ (at|st) π θ old (at|st) andÊ t represents an empirical average over a finite sample set. This objective seeks to maximize the expected return while encouraging a small step in policy space. The clipping within the objective removes any incentive for moving r t (θ) outside the interval [1−, 1+]. The net effect of this objective is a simple, robust policy gradient algorithm that attains or matches state-of-the-art on a wide array of tasks We extend the standard reinforcement learning formulation by considering the space of possible robot designs Ω. Specifically, we assume that for every design E ∈ Ω, we can define common state and action spaces S and A, reward function r(s, a), and initial state distribution p 0 (s), that are meaningful to all designs. The designs differ only in the transition dynamics they induce p E (s |s, a) and share a common action space A-to achieve this, we assume a common morphology for all possible designs. Our goal then is to find the optimal design E * and policy π * E * pair that maximizes the expected value of a given reward function. However, this is a non-linear, non-convex optimization problem over the spaces of all possible designs and all possible policies. Solving it exactly would require enumerating all possible designs (possibly through discretization of the design space), learning policies for each, and comparing the ing expected rewards. This is computationally infeasible for all but the simplest of cases. Instead, we develop a gradient-based approach to solving this optimization problem with the following key components: we maintain a multi-modal distribution over the space of physical designs and update this distribution using policy gradient methods in parameter space, similar to BID44 and BID38; and we train a single controller to act on all sampled designs during training, providing this controller with the design parameters of the specific sample it is controlling. We find that the multi-modal stochastic parameterization of the design space allows our method to explore the space more thoroughly (where a uni-modal Gaussian distribution would frequently get trapped in local minima). Moreover, a common controller makes optimization tractable, allowing efficient evaluation of unseen designs. We additionally benefit from learning common strategies across diverse designs, while still adapting different policies to different parts of the design space given the sample parameters as input. Together, these components enable efficient joint exploration of the design and policy spaces. Formally, let p(r; φ) denote the distribution over designs, and π(a t |s t, r; θ) a stochastic control policy parameterized by φ and θ respectively. In our experiments, we use a neural network to model the control policy π, and a Gaussian mixture model as the parametric form of the distribution p(r; φ). Our goal is then to solve the following optimization problem: DISPLAYFORM1 where E r,t [·] is the expectation over robots and trajectories those robots induce. We use stochastic gradient-based updates to optimize Eqn. 2. Our method (Algorithm 3.2) alternates between updating the parameters of the policy and design distributions, θ and φ, respectively. We empirically find this to yield convergence to better solutions in a reasonable number of iterations compared to performing simultaneous updates. We optimize the policy parameters θ using Proximal Policy Optimization. However, instead of collecting data from a single design, we sample a design r after a fixed number T of timesteps according to the distribution p(r; φ). After n iterations of PPO, we freeze the policy parameters and proceed to optimize the parameters of the design distribution. Without knowledge of the model, we optimize the design parameters φ via policy gradient over parameter space, similar to BID44. This is equivalent to black-box optimization methods that have proven effective for complex learning problems BID38. We sample m different designs and compute their returns for a single episode acting under policy π(·; θ). We then use this data to shift the design distribution p(·; φ) in a direction maximizing expected return. These iterations are repeated until convergence. We validate our approach with three commonly used robots provided in OpenAI's Roboschool. Environments in Roboschool are built on top of Bullet Physics, a popular open-source physics engine. In a series of locomotion experiments, we show that our approach not only discovers novel robot designs and gaits, but also outperforms two out of three of the hand-designed robots trained on the same task and yields performance comparable to the third. We evaluate our approach on three legged locomotion tasks within Roboschool: RoboschoolHopper, RoboschoolWalker2d, and RoboschoolAnt (note that we subsequently drop the Roboschool prefix for brevity). FIG0 depicts the default Roboschool design for each robot. These environments describe a locomotion task that has the robots moving along the positive x-axis (to the right in the figures) towards a distant goal. The reward function defined in these environments is a weighted sum of rewards for forward progress and staying upright, and penalties for applying torques and for joints that reach their rotational limits. The episode ends when the robot falls over or reaches a maximum number of timesteps. We learn robot designs using the morphologies specified by the standard Roboschool designs. For each morphology, we parameterize the robot in terms of the length and radius (e.g., mass) of each link (or just the radius in the case of the sphere for the ant body). We impose symmetry, and share parameters across each leg for the Walker2d and Ant designs. This parameterization permits a wide variety of robots of different shapes and sizes, some of which are better suited to the locomotion objective than others, and some designs that do not permit a control policy that in locomotion. We model the control policy π(a t |s t, E; θ p) as a feed forward neural network consisting of three fully-connected layers with 64 units and tanh activation functions. A final layer maps the output to the robot action space. With the exception of the last robot-specific layer, the architecture is the same for all experiments. Meanwhile, we represent the distribution over the parameterized robot design p(·; θ r) using a Gaussian mixture model with four mixture components (each with a diagonal covariance matrix over the different parameters). We initialize the means of each component randomly within a wide range, and initialize the variances in order to span the range of each parameter. We find that our approach maintains high variance distributions during early iterations-thereby continuing exploration of the design space-before committing to a chosen design. The appendix provides further details regardng the evolution of these distributions. Timesteps Ant Figure 3: Training curves that show the evolution of reward across training iterations for the Hopper, Walker2d, and Ant environments over eight different random seeds. The black dashed line corresponds to the highest achievable performance of the corresponding baseline robot. In other figures, we display the best-performing checkpoint of each run. Therefore, we make each training curve transparent after it stops improving. We train our method with eight threads of Proximal Policy Optimization for a total of 300M environment timesteps across all threads. We use the publicly available implementation of PPO provided in OpenAI Baselines BID13. We alternate between 50 policy PPO iterations and 2 design iterations, finding this ratio to yield convergence to good solutions. We set this and the other hyper-parameters on the Hopper and used the same settings for all experiments. We evaluate the performance of our approach to joint optimization over robot design and control on the Hopper, Walker2d, and Ant robots, and compare against a policy trained for the standard Roboschool design. We evaluate the consistency and robustness of our approach relative to the initial robot parameter distribution using several different random seeds. For each robot morphology, we find that our method learns robot designs and the corresponding control policy that are either comparable (Walker2d) or exceed the performance (Hopper and Ant) achievable using the default designs (Fig. 3). We also see that our method achieves these levels of performance by discovering unique robot designs (Fig. 1) together with novel walking gaits (Fig. 4) that can not be achieved with the default designs. Note that our method was able to learn these designs and control policies from a random initialization, without access to a dynamics model or any expert supervision. Our method learns a joint design and controller for the Hopper that outperforms the baseline by as much as 50%. Our learned robot exploits the small size of the foot to achieve faster, more precise walking gaits. Meanwhile, the longer torso of the learned robot improves stability, allowing it to maintain balance while locomoting at a faster pace. We found this behavior to be consistent across several different Hopper experiments, with the method converging to designs with a small foot and long, thin torso. In the appendix, we explore the stability of this design with respect to variations in the environment using the coefficient of friction as an example, and find the improvement to be consistent. For the ant, our optimization yields a physical design that is significantly different from the default design FIG0. Consequently, the learned Ant drastically outperforms the baseline, improving reward by up to 116%. Our method learns a design with a small, lightweight body and extremely long legs. The long legs enable the ant to apply large torque at contact, allowing it to move at a fast pace. Our framework learned different design-control pairs for the Walker2d that perform similarly to the default design. Across several different experiments, we see two distinct, successful designs and walking gaits. Interestingly, neither agent walks primarily on its feet. The first design has small, thick legs and long feet. The controller is then able to exploit the thick middle leg link, which protrudes slightly past the foot, to push off the ground. The long foot then provides additional balance. The second design is similar in geometry to the baseline Walker2d, but moves in a very different way. By lowering the knee joint and lengthening the foot, the Walker2d is able to efficiently balance on its knees and toes. This low stance allows the Walker2d to fully extend its leg backwards, creating a long, powerful stride, similar to that of a sprinter using starting blocks. Figure 4: Here, we compare locomotion of default and learned robots, visualizing both their physical design and corresponding learned gaits. We pick the best for Hopper and Ant, and two of the best for Walker2d (due to diversity in gaits). Note that for each robot type, we show a blend of a fixed number of frames in the same time interval, allowing direct comparison between the speed with which different designs are able to locomote. We proposed what is, to the best of our knowledge, the first model-free algorithm that jointly optimizes over the physical design of a robot and the corresponding control policy, without any need for expert supervision. Given an arbitrary morphology, our robot maintains a distribution over the robot design parameters and learns these parameters together with a neural network controller using policy gradient-based reinforcement learning. This in an assignment to the policy over robot parameters and the control policy that are jointly optimal. We evaluated our approach on a series of different legged robot morphologies, demonstrating that it in novel robot designs and walking gaits, achieving performance that either matches or exceeds that of manually defined designs. Our findings suggest several avenues for future work. The most direct is extending the current approach to find optimized designs for uneven terrain, the presence of obstacles, changes in slope, variations in friction, etc. We are also interested in extending our framework to relax the assumption that the morphology is pre-defined. Finally, we are investigating applications to different types of agents and design spaces beyond legged robots (e.g., end-effectors), and exploring appropriate stochastic parameterization for such designs. The following provides further evaluation of our framework, including the behavior of the design distribution during training and the robustness of the learned designs to environment variations. To provide insight into the training process of our algorithm, we evaluate the evolution of the Gaussian mixture model distribution throughout training of the best performing Hopper experiment. We initialize each component of the mixture model with random means and a diagonal covariance matrix chosen to cover the parameter space. The initial mixture weights are uniform. As shown in FIG1 (left), roughly one third of the way through training, our algorithm converges to the most successful component. Additionally, we find that modes generally do not collapse (FIG1). We find this behavior to be consistent across random seeds and different robots. It is often desirable for a robot to be able to operate in a variety of environments. In this section, we consider the robustness of a learned design to changes in friction. We conducted experiments on the Hopper in which we first learned the design and controller for one friction setting (0.8). We finetuned the controller in environments with different friction settings, while leaving the design fixed. We find the learned design to be reasonably robust with variability comparable to controllers finetuned for the default hand-crafted design (Fig. 6). Additionally, the learned design outperforms the hand-crafted one across the full range of friction values (although, for very low friction values, both designs essentially were unable to learn a successful gait).Note that our framework can incorporate this goal of generalization by simply sampling from a diverse set of environments during training. But at the same time, it may be useful in some applications to seek out solutions that are specifically adapted to a relatively narrower set of environments, gaining better performance within this set at the cost of more general performance. Figure 6: A visualization of accumulated reward for different friction values. The controllers and learned design were trained in an environment with a friction value of 0.8. We finetuned the controllers (but not the designs) for both the learned and hand-crafted designs for 10M timesteps for each friction value. Rewards are reported as an average over 100 episodes.

represent the tldr for finding the corresponding scientific article.

Use deep reinforcement learning to design the physical attributes of a robot jointly with a control policy.

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.


Question: represent the scientific article to find a short tldr!

Question: Deep learning enables training of large and flexible function approximators from scratch at the cost of large amounts of data. Applications of neural networks often consider learning in the context of a single task. However, in many scenarios what we hope to learn is not just a single task, but a model that can be used to solve multiple different tasks. Such multi-task learning settings have the potential to improve data efficiency and generalization by sharing data and representations across tasks. However, in some challenging multi-task learning settings, particularly in reinforcement learning, it is very difficult to learn a single model that can solve all the tasks while realizing data efficiency and performance benefits. Learning each of the tasks independently from scratch can actually perform better in such settings, but it does not benefit from the representation sharing that multi-task learning can potentially provide. In this work, we develop an approach that endows a single model with the ability to represent both extremes: joint training and independent training. To this end, we introduce matrix-interleaving (Mint), a modification to standard neural network models that projects the activations for each task into a different learned subspace, represented by a per-task and per-layer matrix. By learning these matrices jointly with the other model parameters, the optimizer itself can decide how much to share representations between tasks. On three challenging multi-task supervised learning and reinforcement learning problems with varying degrees of shared task structure, we find that this model consistently matches or outperforms joint training and independent training, combining the best elements of both. While deep learning has enabled remarkable levels of generalization through the use of function approximators, this comes at the cost of large amounts of data, which remains a critical challenge in deploying deep learning to a number of domains. When combined with deep networks, multitask learning offers the promise of building more powerful representations using less data per task, leading to greater performance and data efficiency. However, multi-task deep learning has also posed considerable challenges. Numerous works have observed that joint training on multiple tasks can actually decrease task performance due to the negative influence of other tasks (; a). Indeed, training networks entirely independently on each task has remained a strong approach, to the point that multiple multi-task methods have first trained models independently before using them to train a multi-tasking model (; a; ; ; . Moreover, our experiments in Section 6 indicate that three recently proposed methods for multi-task learning are all surpassed by training models independently per task. However, training independent models will only work well when provided enough data per task, and precludes potential positive data-efficiency gains from multi-task learning, only providing protection against negative transfer. Further, while a number of works have successfully shared parameters, finding an architecture with the appropriate level of parameter sharing for a given problem domain can require a considerable amount of manual engineering. In this work, we aim to develop a multi-task learning method that can perform well both when tasks share very little and when they share a large amount of structure. To address this problem, we consider how a single neural network model can represent two extremes: independent models, when optimization challenges prevail, or a single model with shared weights, when sharing is beneficial. Further, we would like such a model to be able to represent intermediate levels of model sharing, when appliable. One option for performing independent training within a single model is to put separate networks with independent weights into a single model, using the task ID to select which network prediction to output. However, this prevents any sharing. An alternative approach is to condition the model on the task ID, through various conditioning approaches, including additive and multiplicative approaches such as FiLM . In fact, point-wise multiplicative conditioning, as proposed in FiLM, can indeed represent separate networks by selecting which parts of the network to be used for different tasks, as can a number of other approaches in multi-task learning (; 2019;). Yet, these approaches still require an optimization over shared parameters in order to select which parameters are used for each task. These shared parameters can introduce significant optimization challenges. We instead consider how to allow a model to perform optimization on only shared parameters, only disjoint parameters, or any combination thereof. We can achieve this by simply interleaving learned per-task matrices at each layer of a jointly-trained neural network. When optimization over shared parameters is ineffective, the model can still represent a full neural network per task using only the per-task matrices, ing in independent training; while using identical per-task matrices in standard joint training. Intermediately, a mix of shared and per-task parameters may be used. In effect, by incorporating these matrices into the network, the optimizer itself can automatically and dynamically modulate the degree to which a representation is shared between tasks, depending on the problem domain and the optimization progress, and can do so without having to optimize shared parameters. The primary contribution of this paper is a simple yet effective approach for multi-task learning that can represent and smoothly interpolate between independent training and joint training, via matrix interleaving (Mint). We describe how we can implement Mint in deep multi-task models and show its effectiveness in improving data efficiency and generalization in multi-task settings while providing intuition about the reasons why this architecture performs so well. Further, we show that the model can be extended to goal-conditioned reinforcement learning in a straightforward manner by allowing the model to generate the interleaved matrices conditioned on task information such as the goal. We evaluate Mint on sets of tasks with both high and low levels of shared structure and find that it performs well in both settings, performing comparably to or outperforming both joint training and independent training, effectively combining the best elements of both. Further, in comparison to previous methods that use multiplicative interactions for continual learning and for general conditioning , Mint is better able to separate tasks by avoiding the need to optimize over shared parameters and can empirically produce substantially better performance on a range of challenging multi-task problems. Finally, Mint also outperforms state-of-the-art approaches for multi-task learning while being significantly simpler to implement. In multi-task learning, the goal is to find a θ-parameterized model f θ that reaches high performance across all training tasks drawn from a task distribution where L k denotes the loss function for task T k. In this work, we study the setting in which we have a fixed set of K tasks, and we wish to obtain high performance on all of the tasks in this set. In Section 4, we will study this multi-task problem in the context of a reinforcement learning setting. In our multi-task learning setup, we train a model that is conditioned on a task indicator z k which is used to specify a task T k that should be performed. The task indicator can be represented in a variety of ways, from simple categorical variables to learned task embeddings . This formulation can be readily extended to a goal-conditioned reinforcement learning setting, where z k indicates the desired goal. The multi-task learning model learns to optimize objectives of all the tasks T k that the model is trained on. Joint and fully independent training are the two extremes of multi-task learning. Assuming a set of n training tasks, we characterize multi-task training as independent if it optimizes a set of task-specific, disjoint parameters {θ k} n i=1 that parameterize the model f θ k. We define joint training as finding a single set of task-independent, shared parameters θ of f θ. Note that joint training utilizes the fact that the parameters are shared throughout learning. While joint training has a number of data-efficiency and generalization benefits, it can be difficult to train effectively. Considering fully independent and fully joint training as two ends of the spectrum in multi-task learning problems, we want to design an algorithm to get the best of both worlds -the stability of independent training and the parameter-sharing efficiency of jointly trained models. To this end, we propose to build models that allow the neural network to learn how much information should be shared and between which tasks throughout learning, in an adaptive fashion. We describe the details of our approach below. In the case of neural network models, we view representations as network activations at different layers of the network. We aim to introduce a modification to the neural network that would allow the model to either form those activations in a completely task-specific way, in a completely shared way, or in a way that shares to an intermediate degree. To achieve this, we propose a model architecture that transforms the previous layer's representation both in a task-general way and in a task-specific way, in sequence. When two tasks share very little, the network can optimize task-specific weights, while when the tasks share a considerable degree of structure, the network can leverage the shared weights. Since these transformations are task-specific and can be introduced at various layers of the network, they allow for a different amounts of representation shared at different levels of the neural network model. To understand the practical implementation, we consider the activations at layer l of a fully-connected neural network:, where W (l) is the weight matrix for layer l, b (l) is the bias vector for layer l, and σ is the non-linearity at each layer. The Mint layer augments the traditional fully-connected layer with task-specific weight matrix M k and bias vector β k, where k indexes the task. The forward pass of a Mint layer for some vector of activations y is presented in Definition 1. Definition 1. A Mint layer applies an affine transformation to activations y ∈ R n as follows, yielding new activations a: where M k and β k are per-layer task-specific matrix and bias, respectively. A neural network augmented with Mint thus contains parameters that are both shared across all tasks as well as parameters that are only used for a particular task for each layer l, i.e. Figure 1 for a visual depiction of the application of Mint. We show how the regular fully-connected layers and Mint layers can be interleaved in Equation 2 and 3 below: Because this layer is fully-differentiable, we can learn the task-specific matrices M (l) k and biases β k jointly with the shared parameters of the model. When we apply Mint to tasks with very large numbers of tasks, or arbitrary task descriptors (e.g., goal-conditioning), we can train separate neural networks, to output the Mint matrices and biases at every layer, instead of storing independent matrices for each task. In this case, Mint resembles , another method which performs transformation-based task conditioning. In contrast to FiLM, Mint uses a matrix transforming the activations at each layer instead of a point-wise multiplication by a vector. In the next section, we study a theoretical property of Mint that motivates the chosen definition of the Mint layer. We validate the benefits of Mint in our experimental evaluation in Section 6. We next aim to theoretically study the expressive power of multi-task learning architectures when the shared parameters cannot be optimized effectively (e.g. due to optimization challenges arising from learning tasks that share little or no structure). While a viable approach in this regime is to simply use separate task-specific networks f φi with no shared parameters, this approach makes the strong assumption that the tasks indeed share no exploitable structure. This is often not known a priori. We show in this section that in the'worst case' where shared parameters are not useful, the class of functions defined by a Mint network is still as rich as that defined by a set of purely task-specific networks. Thus, Mint makes no sacrifices in terms of expressiveness with respect to the optimal attainable parameters φ * i and retains the possibility of exploiting shared structure if it does exist. Further, we show that the same is not true of existing methods that are related to Mint. Specifically, we compare the class of functions representable by Mint with 1. The class of functions defined by a FiLM network , where a layer's activations are'tilted' through element-wise multiplication with a task-specific vector γ k and addition with a task-specific bias β k. 2. The class of functions expressable by a task indicator concatenation approach, where a one-hot task indicator vector z k for task T k is concatenated to each shared layer's input. We begin by considering the optimal parameters for task-specific networks f φ * i, where φ * i = arg min φ L i (φ), the parameters that minimize the loss function L i for task T i. We assume that f φ * i is an L-layer MLP. We also consider L-layer Mint networks f θi (consisting of L sequential pairs of Mint transform and shared parameter layer as in Equation 1) and L-layer FiLM networks (consisting of L sequential pairs of FiLM transform and shared parameter layer). We state Definition 2 to complete our setup. Definition 2. Let y (l−1) be the activations of layer l − 1, and let α i = {W (l), b (l) } be the shared parameters (weight matrix and bias vector) at the l-th layer. We assume that W (l) is an arbitrary, fixed invertible matrix. With this setup, we state Lemmas 1 and 2, which highlight a key difference in the layer-wise expressiveness of Mint and methods such as FiLM and task indicator concatenation. Lemma 1. For a given α i, applying Mint to y (l−1) can express an arbitrary affine transformation at layer l for each task. Lemma 2. (a) For a given α i, there exist affine transformations that cannot be expressed by applying FiLM to y (l−1). (b) Similarly, for a given α i, there exist affine transformations that cannot be expressed by a network which concatenates the input y (l−1) of each layer l with a task indicator z k. The proof can be found in Appendix A. We assume only that the shared parameter matrices W (l) are invertible, such that they do not lose information about the input. This assumption critically does not require the optimizer to productively update the shared parameters. We use Lemmas 1 and 2 to extend the above layer-wise expressiveness properties of Mint and FiLM to compare the function classes defined by L-layer Mint networks f θi and FiLM networks f ψi in the context of optimal L-layer MLP f φ * i. We state this comparison in Theorem 1. We use α to denote the subset of θ i and ψ i that corresponds to the parameters of the network that are shared across tasks. is an MLP (a composition of affine transformations and activation functions), the proof of the first statement follows from applying Lemma 1 to each layer of f φ * i and f θi (the task specific and Mint networks, respectively). Similarly, Lemma 2 implies that we can construct an α and L i (and therefore φ * i) such that we can find anx satisfying the second statement. That is, for any set of shared parameters in a Mint network and any set of optimal'target' MLP, there exist weights for Mint that are effectively equivalent to the target MLP. On the other hand, there exist settings of the shared parameters that make matching the outputs of the target MLP unattainable in the FiLM and task indicator concatenation regimes. Using Lemma 2, we reach an equivalent regarding the limited expressiveness of the task indicator concatenation approach. The general idea of Mint is implemented as follows in the supervised and reinforcement learning settings. In the case of supervised learning models, we simply apply Mint layers after every fully connected layer of the model. Specifically, for a task identifier z k ∈ R K where K is the number of tasks and for every layer l with activations a (l) ∈ R n, nonlinearity σ and weight W (l) and bias b (l), we represent the transformation using two matrices T M ∈ R n×n×K and β ∈ R n×K that take in the task identifier and output the per-layer task-specific matrices M (l) and biases β l respectively. The transformation can be summarized as follows: Multi-Task Reinforcement Learning For multi-task reinforcement learning, we implement the architecture similarity to the supervised learning case but we combine this with actor-critic RL algorithms by introducing this architecture into both the critic Q(s, a, z k) and the actor π(a|s, z k). For the case of goal conditioned RL, we introduce a slightly modified Mint architecture into both the actor and the critic conditioned on the task goal g. Specifically, for every layer l with activations a (l), nonlinearity σ, weight W (l), and bias b (l), we represent two transformation function T (l) φ and T ψ by two 2-layer ReLU networks that take in the goal and produces a per-layer goal-specific matrix M l (g) and the bias β (l) (g) respectively. The transformation can be summarized as: 5 RELATED WORK Multi-task learning (; ;) focuses on the problem of finding a single model that can solve multiple different tasks. This formulation can be readily applied to a variety of learning domains, such as supervised learning (; ; b; ;), and multi-task (; ;) and goal-conditioned reinforcement learning (; ;). While multi-task learning offers the promise of efficient training of shared representations, naïvely training a single model on multiple tasks often does not in these desired benefits, due to the optimization challenges introduced by the multi-task setting (; ; a). In order to eliminate potential negative interference between different tasks during multi-task learning using a single model, many approaches propose to learn each task separately, to later combine their solutions into a single multi-task model;; a; ). In contrast to these works, we present a method that is able to train a single model on multiple tasks and is able to interpolate between the extremes of joint and independent training. More closely related to our approach, various architectural solutions have been proposed to increase the multi-task learning capability of the model. Example approaches include architectural changes that allow multiple modules or paths within the same network (; ; ; b; ;), transformation-based task conditioning , attention-based architectures , multi-headed network solutions , and a variety of other approaches (; Ruder12 et al., 2017). We demonstrate an approach that allows for a middle ground between the conceptual extremes of fully independent training at one end and single-model joint training at the other. This added flexibility enables us to sidestep the negative effects of potential task interference while at the same time share parameters between the tasks when appropriate. Prior approaches (a;) have investigated this ability by factorizing layers in the neural network across independent and shared components. In contrast, our method is simpler to implement, less computationally intensive, and empirically easier to optimize. In our experiments, we provide direct comparisons of our method to cross-stich , routing networks , the FiLM architecture , rotational superposition , and multi-headed models. The goal of our experimental evaluation is to answer the following questions: does our method enable effective multi-task learning both in settings where there is substantial overlap in tasks and where there is little overlap?, how does our method compare to independent training and joint training in those settings?, how does our method compare to prior state-of-the-art approaches? To answer the above questions, we conduct experiments on multi-task reinforcement learning domains. For multi-task RL doamins, we perform experiments on two multi-task RL benchmark variants MT10 and MT50 (as showcased in Figure 2) proposed by. Finally, to test if Mint can excel in continuous task distributions, we also evaluate the method on a goal-conditioned RL domain where a Sawyer robot arm is required to push a randomly initialized puck to different goal positions. For all RL experiments, we use the popular off-policy RL algorithm, soft actor-critic (SAC) , which has shown to solve many RL benchmarks with great data-efficiency. On the multi-task RL domains, we compare Mint to the following methods: • SAC: train a vanilla SAC agent with task identifier as part of the input. • Multi-head SAC: train a SAC agent where both the actor and critic are represented as multihead feedforward neural networks where the number of heads is the number of tasks. • SAC (concat all fc): train a SAC agent where the task identifier z is concatenated with the activation at each layer and passed as inputs to the next layer. • FiLM : the actor and critic are learned with neural networks combined with FiLM. • Superposition : the actor and critic are learned with neural networks combined with superposition. • independent: learn separate actor and critic per task. We provide details of architecture design in each domain as well as environment set-up in the Appendix B. On the RL domain, we first investigate the ability of Mint to perform a set of distinct RL tasks. As discussed in , MT10 and MT50 serve as representative benchmarks to evaluate multi-task learning algorithms on learning a diverse range of robotics manipulation tasks. We present the in Figure 3. The success rates are averaged across tasks and we adopt the success metrics used in the Meta-World benchmark. We design appropriate architectures of the actor and the critic of the SAC algorithm for each method such that the number of parameters of each approach is around the same magnitude (see Appendix B for details). For MT10, Mint learns all tasks with the best data efficiency, while independent networks also learn all of the tasks with slightly worse data-efficiency. The other methods are unable to acquire half of the skills. Mint, on the other hand, enjoys the expressive power to interpolate between independent learning and sharing, while mitigating optimization challenges, to attain the best between the two extremes. For MT50, where the evaluations are done on all 50 of the Meta-World environments, as shown on the right in Figure 3, Mint quickly learns to solve more than 60% of tasks in 20 million environment steps while SAC and SAC with multi-head architectures struggled in solve 40% of the tasks after 35 million steps. Independent networks learn to solve the tasks slower than Mint but eventually surpasses it. 1 This also validates the expressive power of Mint to represent both separate learning and learning with shared networks. Figure 3: Learning curves on MT10 (left) and MT50 (right). We observe that independent training performs well on both benchmarks. Mint, unlike prior multi-task learning approaches, is able to perform at a similar level to independent training. We also examine the learned Mint layers in the MT10 setting to analyze whether they capture taskspecific information. Specifically, we replace the'close drawer' task in MT10 with a duplicated'press button top' task and thus we have two'press button top' tasks in MT10 (see Figure 2). We train Mint on this new set of tasks. Concretely, if we index the tasks from 1 to 9, then we have two copies of task T 1 and learn two separate Mint matrices: M T1 1 and M T1 2. We then compute the pairwise 1 distance between the Mint layer learned for one of the duplicate'press button top' tasks and the Mint layer learned for any other task such as'insert peg side' in the set. Specifically, we compute d(M T1 1, M Ti) for all i = 1, where d is the l 1 distance. We compute the percent difference between each d(M T1 1, M Ti) and d(M T1 1, M T1 2) and present it in Figure 4 on the left. For each pair M Ti, i = 1, we plot. From the figure, the two Mint layers learned for the duplicated tasks have the smallest relative 1 distance except the distance between the Mint layers of'press button top' and'reach', which is reasonable since'press button top' is structurally similar to'reaching'. Finally, we provide a comparison between Mint and other methods where the non-Mint methods are trained with the same number layers as Mint as opposed to the similar number of parameters in Figure 3. As shown on the right in Figure 4, Mint outperforms all of the other approaches in this setting. Figure 4: On the left, we show the percent increase in 1 distance between Mint layers learned for one of the duplicate tasks and each of the other tasks in MT10 as compared to the distance between the Mint layers learned for the two duplicate tasks. We can see that for most of the tasks, the percent increase in 1 distance is approximately 10%, except that the distance between'reach' and'press button top' is smaller, which could be explained by the fact that'press button top' is inherently just a reaching task. On the right, we compare Mint to a list of other methods with the same number of layers and find that Mint achieves a significantly higher success rate than any of the other approaches. Next, we consider the question if Mint can be applied to a set of RL goals that have considerable shared structure. Hence, we evaluate all methods on the goal-conditioned sawyer pushing domain. In this environment, the goal space consists of the initial positions of the puck and the goal positions. For details of the goal-conditioned environment, see Appendix B. At each policy rollout step, we sample a batch of 9 goals and collect 3 paths for each goal, where all the paths are stored in the task-specific replay buffers. At each training step, we sample a batch of 9 goals and 128 samples per goal from their corresponding replay buffers. To prevent creating infinite number of replay buffers, we discretize the goal space into 200 goals. Given that it is impractical to train 200 independent agents, we sample 10 goals from the goal space and train 10 independent SAC agents for estimating the performance of independent training in goal-conditioned pushing. As shown in Figure 5, Mint outperforms all methods both in terms of data efficiency and distance to the goal. SAC (concat all fc) also achieves comparable performance while independent networks fail to learn the task without sufficient amounts of data, suggesting that the ability of Mint to represent both joint training and independent networks per task is crucial in multi-task learning and can lead to considerable improvement. Simultaneous optimization of multiple, potentially unrelated tasks can prove challenging for deep neural networks. Recent multi-task learning architectures attempt to mitigate this issue by providing alternative pathways for information to flow through a neural network for each task. In this paper, we introduce a new multi-task learning module, Mint, which provides theoretical guarantees of universal approximation even for multi-task settings with no shared structure. We conjecture that this property, not shared by similar multi-task architectures, enables Mint to outperform other multi-task approaches on a variety of reinforcement learning benchmarks. We also observe that Mint is able to match or improve upon the performance of independent training. While Mint exhibits strong performance gains over previous methods, one potential limitation is that the task matrices may introduce a significant number of parameters, particularly as the number of tasks increases. As discussed, this can be alleviated for problem domains with many tasks, by learning a single neural network that produces the matrices and biases conditioned on the task descriptor. Further, in our experiments, we find that Mint-based networks can outperform prior methods while using comparable or fewer parameters. In summary, Mint is a simple, yet effective approach for deep multi-task learning. Its implementation requires minimal modifications over standard deep networks. As a , we expect it to be straightforward for future work to build upon or use Mint for more effective multi-task learning in deep networks. A PROOF OF THEOREM 1 Lemma 1. For a given α i, applying Mint to y (l−1) can express an arbitrary affine transformation at layer l for each task. Let W (l) and b (l) be an arbitrary weight matrix and bias vector. Suppose that for task k we wish to represent the affine transformation W k (l) y (l−1) + b k (l) at layer l of the network using the combination of Mint and the affine transformation described by applying W (l) multiplicatively

Represent the short summary for finding the corresponding scientific article (long)

We propose an approach that endows a single model with the ability to represent both extremes: joint training and independent training, which leads to effective multi-task learning.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the scientific article to find a short TLDR

Question: Training agents to operate in one environment often yields overfitted models that are unable to generalize to the changes in that environment. However, due to the numerous variations that can occur in the real-world, the agent is often required to be robust in order to be useful. This has not been the case for agents trained with reinforcement learning (RL) algorithms. In this paper, we investigate the overfitting of RL agents to the training environments in visual navigation tasks. Our experiments show that deep RL agents can overfit even when trained on multiple environments simultaneously. We propose a regularization method which combines RL with supervised learning methods by adding a term to the RL objective that would encourage the invariance of a policy to variations in the observations that ought not to affect the action taken. The of this method, called invariance regularization, show an improvement in the generalization of policies to environments not seen during training. Learning control policies from high-dimensional sensory input has been gaining more traction lately due to the popularity of deep reinforcement learning (DRL);; Zhang et al. (2018b); , which enables learning the perception and control modules simultaneously. However, most of the work done in RL chooses to evaluate the learned policies in the same environment in which training occurred. Using the same environments to train and test agents does not give any insight into the generalization abilities of the learned policy. There could be a number of changes in the environment at test time that would degrade the agent's performance. Variations could appear in the visual aspects that determine the agent's observation, the physical structure that determines the agent's state and even some aspects that are related to the agent's goal (Figure 1). For example, different observations of the same room are encountered at different times of the day (different lighting conditions). New obstacles could be present. Levels of a game could be different, yet playing a few levels should often be enough to figure out how to play the rest. Such variations might in a new environment where the control model that defined the training environment has changed. A robust policy should generalize from its experience and perform the same skills in the presence of these variations. DRL agents have been notorious for overfitting to their training environments. An agent could have drastically different performance on testing environments even if it manages to maximize the reward during training Zhang et al. (2018a). Supervised learning algorithms have been shown to have some generalization guarantees when adding proper regularization. However, these guarantees are weakened in reinforcement learning algorithms where the source of the data is not i.i.d.. In order to make use of the progress of DRL algorithms in practice we need policies that are robust to possible changes in the sensory inputs, surrounding structure and even some aspects of the task. In this paper we study the notion of generalization that is appropriate for visual navigation control policies that are learned with DRL. We present: a study of the generalization of visual control policies to certain changes in the underlying dynamical system; an alternative training method that combines DRL with supervised learning, thus using DRL to learn a controller while leveraging the generalization properties of supervised learning. In our experiments we use the VizDoom platform which is easily customizable and enables the generation of numerous variants of a given environment. Figure 1: The figure shows how environments may differ in their visual aspects, like textures of the surfaces. The textures provide a differentiator for each environment, where without them the environments would have shared the same state space. Visual navigation for mobile robots combines the domains of vision and control. Navigation can be described as finding a suitable and safe path between a starting state and a goal state. Classical approaches split the problem into a sequence of sub-tasks, such as map construction, localization, planning and path following. However, each sub-task requires some handengineering that is specific to the environment and task which makes it hard to adapt it to different scenarios without performing some tuning. Deep learning approaches enable the use of highly non-linear classifiers that can adapt their inner representations to learn to robustly solve complicated tasks. In this work, we use reinforcement learning algorithms coupled with deep learning approaches to solve the task of navigating an agent towards a goal object using only its visual observations as input. The field of view of the agent is limited, i.e., it does not observe the full environment, and we do not provide an explicit map of the environment to that agent. We model the problem of visual navigation as a partially observed Markov decision process (POMDP) . A POMDP is given by a tuple where S is the set of states, A is the set of actions and Ω is the set of observations, all which are assumed to be finite sets. The reward function is R: S × A → R. The conditional transition probability mass function is T: S × A × S →, with the interpretation that T (s, a, s) = p(s t+1 = s |s t = s, a t = a) is the probability that the next state is s given that the current state is s and that action a is taken. The conditional observation probability mass function is is the probability of observing o in state s when the last action taken was a, and we allow for a special observation probability O(s, o) = p(o 0 = o|s 0 = s) when in the initial state s and no action has yet been taken. Finally, P 0 is the initial state probability mass function, so that P 0 (s) = p(s 0 = s) is the probability that the initial state is s. In DRL, we work with a parameterized policy π θ (h, a) = p θ (a t = a|h t = h) with parameters θ ∈ Θ, giving the probability of taking action a given observation-action history h t:= (o 0, a 0, o 1, a 1, . . ., a t−1, o t). The objective is to adjust the parameters θ to attain a high value for the discounted reward with discount factor γ ∈. The expectation is over state-observation-action sequences where the initial state s 0 is drawn from P 0 and other elements of such a sequence are drawn from T, O and π θ respectively . Many methods for attempting to approximate optimal policies have been proposed. For instance, policy gradient methods perform gradient ascent on estimates of the expected discounted reward. In this work we use the proximal policy optimization (PPO) algorithm, which arguably shows relatively robust performance on a wide range of different tasks . As in classification, we wish to learn from a finite training set but still perform well on previouslyunseen examples from a test set. To formalize this, we have a distribution D over POMDPs, representing multiple environments or tasks, and we sample n train POMDPs from this distribution P 1, P 2,..., P n train. In the context of navigation, these POMDPs might differ in terms of their observation distributions, perhaps representing views of the same environment at different times of day or year, in terms of their transition distributions, perhaps representing maps with different geometries, or in terms of their reward distributions, perhaps corresponding to the specification of different goal states. Given this sample, we then learn a policy π θ from a finite collection of state-observation-action sequences from these POMDPs. In order to have a meaningful common policy across these POMDPs, we require that they have common state, action and observation spaces S, A and Ω. By analogy with the notion of generalization risk in classification , we say that policy π θ generalizes well if it attains a high value for the expectation of the discounted reward over the full distribution of POMDPs, which we call the discounted generalization reward, so that E P∼D J P (θ) is high in some sense. This is our own terminology as we did not find a semantically-equivalent term in the literature. It is not hard to see that the discounted generalization reward is actually the discounted reward J P D (θ) of a single larger POMDP P D, whose state space may however no longer be finite. To see this, let us associate a unique identifier i(P) with any POMDP sampled from D and let I D be the set of all such unique identifiers. In the large POMDP P D, the state space is the Cartesian product S × I D of the original states and these unique identifiers, but the action and observation spaces are just A and Ω. The initial state distribution is obtained by first sampling a POMDP P ∼ D and then sampling s env ∼ P P 0 from that POMDP's initial state distribution. The initial state in the large POMDP is then the concatenation (s env, i(P)). Thus one might succinctly state the problem of generalization in POMDPs as follows: given a distribution D over POMDPs with common state, action and observation spaces and access to a sample of state-observation-action sequences from a sample of POMDPs drawn from D, choose a policy π θ that obtains a high value for the discounted reward J P D (θ). Training in synthetic environments enables the simulation of huge amounts of experience in a span of a few hours. Simulations are convenient to use when training reinforcement learning agents that are often highly sample inefficient. There is, frequently, a gap between the synthetic world and the real-world, mainly due to the manner in which the simulators depict the real-world dynamics and visual appearances. Often, these simulated worlds capture the richness and noise of the real-world with low-fidelity. Many have tried to propose transfer learning techniques to bridge the reality gap in order to still make use of fast simulators for training. One popular method to bridge the reality gap is by randomizing some aspects of the training environment. This domain randomization technique has been shown to be successful for the transfer of grasping policies from simulated training environments to the real-world. However, the learned models ing from that work are not control policies, but perception modules. Previous work has showed some success in transferring the perception module learned in simulation to the real world, but not the controller. conduct a large scale study on generalization using a new environment, that resembles an arcade game, which they call CoinRun. They experiment by training on different images and different level structures. They test with different regularization strategies and network architectures finding that the RL agent has a surprising tendency to overfit even to large training sets. Zhang et al. (2018a) reach a similar , when learning in grid-world environments, and state that the agents have a tendency to memorize levels of the training set. , however, they argue that the methods that inject stochasticity into the dynamics of the system to prevent memorization, such as sticky actions and random initializations , often do not help. In our work we are interested in generalization when navigating under partial observability unlike the fully observable CoinRun or grid-world environments. Domain adaptation methods have also been used for simulated to real transfer. They allow models trained on a source domain to generalize to a target domain. train a generative model to adapt the synthetic images of the simulator to appear like the real environment. It was shown to successfully transfer a grasping policy trained in simulation to the real world. However, they do not discuss whether the policy generalizes when variations happen in the target domain. Another aspect of generalization is the transfer of learned skills to solve different tasks. In other words, generalization to the goal of the trained agent g. Achieving different tasks would require the agent to have the ability to maximize different reward functions. consider working with value functions that contain the goal g as part of the agent's state. They call them universal value functions. The reward will then become a function of a state-action-goal tuple (s, a, g) instead of a classical state-action pair. In the paper, the authors present universal value function approximators (UVFA). A method that attempts to learn a universal value function estimate V θ (s, g). They show that UVFA's can generalize for unseen state-goal pairs in grid-world setup. Deep reinforcement learning has been used to train control policies. These DRL based methods generally propose to learn motor control commands from raw camera images, thus mapping pixels to commands that control the robot's motors. DRL algorithms have been used for various navigation tasks such as goal conditioned navigation; and mapless navigation. Control policies learned from high-dimensional visual input are often brittle and lack the robustness to operate in novel situations. Our main contribution is to propose a regularization term that can be added to the RL objective to improve the robustness of the learned policy to variations in the observations, presented in Section 4.2. However, to motivate the necessity of our proposed method we study domain randomization in Section 4.1; one of the current main practices that aims at learning a policy that generalizes well. Domain randomization is typically used to train policies that can generalize to variations and noise in the observations. It is done by training on several POMDP's that share the same S, A, Ω spaces, however they could in their observation distribution. The motivation behind domain randomization is that it is assumed to be an effective technique to provide a policy that is invariant to the changes that would appear in the observations. We explore the problem of navigating the agent towards a goal object with random noise added to the agent's observations. If the agent is able to perform the task in an environment defined by a POMDP P 1 then it should still be able to perform the task in another POMDP P 2, if certain features f of the environment that are specific to successfully achieving the task exist and are invariant to these variations, i.e., f (P 1) = f (P 2). In Section 5.1, we study domain randomization when added to RL training and the ability of ing policies to generalize in unseen POMDPs. We want to investigate if the policy does in fact overfit to the training POMDPs and whether we mitigate that overfitting by training the policies on multiple POMDPs. In the previous sections, we discussed how overfitting to the training environment can be a big problem in RL. Furthermore, we should be careful not to jump to the that training on different environments will ensure policies that generalize well to new environments. It is merely an assumption that has been shown to empirically hold up when used in a supervised learning context. However, we show in this work that this assumption might not hold for reinforcement learning techniques. This is compatible with the findings in and Zhang et al. (2018a). We reason that in order to generalize well, the training objective should include a term that encourages policy generalization. Therefore, putting the weight of the problem of generalizing explicitly in the objective function itself. Formally, a function h of variable x is invariant to a transformation φ of x if h(x) = h(φ(x)). We can deduce the same definition for the invariance of a policy π to changes in the observation given by some transformation T, π(o) = π(T (o)). We add this regularization penalty term to the RL objective as shown in Equation: where L ppo is the PPO objective , θ is the set of parameters that define the policy π θ, d is a distance function between the two conditional distributions, and λ is a weighting coefficient of the penalty. T is a transformation of the observations. Given an observation o and a transformation on that observation T where the transformation still holds the semantic context of the underlying state, but with added visual variations. We can think of the difference between observing a room with observation o and observing the same room with observation T (o) as the color of the wall for example. Therefore, let us say that we observe o in POMDP P and observe, where f (P) is the set of invariant features of the environment defined by the POMDP P. We further discuss the nature of T in the experiments section. M is the number of transformations of each observations. The penalty d in Equation 1 resembles adding a constraint on the PPO objective, where the new objective dictates that the policy should simultaneously obtain a high reward while behaving similarly for the observations o and T (o). The idea is similar, in spirit, to trust region policy optimization where a penalty term, resembling that which would from imposing a trustregion constraint, is added to ensure monotonic improvement of the average return with each policy update. We call this method in Equation 1 invariance regularization (IR) since the regularization term indicates the invariance of the learned policy to a transformation of given observations. We propose two ways to solve the RL problem in Equation 1. The first is to directly optimize the full objective by adding the penalty to the original PPO loss. The second method splits the training process to two stages of training RL first and then performing a supervised learning step to minimize d(π(o), π(T (o))) which presents an elegant form that combines reinforcement learning with supervised learning, more details of the second method is available in Appendix A. In the next section, we will discuss experiments using both methods. Before that, we will describe a study on the effectiveness of domain randomization as a mean to reducing overfitting in DRL agents. In this section we present the of two experiments. The first is about training RL with domain randomization. We discuss the ability of the learned policies to generalize to unseen environments when trained on variations of the training environment. The next part presents the obtained when using the invariance regularization (IR) method, proposed in Section 4.2, with domain randomization and shows that it improves the success rate considerably. We performed these experiments because we are interested in the following questions: Does training on environments with random variations (as domain randomization suggests) learn a representation of the invariant f with which the policy can generalize to other environments that share the same invariant features? Can we find a training algorithm that would empirically guarantee finding these invariant features f? We leverage the customizability of VizDoom maps with hundreds of unique textures to generate train/test scenarios. The agent is required to reach an object in order to get a reward. We train an actor-critic style agent to solve the task. The network consists of three convolutional layers and 2 fully connected layers, followed by the policy and value function estimator layers. The policy output is a four-dimensional fully-connected layer, where the four dimensions corresponds to four actions; move forward, turn right, turn left and do nothing. The ouput of the policy layer is a log-probability of each action. The value layer is a single unit that predicts the value function. This network architecture was proposed by. ReLUs are used as the non-linear operations in all layers. As mentioned, we optimize the PPO objective with a binary reward function (+1 if goal is reached, 0 otherwise) and a discount factor γ = 0.99. We generate the variations of the training environment by changing the textures on the surfaces using the numerous textures provided by. We train agents on a subset of 1, 10, 50, 100 and 500 rooms from the generated environments and test on 50 rooms with textures from a hold-out set which are not seen in training. We detail this experimental setup in Appendix B. We experiment with different types of visual input; RGB, RGB-D and Grayscale. The number of training iterations is fixed at 5 × 10 6 to ensure repeatability of our experiment. The are therefore potentially pessimistic, and in future work we would like to choose the number of iterations for each network independently so as to maximize generalization performance. The agent and the goal object are initialized at random positions in the environment at the start of each episode. The role of depth. Adding a depth channel to the observation plays a significant role in generalization. Depth is invariant to many changes in the visible spectrum of the observations. This might lead the training agent to partly find an invariance in observations in its implicit perception model, which in this case can be as simple as focusing on the depth channel only. Therefore, it was not surprising to see, in Table 1, that the depth agents (RGB-D) generalize better than the agents without any depth information. Table 1 shows the success rate of the PPO models with respect to the number of training environments used and the input type (RGB, RGB-D). The are averaged over 5 seeds, a standard practice in the RL literature today. We notice the superior performance of the agent with depth than the agent without depth. The fact that the RGB agent is not able to generalize well even when exposed to numerous environments tells us that it might not be learning the invariance relating the environments. On the other hand, the RGB-D agents perform well on the testing environments even when the agents are only exposed to 10 random training environments. Looking at the RGB and RGB-D experiments, the agents trained on 100 and 500 environments generalize worse on average than the ones trained on 10 and 50, which indicates that some agents might be overfitting. This is inspite of the fact that these agents are able to maximize the reward in the training set regardless of the set size. Looking at the max statistic of these (not shown in this paper) the 100 and 500 experiments outperform the rest. However, the 100 and 500 experiments have a higher variance in the success rates of different seeds than the 10 and 50 experiments. High variance in the test of the 100/500 RGB-D experiments shows that some seeds are able to achieve a near perfect score on the testing environment and others completely fail, thus there is then no empirical guarantee that RL agents will generalize when exposed to numerous environments. The average success rate for the RGB input without the depth shows that domain randomization alone might not be an effective method to adapt the policy to variations in the observations, at least not in the context of RL. In fact, it shows little progress, e.g., the RGB agent exposed to one environment achieves around a 20% success on the testing environments and the agents exposed to 50+ environments achieve less than 40% success. These are consistent when running with a grayscale channel (see Table 1). While training by randomizing the environment did show some success in making supervised learning models generalize better, it fails to do so in RL policies. It is clear from these , that adding random variations and relying solely on the RL objective is not enough to ensure generalization. Much of the success of domain randomization in previous works was reported using supervised learning. Also, the generalization abilities of machine learning algorithms have been linked to supervised learning setups. Therefore, it would make sense to adapt supervised learning techniques to regularize the models trained with DRL. In this section we will discuss the obtained from training the agent using the method proposed in Section 4.2. As mentioned in Section 4.2, we propose two methods of using the proposed IR penalty. The first is to add to the PPO objective as in Equation 1, this method is referred to as (full objective) in the . The second, which is referred to as (split) in the , is to split the objective into two parts; RL step and a supervised learning step (more details available in Appendix A). The value of λ in Equation 1 used in all IR (full objective) experiments is 1.0. As for the nature of transformation T of the observations, we tested with the same randomly textured environments from VizDoom, that were used in the previous section, in order to be able to make fair comparisons with the pure RL and domain randomization agents. Regarding the distance penalty term d in Equation 1, we did preliminary experiments with the KL divergence, L1, L2 and cross-entropy losses and the KL divergence returned the best . Table 1 shows the for combining PPO with the IR penalty using the two proposed implementations. Observing the split method's , we see that the proposed training procedure returns stable success rates that are improves as more environments are added. The split version was able to outperform vanilla PPO and substantially improve the generalization especially in the cases of RGB/Grayscale inputs. Training with the full objective, however, returned the best that outperform vanilla PPO with domain randomization and the split version of the IR algorithm. Similar to the split version, training on the full objective shows stable performance for the different inputs across different seeds. The , in Table 1, also show that the trained models, on the full objective, are achieving test success rate, with only 10 training environments, that is close if not identical to the agents trained on 50, 100 and 500 environments. These suggests that training with the full objective version of the IR algorithms does not require a large number of environments to learn the invariant features. Notice the average testing success rate is similar across the different number of training environments since the model learns the invariant features from only 10 environments and adding more environments that share the same invariant features will not make a difference. We can verify that hypothesis when looking at the RGB-D testing in the full objective part. All agents achieve a near perfect score which we attribute to the availability of an invariant feature map in the input (the depth channel) which only the agents trained with the full objective are able to catch. Regularization has been shown to help in the generalization of supervised learning models. Using regularization in supervised learning often improves the performance of the trained models on test sets. However, regularization has not been frequently used in DRL setups, The average success rate show that some of these methods achieve similar to ours in some instances. (Right): The lower SPL of the other regularization methods relative to ours indicates some randomness in their learned policies possibly due to the previously-common poor practise of testing and training in the same environment so there is no generalization gap. We compare our method with some regularization techniques that are frequently used; dropout, batchnorm and L2. The first experiment has a dropout layer added after each convolutional layer , the second has a batchnorm layer added after every convolutional layer and the last uses L2 regularization. We choose the dropout probability to be 0.1 and the L2 weight to be 10 −4, the same values that were proposed by. As in the previous setup, we train five models (different seeds) for each technique and evaluate on 50 environments whose textures are sampled from a hold-out set. We report the experiments done with RGB input only as it poses a harder problem and a larger gap than RGB-D. Figure 2 (left) shows the average success rate over 5 seeds for the four methods. We see that our proposed method is the only one that is steadily improving when more environments are added. The batchnorm models performed worst while dropout and L2 achieved similar success rates to the split version of our method given 50 and 500 training environments. However, the entropy of the learned policies is substantially higher when dropout and L2 are added to the model. We hypothesize that the high entropy policies are able to generalize by acting randomly in some instances and this makes them more robust in certain situations. We show the success weighted shortest path length (SPL) in Figure 2 (right). A random behavior that displays robustness (has a high success probability) would return a relatively lower SPL due to the fact that this random behavior will probably not take the shortest possible path to the goal. Details of the formulation of SPL are available in Appendix C. Figure 2 (right) shows that the dropout and L2 agents have a lower SPL than the IR agents indicating that these policies with higher entropy are inefficient. We present a study of the generalization capabilities of visual navigation agents trained with deep reinforcement learning algorithms. We formalize what it means to generalize in the context of a POMDP. We find that the tendency of RL agent to overfit even when exposed to large training sets is quite visible. We show that using domain randomization with RL, without adding invariant features to the input such as the depth maps, is not enough to generalize. In the second part, we proposed Invariance Regularization (IR), a method that attempts to regularize the RL model with a supervised learning loss. It improves the generalization success and displays stable performance across different seeds. In this work, we focused our experimentation on generalization to changes in the input observation. However, it is also interesting to generalize the learned skills to different architectural designs of the environment, just as one one wishes to generalize to different levels of the game as proposed in the retro competition. Another avenue of future work is to explore the appropriate transformation function T of the observations. One might consider an adaptive form of T learned with data augmentation or adversarial examples Goodfellow et al. (2015 The first part consists of training RL on the observations of the original training environment, while the second part can be seen as a supervised learning objective on the transformed observations, as shown in Algorithm 1. The first step trains RL on one environment and then use the actions that the trained policy would have taken in that environment to tune the model with supervised learning on the textured environments. In the reported experiments using the split version, the model is trained with one iteration of the algorithm. Therefore, the training process has two stages, train RL then train with a supervised learning setup, without iterating between both. As stated in Section 5.1, we run training on a subset of 1, 10, 50, 100 and 500 rooms where the surfaces in each room are sampled from the variety of textures available in Vizdoom. The ing policies are tested on 50 rooms with textures from a hold-out set which are not seen in training. During training we run several agents in parallel to quickly collect observation-action-reward data in multiple environments. Another advantage of this parallelization is the ability to run each agent on a variation of the training environment. Due to hardware limitations, we cannot run one agent for each environment, at least not when we have a large number of training environments, i.e., 100 or 500. Therefore, each agent samples one environment from the training set and runs on it for some n episodes before sampling another one (n = 25 episodes). SPL was proposed by as a way of measuring the navigation agents success rates while taking into account the time it takes agents to succeed 1. where N is the number of runs, S i is the binary indicator of the success of episode i, l i is the length of the shortest possible path and p i is the length of the path taken by the agent.

Represent the TLDR for finding the corresponding scientific article

We propose a regularization term that, when added to the reinforcement learning objective, allows the policy to maximize the reward and simultaneously learn to be invariant to the irrelevant changes within the input..

Represent the short summary for finding the corresponding scientific article (long)

Parameter space noise allows reinforcement learning algorithms to explore by perturbing parameters instead of actions, often leading to significantly improved exploration performance.