Question:
represent the article for finding a single-sentence tldr

------

The query could be ' We give a formal procedure for computing preimages of convolutional network outputs using the dual basis defined from the set of hyperplanes associated with the layers of the network. We point out the special symmetry associated with arrangements of hyperplanes of convolutional networks that take the form of regular multidimensional polyhedral cones. We discuss the efficiency of of large number of layers of nested cones that from incremental small size convolutions in order to give a good compromise between efficient contraction of data to low dimensions and shaping of preimage manifolds. We demonstrate how a specific network flattens a non linear input manifold to an affine output manifold and discuss it's relevance to understanding classification properties of deep networks. Deep convolutional networks for classification map input data domains to output domains that ideally correspond to various classes. The ability of deep networks to construct various mappings has been the subject of several studies over the years (1; 3; 10) and in general ed in various estimates of capacity given a network structure. The actual mappings that are learnt by training a specific network however, often raise a set of questions such as why are increasingly deeper networks advantageous (13; 14)? What are the mechanisms responsible for the successful generalisation properties of deep networks? Also the basic question why deep learning over large datasets is so much more effective than earlier machine learning approaches is still essentially open, BID6. These questions are not in general answered by studies of capacity. A more direct approach based on actual trained networks and the mappings they are efficiently able to produce seems needed in order to answer these questions. It seems ever more likely e.g that the ability of deep networks to generalize is connected with some sort of restriction of mappings that they theoretically can produce and that these mappings are ideally adapted to the problem for which deep learning has proven successful, Due to the complexity of deep networks the actual computation of how input domains are mapped to output classifiers has been considered prohibitively difficult. From general considerations of networks with rectifier (ReLU) non linearities we know that these functions must be piecewise linear BID9 but the relation between network parameters such has convolutional filter weights and fully connected layer parameters and the actual functions remains largely obscure. In general, work has therefore been concentrated on empirical studies of actual trained networks (6; 8; 9) Recently however there have been attempts to understand the relation between networks and their mapping properties from a more general and theoretical point of view. This has included specific procedures for generating preimages of network outputs BID3 and more systematic studies of the nature of piecewise linear functions and mappings involved in deep networks, (2; 11; 15).In this work we will make the assertion that understanding the geometry of deep networks and the manifolds of data they process is an effective way to understand the comparative success of deep networks. We will consider convolutional networks with ReLU non linearities. These can be completely characterised by the corresponding hyperplanes associated with individual convolutional kernels. We will demonstrate that the individual arrangement of hyperplanes inside a layer and the relative arrangement between layers is crucial to the understanding the success of various deep network structures and how they map data from input domains to output classifiers. We will consider only the convolutional part of a deep network with a single channel. We will assume no subsampling or max pooling. This will allow us to get a clear understanding of the role of the convolutional part. A more complete analysis involving multiple channels and fully connected layers is possible but more complex and will be left to future work. The focus of our study is to analyse how domains of input data are mapped through a deep network. A complete understanding of this mapping and its inverse or preimage will give a detailed description of the workings of the network. Since we are not considering the final fully connected layers we will demonstrate how to compute in detail the structure of input data manifold that can be mapped to a specified reduced dimensionality affine manifold in the activity space of the final convolutional output layer. This flattening of input data is often considered as a necessary preprocessing step for efficient classification. The understanding of mappings between layers will be based on the specific understanding of how to compute preimages for networks activities. We will recapitulate and extend the work in based on the construction of a dual basis from an arrangement of hyperplanes. By specialising to convolutional networks we will demonstrate that the arrangement of hyperplanes associated with a specific layer can be effectively described by a regular multidimensional polyhedral cone oriented in the identity direction in the input space of the layer. Cones associated with successive layers are then in general partly nested inside their predecessor. This leads to efficient contraction and shaping of the input domain data manifold. In general however contraction and shaping are in conflict in the sense that efficient contraction implies less efficient shaping. We will argue that this' conflict is resolved by extending the number of layers of the network with small incremental updates of filters at each layer. The main contribution of the paper is the exploitation of the properties of nested cones in order to explain how non linear manifolds can be shaped and contracted in order to comply with the distribution of actual class manifolds and to enable efficient preprocessing for the final classifier stages of the network. We will specifically demonstrate the capability of the convolutional part of the network to flatten non linear input manifolds which has previously been suggested as an important preprocessing step in object recognition, (5; 12) Transformations between layers in a network with ReLU as nonlinear elements can be written as DISPLAYFORM0 Where [] + denotes the ReLU function max(0, x i). applied component wise to the elements of the input vector x which will be confined to the positive orthant of the d-dimensional Euclidean input space. It divides the components of the output vector y into two classes depending on the location of the input x: DISPLAYFORM1 In order to analyse the way domains are mapped through the network we will be interested in the set of inputs x that can generate a specific output y. DISPLAYFORM2 This set, known as the preimage of y can be empty, contain a unique element x or consist of a whole domain of the input space. This last case is quite obvious by considering the ReLU nonlinearity that maps whole half spaces of the input domain to 0 components of the output y. The preimage will depend on the location of the input relative to the arrangement of the hyperplanes defined by the affine part of the mapping: DISPLAYFORM3 These hyperplanes divides the input space into a maximum of 2 d number of different cells with the maximum attained if all hyperplanes cut through the input space which we take as the non negative orthant of the d-dimensional Euclidean space R d +. Understanding the arrangement of these hyperplanes in general and especially in the case of convolutional mappings will be central to our understanding of how input domains are contracted and collapsed through the network. The preimage problem can be treated geometrically using these hyperplanes as well as the constraint input domains defined by these. For a given output y we can denote the components where y j > 0 as y j1, y j1... y jq and the complementary index set where y i = 0 as i 1, i 1... i p With each positive component of y we can associate a hyperplane: DISPLAYFORM4 which is just the hyperplane Π j translated with the output y j For the 0-components of y we can define the half spaces DISPLAYFORM5 I.e the half space cut out by the negative side of the plane Π i. These planes and half spaces together with the general input domain constraint of being inside R + d define the preimage constraints given the output y. If we define the affine intersection subspace: DISPLAYFORM6 and the intersection of half spaces: DISPLAYFORM7 the preimage of y can be defined as: DISPLAYFORM8 The constraint sets and the preimage set is illustrated in FIG0 for the case of d = 3 and various outputs y with different number of 0-components. For fully connected networks, computing the preimage set amounts to finding the intersection of an affine subspace with a polytope in d− dimensional space. This problem is known to be exponential in d and therefore intractable. However, we will see that this situation is changed substantially when we consider convolutional instead of fully connected networks. In order to get more insight into the nature of preimages we will devise a general method of computing that highlights the nature of the arrangement of hyperplanes. The set of hyperplanes Π i, i = 1... d will be assumed to be in general position, i.e. no two planes are parallel. The intersection of all hyperplanes excluding plane i: DISPLAYFORM0 is then a one-dimensional affine subspace S i that is contained in all hyperplanes Π j excluding j = i. For all i we can define vectors e i in R d parallel to S i. The general position of the hyperplanes then guarantees that the set e i is complete in R d. By translating all vectors e i to the point in R d which is the mutual intersection of all planes Π i DISPLAYFORM1 they can therefore be used as a basis that spans R d. This construction also has the property that the intersection of the subset of hyperplanes: DISPLAYFORM2 is spanned by the complementary dual basis set DISPLAYFORM3 The dual basis can now be used to express the solution to the preimage problem. The affine intersection subspace P * associated with the positive components j 1 j 2... j q of the output y is spanned by the complementary vectors associated with the negative components i 1 i 2... i p. These indices also define the hyperplanes Π 1, Π 2... Π p that constrain the preimage to lie in the intersections of half spaces associated with the negative sides. We now define the positive direction of the vector e i as that associated with the negative side of the plane Π i. If we consider the intersection of the subspace P * and the subspace generated by the intersections of the hyperplanes Π i associated with the negative components of y we get: DISPLAYFORM4 Due to complementarity of the positive and negative indices, this is a unique element x * ∈ R d (marked "output" in figure 1 which lies in the affine subspace of the positive output components P * as well as on the intersection of the boundary hyperplanes Π i that make up the half space intersection constraint X − for the preimage. if we take the subset of the dual basis vectors with e i1, e i2 . . . e ip and move them to this intersection element, they will span the part of the negative constraint region X − associated with the preimage. I.e. the preimage of the output y is given by: DISPLAYFORM5 We will now specialise to the standard case of convolutional networks. In order to emphasize the basic role of geometric properties we will consider only a single channel with no subsampling. Most of what we state will generalize to the more general case of multiple channels with different convolutional kernels but needs a more careful analysis we will exploit the fact that convolutional matrices are in most respects asymptotically equivalent to those of circulant matrices where each new row is a one element cyclic shift of the previous. For any convolution matrix we will consider the corresponding circulant matrix that appends rows at the end to make it square and circulant. Especially when the support of the convolution is small relative to the dimension d, typically the order of 10 in relation to 1000, this approximation will be negligible. Except for special cases the corresponding circulant will be full rank, which means that properties about dual basis etc. derived previously will apply also here. As is standard we will assume that the bias b is the same for all applications of the convolution kernels. The first thing to note about hyperplanes associated with circulant matrices is that they all intersect on the identity line going through the origin and the point (1, 1, . . . 1). Denote the circulant matrix as C with elements c i,j. The circulant property implies c i+1, DISPLAYFORM0. Each row is shifted one step cyclically relative to the previous. For the hyperplane corresponding to row i we have: DISPLAYFORM1 It is easy to see that the circulant property implies that the sum of all elements along a row is the same for all rows. Let the sum of the row be a. We then get: x j = −b/a for j = 1... d as a solution for this system of equations which is a point on the identity line in R d.The arrangement of the set of hyperplanes: DISPLAYFORM2 with w T i the i:th row of the circulant augmented convolutional matrix W, will be highly regular. Consider a cyclic permutation P x of the components of the input x described by the single shift matrix P i.e x i is mapped to x i+1 for i = 1... d − 1 and x d is mapped to x 1. We then get: DISPLAYFORM3 which states that points on the hyperplane associated with weights w i are mapped to hyperplane associated with weights w i+1. The hyperplanes associated with the weights w i i = 1... d therefore form a regular multidimensional polyhedral cone in R d around the identity line, with the apex located at x T = (−b/a, −b/a . . . − b/a) controlled by the bias b and the sum of filter weights a. Geometrically, the cone is determined by the apex location, the angle of the planes to the central identity line and its rotation in d-dimensional space. Apex location and angle are two parameters which leaves d − 2 parameters for the multidimensional rotation in R d. This maximum degree of freedom is however attained only for unrestricted circulant transformations. The finite support of the convolution weights in CNN:s will heavily restrict rotations of the cone. The implications of this will be discussed later. Any transformation between two layers in a convolutional network can now be considered as a mapping between two regular multidimensional polyhedral cones that are symmetric around the identity line in R d. The coordinate planes of the input space R d + can be modelled as such a cone as well as the output space given by the convolution. The strong regularity of these cones will of course impose strong regularities on the geometric description of the mapping between layers. Just as in the general case, this transformation will be broken down to transformations between intersection subspaces of the two cones. In order to get an idea of this we will start with a simple multi layer network with two dimensional input and output and a circulant transformation: FIG1 illustrates the mapping of data from the input space (x 1, x 2) to the output space (y 1, y 2) for two networks with 3 and 6 layers respectively. The dashed lines represent successive preimages of data that maps to a specific location at a layer. By connecting them we get domains of input data mapped to the same output at the final layer, i.e they are contraction flows depicting how data is moved through the network. Note that in both layers the major part of the input domain is mapped to output. This is illustrated for the first trivial bias only layer with a = 1, b = 0. The domain of the input that is mapped to output domain is just quite trivial planar manifolds. DISPLAYFORM0 The second network with more varied weights illustrates how input domain manifolds with more structure can be created. It also demonstrates the importance of the concept of "nested cones" and how this affects the input data manifolds. The red lines represent data that is associated with layer cones that are completely nested inside its predecessors, while the black lines represent data where the succeeding cone has a wider angle than its predecessor. When this happens, the hyperplanes associated with the output cone will intersect the hyperplanes of the input cone and input data beyond this intersection is just transformed linearly. Since all data in figure 2 is remapped to the input space this has the effect that data is not transformed at all. This has no effect at all at the shaping of the input manifold. One could say that these layers are "wasted" beyond the location of the intersection as far as network properties are concerned since they neither contribute to the shaping or the contraction of input data manifolds. The effect of this on the input manifold can be seen as a less diverse variation of its shape (black) compared to the previous part associated with the completely nested part of the layer cones. In higher dimensions the effects of nested vs. partially nested cones appear in the same way but more elaborate. In addition to the 2d case we also have to consider rotations of the cone, which as was pointed out earlier, has d − 2 degrees of freedom for cones in d dimensional space. The effects of contraction of data from higher to lower dimensions also become more intricate as the number of different subspace dimensionalities increases. Most of these effects can be illustrated with 3 dimensional input and output spaces. For d = 3 the generic circulant matrix can be used to define a layer transformation: DISPLAYFORM1 The transformation properties of this network are most easily illustrated if we start with the pure bias case with transformation W = I, i.e a = 1, b = 0, c = 0. A specific element in input space is mapped according to its position relative to the hyperplanes. If we use the dual basis to define the coordinates of the output data, the mapping for the input element will be the same in input cells with the same relation to all hyperplanes. In d dimensions, the hyperplanes divide the input space into 2 d cells where elements are mapped to a specific associated intersection subspace in the output domain. In FIG2, the grey shaded boxes indicate two cells with different numbers of negative constraints 1 and 2 respectively. The content of the upper one with one negative constraint including all its bounding faces and their intersections is mapped to a specific 2d hyperplane in the output domain while the content of the lower one with two negative constraints is mapped to the 1d intersection of two hyperplanes. This illustrates the most important property of the nesting of the cones associated with the input and output layer: For a range of transformations in the vicinity of the identity mapping, the input space, properly normalised in range, is divided into cells where the elements of the cells including their bounding faces an their intersections are mapped to output intersection subspaces with equal or lower dimension. This means that the content of the cell is irreversibly contracted to lower dimensions. and the components of the dual bases used to span these. Note that these are affected by changing the angle of the output cone. It introduces a limit of the nesting beyond which the mapping properties of the transformation are changed so that data no longer maps to a manifold of equal or lower dimensionality. I.e the contraction property is lost in those regions of the input space where nesting of cones ceases. We will formally define this important property of nested cones as:Let R We see that this definition implies that the cone formed by planes Π is completely contained in that formed by the planes Π 0 but also that its relative rotation is restricted. We will have reason to relax the condition of inclusion of all elements i 1... i p in the intersection subset and talk about cones with restricted nesting. Complete nesting implies contraction of data from one layer to the next which can be seen from the fact that all elements of the complete intersection subset Π 0 i1 ∩... ∩ Π 0 ip and thereby in each subset are mapped to the intersection subset M i1...ip = Π i1 ∩... ∩ Π ip with same dimensionality. In addition elements from intersection subsets formed by subsets of indexes i 1... i p will also be mapped to this same intersection subset. The subsets associated with these indexes are however of higher dimension. Consequently, mapping of data between layers will be from intersection subsets to intersection subsets with equal or lower dimension. This is the crucial property connecting degree of nesting with degree of contraction. By going further through the network to higher layers this contraction is iterated and data is increasingly concentrated on intersection subspaces with lower dimension which is reflected by the increased sparsity of the nodes of the network. The convolutional part of a deep network can therefore be seen as a component in a metric learning system where the purpose of the training will be to create domains in input space associated with different classes that are mapped to separate low dimensional outputs as a preprocessing for the final fully connected layers that will make possible efficient separation of classes. There is therefore a conflict between the diversity of input manifolds that contract to low dimensional outputs and the degree of contraction that can be generated in a convolutional network. The efficient resolution of this conflict seems to lie in increasing the number of layers in the network in order to be able to shape diverse input manifolds but with small incremental convolution filters that retain the nesting property of the cone in order to preserve the proper degree of contraction. Empirically, this is exactly what has been demonstrated to be the most efficient way to increase performance of deep networks (13; 14). We are now in a position to give a general characterizaton of the preimage corresponding to a specified output domain at the final convolutional output layer assuming the property of nested layer cones. Ideally we would like to include the final fully connected layers in this analysis but it will require a special study since we cannot assume the nesting property to be valid for these. In the end the network should map different classes to linearly separable domains in order to enable efficient classification. It is generally suggested that the preprocessing part of a network corresponds to flattening nonlinear input manifolds in order to achieve this final separation at the output. In order to be able to draw as general as possible we shall demonstrate the exact structure of a nonlinear input manifold that maps to a prespecified affine manifold at the final convolutional layer. We denote this manifold M and the output at the final convolutional layer by x (l). The final layer can be characterised by the set of hyperplanes: DISPLAYFORM0 Let the zero components of the output x (l) be i 1, i 2... i q. It can then be associated with the intersection of the output manifold and the corresponding hyperplanes DISPLAYFORM1 The degree of intersection q will depend on the dimensionality of M. If M is a d − 1 dimensional hyperplane in general position it will intersect any combination of hyperplanes at output level l. This is the maximum complexity situation that will generate a d − 1 -dimensional input manifold. Reducing dimensionality of M means reducing the possible intersection with combinations of hyperplanes. Note that if M intersects the set the intersection of planes i 1, i 2... i p it also intersects the intersection of any subset of these. Intersecting M with each of these subsets will generate pieces of intersections linked together. These are affine subsets with different dimensionality and the preimage of each piece will be generated by complementary dual basis components. This is illustrated by figure 4 for the case of an affine plane in R 3 + intersecting to give a triangular domain. In this case we have three points on the coordinate axis, and three lines connecting these. The three points will all span 2d planes bases on different pairs of complementary dual basis components. In addition to these, the points on the lines of the triangle generated by intersecting M with each of the three individual output planes will generate 1d lines that jointly will span a 2d plane. This plane will connect continuously with the planes spanned from the points on the axis to yield a piecewise planar input manifold to the final layer. Continuing through the network, this piecewise planar manifold will intersect with the planes of layer l − 1 and the procedure is iterated until we reach the input layer. This procedure generalises to arbitrary dimensions but the complexity of course grows with the increasing combinatorics. The basic principle of layer by layer recursively generating piecewise affine manifolds still holds. The complexity lies in the fact that each intersection of the manifold M with every subset of possible hyperplane intersections will generate a seeding hyperplane and and each of these will act as a new manifold M at the next layer. Note however that the nested cone property substantially reduces complexity compared to the general case of arbitrary hyperplanes. Figure 4: Piecewise planar manifold in 3d input space that maps to affine manifold (blue triangle) at the final convolutional layer in a 3-node 3-layer network with circulant transformations. All data is remapped to the input space. Left: Red patches are mapped to 0 dimensional red points at the three output coordinate axis Blue patches are mapped to 1d lines connecting the points. (dark red is outside light red is inside of the manifold Right: Patches that are generated by selective components of the dual basis at each layer. The positive span generated by selective components of the dual basis emanating from the red output points on the triangle as well as from each intersection with coordinate lines in early layers, intersects with the arrangement of hyperplanes representing the preceding layer. The 1-d intersections are then used as seed points for new spans that intersect next preceding layer etc. The 2d intersections together with selective edges from the spans generate linking patches that ensures the continuity of the input manifold. as It should be pointed out that these manifold do not necessarily correspond to actual class manifolds since we are not considering the complete network with fully connected layers. They can however be considered as more elaborate and specific building blocks in order to construct the actual class manifolds of a trained network. We have defined a formal procedure for computing preimages of deep linear transformation networks with ReLU non linearities using the dual basis extracted from the set of hyperplanes representing the transformation. Specialising to convolutional networks we demonstrate that the complexity and the symmetry of the arrangement of corresponding hyperplanes is substantially reduced and we show that these arrangements can be modelled closely with multidimensisional regular polyhedral cones around the identity line in input space. We point out the crucial property of nested cones which guarantees efficient contraction of data to lower dimensions and argue that this property could be relevant in the design of real networks. By increasing the number of layers to shape input manifolds in the form of preimages we can retain the nested cone property that most efficiently exploits network data in order to construct input manifolds that comply with manifolds corresponding to real classes and would explain the success of ever deeper networks for deep learning. The retaining of the nested cone property can be expressed as a limitation of the degrees of freedom of multidimensional rotation of the cones. Since convolutional networks essentially always have limited spatial support convolutions, this is to a high degree built in to existing systems. The desire to retain the property of nesting could however act as an extra constraint to further reduce the complexity of the convolutions. This of course means that the degrees of freedom are reduced for a network which could act as a regularization constraint and potentially explain the puzzling efficiency of generalisation of deep networks in spite of a high number of parameters. We demonstrate that it is in principle possible to compute non linear input manifolds that map to affine output manifolds. This demonstrates the possibility of deep convolutional networks to achieve flattening of input data which is generally considered as an important preprocessing step for classification. Since we do not consider a complete network with fully connected layers at the end we cannot give details how classification is achieved. The explicit demonstration of non linear manifolds that map to affine outputs however indicates a possible basic structure of input manifolds for classes. It is easy to see that a parallel translation of the affine output manifold would in two linearly separable manifolds that would be generated by essentially parallel translated non linear manifolds in the input space. This demonstrates that convolutional networks can be designed to exactly separate sufficiently "covariant " classes. and that this could be the reason for the relative success of convolutional networks over previous machine learning approaches to classification and explain why using a large number of classes for training is advantageous since they all contribute to very similar individual manifolds. Disregarding these speculations the fact remains that these manifolds will always exist since they are derived on purely formal grounds from the structure of the network. If they have no role in classification their presence will have to be explained in other ways.' and should be close to 'Analysis of deep convolutional networks in terms of associated arrangement of hyperplanes' but very far from 'We show that neural networks operate by changing topologly of a data set and explore how architectural choices effect this change.'

Question: Deep reinforcement learning has demonstrated increasing capabilities for continuous control problems, including agents that can move with skill and agility through their environment. An open problem in this setting is that of developing good strategies for integrating or merging policies for multiple skills, where each individual skill is a specialist in a specific skill and its associated state distribution. We extend policy distillation methods to the continuous action setting and leverage this technique to combine \expert policies, as evaluated in the domain of simulated bipedal locomotion across different classes of terrain. We also introduce an input injection method for augmenting an existing policy network to exploit new input features. Lastly, our method uses transfer learning to assist in the efficient acquisition of new skills. The combination of these methods allows a policy to be incrementally augmented with new skills. We compare our progressive learning and integration via distillation (PLAID) method against three alternative baselines. As they gain experience, humans develop rich repertoires of motion skills that are useful in different contexts and environments. Recent advances in reinforcement learning provide an opportunity to understand how motion repertoires can best be learned, recalled, and augmented. Inspired by studies on the development and recall of movement patterns useful for different locomotion contexts BID17, we develop and evaluate an approach for learning multi-skilled movement repertoires. In what follows, we refer to the proposed method as PLAID: Progressive Learning and Integration via Distillation. For long lived applications of complex control tasks a learning system may need to acquire and integrate additional skills. Accordingly, our problem is defined by the sequential acquisition and integration of new skills. Given an existing controller that is capable of one-or-more skills, we wish to: (a) efficiently learn a new skill or movement pattern in a way that is informed by the existing control policy, and (b) to reintegrate that into a single controller that is capable of the full motion repertoire. This process can then be repeated as necessary. We view PLAID as a continual learning method, in that we consider a context where all tasks are not known in advance and we wish to learn any new task in an efficient manner. However, it is also proves surprisingly effective as a multitask solution, given the three specific benchmarks that we compare against. In the process of acquiring a new skill, we also allow for a control policy to be augmented with additional inputs, without adversely impacting its performance. This is a process we refer to as input injection. Understanding the time course of sensorimotor learning in human motor control is an open research problem BID31 ) that exists concurrently with recent advances in deep reinforcement learning. Issues of generalization, context-dependent recall, transfer or "savings" in fast learning, forgetting, and scalability are all in play for both human motor control models and the learning curricula proposed in reinforcement learning. While the development of hierarchical models for skills offers one particular solution that supports scalability and that avoids problems related to forgetting, we eschew this approach in this work and instead investigate a progressive approach to integration into a control policy defined by a single deep network. Distillation refers to the problem of combining the policies of one or more experts in order to create one single controller that can perform the tasks of a set of experts. It can be cast as a supervised regression problem where the objective is to learn a model that matches the output distributions of all expert policies BID13 BID28 BID19. However, given a new task for which an expert is not given, it is less clear how to learn the new task while successfully integrating this new skill in the pre-existing repertoire of the control policy for an agent. One wellknown technique in machine learning to significantly improve sample efficiency across similar tasks is to use Transfer Learning (TL) BID12, which seeks to reuse knowledge learned from solving a previous task to efficiently learn a new task. However, transferring knowledge from previous tasks to new tasks may not be straightforward; there can be negative transfer wherein a previously-trained model can take longer to learn a new task via fine-tuning than would a randomlyinitialized model BID16. Additionally, while learning a new skill, the control policy should not forget how to perform old skills. The core contribution of this paper is a method Progressive Learning and Integration via Distillation (PLAiD) to repeatedly expand and integrate a motion control repertoire. The main building blocks consist of policy transfer and multi-task policy distillation, and the method is evaluated in the context of a continuous motor control problem, that of robust locomotion over distinct classes of terrain. We evaluate the method against three alternative baselines. We also introduce input injection, a convenient mechanism for adding inputs to control policies in support of new skills, while preserving existing capabilities. Transfer learning and distillation are of broad interest in machine learning and RL BID12 BID27 BID28. Here we outline some of the most relevant work in the area of Deep Reinforcement Learning (DRL) for continuous control environments. Distillation Recent works have explored the problem of combining multiple expert policies in the reinforcement learning setting. A popular approach uses supervised learning to combine each policy by regression over the action distribution. This approach yields model compression BID19 as well as a viable method for multi-task policy transfer BID13 on discrete action domains including the Arcade Learning Environment BID0. We adopt these techniques and extend them for the case of complex continuous action space tasks and make use of them as building block. Transfer Learning Transfer learning exploits the structure learned from a previous task in learning a new task. Our focus here is on transfer learning in environments consisting of continuous control tasks. The concept of appending additional network structure while keeping the previous structure to reduce catastrophic forgetting has worked well on Atari games BID19 BID13 BID20 BID2 Other methods reproduce data from all tasks to reduce the possibility of forgetting how to perform previously learned skills e.g, BID23 BID9. Recent work seeks to mitigate this issue using selective learning rates for specific network parameters. A different approach to combining policies is to use a hierarchical structure BID29. In this setting, previously-learned policies are available as options to execute for a policy trained on a new task. However, this approach assumes that the new tasks will be at least a partial composition of previous tasks, and there is no reintegration of newly learned tasks. A recent promising approach has been to apply meta-learning to achieve control policies that can quickly adapt their behaviour according to current rewards BID4. This work is demonstrated on parameterized task domains. The Powerplay method provides a general framework for training an increasingly general problem solver BID21 BID25. It is based on iteratively: inventing a new task using play or invention; solving this task; and, lastly, demonstrating the ability to solve all the previous tasks. The last two stages are broadly similar to our PLAID approach, although to the best of our knowledge, there are no experiments on motor control tasks of comparable complexity to the ones we tackle. In our work, we develop a specific progressive learning-and-distillation methodology for motor skills, and provide a detailed evaluation as compared to three other plausible baselines. We are specifically interested in understanding issues that arise from the interplay between transfer from related tasks and the forgetting that may occur. Hierarchical RL further uses modularity to achieve transfer learning for robotic tasks BID29 This allows for the substitution of network modules for different robot types over a similar tasks BID3. Other methods use Hierarchical Reinforcement Learning (HRL) as a method for simplifying a complex motor control problem, defining a decomposition of the overall task into smaller tasks BID7 BID5 BID15 While these methods examine knowledge transfer, they do not examine the reintegration of policies for related tasks and the associated problems such as catastrophic forgetting. Recent work examines learned motions that can be shaped by prior mocap clips BID11, and that these can then be integrated in a hierarchical controller. In this section we outline the details of the Reinforcement Learning (RL) framework. We also give an introduction to the concepts of TL and distillation. Leveraging the framework of reinforcement learning, we frame the problem as a Markov Decision Processes (MDP): at each time step t, the world (including the agent) is in a state s t ∈ S, wherein the agent is able to perform actions a t ∈ A, sampled from a policy π(s t, a t) = p(a t |s t) and ing in state s t+1 ∈ S according to transition probabilities T (s t, a t, s t+1). Performing action a t from state s t produces a reward r t; the expected cumulative reward earned from following some policy π may then be written as: DISPLAYFORM0 where T is the time horizon, and γ is the discount factor, defining the planning horizon length. The agent's goal is to learn an optimal policy, π *, maximizing J(π). If the policy has parameters θ π, then the goal may be reformulated to identify the optimal parameters θ * π: DISPLAYFORM1 Our policy models a Gaussian distribution with a mean state dependent mean, µ θt (s t). Thus, our stochastic policy may be formulated as follows: DISPLAYFORM2 where Σ is a diagonal covariance matrix with entries σ 2 i on the diagonal, similar to BID15.To optimize our policy, we use stochastic policy gradient methods, which are well-established family of techniques for reinforcement learning BID26. The gradient of the expected reward with respect to the policy parameters, ∇ θπ J(π(·|θ π)), is given by: DISPLAYFORM3 where DISPLAYFORM4 represents the initial state distribution, and p 0 (s 0)(s 0 → s | t, π 0) models the likelihood of reaching state s by starting at state s 0 and following the policy π(a, s|θ π) for T steps BID24. A π (s, a) represents an advantage function BID22. In this work, we use the Positive Temporal Difference (PTD) update proposed by BID30 for A π (s, a): DISPLAYFORM5 DISPLAYFORM6 where V π (s) = E T t=0 γ t r t | s 0 = s is the value function, which gives the expected discounted cumulative reward from following policy π starting in state s. PTD has the benefit of being insensitive to the advantage function scale. Furthermore, limiting policy updates in this way to be only in the direction of actions that have a positive advantage has been found to increase the stability of learning BID30. Because the true value function is unknown, an approximation V π (· | θ v) with parameters θ v is learned, which is formulated as the regression problem: DISPLAYFORM7 3.2 POLICY DISTILLATION Given a set of expert agents that have solved/mastered different tasks we may want to combine the skills of these different experts into a single multi-skilled agent. This process is referred to as distillation. Distillation does not necessarily produce an optimal mix of the given experts but instead tries to produce an expert that best matches the action distributions produced by all experts. This method functions independent of the reward functions used to train each expert. Distillation also scales well with respect to the number of tasks or experts that are being combined. Given an expert that has solved/mastered a task we want to reuse that expert knowledge in order to learn a new task efficiently. This problem falls in the area of Transfer Learning BID12. Considering the state distribution expert is skilled at solving, (D ωi the source distribution) it can be advantageous to start learning a new, target task ω i+1 with target distribution D ωi+1 using assistance from the expert. The agent learning how to solve the target task with domain D ωi+1 is referred to as the student. When the expert is used to assist the student in learning the target task it can be referred to as the teacher. The success of these methods are dependent on overlap between the D ωi and D ωi+1 state distributions. Although we focus on the problem of being presented with tasks sequentially, there exist other methods for learning a multi-skilled character. We considered 4 overall integration methods for learning multiple skills, the first being a controller that learns multiple tasks at the same time (MultiTasker), where a number of skills are learned at the same time. It has been shown that learning many tasks together can be faster than learning each task separately BID13. The curriculum for using this method is shown in Figure 1a were during a single RL simulation all tasks are learned together. It is also possible to randomly initialize controllers and train in parallel (Parallel) and then combine the ing policies Figure 1b. We found that learning many skills from scratch was challenging, we were only able to get fair for the flat task. Also, when a new task is to be learned with the Parallel model it would occur outside of the original parallel learning, leading to a more sequential method. A TL-Only method that uses TL while learning tasks in a sequence Figure 1c, possibly ending with a distillation step to combine the learned policies to decrease forgetting. For more details see Appendix: 8.4. The last version (PLAiD) learns each task sequentially using TL from the previous, most skilled policy, in the end ing in a policy capable of solving all tasks Figure 1d. This method works well for both combining learned skills and learning new skills. DISPLAYFORM0 Figure 1: Different curriculum learning process. The red box with a D in it denotes a distillation step that combines policies. Each gray box denotes one iteration of learning a new policy. The larger red boxes with an L terrain−type denotes a learning step where a new skill is learned. In this section, we detail our proposed learning framework for continual policy transfer and distillation (PLAiD). In the acquisition (TL) step, we are interested in learning a new task ω i+1. Here transfer can be beneficial if the task structure is somewhat similar to previous tasks ω i. We adopt the TL strategy of using an existing policy network and fine-tuning it to a new task. Since we are not concerned with retaining previous skills in this step, we can update this policy without concern for forgetting. As the agent learns it will develop more skills and the addition of every new skill can increase the probability of transferring knowledge to assist the learning of the next skill. In the integration (distillation) step, we are interested in combining all past skills (π 0, . . ., π i) with the newly acquired skill π i+1. Traditional approaches have used policy regression where data is generated by collecting trajectories of the expert policy on a task. Training the student on these trajectories does not always in robust behaviour. This poor behaviour is caused by the student experiencing a different distribution of trajectories than the expert during evaluation. To compensate for this distribution difference, portions of the trajectories should be generated by the student. This allows the expert to suggest behaviour that will pull the state distribution of the student closer to the expert's. This is a common problem in learning a model to reproduce a given distribution of trajectories BID18 BID1 BID10 BID8. We use a method similar to the DAGGER algorithm BID18 which is useful for distilling policies BID13. See Appendix: 8.2.1 for more details. As our RL algorithm is an actor-critic method, we also perform regression on the critic by fitting both in the same step. The presented in this work cover a range of tasks that share a similar action space and state space. Our focus is to demonstrate continual learning between related tasks. In addition, the conceptual framework allows for extensions that would permit differing state spaces, described later in Section: 5.2. In this experiment, our set of tasks consists of 5 different terrains that a 2D humanoid walker (pdbiped) learns to traverse. The humanoid walker is trained to navigate multiple types of terrain including flat in FIG5 ), incline FIG5 ), steps FIG5 ), slopes FIG5 ), gaps FIG5 and a combination of all terrains mixed FIG5 ) on which agents are trained. The goal in these tasks is to maintain a consistent forward velocity traversing various terrains, while also matching a motion capture clip of a natural human walking gait on flat ground, similar to BID14. The pd-biped receives as input both a character and (eventually) a terrain state representation, consisting of the terrains heights of 50 equally-spaced points in front of the character. The action space is 11-dimensional, corresponding to the joints. Reasonable torque limits are applied, which helps produce more natural motions and makes the control problem more difficult. A detailed description of the experimental setup is included in Section: 8.5. The tasks are presented to the agent sequentially and the goal is to progressively learn to traverse all terrain types. We evaluate our approach against three baselines. First, we compare the above learning curriculum from learning new tasks in PLAiD with learning new tasks in Parallel. This will demonstrate that knowledge from previous tasks can be effectively transferred after distillation steps. Second, we compare to the MultiTasker to demonstrate that iterated distillation is effective for the retention of learned skills. The MultiTasker is also used as a baseline for comparing learning speed. Last, a method that performs TL between tasks and concludes with a distillation step is evaluated to illustrate the of different TL and distillation schedules. The of the PLAiD controller are displayed in the accompanying Video 1 5.1 TRANSFER LEARNING First, the pd-biped is trained to produce a walking motion on flat ground (flat). In FIG0 PLAiD is compared to the three baselines for training on incline. The TL-Only method learns fast as it is given significant information about how to perform similar skills. The Parallel method is given no prior information leading to a less skilled policy. The first MultiTasker for the incline task is initialized from a terrain injected controller that was trained to walk on flat ground. Any subsequent MultiTasker is initialized from the final MultiTasker model of the preceding task. This controller has to learn multiple tasks together, which can complicate the learning process, as simulation for each task is split across the training and the overall RL task can be challenging. This is in contrast to using PLAiD, that is also initialized with the same policy trained on flat, that will integrate skills together after each new skill is learned. In FIG0 the MultiTasker is learning the new task (steps) with similar speed to PLAiD. However, after adding more tasks the MultiTasker is beginning to struggle in FIG0 and starts to forget in FIG0, with the number of tasks it must learn at the same time. While PLAiD learns the new tasks faster and is able to integrate the new skill required to solve the task robustly. TL-Only is also able to learn the new tasks very efficiently. An appealing property of using distillation in PLAiD is that the combined policy model need not resemble that of the individual expert controllers. For example, two different experts lacking state features and trained without a local map of the terrain can be combined into a single policy that has new state features for the terrain. These new terrain features can assist the agent in the task domain in which it operates. We introduce the idea of input injection for this purpose. We augment a policy with additional input features while allowing it to retain its original functional behaviour similar to BID2. This is achieved by adding additional inputs to the neural network and initializing the connecting layer weights and biases to 0. By only setting the weights and biases in the layer connecting the new features to the original network to 0, the gradient can still propagate to any lower layers which are initialized random without changing the functional behaviour. This is performed when distilling the flat and incline experts. Further details can be found in Appendix: 8.3. Training over multiple tasks at the same time may help the agent learn skills quicker, but this may not scale with respect to the number of tasks. When training the MultiTasker over two or even three tasks (FIG1) the method displays good , however when learning a fourth or more tasks the method struggles, as shown in FIG1 and 3b. Part of the reason for this struggle is when new Table 1: These values are relative percentage changes in the average reward, where a value of 0 is no forgetting and a value of −1 corresponds to completely forgetting how to perform the task. A value > 0 corresponds to the agent learning how to better perform a task after training on other tasks. Here, the final policy after training on gaps compared to the original polices produced at the end of training for the task noted in the column heading. The TL-Only baseline forgets more than PLAiD. The MultiTasker forgets less than PLAiD but has a lower average reward over the tasks.tasks are added the MultiTasker has to make trade-offs between more tasks to maximizes. As more tasks are added, this trade-off becomes increasingly complex ing in the MultiTasker favouring easier tasks. Using PLAiD to combine the skills of many policies appears to scale better with respect to the number of skills being integrated. This is likely because distillation is a semi-supervised method which is more stable than the un-supervised RL solution. This can be seen in FIG1, 3e and especially in 3f where PLAiD combines the skills faster and can find higher value policies in practice. PLAiD also presents zero-shot training on tasks which it has never been trained on. In Figure 7 this generalization is shown as the agent navigates across the mixed environment. This is also reflected in Table 1, that shows the final average reward when comparing methods before and after distillation. The TL-Only is able to achieve high performance but much is lost when learning new tasks. A final distillation step helps mitigate this issue but does not work as well as PLAiD. It is possible performing a large final distillation step can lead to over-fitting. There are some indications that distillation is hindering training during the initial few iterations. We are initializing the network used in distillation with the most recently learning policy after TL. The large change in the initial state distribution from the previous seen distribution during TL could be causing larger gradients to appear, disrupting some of the structure learned during the TL step, shown in FIG1 and 3e. There also might not exist a smooth transition in policy space between the newly learned policy and the previous policy distribution. MultiTasker vs PLAiD: The MultiTasker may be able to produce a policy that has higher overall average reward, but in practise constraints can keep the method from combining skills gracefully. If the reward functions are different between tasks, the MultiTasker can favour a task with higher rewards, as these tasks may receive higher advantage. It is also a non-trivial task to normalize the reward functions for each task in order to combine them. The MultiTasker may also favour tasks that are easier than other tasks in general. We have shown that the PLAiD scales better with respect to the number of tasks than the MultiTasker. We expect PLAiD would further outperform the MultiTasker if the tasks were more difficult and the reward functions dissimilar. In our evaluation we compare the number of iterations PLAiD uses to the number the MultiTasker uses on only the new task, which is not necessarily fair. The MultiTasker gains its benefits from training on the other tasks together. If the idea is to reduce the number of simulation samples that are needed to learn new tasks then the MultiTasker would fall far behind. Distillation is also very efficient with respect to the number of simulation steps needed. Data could be collected from the simulator in groups and learned from in many batches before more data is needed as is common for behavioural cloning. We expect another reason distillation benefits learning multiple tasks is that the integration process assists in pulling policies out of the local minima RL is prone to. Transfer Learning: Because we are using an actor-critic learning method, we also studied the possibility of using the value functions for TL. We did not discover any empirical evidence that this assisted the learning process. When transferring to a new task, the state distribution has changed and the reward function may be completely different. This makes it unlikely that the value function will be accurate on this new task. In addition, value functions are in general easier and faster to learn than policies, implying that value function reuse is less important to transfer. We also find that helpfulness of TL depends on not only the task difficulty but the reward function as well. Two tasks may overlap in state space but the area they overlap could be easily reachable. In this case TL may not give significant benefit because the overall RL problem is easy. The greatest benefit is gained from TL when the state space that overlaps for two tasks is difficult to reach and in that difficult to reach area is where the highest rewards are achieved. Once integrated, the skills for our locomotion tasks are self-selecting based on their context, i.e., the knowledge of the upcoming terrain. It may be that other augmentation and distillation strategies are better for situations where either the reward functions are different or a one-hot vector is used to select the currently active expert. In our transfer learning we could be over fitting the initial expert for the particular task it was learning. Making it more challenging for the policy to learn a new task, ing in negative transfer. After learning many new tasks the previous tasks may not receive a large enough potion of the distillation training process to preserve the experts skill well enough. How best to chose which data should be trained on next to best preserve the behaviour of experts is a general problem with multi-task learning. Distillation treats all tasks equally independent of their reward. This can in very low value tasks, receiving potentially more distribution than desired and high value tasks receiving not enough. We have not needed the use a one-hot vector to indicate what task the agent is performing. We want the agent to be able to recognize which task it is given but we do realize that some tasks could be too similar to differentiate, such as, walking vs jogging on flat ground. It would be interesting to develop a method to prioritize tasks during the distillation step. This could assist the agent with forgetting issues or help with relearning tasks. While we currently use the Mean Squared Error (MSE) to pull the distributions of student policies in line with expert polices for distillation, better distance metrics would likely be helpful. Previous methods have used KL Divergence in the discrete action space domain where the state-action value function encodes the policy, e.g., as with Deep Q-Network (DQN). In this work we do not focus on producing the best policy from a mixture of experts, but instead we match the distributions from a number of experts. The difference is subtle but in practice it can be more challengine to balance many experts with respect to their reward functions. It could also be beneficial to use a KL penalty while performing distillation, i.e., something similar to the work in BID28 in order to keep the policy from changing too rapidly during training. We have proposed and evaluated a method for the progressive learning and integration (via distillation) of motion skills. The method exploits transfer learning to speed learning of new skills, along with input injection where needed, as well as continuous-action distillation, using DAGGER-style learning. This compares favorably to baselines consisting of learning all skills together, or learning all the skills individually before integration. We believe that there remains much to learned about the best training and integration methods for movement skill repertoires, as is also reflected in the human motor learning literature. We augment the blind network design by adding features for terrain to create an agent with sight. This network with terrain features has a single convolution layer with 8 filters of width 3. This constitutional layer is followed by a dense layer of 32 units. The dense layer is then concatenated twice, once along each of the original two hidden layers in the blind version of the policy. The policy network models a Gaussian distribution by outputting a state dependant mean. We use a state independent standard deviation that normalized with respect to the action space and multiplied by 0.1. We also use a version of epsilon greedy exploration where with probability an exploration action is generated. For all of our experiments we linearly anneal from 0.2 to 0.1 in 100, 000 iterations and leave it from that point on. Each training simulation takes approximately 5 hours across 8 threads. For network training we use Stochastic Gradient Decent (SGD) with momentum. During the distillation step we use gradually anneal the probability of selecting an expert action from 1 to 0 over 10, 000 iterations. For the evaluation of each model on a particular task we use the average reward achieved by the agent over at most 100 seconds of simulation time. We average this over running the agent over a number of randomly generated simulation runs. For each of the distillation steps we initialize the policy from the most recently trained policy. This policy has seen all of the tasks thus far but may have overfit the most recent tasks. We us a version of the DAGGER algorithm for the distillation process BID18. We anneal from selecting actions from the expert polices to selecting actions from the student policy The probability of selecting an action from the expert is annealed to near zero after 10, 000 training updates. We still add exploration noise to the policies when generating actions to take in the simulation. This is also annealed along with the probability of selecting from the expert policy. The actions used for training always come from the expert policy. Although some actions are applied in the simulation from the student, during a training update those actions will be replaced with ones from the proper expert. The expert used to generate actions for tasks 0 − i is π i and the expert used to generate action for task i + 1 is π i+1. We keep around at most 2 policies at any time. In order to add additional input features to the policy network we construct a new network. This new network has a portion of it that is the same design as the previous network plus additional parameters. First we initialize the new network with random parameters. Then we copy over the values from the previous network into the new one for the portion of the network design that matches the old. Then the weight for the layers that connect the old portion of the network to the new are set to 0. This will allow the network to preserve the previous distribution it modeled. Having the parameters from the old network will also help generate gradients to train the new 0 valued network parameters. We use feature injection to assist the learning method with differentiating between different states. For example, it could be challenging to discover the difference between the flat and incline tasks using only the character features. Therefore, we add new terrain features to allow the controller to better differentiate between these two different tasks. We also evaluate a baseline where we TL for all tasks. In this baseline TL is performed for a number of tasks and then distillation is used to combined these many learned skills. This method can be considered a version of PLAiD where tasks are learned in groups and after some number of tasks, a collection of policies/skills are distilled together. In FIG3 the learning curves for the TL-Only baseline are given. The TL-Only method learns new tasks well. We do not show the incline tasks as the two methods are the same up to starting the steps tasks. In Table 1 the amount of forgetting is compared between methods. To compare the amount of forgetting between TL-Only and PLAiD we show the relative loss in average reward between the original policy trained for the tasks steps and slopes and the final polices for each method on gaps. The TL-Only method shows a larger drop in policy performance corresponding to a large amount of forgeting compared to PLAiD, in particular for the more complex tasks steps and slopes. Interestingly, the final distllation step for TL-Only appears to reduce the performance of the policy. We believe this is related to the final distillation step being more challenging than performing a simpler distillation after each new task. Note that we only compare these two steps because the process for the first two tasks for PLAiD and TL-Only are the same. A comparison of the average rewards for the final policies are given in TAB1. The agent used in the simulation models the dimensions and masses of the average adult. The size of the character state is 50 parameters that include the relative position and velocity of the links in the agent FIG2 ). The action space consists of 11 parameters that indicate target joint positions for the agent. The target joint positions (pd-targets) are turned into joint torques via proportional derivative controllers at each joint. The reward function for the agent consists of 3 primary terms. The first is a velocity term the rewards the agent for going at velocity of 1 m/s The second term is the difference between the pose of the agent and the current pose of a kinematic character controlled via a motion capture clip. The difference between the agent and the clip consists of the rotational difference between each corresponding joint and the difference in angular velocity. The angular velocity for the clip is approximated via finite differences between the current pose of the clip and it's last pose. The last term is an L2 penalty on the torques generated by the agent to help reduce spastic motions. We also impose torque limits on the joints to reduce unrealistic behaviour, limits: Hips 150, knees 125, ankles 100, shoulders 100, elbows 75 and neck 50 N/m. Terrain Types All terrain types are randomly generated per episode, except for the flat terrain. The incline terrain is slanted and the slant of the terrain is randomly sampled between 20 and 25 degrees. The steps terrain consists of flat segments with widths randomly sampled from 1.0 m to 1.5 m followed by sharp steps that have randomly generated heights between 5 cm and 15 cm. The slopes terrain is randomly generated by updating the slope of the previous point in the ground with a value sampled from −20 and 20 degrees to generate a new portion of the ground every 10 cm. The gaps terrain generate gaps of width 25 -30 cm separated by flat segments of widths sampled from 2.0 m to 2.5 m. The mixed terrain is a combination of the above terrains where a portion is randomly chosen from the above terrain types. Figure 7: Still frame shots of the pd-biped traversing the mixed environment. In certain cases the MultiTasker can learn new task faster than PLAiD. In Figure 8a we present the MultiTasker and compare it to PLAiD. In this case the MultiTasker splits its training time across multiple tasks, here we compare the two methods with respect to the time spent learning on the single new task. This is a good baseline to compare our method against but in some ways this is not fair. If the real measure of how efficient a learning method is the number of simulation samples that are needed to learn would fall far behind as the MultiTasker needs to train across all tasks to gain the benefits of improving a single task without forgetting the old tasks.(a) Figure 8: (a) Shows that the MultiTasker can learn faster on steps, flat and incline than PLAiD (expert) learning the single task steps with TL.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A continual learning method that uses distillation to combine expert policies and transfer learning to accelerate learning new skills.

Represent the short summary for finding the corresponding scientific article (long):

We propose a novel Intrinsically Motivated Goal Exploration architecture with unsupervised learning of goal space representations, and evaluate how various implementations enable the discovery of a diversity of policies.