Question: represent the article for finding a single-sentence tldr

Question: Many recently trained neural networks employ large numbers of parameters to achieve good performance. One may intuitively use the number of parameters required as a rough gauge of the difficulty of a problem. But how accurate are such notions? How many parameters are really needed? In this paper we attempt to answer this question by training networks not in their native parameter space, but instead in a smaller, randomly oriented subspace. We slowly increase the dimension of this subspace, note at which dimension solutions first appear, and define this to be the intrinsic dimension of the objective landscape. The approach is simple to implement, computationally tractable, and produces several suggestive . Many problems have smaller intrinsic dimensions than one might suspect, and the intrinsic dimension for a given dataset varies little across a family of models with vastly different sizes. This latter has the profound implication that once a parameter space is large enough to solve a problem, extra parameters serve directly to increase the dimensionality of the solution manifold. Intrinsic dimension allows some quantitative comparison of problem difficulty across supervised, reinforcement, and other types of learning where we conclude, for example, that solving the inverted pendulum problem is 100 times easier than classifying digits from MNIST, and playing Atari Pong from pixels is about as hard as classifying CIFAR-10. In addition to providing new cartography of the objective landscapes wandered by parameterized models, the method is a simple technique for constructively obtaining an upper bound on the minimum description length of a solution. A byproduct of this construction is a simple approach for compressing networks, in some cases by more than 100 times. Training a neural network to model a given dataset entails several steps. First, the network designer chooses a loss function and a network architecture for a given dataset. The architecture is then initialized by populating its weights with random values drawn from some distribution. Finally, the network is trained by adjusting its weights to produce a loss as low as possible. We can think of the training procedure as traversing some path along an objective landscape. Note that as soon as a dataset and network architecture are specified, the landscape in its entirety is completely determined. It is instantiated and frozen; all subsequent parameter initialization, forward and backward propagation, and gradient steps taken by an optimizer are just details of how the frozen space is explored. Consider a network parameterized by D weights. We can picture its associated objective landscape as a set of "hills and valleys" in D dimensions, where each point in R D corresponds to a value of the loss, i.e., the elevation of the landscape. If D = 2, the map from two coordinates to one scalar loss can be easily imagined and intuitively understood by those living in a three-dimensional world with similar hills. However, in higher dimensions, our intuitions may not be so faithful, and generally we must be careful, as extrapolating low-dimensional intuitions to higher dimensions can lead to unreliable . The difficulty of understanding high-dimensional landscapes notwithstanding, it is the lot of neural network researchers to spend their efforts leading (or following?) networks over these multi-dimensional surfaces. Therefore, any interpreted geography of these landscapes is valuable. Several papers have shed valuable light on this landscape, particularly by pointing out flaws in common extrapolation from low-dimensional reasoning. BID4 showed that, in contrast to conventional thinking about getting stuck in local optima (as one might be stuck in a valley in our familiar D = 2), local critical points in high dimension are almost never valleys but are instead saddlepoints: structures which are "valleys" along a multitude of dimensions with "exits" in a multitude of other dimensions. The striking is that one has less to fear becoming hemmed in on all sides by higher loss but more to fear being waylaid nearly indefinitely by nearly flat regions. BID9 showed another property: that paths directly from the initial point to the final point of optimization are often monotonically decreasing. Though dimension is high, the space is in some sense simpler than we thought: rather than winding around hills and through long twisting corridors, the walk could just as well have taken a straight line without encountering any obstacles, if only the direction of the line could have been determined at the outset. In this paper we seek further understanding of the structure of the objective landscape by restricting training to random slices through it, allowing optimization to proceed in randomly generated subspaces of the full parameter space. Whereas standard neural network training involves computing a gradient and taking a step in the full parameter space (R D above), we instead choose a random d-dimensional subspace of R D, where generally d < D, and optimize directly in this subspace. By performing experiments with gradually larger values of d, we can find the subspace dimension at which solutions first appear, which we call the measured intrinsic dimension of a particular problem. Examining intrinsic dimensions across a variety of problems leads to a few new intuitions about the optimization problems that arise from neural network models. We begin in Sec. 2 by defining more precisely the notion of intrinsic dimension as a measure of the difficulty of objective landscapes. In Sec. 3 we measure intrinsic dimension over a variety of network types and datasets, including MNIST, CIFAR-10, ImageNet, and several RL tasks. Based on these measurements, we draw a few insights on network behavior, and we conclude in Sec. 4. We introduce the intrinsic dimension of an objective landscape with an illustrative toy problem. Let θ (D) ∈ R D be a parameter vector in a parameter space of dimension D, let θ (D) 0 be a randomly chosen initial parameter vector, and let θ (D) * be the final parameter vector arrived at via optimization. Consider a toy optimization problem where D = 1000 and where θ (D) optimized to minimize a squared error cost function that requires the first 100 elements to sum to 1, the second 100 elements to sum to 2, and so on until the vector has been divided into 10 groups with their requisite 10 sums. We may start from a θ Solutions to this problem are highly redundant. With a little algebra, one can find that the manifold of solutions is a 990 dimensional hyperplane: from any point that has zero cost, there are 990 orthogonal directions one can move and remain at zero cost. Denoting as s the dimensionality of the solution set, we define the intrinsic dimensionality d int of a solution as the codimension of the solution set inside of R D: DISPLAYFORM0 Here the intrinsic dimension d int is 10 (1000 = 10 + 990), with 10 corresponding intuitively to the number of constraints placed on the parameter vector. Though the space is large (D = 1000), the number of things one needs to get right is small (d int = 10). The above example had a simple enough form that we obtained d int = 10 by calculation. But in general we desire a method to measure or approximate d int for more complicated problems, including problems with data-dependent objective functions, e.g. neural network training. Random subspace optimization provides such a method. Standard optimization, which we will refer to hereafter as the direct method of training, entails evaluating the gradient of a loss with respect to θ (D) and taking steps directly in the space of θ (D). To train in a random subspace, we instead define θ (D) in the following way: DISPLAYFORM0 where P is a randomly generated D × d projection matrix 1 and θ (d) is a parameter vector in a gen- DISPLAYFORM1 and P are randomly generated and frozen (not trained), so the system has only d degrees of freedom. We initialize θ (d) to a vector of all zeros, so initially DISPLAYFORM2 0. This convention serves an important purpose for neural network training: it allows the network to benefit from beginning in a region of parameter space designed by any number of good initialization schemes BID8 BID11 to be well-conditioned, such that gradient descent via commonly used optimizers will tend to work well. Training proceeds by computing gradients with respect to θ (d) and taking steps in that space. Columns of P are normalized to unit length, so steps of unit length in θ (d) chart out unit length motions of θ (D). Columns of P may also be orthogonalized if desired, but in our experiments we relied simply on the approximate orthogonality of high dimensional random vectors. By this construction P forms an approximately orthonormal basis for a randomly oriented d dimensional subspace of R D, with the origin of the new coordinate system at θ0. FIG1 (left and middle) shows an illustration of the related vectors. Consider a few properties of this training approach. If d = D and P is a large identity matrix, we recover exactly the direct optimization problem. If d = D but P is instead a random orthonormal basis for all of R D (just a random rotation matrix), we recover a rotated version of the direct problem. Note that for some "rotation-invariant" optimizers, such as SGD and SGD with momentum, rotating the basis will not change the steps taken nor the solution found, but for optimizers with axis-aligned assumptions, such as RMSProp and Adam BID16, the path taken through θ (D) space by an optimizer will depend on the rotation chosen. Finally, in the general case where d < D and solutions exist in D, solutions will almost surely (with probability 1) not be found if d is less than the codimension of the solution. On the other hand, when d ≥ D −s, if the solution set is a hyperplane, the solution will almost surely intersect the subspace, but for solution sets of arbitrary topology, intersection is not guaranteed. Nonetheless, by iteratively increasing d, re-running optimization, and checking for solutions, we obtain one estimate of d int. We try this sweep of d for our toy problem laid out in the beginning of this section, measuring (by convention as described in the next section) the positive performance (higher is better) instead of loss.3 As expected, the solutions are first found at d = 10 (see FIG1, right), confirming our intuition that for this problem, d int = 10. In the rest of this paper, we measure intrinsic dimensions for particular neural network problems and draw about the associated objective landscapes and solution sets. Because modeling real data is more complex than the above toy example, and losses are generally never exactly zero, we first choose a heuristic for classifying points on the objective landscape as solutions vs. non-solutions. The heuristic we choose is to threshold network performance at some level relative to a baseline model, where generally we take as baseline the best directly trained model. In supervised classification settings, validation accuracy is used as the measure of performance, and in reinforcement learning scenarios, the total reward (shifted up or down such that the minimum reward is 0) is used. Accuracy and reward are preferred to loss to ensure are grounded to real-world performance and to allow comparison across models with differing scales of loss and different amounts of regularization included in the loss. We define d int100 as the intrinsic dimension of the "100%" solution: solutions whose performance is statistically indistinguishable from baseline solutions. However, when attempting to measure d int100, we observed it to vary widely, for a few confounding reasons: d int100 can be very high -nearly as high as D -when the task requires matching a very well-tuned baseline model, but can drop significantly when the regularization effect of restricting parameters to a subspace boosts performance by tiny amounts. While these are interesting effects, we primarily set out to measure the basic difficulty of problems and the degrees of freedom needed to solve (or approximately solve) them rather than these subtler effects. Thus, we found it more practical and useful to define and measure d int90 as the intrinsic dimension of the "90%" solution: solutions with performance at least 90% of the baseline. We chose 90% after looking at a number of dimension vs. performance plots (e.g. FIG2) as a reasonable trade off between wanting to guarantee solutions are as good as possible, but also wanting measured d int values to be robust to small noise in measured performance. If too high a threshold is used, then the dimension at which performance crosses the threshold changes a lot for only tiny changes in accuracy, and we always observe tiny changes in accuracy due to training noise. If a somewhat different (higher or lower) threshold were chosen, we expect most of in the rest of the paper to remain qualitatively unchanged. In the future, researchers may find it useful to measure d int using higher or lower thresholds. We begin by analyzing a fully connected (FC) classifier trained on MNIST. We choose a network with layer sizes 784-200-200-10, i.e. a network with two hidden layers of width 200; this in a total number of parameters D = 199, 210. A series of experiments with gradually increasing subspace dimension d produce monotonically increasing performances, as shown in FIG2. By checking the subspace dimension at which performance crosses the 90% mark, we measure this network's intrinsic dimension d int90 at about 750.Some networks are very compressible. A salient initial is that 750 is quite low. At that subspace dimension, only 750 degrees of freedom (0.4%) are being used and 198,460 (99.6%) unused to obtain 90% of the performance of the direct baseline model. A compelling corollary of this is a simple, new way of creating and training compressed networks, particularly networks for applications in which the absolute best performance is not critical. To store this network, one need only store a tuple of three items: (i) the random seed to generate the frozen θ This compression approach differs from other neural network compression methods in the following aspects. (i) While it has previously been appreciated that large networks waste parameters BID3 and weights contain redundancy BID5 ) that can be exploited for posthoc compression , this paper's method constitutes a much simpler approach to compression, where training happens once, end-to-end, and where any parameterized model is an allowable base model. (ii) Unlike layerwise compression models BID5 ), we operate in the entire parameter space, which could work better or worse, depending on the network. (iii) Compared to methods like that of , who take a Bayesian perspective and consider redundancy on the level of groups of parameters (input weights to a single neuron) by using group-sparsity-inducing hierarchical priors on the weights, our approach is simpler but not likely to lead to compression as high as the levels they attain. (iv) Our approach only reduces the number of degrees of freedom, not the number of bits required to store each degree of freedom, e.g. as could be accomplished by quantizing weights. Both approaches could be combined. (v) There is a beautiful array of papers on compressing networks such that they also achieve computational savings during the forward pass ; subspace training does not speed up execution time during inference. (vi) Finally, note the relationships between weight pruning, weight tying, and subspace training: weight pruning is equivalent to finding, post-hoc, a subspace that is orthogonal to certain axes of the full parameter space and that intersects those axes at the origin. Weight tying, e.g. by random hashing of weights into buckets BID2, is equivalent to subspace training where the subspace is restricted to lie along the equidistant "diagonals" between any axes that are tied together. Robustness of intrinsic dimension. Next, we investigate how intrinsic dimension varies across FC networks with a varying number of layers and varying layer width. 4 We perform a grid sweep of networks with number of hidden layers L chosen from {1, 2, 3, 4, 5} and width W chosen from {50, 100, 200, 400}. FIG7 in the Supplementary Information shows performance vs. subspace dimension plots in the style of FIG2 for all 20 networks, and FIG3 shows each network's d int90 plotted against its native dimension D. As one can see, D changes by a factor of 24.1 between the smallest and largest networks, but d int90 changes over this range by a factor of only 1.33, with much of this possibly due to noise. Thus it turns out that the intrinsic dimension changes little even as models grown in width or depth! The striking is that every extra parameter added to the network -every extra dimension added to D -just ends up adding one dimension to the redundancy of the solution, s. Often the most accurate directly trained models for a problem have far more parameters than needed ; this may be because they are just easier to train, and our observation suggests a reason why: with larger models, solutions have greater redundancy and in a sense "cover" more of the space.5 To our knowledge, this is the first time this phenomenon has been directly measured. We should also be careful not to claim that all FC nets on MNIST will have an intrinsic dimension of around 750; instead, we should just consider that we have found for this architecture/dataset combination a wide plateau of hyperparamter space over which intrinsic dimension is approximately constant. Are random subspaces really more parameter-efficient for FC nets? One might wonder to what extent claiming 750 parameters is meaningful given that performance achieved (90%) is far worse than a state of the art network trained on MNIST. With such a low bar for performance, could a directly trained network with a comparable number of trainable parameters be found that achieves the same performance? We generated 1000 small networks (depth randomly chosen from {1, 2, 3, 4, 5}, layer width randomly from {2, 3, 5, 8, 10, 15, 20, 25}, seed set randomly) in an attempt to find high-performing, small FC networks, but as FIG4 (left) shows, a gap still exists between the subspace dimension and the smallest direct FC network giving the same performance at most levels of performance. Measuring d int90 on a convolutional network. Next we measure d int90 of a convolutional network, LeNet (D=44,426). FIG2 (right) shows validation accuracy vs. subspace dimension d, and we find d int90 = 290, or a compression rate of about 150× for this network. As with the FC case above, we also do a sweep of random networks, but notice that the performance gap of convnets between direct and subspace training methods becomes closer for fixed budgets, i.e., the number of trainable parameters. Further, the performance of direct training varies significantly, depending on the extrinsic design of convet architectures. We interpret these in terms of the Minimum Description Length below. Relationship between Intrinsic Dimension and Minimum Description Length (MDL). As discussed earlier, the random subspace training method leads naturally to a compressed representation of a network, where only d floating point numbers need to be stored. We can consider this d as an upper bound on the MDL of the problem solution. 6 We cannot yet conclude the extent to which this bound is loose or tight, and tightness may vary by problem. However, to the extent that it is tighter than previous bounds (e.g., just the number of parameters D) and to the extent that it is correlated with the actual MDL, we can use this interpretation to judge which solutions are more well-suited to the problem in a principled way. As developed by and further by BID12, holding accuracy constant, the best model is the one with the shortest MDL.Thus, there is some rigor behind our intuitive assumption that LeNet is a better model than an FC network for MNIST image classification, because its intrinsic dimension is lower (d int90 of 290 vs. 750). In this particular case we are lead to a predictable , but as models become larger, more complex, and more heterogeneous, of this type will often not be obvious. Having a simple method of approximating MDL may prove extremely useful for guiding model exploration, for example, for the countless datasets less well-studied than MNIST and for models consisting of separate sub-models that may be individually designed and evaluated BID11 BID14. In this latter case, considering the MDL for a sub-model could provide a more detailed view of that sub-model's properties than would be available by just analyzing the system's overall validation performance. Finally, note that although our approach is related to a rich body of work on estimating the "intrinsic dimension of a dataset" BID1 BID15 BID6; ), it differs in a few respects. Here we do not measure the number of degrees of freedom necessary to represent a dataset (which requires representation of a global p(X) and per-example properties and thus grows with the size of the dataset), but those required to represent a model for part of the dataset (here p(y|X), which intuitively might saturate at some complexity even as a dataset grows very large). That said, in the following section we do show measurements for a corner case where the model must memorize per-example properties. Are convnets always better on MNIST? Measuring d int90 on shuffled data. Zhang et al. FORMULA0 provocatively showed that large networks normally thought to generalize well can nearly as easily be trained to memorize entire training sets with randomly assigned labels or with input pixels provided in random order. Consider two identically sized networks: one trained on a real, non-shuffled dataset and another trained with shuffled pixels or labels. As noted by , externally the networks are very similar, and the training loss may even be identical at the final epoch. However, the intrinsic dimension of each may be measured to expose the differences in problem difficulty. When training on a dataset with shuffled pixels -pixels for each example in the dataset subject to a random permutation, chosen once for the entire dataset -the intrinsic dimension of an FC network remains the same at 750, because FC networks are invariant to input permutation. But the intrinsic dimension of a convnet increases from 290 to 1400, even higher than an FC network. Thus while convnets are better suited to classifying digits given images with local structure, when this structure is removed, violating convolutional assumptions, our measure can clearly reveal that many more degrees of freedom are now required to model the underlying distribution. When training on MNIST with shuffled labels -the label for each example is randomly chosen -we redefine our measure of d int90 relative to training accuracy (validation accuracy is always at chance). We find that memorizing random labels on the 50,000 example MNIST training set requires a very high dimension, d int90 = 190, 000, or 3.8 floats per memorized label. Sec. S5.2 gives a few further , in particular that the more labels are memorized, the more efficient memorization is in terms of floats per label. Thus, while the network obviously does not generalize to an unseen validation set, it would seem "generalization" within a training set may be occurring as the network builds a shared infrastructure that makes it possible to more efficiently memorize labels. We scale to larger supervised classification problems by considering CIFAR-10 and ImageNet . When scaling beyond MNIST-sized networks with D on the order of 200k and d on the order of 1k, we find it necessary to use more efficient methods of generating and projecting from random subspaces. This is particularly true in the case of ImageNet, where the direct network can easily require millions of parameters. In Sec. S7, we describe and characterize scaling properties of three methods of projection: dense matrix projection, sparse matrix projection , and the remarkable Fastfood transform . We generally use the sparse projection method to train networks on CIFAR-10 and the Fastfood transform for ImageNet. Measuring intrinsic dimension allows us to perform some comparison across the divide between supervised learning and reinforcement learning. In this section we measure the intrinsic dimension of three control tasks of varying difficulties using both value-based and policy-based algorithms. The value-based algorithm we evaluate is the Deep Q-Network (DQN) , and the policy-based algorithm is Evolutionary Strategies (ES) . Training details are given in Sec. S6.2. For all tasks, performance is defined as the maximum-attained (over training iterations) mean evaluation reward (averaged over 30 evaluations for a given parameter setting). In horizontal line). A dot is darkened signifying the first d that allows a satisfactory performance. We find that the inverted pendulum task is surprisingly easy, with d int100 = d int90 = 4, meaning that only four parameters are needed to perfectly solve the problem (see for a similarly small solution found via evolution). The walking humanoid task is more difficult: solutions are found reliably by dimension 700, a similar complexity to that required to model MNIST with an FC network, and far less than modeling CIFAR-10 with a convnet. Finally, to play Pong on Atari (directly from pixels) requires a network trained in a 6k dimensional subspace, making it on the same order of modeling CIFAR-10. For an easy side-by-side comparison we list all intrinsic dimension values found for all problems in In this paper, we have defined the intrinsic dimension of objective landscapes and shown a simple method -random subspace training -of approximating it for neural network modeling problems. We use this approach to compare problem difficulty within and across domains. We find in some cases the intrinsic dimension is much lower than the direct parameter dimension, and hence enable network compression, and in other cases the intrinsic dimension is similar to that of the best tuned models, and suggesting those models are better suited to the problem. Further work could also identify better ways of creating subspaces for reparameterization: here we chose random linear subspaces, but one might carefully construct other linear or non-linear subspaces to be even more likely to contain solutions. Finally, as the field departs from single stackof-layers image classification models toward larger and more heterogeneous networks BID11 BID14 often composed of many modules and trained by many losses, methods like measuring intrinsic dimension that allow some automatic assessment of model components might provide much-needed greater understanding of individual black-box module properties. In the main paper, we attempted to find d int90 across 20 FC networks with various depths and widths. A grid sweep of number of hidden layers from {1,2,3,4,5} and width of each hidden layer from {50,100,200,400} is performed, and all 20 plots are shown in FIG7. For each d we take 3 runs and plot the mean and variance with blue dots and blue error bars. d int90 is indicated in plots (darkened blue dots) by the dimension at which the median of the 3 runs passes 90% performance threshold. The variance of d int90 is estimated using 50 bootstrap samples. Note that the variance of both accuracy and measured d int90 for a given hyper-parameter setting are generally small, and the mean of performance monotonically increases (very similar to the single-run ) as d increases. This illustrates that the difference between lucky vs. unlucky random projections have little impact on the quality of solutions, while the subspace dimensionality has a great impact. We hypothesize that the variance due to different P matrices will be smaller than the variance due to different random initial parameter vectors θ 0, and aspects of the network depending on smaller numbers of random samples will exhibit greater variance. Hence, in some other experiments we rely on single runs to estimate the intrinsic dimension, though slightly more accurate estimates could be obtained via multiple runs. In similar manner to the above, in FIG8 we show the relationship between d int90 and D across 20 networks but using a per-model, directly trained baseline. Most baselines are slightly below 100% accuracy. This is in contrast to FIG3, which used a simpler global baseline of 100% across all models. Results are qualitatively similar but with slightly lower intrinsic dimension due to slightly lower thresholds. Two kinds of shuffled MNIST datasets are considered:• The shuffled pixel dataset: the label for each example remains the same as the normal dataset, but a random permutation of pixels is chosen once and then applied to all images in the training and test sets. FC networks solve the shuffled pixel datasets exactly as easily as the base dataset, because there is no privileged ordering of input dimension in FC networks; all orderings are equivalent.• The shuffled label dataset: the images remain the same as the normal dataset, but labels are randomly shuffled for the entire training set. Here, as in , we only evaluate training accuracy, as test set accuracy remains forever at chance level (the training set X and y convey no information about test set p(y|X), because the shuffled relationship in test is independent of that of training).On the full shuffled label MNIST dataset (50k images), we trained an FC network (L = 5, W = 400, which had d int90 = 750 on standard MNIST), it yields d int90 = 190k. We can interpret this as requiring 3.8 floats to memorize each random label (at 90% accuracy). Wondering how this scales with dataset size, we estimated d int90 on shuffled label versions of MNIST at different scales and found curious , shown in TAB4 and Fig. S8. As the dataset memorized becomes smaller, the number of floats required to memorize each label becomes larger. Put another way, as dataset size Figure S8: Training accuracy vs. subspace dimension d for a FC networks (W =400, L=5) trained on a shuffled label version of MNIST containing 100%, 50%, and 10% of the dataset.increases, the intrinsic dimension also increases, but not as fast as linearly. The best interpretationis not yet clear, but one possible interpretation is that networks required to memorize large training sets make use of shared machinery for memorization. In other words, though performance does not generalize to a validation set, generalization within a training set is non-negligible even though labels are random. An interesting tangential observation is that random subspace training can in some cases make optimization more stable. First, it helps in the case of deeper networks. FIG9 shows training for FC networks with up to 10 layers. SGD with step 0.1, and ReLUs with He initialization is used. Multiple networks failed at depths 4, and all failed at depths higher than 4, despite the activation function and initialization designed to make learning stable BID11. Second, for MNIST with shuffled labels, we noticed that it is difficult to reach high training accuracy using the direct training method with SGD, though both subspace training with SGD and either type of training with Adam reliably reach 100% memorization as d increases (see Fig. S8).Because each random basis vector projects across all D direct parameters, the optimization problem may be far better conditioned in the subspace case than in the direct case. A related potential downside is that projecting across D parameters which may have widely varying scale could in ignoring parameter dimensions with tiny gradients. This situation is similar to that faced by methods like SGD, but ameliorated by RMSProp, Adam, and other methods that rescale per-dimension step sizes to account for individual parameter scales. Though convergence of the subspace approach seems robust, further work may be needed to improve network amenability to subspace training: for example by ensuring direct parameters are similarly scaled by clever initialization or by inserting a pre-scaling layer between the projected subspace and the direct parameters themselves. Another finding through our experiments with MNIST FC networks has to do with the role of optimizers. The same set of experiments are run with both SGD (learning rate 0.1) and ADAM (learning rate 0.001), allowing us to investigate the impact of stochastic optimizers on the intrinsic dimension achieved. The intrinsic dimension d int90 are reported in FIG1 (a)(b). In addition to two optimizers we also use two baselines: Global baseline that is set up as 90% of best performance achieved across all models, and individual baseline that is with regards to the performance of the same model in direct training. DQN on Cartpole We start with a simple classic control game CartPole−v0 in OpenAI Gym BID0. A pendulum starts upright, and the goal is to prevent it from falling over. The system is controlled by applying a force of LEFT or RIGHT to the cart. The full game ends when one of two failure conditions is satisfied: the cart moves more than 2.4 units from the center (where it started), or the pole is more than 15 degrees from vertical (where it started). A reward of +1 is provided for every time step as long as the game is going. We further created two easier environments Pole and Cart, each confined by one of the failure modes only. A DQN is used, where the value network is parameterized by an FC (L = 2, W = 400). For each subspace d at least 5 runs are conducted, the mean of which is used to computed d int90, and the baseline is set as 195.0 7. The are shown in FIG1. The solid line connects mean rewards within a run over the last 100 episodes, across different ds. Due to the noise-sensitiveness of RL games the course is not monotonic any more. The intrinsic dimension for CartPole, Pole and Cart is d int90 = 25, 23 and 7, respectively. This reveals that the difficulty of optimization landscape of these games is remarkably low, as well as interesting insights such as driving a cart is much easier than keeping a pole straight, the latter being the major cause of difficulty when trying to do both. We carry out with ES 3 RL tasks: InvertedPendulum−v1, Humanoid−v1, Pong−v0. The hyperparameter settings for training are in Table S3.Inverted pendulum The InvertedPendulum−v1 environment uses the MuJoCo physics simulator to instantiate the same problem as CartPole−v0 in a realistic setting. We expect that even with richer environment dynamics, as well as a different RL algorithm -ESthe intrinsic dimensionality should be similar. As seen in Fig. 5, the measured intrinsic dimensionality d int90 = 4 is of the same order of magnitude, but smaller. Interestingly, although the environment FIG3 and FIG8, because in the former we average over three runs, and in the latter we show one run each for all optimization methods. DISPLAYFORM0 Figure S11: Subspace training of DQN on CartPole game. Shown as dots are rewards collected through a game run averaged over the last 100 episodes, under each subspace training of DQN, and each game environment. The line connects mean rewards across different ds.dynamics are more complex than in CartPole−v0, using ES rather than DQN seems to induce a simpler objective landscape. Learning to walk A more challenging problem is Humanoid−v1 in MuJoCo simulator. Intuitively, one might believe that learning to walk is a more complex task than classifying images. Our show the contrary -that the learned intrinsic dimensionality of d int90 = 700 is similar to that of MNIST on a fully-connected network (d int90 = 650) but significantly less than that of even a convnet trained on CIFAR-10 (d int90 = 2, 500). Table S3: Hyperparameters used in training RL tasks using ES. σ refers to the parameter perturbation noise used in ES. Default Adam parameters of β 1 = 0.9, β 2 = 0.999, = 1 × 10 −7 were used.begin to see training runs reach the threshold as early as d = 400, with the median performance steadily increasing with d. Atari Pong Finally, using a base convnet of approximately D = 1M in the Pong−v0 pixels-toactions environment (using 4-frame stacking). The agent receives an image frame (size of 210 × 160 × 3) and the action is to move the paddle UP or DOWN. We were able to determine d int90 = 6, 000. Scaling the random subspace training procedure to large problems requires an efficient way to map from R d into a random d-dimensional subspace of R D that does not necessarily include the origin. Algebraically, we need to left-multiply a vector of parameters v ∈ R d by a random matrix M ∈ R D×d, whose columns are orthonormal, then add an offset vector θ 0 ∈ R D. If the low-dimensional parameter vector in R d is initialized to zero, then specifying an offset vector is equivalent to choosing an initialization point in the original model parameter space R D.A naïve approach to generating the random matrix M is to use a dense D × d matrix of independent standard normal entries, then scale each column to be of length 1. The columns will be approximately orthogonal if D is large because of the independence of the entries. Although this approach is sufficient for low-rank training of models with few parameters, we quickly run into scaling limits because both matrix-vector multiply time and storage of the matrix scale according to O(Dd). We were able to successfully determine the intrinsic dimensionality of MNIST (d=225) using a LeNet (D=44,426), but were unable to increase d beyond 1,000 when applying a LeNet (D=62,006) to CIFAR-10, which did not meet the performance criterion to be considered the problems intrinsic dimensionality. Random matrices need not be dense for their columns to be approximately orthonormal. In fact, a method exists for "very sparse" random projections , which achieves a density of DISPLAYFORM0 To construct the D × d matrix, each entry is chosen to be nonzero with probability DISPLAYFORM1. If chosen, then with equal probability, the entry is either positive or negative with the same magnitude in either case. The density of DISPLAYFORM2 time and space complexity. Implementing this procedure allowed us to find the intrinsic dimension of d=2,500 for CIFAR-10 using a LeNet mentioned above. Unfortunately, when using Tensorflow's SparseTensor implementation we did not achieve the theoretical √ D-factor improvement in time complexity (closer to a constant 10x). Nonzero elements also have a significant memory footprint of 24 bytes, so we could not scale to larger problems with millions of model parameters and large intrinsic dimensionalities. We need not explicitly form and store the transformation matrix. The Fastfood transform was initially developed as an efficient way to compute a nonlinear, high-dimensional feature map φ(x) for a vector x. A portion of the procedure involves implicitly generating a D × d matrix with approximately uncorrelated standard normal entries, using only O(D) space, which can be multiplied by v in O(D log d) time using a specialized method. The method relies on the fact that Hadamard matrices multiplied by Gaussian vectors behave like dense Gaussian matrices. In detail, to implicitly multiply v by a random square Gaussian matrix M with side-lengths equal to a power of two, the matrix is factorized into multiple simple matrices: M = HGΠHB, where B is a random diagonal matrix with entries +-1 with equal probability, H is a Hadamard matrix, Π is a random permutation matrix, and G is a random diagonal matrix with independent standard normal entries. Multiplication by a Hadamard matrix can be done via the Fast Walsh-Hadamard Transform In practice, the reduction in space footprint allowed us to scale to much larger problems, including the Pong RL task using a 1M parameter convolutional network for the policy function. FIG4 compares the computational time for direct and subspace training (various projections) methods for each update. Our subspace training is more computational expensive, because the subspace training method has to propagate the signals through two modules: the layers of neural networks, and the projection between two spaces. The direct training only propagates signals in the layers of neural networks. We have made efforts to reduce the extra computational cost. For example, the sparse projection less than doubles the time cost for a large range of subspace dimensions. We consider the CIFAR-10 dataset and test the same set of FC and LeNet architectures as on MNIST. For FC networks, d int90 values for all 20 networks are shown in FIG1 plotted against the native dimension D of each network; D changes by a factor of 12.16 between the smallest and largest networks, but d int90 changes over this range by a factor of 5.0. However, much of this change is due to change of baseline performance. In FIG1, we instead compute the intrinsic dimension with respect to a global baseline: 50% validation accuracy. d int90 changes over this range by a factor of 1.52. This indicates that various FC networks share similar intrinsic dimension (d int90 = 5000 ∼ 8000) to achieve the same level of task performance. For LeNet (D = 62, 006), the validation accuracy vs. subspace dimension d is shown in FIG1, the corresponding d int90 = 2900. It yields a compression rate of 5%, which is 10 times larger than LeNet on MNIST. It shows that CIFAR-10 images are significantly more difficult to be correctly classified than MNIST. In another word, CIFAR-10 is a harder problem than MNIST, especially given the fact that the notion of "problem solved" (baseline performance) is defined as 99% accuracy on MNIST and 58% accuracy on CIFAR-10. On the CIFAR-10 dataset, as d increases, subspace training tends to overfitting; we study the role of subspace training as a regularizer below. ResNet vs. LeNet We test ResNets, compare to LeNet, and find they make efficient use of parameters. We adopt the smallest 20-layer structure of ResNet with 280k parameters, and find out in FIG1 (b) that it reaches LeNet baseline with d int90 = 1000 ∼ 2000 (lower than the d int90 of LeNet), while takes a larger d int90 (20, 000 ∼ 50, 000) to reach reach its own, much higher baseline. The role of regularizers Our subspace training can be considered as a regularization scheme, as it restricts the solution set. We study and compare its effects with two traditional regularizers with an FC network (L=2, W=200) on CIFAR-10 dataset, including 2 penalty on the weights (i.e., weight decay) and Dropout.• 2 penalty Various amount of 2 penalty from {10 −2, 10 −3, 5 × 10 −4, 10 −4, 10 −5, 0} are considered. The accuracy and negative log-likelihood (NLL) are reported in FIG1 (a) (b), respectively. As expected, larger amount of weight decay reduces the gap between training and testing performance for both direct and subspace training methods, and eventually closes the gap (i.e., 2 penalty = 0.01). Subspace training itself exhibits strong regularization ability, especially when d is small, at which the performance gap between training and testing is smaller.• Dropout Various dropout rates from {0.5, 0.4, 0.3, 0.2, 0.1, 0} are considered. The accuracy and NLL are reported in FIG1. Larger dropout rates reduce the gap between training and testing performance for both direct and subspace training methods. When observing testing NLL, subspace training tends to overfit the training dataset less.• Subspace training as implicit regularization Subspace training method performs implicit regularization, as it restricts the solution set. We visualized the testing NLL in FIG1. Subspace training method outperforms direct method when d is properly chosen (when 2 penalty< 5×10 −4, or dropout rate < 0.1), suggesting the potential of this method as a better alternative to traditional regularizers. When d is large, the method also overfits the training dataset. Note that the these methods perform regularization in different ways: weight decay enforces the learned weights concentrating around zeros, while subspace training directly reduces the number of dimensions of the solution space. To investigate even larger problems, we attempted to measure d int90 for an ImageNet classification network. We use a relatively smaller network, SqueezeNet by BID13, with 1.24M parameters. Larger networks suffered from memory issues. A direct training produces Top-1 accuracy of 55.5%. We vary intrinsic dimension from 50k, 100k, 200k, 500k, 800k, and record the validation accuracies as shown in FIG1. The training of each intrinsic dimension takes about 6 to 7 days, distributed across 4 GPUs. Due to limited time, training on ImageNet has not yet produced a reliable estimate for d int90 except that it is over 500k. Since the learned d int90 can be used as a robust measure to study the fitness of neural network architectures for specific tasks, we further apply it to understand the contribution of each component in convolutional networks for image classification task. The convolutional network is a special case of FC network in two aspects: local receptive fields and weight-tying. Local receptive fields force each filter to "look" only at a small, localized region of the image or layer below. Weight-tying enforces that each filter shares the same weights, which reduces the number of learnable parameters. We performed control experiments to investigate the degree to which each component contributes. Four variants of LeNet are considered:• Standard LeNet 6 kernels (5×5) -max-pooling (2×2) -16 kernels (5×5) -max-pooling (2 × 2) -120 FC -84 FC -10 FC • Untied-LeNet The same architecture with the standard LeNet is employed, except that weights are unshared, i.e., a different set of filters is applied at each different patch of the input. For example in Keras, the LocallyConnected2D layer is used to replace the Conv2D layer.• FCTied-LeNet The same set of filters is applied at each different patch of the input. we break local connections by applying filters to global patches of the input. Assume the image size is H × H, the architecture is 6 kernels ((2H −1) × (2H −1)) -max-pooling (2 × 2) -16 kernels ((H −1) × (H −1)) -max-pooling (2 × 2) -120 FC -84 FC -10 FC. The padding type is same.• FC-LeNet Neither local connections or tied weights is employed, we mimic LeNet with its FC implementation. The same number of hidden units as the standard LeNet are used at each layer. The are shown in FIG1. We set a crossing-line accuracy (i.e., threshold) for each task, and investigate d int90 needed to achieve it. For MNIST and CIFAR-10, the threshold is 90% and 45%, respectively. For the above LeNet variants, d int90 = 290, 600, 425, 2000 on MNIST, and d int90 = 1000, 2750, 2500, 35000 on CIFAR-10. Experiments show both tied-weights and local connections are important to the model. That tied-weights should matter seems sensible. However, models with maximal convolutions (convolutions covering the whole image) may have had the same intrinsic dimension as smaller convolutions, but this turns out not to be the case. The is that convnets are more efficient than FC nets both due to local connectivity and due to weight tying. We summarize d int90 of the objective landscape on all different problems and neural network architectures in TAB7 and FIG2. "SP" indicates shuffled pixel, and "SL" for shuffled label, and "FC-5" for a 5-layer FC. d int90 indicates the minimum number of dimensions of trainable parameters required to properly solve the problem, and thus reflects the difficulty level of problems. C a r t P o le -F C P o le -F C C a r t -F C I n v e r t e d P e n d u lu m -F C H u m a n o id -F C A t a r i P o n g -C o n v N e t I m a g e N e t -S q u e e z e N e t FIG2: Intrinsic dimension of the objective landscapes created by all combinations of dataset and network we tried in this paper.

represent this text to find a much much longer article about it with say hundreds-thousands of words Given A model-based meta-RL algorithm that enables a real robot to adapt online in dynamic environments, a positive would be Although reinforcement learning methods can achieve impressive in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot: We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads. Both model-based and model-free reinforcement learning (RL) methods generally operate in one of two regimes: all training is performed in advance, producing a model or policy that can be used at test-time to make decisions in settings that approximately match those seen during training; or, training is performed online (e.g., as in the case of online temporal-difference learning), in which case the agent can slowly modify its behavior as it interacts with the environment. However, in both of these cases, dynamic changes such as failure of a robot's components, encountering a new terrain, environmental factors such as lighting and wind, or other unexpected perturbations, can cause the agent to fail. In contrast, humans can rapidly adapt their behavior to unseen physical perturbations and changes in their dynamics BID6: adults can learn to walk on crutches in just a few seconds, people can adapt almost instantaneously to picking up an object that is unexpectedly heavy, and children that can walk on carpet and grass can quickly figure out how to walk on ice without having to relearn how to walk. How is this possible? If an agent has encountered a large number of perturbations in the past, it can in principle use that experience to learn how to adapt. In this work, we propose a meta-learning approach for learning online adaptation. Motivated by the ability to tackle real-world applications, we specifically develop a model-based meta-reinforcement learning algorithm. In this setting, data for updating the model is readily available at every timestep in the form of recent experiences. But more crucially, the meta-training process for training such an adaptive model can be much more sample efficient than model-free meta-RL approaches BID11 BID55. Further, our approach foregoes the episodic framework on which model-free meta-RL approaches rely on, where tasks are pre-defined to be different rewards or environments, and tasks exist at the trajectory level only. Instead, our method considers each timestep to potentially be a new "task, " where any detail or setting could have changed at any timestep. This view induces a more general meta-RL problem setting by allowing the notion of a task to represent anything from existing in a different part of the state space, to experiencing disturbances, or attempting to achieve a new goal. Learning to adapt a model alleviates a central challenge of model-based reinforcement learning: the problem of acquiring a global model that is accurate throughout the entire state space. Furthermore, even if it were practical to train a globally accurate dynamics model, the dynamics inherently change as a function of uncontrollable and often unobservable environmental factors, such as those mentioned above. If we have a model that can adapt online, it need not be perfect everywhere a priori. This property has previously been exploited by adaptive control methods BID2 BID45 BID38; but, scaling such methods to complex tasks and nonlinear systems is exceptionally difficult. Even when working with deep neural networks, which have been used to model complex nonlinear systems BID21, it is exceptionally difficult to enable adaptation, since such models typically require large amounts of data and many gradient steps to learn effectively. By specifically training a neural network model to require only a small amount of experience to adapt, we can enable effective online adaptation in complex environments while putting less pressure on needing a perfect global model. The primary contribution of our work is an efficient meta reinforcement learning approach that achieves online adaptation in dynamic environments. To the best knowledge of the authors, this is the first meta-reinforcement learning algorithm to be applied in a real robotic system. Our algorithm efficiently trains a global model that is capable to use its recent experiences to quickly adapt, achieving fast online adaptation in dynamic environments. We evaluate two versions of our approach, recurrence-based adaptive learner (ReBAL) and gradient-based adaptive learner (GrBAL) on stochastic and simulated continuous control tasks with complex contact dynamics (Fig. 2). In our experiments, we show a quadrupedal "ant" adapting to the failure of different legs, as well as a "half-cheetah" robot adapting to the failure off different joints, navigating terrains with different slopes, and walking on floating platforms of varying buoyancy. Our model-based meta RL method attains substantial improvement over prior approaches, including standard model-based methods, online model-adaptive methods, model-free methods, and prior meta-reinforcement learning methods, when trained with similar amounts of data. In all experiments, meta-training across multiple tasks is sample efficient, using only the equivalent of 1.5 − 3 hours of real-world experience, roughly 10× less than what model-free methods require to learn a single task. Finally, we demonstrate GrBAL on a real dynamic legged millirobot (see Fig 2). To highlight not only the sample efficiency of our meta model-based reinforcement learning approach, but also the importance of fast online adaptation in the real world, we show the agent's learned ability to adapt online to tasks such as a missing leg, novel terrains and slopes, miscalibration or errors in pose estimation, and new payloads to be pulled. Advances in learning control policies have shown success on numerous complex and high dimensional tasks BID48 BID27 BID32 BID49. While reinforcement learning algorithms provide a framework for learning new tasks, they primarily focus on mastery of individual skills, rather than generalizing and quickly adapting to new scenarios. Furthermore, model-free approaches BID39 require large amounts of system interaction to learn successful control policies, which often makes them impractical for realworld systems. In contrast, model-based methods attain superior sample efficiency by first learning a model of system dynamics, and then using that model to optimize a policy BID9 BID23 BID36 BID58. Our approach alleviates the need to learn a single global model by allowing the model to be adapted automatically to different scenarios online based on recent observations. A key challenge with model-based RL approaches is the difficulty of learning a global model that is accurate for the entire state space. Prior model-based approaches tackled this problem by incorporating model uncertainty using Gaussian Processes (GPs) BID18 BID8 BID10. However, these methods make additional assumptions on the system (such as smoothness), and does not scale to high dimensional environments. BID7 has recently showed that neural networks models can also benefit from incorporating uncertainty, and it can lead to model-based methods that attain model-free performance with a significant reduction on sample complexity. Our approach is orthogonal to theirs, and can benefit from incorporating such uncertainty. Prior online adaptation approaches BID51 BID3 have aimed to learn an approximate global model and then adapt it at test time. Dynamic evaluation algorithms BID42 BID20 BID14, for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent. There exists extensive prior work on online adaptation in modelbased reinforcement learning and adaptive control BID45. In contrast from inverse model adaptation BID17 BID54 BID38 BID40, we are concerned in the problem of adapting the forward model, closely related to online system identification BID28. Work in model adaptation BID24 BID16 BID15 BID56 has shown that a perfect global model is not necessary, and prior knowledge can be fine-tuned to handle small changes. These methods, however, face a mismatch between what the model is trained for and how it is used at test time. In this paper, we bridge this gap by explicitly training a model for fast and effective adaptation. As a , our model achieves more effective adaptation compared to these prior works, as validated in our experiments. Our problem setting relates to meta-learning, a long-standing problem of interest in machine learning that is concerned with enabling artificial agents to efficiently learn new tasks by learning to learn BID52 BID47 BID37 BID22. A meta-learner can control learning through approaches such as deciding the learner's architecture BID4, or by prescribing an optimization algorithm or update rule for the learner BID5 BID46 BID59 BID1 BID26 BID41. Another popular meta-learning approach involves simply unrolling a recurrent neural network (RNN) that ingests the data BID44 BID33 BID31 and learns internal representations of the algorithms themselves, one instantiation of our approach (ReBAL) builds on top of these methods. On the other hand, the other instantiation of our method (GrBAL) builds on top of MAML. GrBAL differs from the supervised version of MAML in that MAML assumes access to a hand-designed distribution of tasks. Instead, one of our primary contributions is the online formulation of meta-learning, where tasks correspond to temporal segments, enabling "tasks" to be constructed automatically from the experience in the environment. Meta-learning in the context of reinforcement learning has largely focused on model-free approaches BID11 BID55 BID50 BID0. However, these algorithms present even more (meta-)training sample complexity than non-meta model-free RL methods, which precludes them from real-world applications. Recent work BID43 ) has developed a model-based meta RL algorithm, framing meta-learning as a hierarchical latent variable model, training for episodic adaptation to dynamics changes; the modeling is done with GPs, and are shown on the cart-pole and double-pendulum agents. In contrast, we propose an approach for learning online adaptation of high-capacity neural network dynamics models; we present two instantiations of this general approach and show on both simulated agents and a real legged robot. In this section, we present model-based reinforcement learning, introduce the meta-learning formulation, and describe the two main meta-learning approaches. Reinforcement learning agents aim to perform actions that maximize some notion of cumulative reward. Concretely, consider a Markov decision process (MDP) defined by the tuple (S, A, p, r, γ, ρ 0, H). Here, S is the set of states, A is the set of actions, p(s |s, a) is the state transition distribution, r: S × A → R is a bounded reward function, ρ 0: S → R + is the initial state distribution, γ is the discount factor, and H is the horizon. A trajectory segment is denoted by τ (i, j):= (s i, a i, ..., s j, a j, s j+1). Finally, the sum of expected rewards from a trajectory is the return. In this framework, RL aims to find a policy π: S → A that prescribes the optimal action to take from each state in order to maximize the expected return. Model-based RL aims to solve this problem by learning the transition distribution p(s |s, a), which is also referred to as the dynamics model. This can be done using a function approximatorp θ (s |s, a) to approximate the dynamics, where the weights θ are optimized to maximize the log-likelihood of the observed data D. In practice, this model is then used in the process of action selection by either producing data points from which to train a policy, or by producing predictions and dynamics constraints to be optimized by a controller.3.2 META-LEARNING Meta-learning is concerned with automatically learning learning algorithms that are more efficient and effective than learning from scratch. These algorithms leverage data from previous tasks to acquire a learning procedure that can quickly adapt to new tasks. These methods operate under the assumption that the previous meta-training tasks and the new meta-test tasks are drawn from the same task distribution ρ(T) and share a common structure that can be exploited for fast learning. In the supervised learning setting, we aim to learn a function f θ with parameters θ that minimizes a supervised loss L T. Then, the goal of meta-learning is to find a learning procedure, denoted as DISPLAYFORM0 We can formalize this meta-learning problem setting as optimizing for the parameters of the learning procedure θ, ψ as follows: min DISPLAYFORM1 where DISPLAYFORM2 T are sampled without replacement from the meta-training dataset D T. Once meta-training optimizes for the parameters θ *, ψ *, the learning procedure u ψ (·, θ) can then be used to learn new held-out tasks from small amounts of data. We will also refer to the learning procedure u as the update function. Gradient-based meta-learning. Model-agnostic meta-learning (MAML) aims to learn the initial parameters of a neural network such that taking one or several gradient descent steps from this initialization leads to effective generalization (or few-shot generalization) to new tasks. Then, when presented with new tasks, the model with the meta-learned initialization can be quickly fine-tuned using a few data points from the new tasks. Using the notation from before, MAML uses gradient descent as a learning algorithm: DISPLAYFORM3 The learning rate α may be a learnable paramter (in which case ψ = α) or fixed as a hyperparameter, leading to ψ = ∅. Despite the update rule being fixed, a learned initialization of an overparameterized deep network followed by gradient descent is as expressive as update rules represented by deep recurrent networks.Recurrence-based meta-learning. Another approach to meta-learning is to use recurrent models. In this case, the update function is always learned, and ψ corresponds to the weights of the recurrent model that update the hidden state. The parameters θ of the prediction model correspond to the remainder of the weights of the recurrent model and the hidden state. Both gradient-based and recurrence-based meta-learning methods have been used for meta model-free RL BID11. We will build upon these ideas to develop a meta model-based RL algorithm that enables adaptation in dynamic environments, in an online way. In this section, we present our approach for meta-learning for online model adaptation. As explained in Section 3.2, standard meta-learning formulations require the learned model θ *, ψ * to learn using M data points from some new "task." In prior gradient-based and model-based meta-RL approaches BID43, the M has corresponded to M trajectories, leading to episodic adaptation. Our notion of task is slightly more fluid, where every segment of a trajectory can be considered to be a different "task," and observations from the past M timesteps (rather than the past M episodes) can be considered as providing information about the current task setting. Since changes in system dynamics, terrain details, or other environmental changes can occur at any time, we consider (at every time step) the problem of adapting the model using the M past time steps to predict the next K timesteps. In this setting, M and K are pre-specified hyperparameters; see appendix for a sensitivity analysis of these parameters. In this work, we use the notion of environment E to denote different settings or configurations of a particular problem, ranging from malfunctions in the system's joints to the state of external disturbances. We assume a distribution of environments ρ(E) that share some common structure, such as the same observation and action space, but may differ in their dynamics p E (s |s, a). We denote a trajectory segment by τ E (i, j), which represents a sequence of states and actions (s i, a i, ..., s j, a j, s j+1) sampled within an environment E. Our algorithm assumes that the environment is locally consistent, in that every segment of length j − i has the same environment. Even though this assumption is not always correct, it allows us to learn to adapt from data without knowing when the environment has changed. Due to the fast nature of our adaptation (less than a second), this assumption is seldom violated. We pose the meta-RL problem in this setting as an optimization over (θ, ψ) with respect to a maximum likelihood meta-objective. The meta-objective is the likelihood of the data under a predictive modelp θ (s |s, a) with parameters θ, where θ = u ψ (τ E (t − M, t − 1), θ) corresponds to model parameters that were updated using the past M data points. Concretely, this corresponds to the following optimization: DISPLAYFORM0 In that τ E (t − M, t + K) ∼ D corresponds to trajectory segments sampled from our previous experience, and the loss L corresponds to the negative log likelihood of the data under the model: DISPLAYFORM1 In the meta-objective in Equation 3, note that the past M points are used to adapt θ into θ, and the loss of this θ is evaluated on the future K points. Thus, we use the past M timesteps to provide insight into how to adapt our model to perform well for nearby future timesteps. As outlined in Algorithm 1, the update rule u ψ for the inner update and a gradient step on θ for the outer update allow us to optimize this meta-objective of adaptation. Thus, we achieve fast adaptation at test time by being able to fine-tune the model using just M data points. While we focus on reinforcement learning problems in our experiments, this meta-learning approach could be used for a learning to adapt online in a variety of sequence modeling domains. We present our algorithm using both a recurrence and a gradient-based meta-learner, as we discuss next. Gradient-Based Adaptive Learner (GrBAL). GrBAL uses a gradient-based meta-learning to perform online adaptation; in particular, we use MAML. In this case, our update rule is prescribed by gradient descent (5.) DISPLAYFORM2 Recurrence-Based Adaptive Learner (ReBAL). ReBAL, instead, utilizes a recurrent model, which learns its own update rule (i.e., through its internal gating structure). In this case, ψ and u ψ correspond to the weights of the recurrent model that update its hidden state. Sample E ∼ ρ(E) Collect τ E using Alg. 2 6: DISPLAYFORM0 end if 8: DISPLAYFORM1 end for 13: DISPLAYFORM2 DISPLAYFORM3 a ← controller(θ *, r, H, n A) Execute a, add to D 6: end for 7: Return rollout D Figure 2: Two real-world and four simulated environments on which our method is evaluated and adaptation is crucial for success (e.g., adapting to different slopes and leg failures) Now that we have discussed our approach for enabling online adaptation, we next propose how to build upon this idea to develop a model-based meta-reinforcement learning algorithm. First, we explain how the agent can use the adapted model to perform a task, given parameters θ * and ψ * from optimizing the meta-learning objective. Given θ * and ψ *, we use the agent's recent experience to adapt the model parameters: θ * = u ψ * (τ (t − M, t), θ * ). This in a modelp θ * that better captures the local dynamics in the current setting, task, or environment. This adapted model is then passed to our controller, along with the reward function r and a planning horizon H. We use a planning H that is smaller than the adaptation horizon K, since the adapted model is only valid within the current context. We use model predictive path integral control (MPPI) BID57, but, in principle, our model adaptation approach is agnostic to the model predictive control (MPC) method used. The use of MPC compensates for model inaccuracies by preventing accumulating errors, since we replan at each time step using updated state information. MPC also allows for further benefits in this setting of online adaptation, because the modelp θ E itself will also improve by the next time step. After taking each step, we append the ing state transition onto our dataset, reset the model parameters back to θ *, and repeat the entire planning process for each timestep. See Algorithm 2 for this adaptation procedure. Finally, in addition to test-time, we also perform this online adaptation procedure during the meta-training phase itself, to provide on-policy rollouts for meta-training. For the complete meta-RL algorithm, see Algorithm 1. Our evaluation aims to answer the following questions: Is adaptation actually changing the model? Does our approach enable fast adaptation to varying dynamics, tasks, and environments, both inside and outside of the training distribution? How does our method's performance compare to that of other methods? How do GrBAL and ReBAL compare? How does meta model-based RL compare to meta model-free RL in terms of sample efficiency and performance for these experiments? Can our method learn to adapt online on a real robot, and if so, how does it perform? We next present our set-up and , motivated by these questions. Videos are available online 2, and further analysis is provided in the appendix. We first conduct a comparative evaluation of our algorithm, on a variety of simulated robots using the MuJoCo physics engine BID53. For all of our environments, we model the transition probabilities as Gaussian random variables with mean parameterized by a neural network model (3 hidden layers of 512 units each and ReLU activations) and fixed variance. In this case, maximum likelihood estimation corresponds to minimizing the mean squared error. We now describe the setup of our environments (Fig. 2), where each agent requires different types of adaptation to succeed at run-time:Half-cheetah (HC): disabled joint. For each rollout during meta-training, we randomly sample a joint to be disabled (i.e., the agent cannot apply torques to that joint). At test time, we evaluate performance in two different situations: disabling a joint unseen during training, and switching between disabled joints during a rollout. The former examines extrapolation to out-of-distribution environments, and the latter tests fast adaptation to changing dynamics. HC: sloped terrain. For each rollout during meta-training, we randomly select an upward or downward slope of low steepness. At test time, we evaluate performance on unseen settings including a gentle upward slope, a steep upward slope, and a steep hill that first goes up and then down. HC: pier. In this experiment, the cheetah runs over a series of blocks that are floating on water. Each block moves up and down when stepped on, and the changes in the dynamics are rapidly changing due to each block having different damping and friction properties. The HC is meta-trained by varying these block properties, and tested on a specific (randomly-selected) configuration of properties. Ant: crippled leg. For each meta-training rollout, we randomly sample a leg to cripple on this quadrupedal robot. This causes unexpected and drastic changes to the underlying dynamics. We evaluate this agent at test time by crippling a leg from outside of the training distribution, as well as transitioning within a rollout from normal operation to having a crippled leg. In the following sections, we evaluate our model-based meta-RL methods (GrBAL and ReBAL) in comparison to several prior methods:• Model-free RL (TRPO): To evaluate the importance of adaptation, we compare to a model-free RL agent that is trained across environments E ∼ ρ(E) using TRPO BID48.• Model-free meta-RL (MAML-RL): We compare to a state-of-the-art model-free meta-RL method, MAML-RL ).• Model-based RL (MB): Similar to the model-free agent, we also compare to a single modelbased RL agent, to evaluate the importance of adaptation. This model is trained using supervised model-error and iterative model bootstrapping.• Model-based RL with dynamic evaluation (MB+DE): We compare to an agent trained with model-based RL, as above. However, at test time, the model is adapted by taking a gradient step at each timestep using the past M observations, akin to dynamic evaluation BID20. This final comparison evaluates the benefit of explicitly training for adaptability. All model-based approaches (MB, MB+DE, GrBAL, and ReBAL) use model bootstrapping, use the same neural network architecture, and use the same planner within experiments: MPPI BID57 for the simulated experiments and random shooting (RS) BID35 for the real-world experiments. First, we analyze the effect of the model adaptation, and show from test-time runs on three environments: HC pier, HC sloped terrain with a steep up/down hill, and ant crippled leg with the chosen leg not seen as crippled during training. FIG1 displays the distribution shift between the pre-update and post-update model prediction errors of three GrBAL runs, showing that using the past M timesteps to update θ * (pre) into θ * (post) does indeed reduce model error on predicting the following K timesteps. We first study the sample efficiency of the meta-training process. FIG2 shows the average return across test environments w.r.t. the amount of data used for metatraining. We (meta-)train the model-free methods (TRPO and MAML-RL) until convergence, using the equivalent of about two days of real-world experience. In contrast, we meta-train the model-based methods (including our approach) using the equivalent of 1.5-3 hours of real-world experience. Our methods in superior or equivalent performance to the model-free agent that is trained with 1000 times more data. Our methods also surpass the performance of the non-meta-learned model-based approaches. Finally, our performance closely matches the high asymptotic performance of the modelfree meta-RL method for half-cheetah disabled, and achieves a suboptimal performance for ant crippled but, again, it does so with the equivalent of 1000 times less data. Note that this suboptimality in asymptotic performance is a known issue with model-based methods, and thus an interesting direction for future efforts. The improvement in sample efficiency from using model-based methods matches prior findings BID8 BID35 BID21; the most important evaluation, which we discuss in more detail next, is the ability for our method to adapt online to drastic dynamics changes in only a handful of timesteps. In our second comparative evaluation, we evaluate final test time performance both GrBAL and Re-BAL in comparison to the aforementioned methods. In the interest of developing efficient algorithms for real-world applications, we operate all methods in the low data regime for all experiments: the amount of data available (meta-)training is fixed across methods, and roughly corresponds to 1.5-3 hours of real-world experience depending on the domain. We also provide the performance of a MB oracle, which is trained using unlimited data from only the given test environment (rather than needing to generalize to various training environments).In these experiments, note that all agents were meta-trained on a distribution of tasks/environments (as detailed above), but we then evaluate their adaptation ability on unseen environments at test time. We test the ability of each approach to adapt to sudden changes in the environment, as well as to generalize beyond the training environments. We evaluate the fast adaptation (F.A.) component on the HC disabled joint, ant crippled leg, and the HC pier. On the first two, we cause a joint/leg of the robot to malfunction in the middle of a rollout. We evaluate the generalization component also on the tasks of HC disabled joint and ant crippled leg, but this time, the leg/joint that malfunctions has not been seen as crippled during training. The last environment that we test generalization on is the HC slopped terrain for a hill, where the agent has to run up and down a steep slope, which is outside of the gentle slopes that it experienced during training. The , shown in FIG3, show returns that are normalized such that the MB oracle achieves a return of 1. In all experiments, due to low quantity of training data, TRPO performs poorly. Although MB+DE achieves better generalization than MB, the slow nature of its adaptation causes it to fall behind MB in the environments that require fast adaptation. On the other hand, our approach surpasses the other approaches in all of the experiments. In fact, in the HC pier and the fast adaptation of ant environments, our approach surpasses the model-based oracle. This showcases the importance of adaptation in stochastic environments, where even a model trained with a lot of data cannot be robust to unexpected occurrences or disturbances. ReBAL displays its strengths on scenarios where longer sequential inputs allow it to better asses current environment settings, but overall, GrBAL seems to perform better for both generalization and fast adaptation. To test our meta model-based RL method's sample efficiency, as well as its ability to perform fast and effective online adaptation, we applied GrBAL to a real legged millirobot, comparing it to model-based RL (MB) and model-based RL with dynamic evaluation (MB+DE). Due to the cost of running real robot experiments, we chose the better performing method (i.e., GrBAL) to evaluate on the real robot. This small 6-legged robot, as shown in Fig. 1 and Fig. 2, presents a modeling and control challenge in the form of highly stochastic and dynamic movement. This robot is an excellent candidate for online adaptation for many reasons: the rapid manufacturing techniques and numerous custom-design steps used to construct this robot make it impossible to reproduce the same dynamics each time, its linkages and other body parts deteriorate over time, and it moves very quickly and dynamically withThe state space of the robot is a 24-dimensional vector, including center of mass positions and velocities, center of mass pose and angular velocities, back-EMF readings of motors, encoder readings of leg motor angles and velocities, and battery voltage. We define the action space to be velocity setpoints of the rotating legs. The action space has a dimension of two, since one motor on each side is coupled to all three of the legs on that side. All experiments are conducted in a motion capture room. Computation is done on an external computer, and the velocity setpoints are streamed over radio at 10 Hz to be executed by a PID controller on the microcontroller on-board of the robot. We meta-train a dynamics model for this robot using the meta-objective described in Equation 3, and we train it to adapt on entirely real-world data from three different training terrains: carpet, styrofoam, and turf. We collect approximately 30 minutes of data from each of the three training terrains. This data was entirely collected using a random policy, in conjunction with a safety policy, whose sole purpose was to prevent the robot from exiting the area of interest. Our first group of (Table 1) show that, when data from a random policy is used to train a dynamics model, both a model trained with a standard supervised learning objective (MB) and a GrBAL model achieve comparable performance for executing desired trajectories on terrains from the training distribution. Next, we test the performance of our method on what it is intended for: fast online adaptation of the learned model to enable successful execution of new, changing, or out-of-distribution environments at test time. Similar to the comparisons above, we compare GrBAL to a model-based method (MB) that involves neither meta-training nor online adaptation, as well as a dynamic evaluation method that involves online adaptation of that MB model (MB+DE). Our (Fig. 6) demonstrate that GrBAL substantially outperforms MB and MB+DE, and, unlike MB and MB+DE, and that GrBAL can quickly 1) adapt online to a missing leg, 2) adjust to novel terrains and slopes, 3) account for miscalibration or errors in pose estimation, and 4) compensate for pulling payloads. Figure 6: GrBAL clearly outperforms both MB and MB+DE, when tested on environments that require online adaptation, and/or were never seen during training.these environments were seen during training time, but the agent's ability to learn how to learn enables it to quickly leverage its prior knowledge and fine-tune to adapt to new environments online. Furthermore, the poor performance of the MB and MB+DE baselines demonstrate not only the need for adaptation, but also the importance of good initial parameters to adapt from (in this case, meta-learned parameters). The qualitative of these experiments in Fig. 7 show that the robot is able to use our method to adapt online and effectively follow the target trajectories, even in the presence of new environments and unexpected perturbations at test time.bounding-style gaits; hence, its dynamics are strongly dependent on the terrain or environment at hand. Figure 7: The dotted black line indicates the desired trajectory in the xy plane. By effectively adapting online, our method prevents drift from a missing leg, prevents sliding sideways down a slope, accounts for pose miscalibration errors, and adjusts to pulling payloads (left to right). Note that none of these tasks/environments were seen during training time, and they require fast and effective online adaptation for success. In this work, we present an approach for model-based meta-RL that enables fast, online adaptation of large and expressive models in dynamic environments. We show that meta-learning a model for online adaptation in a method that is able to adapt to unseen situations or sudden and drastic changes in the environment, and is also sample efficient to train. We provide two instantiations of our approach (ReBAL and GrBAL), and we provide a comparison with other prior methods on a range of continuous control tasks. Finally, we show that (compared to model-free meta-RL approaches), our approach is practical for real-world applications, and that this capability to adapt quickly is particularly important under complex real-world dynamics. In this section, we show the effect of adaptation in the case of GrBAL. In particular, we show the histogram of the K step normalized error, as well as the per-timestep visualization of this error during a trajectory. Across all tasks and environments, the post-updated modelp θ * achieves lower prediction error than the pre-updted modelp θ *. To see how training distribution affects test performance, we ran an experiment that used GrBAL to train models of the 7-DOF arm, where each model was trained on the same number of datapoints during meta-training, but those datapoints came from different ranges of force perturbations. We observe (in the plot below) that 1. Seeing more during training is helpful during testing -a model that saw a large range of force perturbations during training performed the best 2. A model that saw no perturbation forces during training did the worst 3. The middle 3 models show comparable performance in the "constant force = 4" case, which is an out-of-distribution task for those models. Thus, there is not actually a strong restriction on what needs to be seen during training in order for adaptation to occur at train time (though there is a general trend that more is better) In this section we analyze how sensitive is our algorithm w.r.t the hyperparameters K and M. In all experiments of the paper, we set K equal to M. FIG6 shows the average return of GrBAL across meta-training iterations of our algorithm for different values of K = M. The performance of the agent is largely unaffected for different values of these hyperparameters, suggesting that our algorithm is not particularly sensitive to these values. For different agents, the optimal value for these hyperparameters depends on various task details, such as the amount of information present in the state (a fully-informed state variable precludes the need for additional past timesteps) and the duration of a single timestep (a longer timestep duration makes it harder to predict more steps into the future). terrain environments. The x-axis shows data aggreation iterations during meta-training, whereas the y-axis shows the average return achieved when running online adaptation with the meta-learned model from the particular iteration. The curves suggest that GrBAL performance is fairly robust to the values of these hyperparameters. For each MuJoCo agent, the same reward function is used across its various tasks. TAB2 shows the reward functions used for each agent. We denote by x t the x-coordinate of the agent at time t, ee t refers to the position of the end-effector of the 7-DoF arm, and g corresponds to the position of the desired goal.

We train in random subspaces of parameter space to measure how many dimensions are really needed to find a solution.

Represent the short summary for finding the corresponding scientific article (long)

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.