Question: represent the scientific article to find a short tldr

Question: The dependency of the generalization error of neural networks on model and dataset size is of critical importance both in practice and for understanding the theory of neural networks. Nevertheless, the functional form of this dependency remains elusive. In this work, we present a functional form which approximates well the generalization error in practice. Capitalizing on the successful concept of model scaling (e.g., width, depth), we are able to simultaneously construct such a form and specify the exact models which can attain it across model/data scales. Our construction follows insights obtained from observations conducted over a range of model/data scales, in various model types and datasets, in vision and language tasks. We show that the form both fits the observations well across scales, and provides accurate predictions from small- to large-scale models and data. With the success and heightened adoption of neural networks for real world tasks, some questions remain poorly answered. For a given task and model architecture, how much data would one require to reach a prescribed performance level? How big a model would be needed? Addressing such questions is made especially difficult by the mounting evidence that large, deep neural networks trained on large-scale data outperform their smaller counterparts, rendering the training of high performance models prohibitively costly. Indeed, in the absence of practical answers to the above questions, surrogate approaches have proven useful. One such common approach is model scaling, where one designs and compares small-scale models, and applies the obtained architectural principles at a larger scale (e.g., ; ;). Despite these heuristics being widely used to various degrees of success, the relation between the performance of a model in the small-and large-scale settings is not well understood. Hence, exploring the limitations or improving the efficiency of such methods remains subject to trial and error. In this work we circle back to the fundamental question: what is the (functional) relation between generalization error and model and dataset sizes? Critically, we capitalize on the concept of model scaling in its strictest form: we consider the case where there is some given scaling policy that completely defines how to scale up a model from small to large scales. We include in this context all model parameters, such that traversing from one scale (in which all parameters are known) to another requires no additional resources for specifying the model (e.g., architecture search/design). We empirically explore the behavior of the generalization error over a wide range of datasets and models in vision and language tasks. While the error landscape seems fairly complex at first glance, we observe the emergence of several key characteristics shared across benchmarks and domains. Chief among these characteristics is the emergence of regions where power-law behavior approximates the error well both with respect to data size, when holding model size fixed, and vice versa. Motivated by these observations, we establish criteria which a function approximating the error landscape should meet. We propose an intuitive candidate for such a function and evaluate its quality, both in explaining the observed error landscapes and in extrapolating from small scale (seen) to large scale (unseen) errors. Critically, our functional approximation of the error depends on both model and data sizes. We find that this function leads to a high quality fit and extrapolation. For instance, the mean and standard deviation of the relative errors are under 2% when fitting across all scales investigated and under 5% when extrapolating from a slimmed-down model (1/16 of the parameters) on a fraction of the training data (1/8 of the examples) on the ImageNet and WikiText-103 datasets, with similar for other datasets. To the best of our knowledge, this is the first work that provides simultaneously: • A joint functional form of the generalization error landscape-as dependent on both data and model size-with few, interpretable degrees of freedom (section 5). • Direct and complete specification (via the scaling policy) of the model configuration attaining said generalization error across model and dataset sizes. • Highly accurate approximation of error measurements across model and data scales via the functional form, evaluated on different models, datasets, and tasks (section 6). • Highly accurate error prediction from small to large model and data (section 7). We conclude with a discussion of some implications of our findings as a practical and principled tool for understanding network design at small scale and for efficient computation and trade-off design in general. We hope this work also provides a useful empirical leg to stand on and an invitation to search for a theory of generalization error which accounts for our findings. Model scaling: A number of studies have explored the effect of model scaling on performance. For instance, image classification networks can be scaled by depth (number of layers;) or width (number of channels; ;). More recently, demonstrated how scaling width, depth, and input resolution has combined positive effects larger than scaling each factor in isolation. However, this relationship has yet to be quantified in a predictive form -by how much will error change with model scaling? In this work, we focus on finding a constructive functional form for determining the model given a specified performance. Data scaling: It has long been recognized that more data improves performance, and various studies report such trends in both computer vision (e.g., ;) and language processing tasks (e.g., ;). A number of prior studies observed power-law relations between the generalization error and training data size (; ;). Most relevant to our work, explored the effect of data size on the generalization error in vision, language, and speech tasks, and observed a strikingly consistent power-law behavior in a large set of experiments. However, while these studies point to the empirical existence of a power law in terms of data, they do not offer tools for predicting the performance given a specified model. Nor do they offer low-cost methods to specify the model configuration which would attain the power law with data dependency. Indeed, Hestness et al. had to search over models and their configurations at large scale to exhibit their findings, incurring prohibitive computational costs. In contrast, we demonstrate a constructive recipe, where we directly predict the test performance at large scale and specify the full model configuration which attains it (with no need for large-scale search), given performance at small scale. (a) Training data size (number of words) and model size (number of parameters excluding word embeddings) for language modeling tasks. Size (N) (b) Training data size (number of images) and model size (number of parameters) for image classification tasks. Theoretical error bounds: Much attention has been given to theoretical explanations of the generalization capabilities of deep neural networks (a; b; a; b;). While fully engaging with this literature is beyond our scope, we note that recent studies have derived bounds involving power-law dependencies in both model and data size . We leave it as an open question for future work to find theoretical explanations for the empirical behavior and the functional form we investigate in this work. Notation: denote a labeled (training) dataset with n samples or datapoints. Let f m denote a neural network whose size is the number of parameters m, such thatŷ = f m (x) is the predicted label. Let (n, m) be the generalization error as a function of n and m, measured by a performance metric (e.g., top-1 accuracy or cross-entropy loss) on a held-out test set. We refer to this error function as the error landscape. Dataset scaling: We wish to scale datasets while preserving the original distribution. For image classification, we uniformly subsample all classes by a constant ratio, thus preserving the relative sample size per class. We limit the maximal sub-sampling to avoid eradicating any class. For language modeling, where the number of classes (vocabulary items) has a very long tail distribution, we randomly sample sentences such that the total number of sampled words will be a certain fraction of the original dataset. Table 1 reports the data scales we use. In all tasks the held-out test set remains untouched for evaluating the error. Model scaling: We are critically interested in a method where moving across scales is defined by some scaling function, such that no additional significant computation would be incurred. We thus consider the case where the model architecture is given and the model size determines how to scale it. For instance, one may scale width (number of channels in convolutional networks, hidden state size in recurrent networks), depth (number of layers), do compound scaling , or more generally define a function tying the model degrees of freedom and size. We focus primarily on width scaling in our experiments; the model scales are reported in Table 1. We also perform selected depth scaling to demonstrate flexibility with respect to the scaling method. Hyper-parameters: For similar reasons we wish to avoid hyper-paramater search at large scales, and thus avoid the temptation to tune hyper-parameters accordingly (learning rate, regularization, etc.). Therefore, we hold all hyper-parameters fixed. This enables us to construct a functional form that fits the error landscape and can be used to predict the error across scales while completely defining the model attaining it. We consider pros and cons of this approach in the discussion (section 8). We experiment with both vision and language tasks. We use 6 benchmark datasets for image classification and 3 for language modeling. For image classification, we train ResNet and WRN models with stochastic gradient decent (SGD). In section 6.2 we explore the effect of varying architectures and optimizers for a fixed task (CIFAR100), adding VGG16 These two observations together can be summarized as the following relation: These two observations together can be summarized as the following relation: where a, α, c n may depend on the model size m, s.t. as n grows, → c n. Example fits to this form (allowing a, α, c n to be fit per m) are seen in figure 2a (left) and figure 2b (left). O3 Joint properties The behavior of the error when scaling model size while holding data size fixed, and vice versa, extends to the entire error landscape in a well-behaved manner, such that the manifold (m, n) is smooth everywhere as a function of both model and data scales. 5.1 CRITERIA Motivated by the above observations, we now consider a functional approximation for the error landscape. In particular, let us consider function families meeting the following criteria which augment and restrict our observations: C1 As either model or dataset size goes to zero, the expected performance is equivalent to a random-guess error level 0. 2 C2 For a given dataset size, scaling up the model will in an initial increase in performance, which will then saturate, taking the form in equation 1. C3 For a given model size, scaling up the dataset will in an initial increase in performance, which will then saturate, taking the form in equation 2. C4 There exists an irreducible error ∞, intrinsic to the dataset. C5 The function must be smooth everywhere and monotonic non-increasing in terms of model and data size (observation O3). While there are many possible function families meeting the above criteria, below we propose a simple function family for our evaluation. We do not claim that this is in fact the true underlying dependency, but rather that it serves as a good approximation of the error landscape-consistent with these criteria. As a first insightful step, consider the implications of satisfying C2 and C3 simultaneously. By examining the limiting behavior as m or n grow, we have: As m grows large: As n grows large: Thus, a consistent form satisfying C2 and C3 simultaneously is: where c ∞ is a constant not dependent on either m or n. Let us now examine the simplified case where a, b, α, β are constant: where α ≥ 0 and β ≥ 0 control the global rate at which error decreases with data and model size, respectively, a > 0 and b > 0 are a form of unit conversion between data and model sizes and error, and c ∞ > 0 is the asymptotic lower value attainable. This function is a special case of equation 3 and meets criteria C2 and C3 by construction. Importantly C4 and C5 are also met. However, by giving up the dependence of a, b, α, β on m, n, this function does not meet criterion C1. We thus need to model the transition from the initial random-guess level to the power-law region. We propose to parameterize the transition using the following envelope (complex) function: where i = √ −1. Here the simple pole at η controls the transition point from the initial random-guess level 0 as (m, n) increase. As (m, n) grow,˜ → c ∞ and the final irreducible error is approached. The random-guess error, 0, is a known parameter determined by dataset statistics (e.g, (N classes −1)/N classes for a balanced dataset). Note that due to our choice of rational envelope, we can divide by a constant the form in equation 4. Without loss of generality, let us choose a = 1. Note that while the forms in equations 3 and 4 are well motivated, the approach taken for modeling the transition is solely a convenience one. In fact, the transition(s) as function of m and n may be captured in the functional forms of a, b, α, β or another envelope mechanism. We leave a more refined investigation of the nature of the transitions to future work. We wish to empirically estimate the quality of the proposed functional parameterization as a fit to the true error landscape. Letˆ (n, m; θ) be the parametric function family (equation 5) approximating the error landscape (n, m), where θ = {α, β, b, c ∞, η}. 3 Define the divergence δ(n, m; θ) as the relative difference between the estimated errorˆ (m, n; θ) and the true error (m, n): We fit a least squares regression model to find the best parameters minimizing the divergence. In this section, we fit the function using 10-fold cross-validation across all model/data configurations m, n (see Table 1) and evaluate the fit quality. (In the next section, we perform extrapolation experiments, from seen to unseen points.) We perform the fit separately for each dataset and evaluate its quality by the mean µ and standard deviation σ of the divergence δ over all points (m, n). See Appendix B.1 for experimental details. As figure 3 shows, estimated test accuracy is highly correlated with actual test accuracy for various datasets, with worst-case values µ < 1% and σ < 5%. Note that the number of free parameters is small (|θ| ≤ 6) compared to the number of points (42-49 model-data configurations), demonstrating the appropriateness of the proposed function for modeling the complex error landscape. Here we verify that our extend to another canonical scaling policy, namely depth scaling. Figure 4a shows the error landscape with depth scaling on CIFAR10, exhibiting the same characteristics as width scaling. Figures 4b and 4c show error landscape estimation for both cases of width and depth scaling, exhibiting small and comparable fit errors (confidence intervals < 3%). Since the difference in approximation quality is effectively indistinguishable when scaling depth or width orthogonally, we expect compound scaling to adhere to the same functional form. Indeed, we verified this on the publicly available (model scaling only) for EfficientNet (However, the model/optimizer settings differ in multiple aspects across the different tasks, rendering the comparison of, say, different optimizers, challenging. In this section we verify that the functional form holds when varying the optimizer and/or the architecture on the same task, namely image classification on CIFAR100. In addition to the previously examined setting of WRN with SGD, we add four more settings: two well known architectures (VGG and DenseNet), each trained with both SGD and Adam optimizers. See Appendix A for experimental details. Figure 5 exhibits consistent, accurate, fit values across all architecture/optimizer settings, with mean divergence of µ < 1% (std: σ < 6%; confidence intervals < 4%). In this section, we evaluate the ability of our functional approximation to extrapolate beyond seen model/data configurations. The primary question we ask is: can we predict the error of a large model/data configuration from the errors of smaller-scale model/data configurations? To do this, we fit the least squares regression on a subset of the configurations and predict the error on larger, unseen configurations. More formally, let (m i, n j) denote a given model/data configuration. We first estimate parameters θ ij by fitting the function in equation 5 on all points of at most that size (m ≤ m i, n ≤ n j). Then we predict the error (m, n) in all points corresponding to larger configurations (m > m i, n > n j) using estimated θ ij. Finally, we measure the divergence δ(m, n) between the estimated error and the actual error at all larger configurations. This process is illustrated in figure 6a. Figure 6b shows the of one such extrapolation experiment, on ImageNet. In this case, we have fit the functional form on all configurations of model size m ≤ m i = M/16 and data size n ≤ n j = N/8, and predicted the error on all larger configurations. As the figure shows, the extrapolation is highly accurate, with a mean divergence of µ = 4.5% (std: σ = 4.7%). Figure 6c reports a similar experiment on WikiText-103. Here, again, we see very good extrapolation, with a mean divergence of µ = 0.5% (std: σ = 1.7%). Note that each extrapolation is run 10 times with different random initializations of θ ij in the least squares with negligible effect on the prediction. In practice, we may be interested in extrapolation quality with different subsets of configurations. Appendix D provides detailed extrapolation on multiple subsets of configurations, for both vision and language datasets. Generally, the extrapolation performs well once not ill-posed, which may be caused by lack of signal in the region of the initial "random-guess" level, or in degenerate cases like having fewer measurements than the number of free parameters in θ. In this work, through insights gained by the joint examination of the dependencies of generalization error on both model and data size, we arrive at criteria for functions consistent with the form of the generalization error under a given scaling policy. We consider one such function and find it to be in very good agreement with the actual behavior of the error landscape. Indeed, the agreement is strong enough that extrapolation from small to large scale becomes feasible: the function predicts the behavior of the generalization error in practice for the practical case of scaling models and data. We discuss several example implications of knowing such a functional form. Small-scale network development: At the core of small fidelity searches is the notion of performance rank comparison between models. However, small scale and large scale ranks are not assured to be consistent. If indeed a functional form such as empirically found in this work holds very generally, then in contrast, one can safely assess scaling rank between models at small scale, with the assurance that it remains consistent. This suggests that one would be well served by searching over scaling policies; a pertinent example of such a success is. The functional form also explains the limitation of small-scale search: once reaching the random-guess error level, where the sensitivity to scaling vanishes, the informativeness of ranking diminishes. Finally, the functional form allows direct usage of differentiable methods for NAS. Principled design: Knowing the error landscape function facilitates reasoning about the choice of (m, n) attaining a specified error level. In other words, for any given error level, one can solve Eq. 5 for m, n based on small-scale measurements. Thus, one can quantitatively answer design questions regarding the expected (in particular, large-scale) relations between m, n, and. In fact, Eq. 5 provides direct ansewrs to questions such as "how much data would one require to reach a prescribed performance level?" or "how big a model would be needed?" Imposing constraints is also straightforward. For instance, consider the following question: "What is the maximal model size possibly needed (useful), when the data is limited in size, n = n lim (for a given model architecture and scaling policy)?" For a fixed dataset size, model scaling eventually contributes marginally to error reduction and becomes negligible when bm Similarly, The maximal useful amount of data for a limited sized model m lim is: Moreover, Eq. 5 allows for complex design trade-offs. Generally, given some design-tradeoff cost function C(m, n,), one can minimize such cost s.t. Eq. 5. For example, consider the case of optimizing for efficient computation which has both practical and environmental importance . Since the number of FLOPs during training is ∝ m · n (for constant epoch budget), the trade-off cost function may be formulated as C(FLOPS,) = C(mn,). Further, since constant error contour is very well approximated by c = 1 n α + b m β (Eq. 5), dataset and models may be scaled with optimal resource efficiency with no effect on performance by solving for: The solution gives us the optimal-computational-efficiency ratio of model to data size: Limitations: We have made a few simplifying assumptions in our choice of approximating function, in particular in how to model the transition from the initial random-guess error level and the union of the random-guess level of the two scenarios (small model with large data and large model with small data). We leave a more detailed examination of the behavior of the transitions from random-guess error levels and refinements of the functional form to future work. Critically, the restrictive nature of our scaling framework (all parameters and hyperparameters described by a policy) is both a blessing and a challenge. The blessing comes in fulfilling the goal of finding simultaneously both the form of the generalization error and the full specification of the model and hyperparameters that attain it across scales. The challenge is that we have demonstrated in this work only the case of constant hyper-parameters. We conjecture that the relation between model configuration and hyperparameter choice may entail the potential to formulate hyperparameter-scaling policies similar in nature to the model-scaling polices, and that these too fall under the scope of the form we find in this work. This too will be the subject of future work. We hope that this work will bring the actual functional form of the generalization error in this practical case of scaling to the fore, both in practice and as an empirical leg to stand on in the quest for its theoretical origins. Scaling the models' width is performed by multiplying the number of channels in each convolutional layer and the width of the hidden linear layers by a constant factor and rounding to the nearest integer. The ranges of width scales (and data scales) for the main experiments are detailed in Table 1b. In section 6.2, we perform width scaling for two additional architectures, VGG16bn and DenseNet (L=40, k=32) . The VGG and DenseNet models were also modified for width scaling from the implementation of. The model scales in this case are 4 −k, 0 ≤ k ≤ 5, for both VGG and DenseNEt. Depth-scaling, in the CIFAR10 case (section 6.1), is performed by appending extra layers within each block. In the main experiments, training is done via SGD with a momentum of 0.9, weight decay of 1e-4 and initial learning rate of 0.1. For ImageNet we train for 90 epochs, decreasing the learning rate by a multiplicative factor of 0.1 after and 30 and after 60 epochs. We use a batch size of 16. For all other vision datasets we use a batch-size of 128. We begin training with a learning rate of 0.1, run for 200 epochs, and reduce by a multiplicative factor of 0.1 after 80, 120, and 160 epochs. For the VGG and DenseNet experiments on CIFAR100 in section 6.2, we train with both SGD and Adam optimizers. We train VGG for 170 epochs and Densenet for 300 epochs. Adam hyperparameters are default, with an initial learning rate of 1e-3. When training with SGD, we retain initial learning rate, batch size, momentum, and weight-decay, as in the main experiment (at 0.1, 128, 0.9, and 1e-4 respectively) and follow standard stepped learning rate schedules: For VGG, learning rate multiplicative factor of 0.1 after 80, 120, and 160 epochs; For DenseNet, learning rate multiplicative factor of 0.1 after 150 and 225 epochs. We evaluate on several datasets commonly used for (word-level) language modeling: Penn Treebank , WikiText-2 , and WikiText-103 . The PTB is a relatively small language modeling dataset of news texts, with a vocabu-lary of 10K unique words and about 900K/70K/80K training/validation/test words. WikiText-2 is drawn from Wikipedia articles and it is both larger and richer, with a vocabulary of 33K words and 2M/210K/240K training/validation/test words. WikiText-103 is also based on Wikipedia, but larger still, with a vocabulary of 270K words and 100M training words (and the same validation and test sets as WikiText-2). We experiment with two standard models for language modeling: Transformer-XL and AWD-LSTM . Transformer-XL is a recent language modeling architecture that is based on transformer self-attention , but modified to better learn dependencies beyond a fixed length by adding a segment-level recurrence mechanism. It has achieved state-of-the-art on multiple benchmarks. We use the official PyTorch implementation 4 with their base configuration: 16 layers, embedding size of 410, inner dimension of 2100 in the fullyconnected layers, and 10 attention heads. Training is done with Adam. See the implementation for other details. For scaling experiments, we decimate the inner dimension. We use Transformer-XL for WikiText-103. AWD-LSTM is a long short-term memory language model with adaptive weight averaging. We use the official implementation 5 with the recommended configuration: 3 layers, embedding size of 400, and hidden state size of 1150. Training is done with SGD. We use AWD-LSTM for PTB and WikiText-2 and follow the recommended settings for these two datasets. For scaling experiments, we decimate the hidden state size. In the experiment described in section 6, we fit a least squares regression model to find the best parameters minimizing the divergence δ(m, n) -evaluated at configurations m, n as in Table 1: We quantify the quality of the fit by the mean µ and standard deviation σ of the fitted divergence by performing standard 10-fold cross validation over all points (m, n) with confidence intervals reported as ±1 std over the folds. In this appendix, we provide error landscape measurements and estimations for all datasets, corresponding to the experiment in section 6. The are shown in 3D graphs similar to figure 1. In each such graph, the z-axis is the logarithm of the generalization error as a function of two independent variables: the model size m and the data size n. The 3D graph is deliberately portrayed in log-log-log scale, as we cover a very large range of data scales and model scales and a correspondingly wide range of errors. This view is a useful one when one wishes to evaluate both large dynamic ranges (simultaneously both very large and very small values) and is especially vivid in portraying power-law like dependencies; a power-law naturally forms a straight line in a log-log view. In each figure, subfigure (a) shows the measured error landscape is in log-log-log scale, where each point (blue dot) is the error ing from training with a model/data configuration m, n. Subfigure (b) shows the best-fit estimated error landscape. The surface is a linear interpolation between the points, which is then projected on the model-error (m,), data-error (n,), and model-data (m, n) planes. The contour plots on each one of these planes are the projections of the error landscape surface, and are useful in considering the behavior of the surface when holding one dimension constant. We call to attention several interesting observations on the datasets explored: • As quantified rigorously in section 6, the fits perform well across error ranges. In these surfaces, one also gets qualitative sense of the fit adequacy across the wide ranges of the dataset and model scales directly. While perhaps slightly difficult to asses the surface directly, a helpful view is to consider the similarity between the projections of the actual and projected surfaces. • With increasing model size, indeed typically the error does remain saturated. However, in one of our tested datasets (figure 12) there was a renewed slight increase. We verify that this is indeed over-fitting, in the sense that there is no corresponding increase in the training error. We note that the functional form we find can actually be used to veer clear of the m, n regions where such over-fitting may occur. • The simplifying approach taken by considering the random guess levels (and associated transitions) for small models or small data as identical, seems to work fairly well with some deviation apparent by examining figure 15. Indeed the simplification can hold well for balanced datasets, but need not for imbalanced ones such as in the task of language modeling. Thus, a relaxation of this simplification is expected to be important conceptually and practically. Here we provide detailed extrapolation , for all datasets. All figures are structured in a similar way. Each subplot shows estimated (y-axis) vs. actual error (x-axis) (0 to 1 scale on both axes). Each subplot is located at the coordinate of the maximal data and model given for the task of performing the fit to the functional form in equation 5. This is the point at the top-right corner of the green dots in the illustration in figure 6a. The target is to find the error-landscape values for unseen, larger scales of both model and data (red points in the same illustration). Going from left to right in each figure indicates observed measurements of the error from models of an increasing fraction w.r.t the full size. Going from bottom-to top indicates observed measurements of the error from dataset sizes of an increasingly large fraction of the full dataset. In each subplot, every point shows the estimated vs. actual error on a model-data configuration. Points that were given for fitting the function are colored in green, while unseen points that were not used are in red. The red points show the estimation error vs. actual error when extrapolating to all larger models and data sizes. In each subplot, the mean and standard deviation over all divergences δ at target points are given in text. Each experiment fit of the parameters was repeated 100 times, with different random initializations of θ. The shaded bands show one standard deviation across these runs. The quality of the extrapolation is critically dependent on the signal provided in the (green) fitted points. Two limiting factors are evident by examining the figures below, which both play a role in the well-posedness of the solution: • The proximity to the initial random guess level. Only upon transitioning from the initial error plateau, does meaningful signal about the scaling rates become available. Indeed, for scales prior still in the region or close to the initial error level, one sees poor extrapolation ; see figures 18, 19, and 21, and the vivid origin of this phenomena by examining figures 11, 10, and 12. • A second source of ill-posedness is tied to the number of configurations used for the estimation of θ. Clearly, when this is small, one cannot expect the extrapolation to be stable. In fact, at least two measurements in each scaling dimension (model/data) are needed, and no less than the number of parameters in θ in total. Indeed, for all the plots in this appendix, the smallest scale of m, n is omitted form the graph such that the lowermost row and leftmost column span exactly two model and data scales correspondingly. Of course, there is nothing tying directly the number of points and scale of configurations measured, and one can decouple these two factors by taking closer spaced samples at small scale. • When both the above factors are not limiting the measurement, one readily sees that for divergences of no more than a few percent, it is sufficient to measure model/data configurations which are far-ranged from the configurations which one wishes to extrapolate to.

represent this text to find a much much longer article about it with say hundreds-thousands of words
------
E.g. Proposing locally adaptive activation functions in deep and physics-informed neural networks for faster convergence == We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing scalable hyper-parameters in each layer (layer-wise) and for every neuron separately (neuron-wise), and then optimizing it using the stochastic gradient descent algorithm. Introduction of neuron-wise activation function acts like a vector activation function as opposed to the traditional scalar activation function given by fixed, global and layer-wise activations. In order to further increase the training speed, an activation slope based slope recovery term is added in the loss function, which further accelerate convergence, thereby reducing the training cost. For numerical experiments, a nonlinear discontinuous function is approximated using a deep neural network with layer-wise and neuron-wise locally adaptive activation functions with and without the slope recovery term and compared with its global counterpart. Moreover, solution of the nonlinear Burgers equation, which exhibits steep gradients, is also obtained using the proposed methods. On the theoretical side, we prove that in the proposed method the gradient descent algorithms are not attracted to sub-optimal critical points or local minima under practical conditions on the initialization and learning rate. Furthermore, the proposed adaptive activation functions with the slope recovery are shown to accelerate the training process in standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets with and without data augmentation. In recent years, research on neural networks (NNs) has intensified around the world due to their successful applications in many diverse fields such as speech recognition, computer vision , natural language translation , etc. Training of NN is performed on data sets before using it in the actual applications. Various data sets are available for applications like image classification, which is a subset of computer vision. MNIST and their variants like, Fashion-MNIST , and KMNIST are the data sets for handwritten digits, images of clothing and accessories, and Japanese letters, respectively. Apart from MNIST, Semeion is a handwritten digit data set that contains 1593 digits collected from 80 persons. SVHN is another data set for street view house numbers obtained from house numbers in Google Street View images. CI-FAR is the popular data set containing color images commonly used to train machine learning algorithms. In particular, the CIFAR-10 data set contains 50000 training and 10000 testing images in 10 classes with image resolution of 32x32. CIFAR-100 is similar to the CIFAR-10, except it has 100 classes with 600 images in each class, which is more challenging than the CIFAR-10 data set. problems, where the approximate solutions of governing equations are obtained, as well as inverse problems, where parameters involved in the governing equation are inferred from the training data. Highly efficient and adaptable algorithms are important to design the most effective NN which not only increases the accuracy of the solution but also reduces the training cost. Various architectures of NN like Dropout NN are proposed in the literature, which can improve the efficiency of the algorithm for specific applications. Activation function plays an important role in the training process of NN. In this work, we are particularly focusing on adaptive activation functions, which adapt automatically such that the network can be trained faster. Various methods are proposed in the literature for adaptive activation function, like the adaptive sigmoidal activation function proposed by for multilayer feedforward NNs, while focuses on learning activation functions in convolutional NNs by combining basic activation functions in a data-driven way. Multiple activation functions per neuron are proposed , where individual neurons select between a multitude of activation functions. proposed a tunable activation function, where only a single hidden layer is used and the activation function is tuned. , used a similar idea of tunable activation function but with multiple outputs. Recently, Kunc and Kléma proposed a transformative adaptive activation functions for gene expression inference, see (Kunc & Kléma, 2019). One such adaptive activation function is proposed by introducing scalable hyper-parameter in the activation function, which can be optimized. Mathematically, it changes the slope of activation function thereby increasing the learning process, especially during the initial training period. Due to single scalar hyper-parameter, we call such adaptive activation functions globally adaptive activations, meaning that it gives an optimized slope for the entire network. One can think of doing such optimization at the local level, where the scalable hyper-parameter are introduced hidden layer-wise or even for each neuron in the network. Such local adaptation can further improve the performance of the network. Figure 1 shows a sketch of a neuron-wise locally adaptive activation function based physics-informed neural network (LAAF-PINN), where both the NN part along with the physicsinformed part can be seen. In this architecture, along with the output of NN and the residual term from the governing equation, the activation slopes from every neuron are also contributing to the loss function in the form of slope recovery term. The rest of the paper is organized as follows. Section 2 presents the methodology of the proposed layer-wise and neuron-wise locally adaptive activations in detail. This also includes a discussion on the slope recovery term, expansion of parametric space due to layer-wise and neuron-wise introduction of hyper-parameters, its effect on the overall training cost, and a theoretical for gradient decent algorithms. Section 3 gives numerical experiments, where we approximate a nonlinear discontinuous function using deep NN by the proposed approaches. We also solve the Burgers equation using the proposed algorithm and present various comparisons in appendix B. Section 4 presents numerical with various standard deep learning benchmarks using CIFAR-10, CIFAR-100, SVHN, MNIST, KMNIST, Fashion-MNIST, and Semeion data sets. Finally, in section 5, we summarize the of our work. We use a NN of depth D corresponding to a network with an input layer, D − 1 hidden-layers and an output layer. In the k th hidden-layer, N k number of neurons are present. Each hidden-layer of the network receives an output z k−1 ∈ R N k−1 from the previous layer where an affine transformation of the form is performed. The network weights w k ∈ R N k ×N k−1 and bias term b k ∈ R N k associated with the k th layer are chosen from independent and identically distributed sampling. The nonlinearactivation function σ(·) is applied to each component of the transformed vector before sending it as an input to the next layer. The activation function is an identity function after an output layer. Thus, the final neural network representation is given by the composition where the operator • is the composition operator, represents the trainable parameters in the network, u is the output and z 0 = z is the input. In supervised learning of solution of PDEs, the training data is important to train the neural network, which can be obtained from the exact solution (if available) or from high-resolution numerical solution given by efficient numerical schemes and it can be even obtained from carefully performed experiments, which may yield both high-and low-fidelity data sets. We aim to find the optimal weights for which the suitably defined loss function is minimized. In PINN the loss function is defined as where the mean squared error (MSE) is given by Here {x represents the residual training points in space-time domain, while {x represents the boundary/initial training data. The neural network solution must satisfy the governing equation at randomly chosen points in the domain, which constitutes the physicsinformed part of neural network given by first term, whereas the second term includes the known boundary/initial conditions, which must be satisfied by the neural network solution. The ing optimization problem leads to finding the minimum of a loss function by optimizing the parameters like, weights and biases, i.e., we seek to find w *, b * = arg min w,b∈Θ (J(w, b)). One can approximate the solutions to this minimization problem iteratively by one of the forms of gradient descent algorithm. The stochastic gradient descent (SGD) algorithm is widely used in machine learning community see, for a complete survey. In SGD the weights are updated as, where η l > 0 is the learning rate. SGD methods can be initialized with some starting value w 0. In this work, the ADAM optimizer , which is a variant of the SGD method is used. A deep network is required to solve complex problems, which on the other hand is difficult to train. In most cases, a suitable architecture is selected based on the researcher's experience. One can also think of tuning the network to get the best performance out of it. In this regard, we propose the following two approaches to optimize the adaptive activation function. Instead of globally defining the hyper-parameter a for the adaptive activation function, let us define this parameter hidden layer-wise as This gives additional D − 1 hyper-parameters to be optimized along with weights and biases. Here, every hidden-layer has its own slope for the activation function. One can also define such activation function at the neuron level as This gives additional k=1 N k hyper-parameters to be optimized. Neuron-wise activation function acts as a vector activation function as opposed to scalar activation function given by L-LAAF and global adaptive activation function (GAAF) approaches, where every neuron has its own slope for the activation function. The ing optimization problem leads to finding the minimum of a loss function by optimizing a k i along with weights and biases, i.e., we seek to find (a The process of training NN can be further accelerated by multiplying a with scaling factor n > 1. The final form of the activation function is given by σ(na It is important to note that the introduction of the scalable hyper-parameter does not change the structure of the loss function defined previously. Then, the final adaptive activation function based neural network representation of the solution is given by In this case, the set of trainable parametersΘ consists of {w and {a In all the proposed methods, the initialization of scalable hyper-parameters is done such that na k i = 1, ∀n. The main motivation of adaptive activation function is to increase the slope of activation function, ing in non-vanishing gradients and fast training of the network. It is clear that one should quickly increase the slope of activation in order to improve the performance of NN. Thus, instead of only depending on the optimization methods, another way to achieve this is to include the slope recovery term based on the activation slope in the loss function as where the slope recovery term S(a) is given by where N is a linear/nonlinear operator. Although, there can be several choices of this operator, including the linear identity operator, in this work we use the exponential operator. The main reason behind this is that such term contributes to the gradient of the loss function without vanishing. The overall effect of inclusion of this term is that it forces the network to increase the value of activation slope quickly thereby increasing the training speed. We now provide a theoretical regarding the proposed methods. The following theorem states that a gradient descent algorithm minimizing our objective functionJ(Θ) in equation 3 does not converge to a sub-optimal critical point or a sub-optimal local minimum, for neither L-LAAF nor N-LAAF, given appropriate initialization and learning rates. In the following theorem, we treatΘ as a real-valued vector. Let Jc = M SE F + M SE u with the constant network u Θ (z) = u Θ (z) = c ∈ R N D for all z, z where c is a constant. Theorem 2.1. Let (Θ m) m∈N be a sequence generated by a gradient descent algorithm asΘ m+1 = Θ m − η m ∇J(Θ). Assume that J(Θ 0) < Jc + S for any c ∈ R N D, J is differentiable, and that for each i ∈ {1, . . ., N f}, there exist differentiable function ϕ i and input. Assume that at least one of the following three conditions holds. (i) (constant learning rate) ∇J is Lipschitz continuous with Lipschitz constant C (i.e., ∇J(Θ) − ∇J(Θ) 2 ≤ C Θ −Θ 2 for allΘ,Θ in its domain), and ≤ η m ≤ (2 −)/C, where is a fixed positive number. (ii) (diminishing learning rate) ∇J is Lipschitz continuous, η m → 0 and ∞ m=0 η m = ∞. (iii) (adaptive learning rate) the learning rate η m is chosen by the minimization rule, the limited minimization rule, the Armjio rule, or the Goldstein rule . Then, for both L-LAAF and N-LAAF, no limit point of (Θ m) m∈N is a sub-optimal critical point or a sub-optimal local minimum. The initial condition J(Θ 0) < Jc + S means that the initial value J(Θ 0) needs to be less than that of a constant network plus the highest value of the slope recovery term. Here, note that S < S. The proof of Theorem 2.1 is included in appendix A. In this section, we shall solve a regression problem of a nonlinear function approximation using deep neural network. The Burgers equation using physics-informed neural network is solved in appendix B. In this test case, a standard neural network (without physics-informed part) is used to approximate a discontinuous function. In this case, the loss function consists of the data mismatch and the slope recovery term as The following discontinuous function with discontinuity at x = 0 location is approximated by a deep neural network. Here, the domain is [−3, 3] and the number of training points used is 300. The activation function is tanh, learning rate is 2.0e-4 and the number of hidden layers are four with 50 neurons in each layer. Figure 2 shows the solution (first column), solution in frequency domain (second column) and pointwise absolute error in log scale (third column). The solution by standard fixed activation function is given in the first row, GAAF solution is given in second row, whereas the third row shows the solution given by L-LAAF without and with (fourth row) slope recovery term. The solution given by N-LAAF without slope recovery term is shown in the fifth row and with slope recovery term in the sixth row. We see that the NN training speed increases for the locally adaptive activation functions compared to fixed and globally adaptive activations. Moreover, both L-LAAF and N-LAAF with slope recovery term accelerate training and yield the least error as compared to other methods. Figure 3 (top) shows the variation of na for GAAF, whereas the second row, left and right shows the layer-wise variation of na k for L-LAAF without and with the slope recovery term respectively. The third row, left and right shows the variation of scaled hyper-parameters for N-LAAF without and with the slope recovery term respectively, where the mean value of na k i along with its standard deviation (Std) are plotted for each hidden-layer. We see that the value of na is quite large with the slope recovery term which shows the rapid increase in the activation slopes. Finally, the comparison of the loss function is shown in figure 4 for standard fixed activation, GAAF, L-LAAF and N-LAAF without the slope recovery (left) and for L-LAAF and N-LAAF with the slope recovery (right) using a scaling factor of 10. The Loss function for both L-LAAF and N-LAAF without the slope recovery term decreases faster, especially during the initial training period compared to the fixed and global activation function based algorithms. The previous sections demonstrated the advantages of adaptive activation functions with PINN for physics related problems. One of the remaining questions is whether or not the advantage of adaptive activations remains with standard deep neural networks for other types of deep learning applications. To explore the question, this section presents numerical with various standard benchmark problems in deep learning. Figures 5 and 6 shows the mean values and the uncertainty intervals Figure 2: Discontinuous function: Neural network solution using standard fixed activation (first row), GAAF (second row), L-LAAF without (third row) and with (fourth row) slope recovery term, and N-LAAF without (fifth row) and with (sixth row) slope recovery term using the tanh activation. First column shows the solution which is also plotted in frequency domain (zoomed-view) as shown by the corresponding second column. Third column gives the point-wise absolute error in the log scale for all the cases. accelerates the minimization process of the training loss values. Here, all of GAAF, L-LAAF and N-LAAF use the slope recovery term, which improved the methods without the recovery term. Accordingly, the of GAAF are also new contributions of this paper. In general, L-LAAF improved against GAAF as expected. The standard cross entropy loss was used for training and plots. We used pre-activation ResNet with 18 layers for CIFAR-10, CIFAR-100, and SVHN data sets, whereas we used a standard variant of LeNet with ReLU for other data sets; i.e., the architecture of the variant of LeNet consists of the following five layers (with the three hidden layers): input layer, convolutional layer with 64 5 × 5 filters, followed by max pooling of size of 2 by 2 and ReLU, convolutional layer with 64 5 × 5 filters, followed by max pooling of size of 2 by 2 and ReLU, fully connected layer with 1014 output units, followed by ReLU, and Fully connected layer with the number of output units being equal to the number of target classes. All hyper-parameters were fixed a priori across all different data sets and models. We fixed the mini-batch size s to be 64, the initial learning rate to be 0.01, the momentum coefficient to be 0.9 and we use scaling factor n = 1 and 2. The learning rate was divided by 10 at the beginning of 10th epoch for all experiments (with and without data augmentation), and of 100th epoch for those with data augmentation. In this paper, we present two versions of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions. Such local activation functions further improve the training speed of the neural network compared to its global predecessor. To further accelerate the training process, an activation slope based slope recovery term is added in the loss function for both layer-wise and neuron-wise activation functions, which is shown to enhance the performance of the neural network. Various NN and PINN test cases like nonlinear discontinuous function approximation and Burgers equation respectively, and benchmark deep learning problems like MNIST, CIFAR, SVHN etc are solved to verify our claim. Moreover, we theoretically prove that no sub-optimal critical point or local minimum attracts gradient descent algorithms in the proposed methods (L-LAAF and N-LAAF) with the slope recovery term under only mild assumptions. k=1 is a limit point of (Θ m) m∈N and a sub-optimal critical point or a sub-optimal local minimum. and h Following the proofs in (, Propositions 1.2.1-1.2.4), we have that ∇J(Θ) = 0 and J(Θ) < Jc + S, for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . ., D − 1}, Furthermore, we have that for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k}, By combining equation 5-equation 7, for all k ∈ {1, . . ., D − 1}, which implies that for all a k = 0 since (D − 1) exp(a k) = 0. This implies that J(Θ) = Jc + S, which contradicts with J(Θ) < Jc + S. This proves the desired statement for L-LAAF. For N-LAAF, we prove the statement by contradiction. Suppose that the parameter vectorΘ consisting of {w k=1 ∀j = 1, 2, · · ·, N k is a limit point of (Θ m) m∈N and a suboptimal critical point or a sub-optimal local minimum. Redefine and h for all j ∈ {1, . . ., N k}, where w k,j ∈ R 1×N k−1 and b k,j ∈ R. Then, by the same proof steps, we have that ∇J(Θ) = 0 and J(Θ) < Jc + S, for all three cases of the conditions corresponding the different rules of the learning rate. Therefore, we have that for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k}, By combining equation 6-equation 8, for all k ∈ {1, . . ., D − 1} and all j ∈ {1, . . ., N k},, which implies that for all a This implies that J(Θ) = Jc + S, which contradicts with J(Θ) < Jc + S. This proves the desired statement for N-LAAF. The Burgers equation is one of the fundamental partial differential equation arising in various fields such as nonlinear acoustics, gas dynamics, fluid mechanics etc, see for more details. The Burgers equation was first introduced by H. and later studied by J.M. in the context of theory of turbulence. Here, we consider the viscous Burgers equation given by equation equation 9 along with its initial and boundary conditions. The non-linearity in the convection term develops very steep solution due to small˜ value. We consider the Burgers equation given by with initial condition u(x, 0) = − sin(πx), boundary conditions u(−1, t) = u(1, t) = 0 and˜ = 0.01/π. The analytical solution can be obtained using the Hopf-Cole transformation, see for more details. The number of boundary and initial training points is 400, whereas the number of residual training points is 10000. The activation function is tanh, learning rate is 0.0008 and the number of hidden layers are 6 with 20 neurons in each layer. Figure 7 shows the evolution of frequency plots of the solution at three different times using the standard fixed activation function (first row), global adaptive activation function (second row), L-LAAF without (third row) and with (fourth row) slope recovery term, N-LAAF without (fifth row) and with (sixth row) slope recovery term using scaling factor n = 10. Again, for both L-LAAF and N-LAAF the frequencies are converging faster towards the exact solution (shown by black line) with and without slope recovery term as compared to the fixed and global activation function based algorithms. decreases faster for all adaptive activations, in particular GAAF. Even though it is difficult to see from the actual solution plots given by figure 8, one can see from the Figure 10: Burgers equation: comparison of na k for L-LAAF for all six hidden-layers. First three columns represent for L-LAAF without slope recovery term whereas the last three columns are with slope recovery term. In all simulations, the scaling factor n is 10. Figure 10 shows the comparison of layer-wise variation of na k for L-LAAF with and without slope recovery term. It can be seen that, the presence of slope recovery term further increases the slope of activation function thereby increasing the training speed. Similarly, figure 11 shows the mean and standard deviation of na k i for N-LAAF with and without slope recovery term, which again validates that with slope recovery term network training speed increases. for N-LAAF for all six hidden-layers. First three columns represent resuls for N-LAAF without the slope recovery term whereas the last three columns are with slope recovery term. In all simulations, the scaling factor n is 10.

We predict the generalization error and specify the model which attains it across model/data scales.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -- Atari, the DeepMind Control Suite and DeepMind Lab. Currently, the best performing methods on many reinforcement learning benchmark problems combine model-free reinforcement learning methods with policies represented using deep neural networks BID18 BID8. Despite reaching or surpassing human-level performance on many challenging tasks, deep model-free reinforcement learning methods that learn purely from the reward signal learn in a way that differs greatly from the manner in which humans learn. In the case of learning to play a video game, a human player not only acquires a strategy for achieving a high score, but also gains a degree of mastery of the environment in the process. Notably, a human player quickly learns which aspects of the environment are under their control as well as how to control them, as evidenced by their ability to rapidly adapt to novel reward functions BID22.Focusing learning on mastery of the environment instead of optimizing a single scalar reward function has many potential benefits. One benefit is that learning is possible even in the absence of an extrinsic reward signal or with an extrinsic reward signal that is very sparse. Another benefit is that an agent that has fully mastered its environment should be able to reach arbitrary achievable goals, which would allow it to generalize to tasks on which it wasn't explicitly trained. Building reinforcement learning agents that aim for environment mastery instead of or in addition to learning about a scalar reward signal is currently an open challenge. One way to represent such knowledge about an environment is using an environment model. Modelbased reinforcement learning methods aim to learn accurate environment models and use them either for planning or for training a policy. While learning accurate environment models of some visually rich environments is now possible BID33 BID7 BID15 using learned models in model-based reinforcement learning has proved to be challenging and model-free approaches still dominate common benchmarks. We present a new model-free agent architecture of Discriminative Embedding Reward Networks, or DISCERN for short. DISCERN learns to control an environment in an unsupervised way by learning purely from the stream of observations and actions. The aim of our agent is to learn a goal-conditioned policy π θ (a|s; s g) BID19 BID37 which can reach any goal state s g that is reachable from the current state s. We show how to learn a goal achievement reward function r(s; s g) that measures how similar state s is to state s g using a mutual information objective at the same time as learning π θ (a|s; s g). The ing learned reward function r(s; s g) measures similarity in the space of controllable aspects of the environment instead of in the space of raw observations. Crucially, the DISCERN architecture is able to deal with goal states that are not perfectly reachable, for example, due to the presence of distractor objects that are not under the agent's control. In such cases the goal-conditioned policy learned by DISCERN tends to seek states where the controllable elements match those in the goal state as closely as possible. We demonstrate the effectiveness of our approach on three domains -Atari games, continuous control tasks from the DeepMind Control Suite, and DeepMind Lab. We show that our agent learns to successfully achieve a wide variety of visually-specified goals, discovering underlying degrees of controllability of an environment in a purely unsupervised manner and without access to an extrinsic reward signal. In the standard reinforcement learning setup an agent interacts with an environment over discrete time steps. At each time step t the agent observes the current state s t and selects an action a t according to a policy π(a t |s t). The agent then receives a reward r t = r(s t, a t) and transitions to the next state s t+1. The aim of learning is to maximize the expected discounted return R = ∞ t=0 γ t r t of policy π where γ ∈ is a discount factor. In this work we focus on learning only from the stream of actions and observations in order to forego the need for an extrinsic reward function. Motivated by the idea that an agent capable of reaching any reachable goal state s g from the current state s has complete mastery of its environment, we pose the problem of learning in the absence of rewards as one of learning a goal-conditioned policy π θ (a|s; s g) with parameters θ. More specifically, we assume that the agent interacts with an environment defined by a transition distribution p(s t+1 |s t, a t). We define a goal-reaching problem as follows. At the beginning of each episode, the agent receives a goal s g sampled from a distribution over possible goals p goal. For example, p goal could be the uniform distribution over all previously visited states. The agent then acts for T steps according to the goal-conditioned policy π θ (a|s; s g) receiving a reward of 0 for each of the first T − 1 actions and a reward of r(s T ; s g) after the last action, where r(s; s g) ∈ for all s and s g 1. The goal achievement reward function r(s; s g) measures the degree to which being in state s achieves goal s g. The episode terminates upon the agent receiving the reward r(s T ; s g) and a new episode begins. It is straightforward to train π θ (a|s; s g) in a tabular environment using the indicator reward r(s; s g) = 1{s = s g}. We are, however, interested in environments with continuous highdimensional observation spaces. While there is extensive prior work on learning goal-conditioned policies BID19 BID37 BID0 BID16 BID34, the reward function is often hand-crafted, limiting generality of the approaches. In the few cases where the reward is learned, the learning objective is typically tied to a pre-specified notion of visual similarity. Learning to achieve goals based purely on visual similarity is unlikely to work in complex, real world environments due to the possible variations in appearance of objects, or goal-irrelevant perceptual context. We now turn to the problem of learning a goal achievement reward function r φ (s; s g) with parameters φ for high-dimensional state spaces. We aim to simultaneously learn a goal-conditioned policy π θ and a goal achievement reward function r φ by maximizing the mutual information between the goal state s g and the achieved state s T as shown in. DISPLAYFORM0 Note that we are slightly overloading notation by treating s g as a random variable distributed according to p goal. Similarly, s T is a random variable distributed according to the state distribution induced by running π θ for T steps for goal states sampled from p goal.The prior work of BID13 showed how to learn a set of abstract options by optimizing a similar objective, namely the mutual information between an abstract option and the achieved state. Following their approach, we simplify in two ways. First, we rewrite the expectation in terms of the goal distribution p goal and the goal conditioned policy π θ. Second, we lower bound the expectation term by replacing p(s g |s T) with a variational distribution q φ (s g |s T) with parameters φ following BID3, leading to DISPLAYFORM1 Finally, we discard the entropy term H(s g) from because it does not depend on either the policy parameters θ or the variational distribution parameters φ, giving our overall objective DISPLAYFORM2 This objective may seem difficult to work with because the variational distribution q φ is a distribution over possible goals s g, which in our case are high-dimensional observations, such as images. We sidestep the difficulty of directly modelling the density of high-dimensional observations by restricting the set of possible goals to be a finite subset of previously encountered states that evolves over time BID25. Restricting the support of q φ to a finite set of goals turns the problem of learning q φ into a problem of modelling the conditional distribution of possible intended goals given an achieved state, which obviates the requirement of modelling arbitrary statistical dependencies in the observations. Optimization: The expectation in the DISCERN objective is with respect to the distribution of trajectories generated by the goal-conditioned policy π θ acting in the environment against goals drawn from the goal distribution p goal. We can therefore optimize this objective with respect to policy parameters θ by repeatedly generating trajectories and performing reinforcement learning updates on π θ with a reward of log q φ (s g |s T) given at time T and 0 for other time steps. Optimizing the objective with respect to the variational distribution parameters φ is also straightforward since it is equivalent to a maximum likelihood classification objective. As will be discussed in the next section, we found that using a reward that is a non-linear transformation mapping log q φ (s g |s T) to worked better in practice. Nevertheless, since the reward for the goal conditioned-policy is a function of log q φ (s g |s T), training the variational distribution function q φ amounts to learning a reward function. Communication Game Interpretation: Dual optimization of the DISCERN objective has an appealing interpretation as a cooperative communication game between two players -an imitator that corresponds to the goal-conditioned policy and a teacher that corresponds to the variational distribution. At the beginning of each round or episode of the game the imitator is provided with a goal state. The aim of the imitator is to communicate the goal state to the teacher by taking T actions in the environment. After the imitator takes T actions, the teacher has to guess which state from a set of possible goals was given to the imitator purely from observing the final state s T reached by the imitator. The teacher does this by assigning a probability to each candidate goal state that it was the goal given to the imitator at the start of the episode, i.e. it produces a distribution p(s g |s T). The objective of both players is for the teacher to guess the goal given to the imitator correctly as measured by the log probability assigned by the teacher to the correct goal. We now describe the DISCERN algorithm -a practical instantiation of the approach for jointly learning π θ (a|s; s g) and r(s; s g) outlined in the previous section. Goal distribution: We adopt a non-parametric approach to the problem of proposing goals, whereby we maintain a fixed size buffer G of past observations from which we sample goals during training. We update G by replacing the contents of an existing buffer slot with an observation from the agent's recent experience according to some substitution strategy; in this work we considered two such strategies, detailed in Appendix A3. This means that the space of goals available for training drifts as a function of the agent's experience, and states which may not have been reachable under a poorly trained policy become reachable and available for substitution into the goal buffer, leading to a naturally induced curriculum. In this work, we sample training goals for our agent uniformly at random from the goal buffer, leaving the incorporation of more explicitly instantiated curricula to future work. We train a goal achievement reward function r(s; s g) used to compute rewards for the goal-conditioned policy based on a learned measure of state similarity. We parameterize r(s; s g) as the positive part of the cosine similarity between s and s g in a learned embedding space, although shaping functions other than rectification could be explored. The state embedding in which we measure cosine similarity is the composition of a feature transformation h(·) and a learned L 2 -normalized mapping ξ φ (·). In our implementation, where states and goals are represented as 2-D RGB images, we take h(·) to be the final layer features of the convolutional network learned by the policy in order to avoid learning a second convolutional network. We find this works well provided that while training r, we treat h(·) as fixed and do not adapt the convolutional network's parameters with respect to the reward learner's loss. This has the effect of regularizing the reward learner by limiting its adaptive capacity while avoiding the need to introduce a hyperparameter weighing the two losses against one another. We train ξ φ (·) according to a goal-discrimination objective suggested by. However, rather than using the set of all goals in the buffer G as the set of possible classes in the goal discriminator, we sample a small subset for each trajectory. Specifically, the set of possible classes includes the goal g for the trajectory and DISPLAYFORM0 we maximize the log likelihood given by DISPLAYFORM1 where β is an inverse temperature hyperparameter which we fix to K + 1 in all experiments. Note that is a maximum log likelihood training objective for a softmax nearest neighbour classifier in a learned embedding space, making it similar to a matching network BID47. Intuitively, updating the embedding ξ φ using the objective in aims to increase the cosine similarity between e(s T) and e(g) and to decrease the cosine similarity between e(s T) and the decoy embeddings e(d),..., e(d K). Subsampling the set of possible classes as we do is a known method for approximate maximum likelihood training of a softmax classifier with many classes BID6.We use max(0, g) as the reward for reaching state s T when given goal g. We found that this reward function is better behaved than the reward logq(s g = g|s T ; d 1, . . . d K, π θ) suggested by the DISCERN objective in Section 3 since it is scaled to lie in. The reward we use is also less noisy since, unlike logq, it does not depend on the decoy states. Goal-conditioned policy: The goal-conditioned policy π θ (a|s; s g) is trained to optimize the goal achievement reward r(s; s g). In this paper, π θ (a|s; s g) is an -greedy policy of a goal-conditioned action-value function Q with parameters θ. Q is trained using Q-learning and minibatch experience replay; specifically, we use the variant of Q(λ) due to Peng (see Chapter 7, BID40). We use a form of goal relabelling BID19 or hindsight experience replay BID0 BID32 as a source successfully achieved goals as well as to regularize the embedding e(·). Specifically, for the purposes of parameter updates (in both the policy and the reward learner) we substitute, with probability p HER the goal with an observation selected from the final H steps of the trajectory, and consider the agent to have received a reward of 1. The motivation, in the case of the policy, is similar to that of previous work, i.e. that being in state s t should correspond to having achieved the goal of reaching s t. When employed in the reward learner, it amounts to encouraging temporally consistent state embeddings BID29 BID38, i.e. encouraging observations which are nearby in time to have similar embeddings. Pseudocode for the DISCERN algorithm, decomposed into an experience-gathering (possibly distributed) actor process and a centralized learner process, is given in Algorithm 1. The problem of reinforcement learning in the context of multiple goals dates at least to BID19, where the problem was examined in the context of grid worlds where the state space is DISPLAYFORM0 /* See Appendix A3 */ end with probability p HER, Sample s HER uniformly from {s T −H, . . ., s T} and set g ← s HER, r T ← 1 otherwise Compute g using r T ← max(0, g) Send (s 1:T, a 1:T, r 1:T, g) to the learner. Poll the learner periodically for updated values of θ,φ. Reset the environment if the episode has terminated. until termination procedure LEARNER Input:Batch size B, number of decoys K, initial policy parameters θ, initial goal embedding parameters φ repeat Assemble batch of experience B = {(s DISPLAYFORM1 Use an off-policy reinforcement learning algorithm to update θ based on B Update φ to maximize DISPLAYFORM2 small and enumerable. BID41 proposed generalized value functions (GVFs) as a way of representing knowledge about sub-goals, or as a basis for sub-policies or options. Universal Value Function Approximators (UVFAs) BID37 extend this idea by using a function approximator to parameterize a joint function of states and goal representations, allowing compact representation of an entire class of conditional value functions and generalization across classes of related goals. While the above works assume a goal achievement reward to be available a priori, our work includes an approach to learning a reward function for goal achievement jointly with the policy. Several recent works have examined reward learning for goal achievement in the context of the Generative Adversarial Networks (GAN) paradigm BID12. The SPIRAL BID11 algorithm trains a goal conditioned policy with a reward function parameterized by a Wasserstein GAN BID1 discriminator. Similarly, AGILE BID2 learns an instruction-conditional policy where goals in a grid-world are specified in terms of predicates which should be satisfied, and a reward function is learned using a discriminator trained to distinguish states achieved by the policy from a dataset of instruction, goal state pairs. Reward learning has also been used in the context of imitation. BID17 derives an adversarial network algorithm for imitation, while time-contrastive networks BID38 leverage pre-trained ImageNet classifier representations to learn a reward function for robotics skills from video demonstrations, including robotic imitation of human poses. Universal Planning Networks (UPNs) BID39 ) learn a state representation by training a differentiable planner to imitate expert trajectories. Experiments showed that once a UPN is trained the state representation it learned can be used to construct a reward function for visually specified goals. Bridging goal-conditioned policy learning and imitation learning, BID34 learns a goal-conditioned policy and a dynamics model with supervised learning without expert trajectories, and present zero-shot imitation of trajectories from a sequence of images of a desired task. A closely related body of work to that of goal-conditioned reinforcement learning is that of unsupervised option or skill discovery. BID26 proposes a method based on an eigendecomposition of differences in features between successive states, further explored and extended in BID27. Variational Intrinsic Control (VIC) BID13 leverages the same lower bound on the mutual information as the present work in an unsupervised control setting, in the space of abstract options rather than explicit perceptual goals. VIC aims to jointly maximize the entropy of the set of options while making the options maximally distinguishable from their final states according to a parametric predictor. Recently, BID9 showed that a special case of the VIC objective can scale to significantly more complex tasks and provide a useful basis for low-level control in a hierarchical reinforcement learning context. Other work has explored learning policies in tandem with a task policy, where the task or environment rewards are assumed to be sparse. propose a framework in which low-level skills are discovered in a pre-training phase of a hierarchial system based on simple-to-design proxy rewards, while BID36 explore a suite of auxiliary tasks through simultaneous off-policy learning. Several authors have explored a pre-training stage, sometimes paired with fine-tuning, based on unsupervised representation learning. BID35 and BID23 employ a two-stage framework wherein unsupervised representation learning is used to learn a model of the observations from which to sample goals for control in simple simulated environments. BID31 propose a similar approach in the context of model-free Q-learning applied to 3-dimensional simulations and robots. Goals for training the policy are sampled from the model's prior, and a reward function is derived from the latent codes. This contrasts with our non-parametric approach to selecting goals, as well as our method for learning the goal space online and jointly with the policy. An important component of our method is a form of goal relabelling, introduced to the reinforcement learning literature as hindsight experience replay by BID0, based on the intuition that any trajectory constitutes a valid trajectory which achieves the goal specified by its own terminal observation. Earlier, BID32 employed a related scheme in the context of supervised learning of motor programs, where a program encoder is trained on pairs of trajectory realizations and programs obtained by expanding outwards from a pre-specified prototypical motor program through the addition of noise. BID45 expands upon hindsight replay and the all-goal update strategy proposed by BID19, generalizing the latter to non-tabular environments and exploring related strategies for skill discovery, unsupervised pre-training and auxiliary tasks. BID24 propose a hierarchical Q-learning system which employs hindsight replay both conventionally in the lower-level controller and at higher levels in the hierarchy. BID31 also employ a generalized goal relabeling scheme whereby the policy is trained based on a trajectory's achievement not just of its own terminal observation, but a variety of retrospectively considered possible goals. We evaluate, both qualitatively and quantitatively, the ability of DISCERN to achieve visuallyspecified goals in three diverse domains -the Arcade Learning Environment BID5, continuous control tasks in the DeepMind Control Suite BID42, and DeepMind Lab, a 3D first person environment BID4. Experimental details including architecture details, details of distributed training, and hyperparameters can be found in the Appendix. We compared DISCERN to several baseline methods for learning goal-conditioned policies:Conditioned Autoencoder (AE): In order to specifically interrogate the role of the discriminative reward learning criterion, we replace the discriminative criterion for embedding learning with an L 2 reconstruction loss on h t; that is, in addition to ξ φ (·), we learn an inverse mapping ξ −1 φ (·) with a separate set of parameters, and train both with the criterion h t − ξ DISPLAYFORM0 Conditioned WGAN Discriminator: We compare to an adversarial reward on the domains considered according to the protocol of BID11, who successfully used a WGAN discriminator as a reward for training agents to perform inverse graphics tasks. The discriminator takes two pairs of images - a real pair of goal images (s g, s g) and a fake pair consisting of the terminal state of the agent and the goal frame (s t, s g). The output of the discriminator is used as the reward function for the policy. Unlike our DISCERN implementation and the conditioned autoencoder baseline, we train the WGAN discriminator as a separate convolutional network directly from pixels, as in previous work. Pixel distance reward (L2): Finally, we directly compare to a reward based on L 2 distance in pixel space, equal to exp − s t − s g 2 /σ pixel where σ pixel is a hyperparameter which we tuned on a per-environment basis. All the baselines use the same goal-conditioned policy architecture as DISCERN. The baselines also used hindsight experience replay in the same way as DISCERN. They can therefore be seen as ablations of DISCERN's goal-achievement reward learning mechanism. The suite of 57 Atari games provided by the Arcade Learning Environment is a widely used benchmark in the deep reinforcement learning literature. We compare DISCERN to other methods on the task of achieving visually specified goals on the games of Seaquest and Montezuma's Revenge. The relative simplicity of these domains makes it possible to handcraft a detector in order to localize the controllable aspects of the environment, namely the submarine in Seaquest and Panama Joe, the character controlled by the player in Montezuma's Revenge. We evaluated the methods by running the learned goal policies on a fixed set of goals and measured the percentage of goals it was able to reach successfully. We evaluated both DISCERN and the baselines with two different goal buffer substitution strategies, uniform and diverse, which are described in the Appendix. A goal was deemed to be successfully achieved if the position of the avatar in the last frame was within 10% of the playable area of the position of the avatar in the goal for each controllable dimension. The controllable dimensions in Atari were considered to be the x-and y-coordinates of the avatar. The are displayed in FIG0. DISCERN learned to achieve a large fraction of goals in both Seaquest and Montezuma's Revenge while none of the baselines learned to reliably achieve goals in either game. We hypothesize that the baselines failed to learn to control the avatars because their objectives are too closely tied to visual similarity. FIG0 shows examples of goal achievement on Seaquest and Montezuma's Revenge. In Seaquest, DISCERN learned to match the position of the submarine in the goal image while ignoring the position of the fish, since the fish are not directly controllable. We have provided videos of the goal-conditioned policies learned by DISCERN on Seaquest and Montezuma's Revenge at the following anonymous URL https://sites.google.com/view/discern-anonymous/home. The DeepMind Control Suite BID42 ) is a suite of continuous control tasks built on the MuJoCo physics engine BID44. While most frequently used to evaluate agents which receive the underlying state variables as observations, we train our agents on pixel renderings of the scene using the default environment-specified camera, and do not directly observe the state variables. Agents acting greedily with respect to a state-action value function require the ability to easily maximize Q over the candidate actions. For ease of implementation, as well as comparison to other considered environments, we discretize the space of continuous actions to no more than 11 unique actions per environment (see Appendix A4.1).The availability of an underlying representation of the physical state, while not used by the learner, provides a useful basis for comparison of achieved states to goals. We mask out state variables relating to entities in the scene not under the control of the agent; for example, the position of the target in the reacher or manipulator domains. DISCERN is compared to the baselines on a fixed set of 100 goals with 20 trials for each goal. The goals are generated by acting randomly for 25 environment steps after initialization. In the case of R a n d o m a g e n t P i x e l d i s t a n c e Figure 2: Average achieved frames for point mass (task easy), reacher (task hard), manipulator (task bring ball), pendulum (task swingup), finger (task spin) and ball in cup (task catch) environments. The goal is shown in the top row and the achieved frame is shown in the bottom row.cartpole, we draw the goals from a random policy acting in the environment set to the balance task, where the pole is initialized upwards, in order to generate a more diverse set of goals against which to measure. FIG1 compares learning progress of 5 independent seeds for the "uniform" goal replacement strategy (see Appendix A5 for with "diverse" goal replacement) for 6 domains. We adopt the same definition of achievement as in Section 6.1. Figure 2 summarizes averaged goal achievement frames on these domains except for the cartpole domain for policies learned by DISCERN. Performance on cartpole is discussed in more detail in FIG4 of the Appendix. The show that in aggregate, DISCERN outperforms baselines in terms of goal achievement on several, but not all, of the considered Control Suite domains. In order to obtain a more nuanced understanding of DISCERN's behaviour when compared with the baselines, we also examined achievement in terms of the individual dimensions of the controllable state. Figure 4 shows goal achievement separately for each dimension of the underlying state on four domains. The perdimension show that on difficult goal-achievement tasks such as those posed in cartpole (where most proposed goal states are unstable due to the effect of gravity) and finger (where a free-spinning piece is only indirectly controllable) DISCERN learns to reliably match the major dimensions of controllability such as the cart position and finger pose while ignoring the other Actor steps Figure 4: Per-dimension quantitative evaluation of goal achievement on continuous control domains using the "uniform" goal substitution scheme (Appendix A3). Each subplot corresponds to a domain, with each group of colored rows representing a method. Each individual row represents a dimension of the controllable state (such as a joint angle). The color of each cell indicates the fraction of goal states for which the method was able to match the ground truth value for that dimension to within 10% of the possible range. The position along the x-axis indicates the point in training in millions of frames. For example, on the reacher domain DISCERN learns to match both dimensions of the controllable state, but on the cartpole domain it learns to match the first dimension (cart position) but not the second dimension (pole angle).dimensions, whereas none of the baselines learned to reliably match any of the controllable state dimensions on the difficult tasks cartpole and finger. We omitted the manipulator domain from these figures as none of the methods under consideration achieved non-negligible goal achievement performance on this domain, however a video showing the policy learned by DISCERN on this domain can be found at https://sites.google.com/view/discern-anonymous/home. The policy learned on the manipulator domain shows that DISCERN was able to discover several major dimensions of controllability even on such a challenging task, as further evidenced by the per-dimension analysis on the manipulator domain in Figure 8 in the Appendix. DeepMind Lab BID4 ) is a platform for 3D first person reinforcement learning environments. We trained DISCERN on the watermaze level and found that it learned to approximately achieve the same wall and horizon position as in the goal image. While the agent did not learn to achieve the position and viewpoint shown in a goal image as one may have expected, it is encouraging that our approach learns a reasonable space of goals on a first-person 3D domain in addition to domains with third-person viewpoints like Atari and the DM Control Suite. We have presented a system that can learn to achieve goals, specified in the form of observations from the environment, in a purely unsupervised fashion, i.e. without any extrinsic rewards or expert demonstrations. Integral to this system is a powerful and principled discriminative reward learning objective, which we have demonstrated can recover the dominant underlying degrees of controllability in a variety of visual domains. In this work, we have adopted a fixed episode length of T in the interest of simplicity and computational efficiency. This implicitly assumes not only that all sampled goals are approximately achievable in T steps, but that the policy need not be concerned with finishing in less than the allotted number of steps. Both of these limitations could be addressed by considering schemes for early termination based on the embedding, though care must be taken not to deleteriously impact training by terminating episodes too early based on a poorly trained reward embedding. Relatedly, our goal selection strategy is agnostic to both the state of the environment at the commencement of the goal episode and the current skill profile of the policy, utilizing at most the content of the goal itself to drive the evolution of the goal buffer G. We view it as highly encouraging that learning proceeds using such a naive goal selection strategy, however more sophisticated strategies, such as tracking and sampling from the frontier of currently achievable goals BID16, may yield substantial improvements. DISCERN's ability to automatically discover controllable aspects of the observation space is a highly desirable property in the pursuit of robust low-level control. A natural next step is the incorporation of DISCERN into a deep hierarchical reinforcement learning setup BID46 BID24 BID30 where a meta-policy for proposing goals is learned after or in tandem with a low-level controller, i.e. by optimizing an extrinsic reward signal. We employ a distributed reinforcement learning architecture inspired by the IMPALA reinforcement learning architecture BID8, with a centralized GPU learner batching parameter updates on experience collected by a large number of CPU-based parallel actors. While BID8 learns a stochastic policy through the use of an actor-critic architecture, we instead learn a goal-conditioned state-action value function with Q-learning. Each actor acts -greedily with respect to a local copy of the Q network, and sends observations s t, actions a t, rewards r t and discounts γ t for a trajectory to the learner. Following BID18, we use a different value of for each actor, as this has been shown to improve exploration. The learner batches re-evaluation of the convolutional network and LSTM according to the action trajectories supplied and performs parameter updates, periodically broadcasting updated model parameters to the actors. As Q-learning is an off-policy algorithm, the experience traces sent to the learner can be used in the usual n-step Q-learning update without the need for an off-policy correction as in BID8. We also maintain actor-local replay buffers of previous actor trajectories and use them to perform both standard experience replay BID25 and our variant of hindsight experience replay BID0. Our network architectures closely resemble those in BID8, with policy and value heads replaced with a Q-function. We apply the same convolutional network to both s t and s g and concatenate the final layer outputs. Note that the convolutional network outputs for s g need only be computed once per episode. We include a periodic representation (sin(2πt/T), cos(2πt/T)) of the current time step, with period equal to the goal length achievement period T, as an extra input to the network. The periodic representation is processed by a single hidden layer of rectified linear units and is concatenated with the visual representations fed to the LSTM. While not strictly necessary, we find that this allows the agent to become better at achieving goal states which may be unmaintainable due to their instability in the environment dynamics. The output of the LSTM is the input to a dueling action-value output network BID48. In all of our experiments, both branches of the dueling network are linear mappings. That is, given LSTM outputs ψ t, we compute the action values for the current time step t as DISPLAYFORM0 We experimented with two strategies for updating the goal buffer. In the first strategy, which we call uniform, the current observation replaces a uniformly selected entry in the goal buffer with probability p replace. The second strategy, which we refer to as diverse goal sampling attempts to maintain a goal buffer that more closely approximates the uniform distribution over all observation. In the diverse goal strategy, we consider the current observation for addition to the goal buffer with probability p replace at each step during acting. If the current observation s is considered for addition to the goal buffer, then we select a random removal candidate s r by sampling uniformly from the goal buffer and replace it with s if s r is closer to the rest of the goal buffer than s. If s is closer to the rest of the goal buffer than s r then we still replace s r with s with probability p add−non−diverse. We used L 2 distance in pixel space for the diverse sampling strategy and found it to greatly increase the coverage of states in the goal buffer, especially early during training. This bears some relationship to Determinantal Point Processes BID20, and goal-selection strategies with a more explicit theoretical foundation are a promising future direction. The following hyper-parameters were used in all of the experiments described in Section 6. All weight matrices are initialized using a standard truncated normal initializer, with the standard deviation inversely proportional to the square root of the fan-in. We maintain a goal buffer of size 1024 and use p replace = 10 −3. We also use p add−non−diverse = 10 −3. For the teacher, we choose ξ φ (·) to be an L 2 -normalized single layer of 32 tanh units, trained in all experiments with 4 decoys (and thus, according to our heuristic, β equal to 5). For hindsight experience replay, a highsight goal is substituted 25% of the time. These goals are chosen uniformly at random from the last 3 frames of the trajectory. Trajectories were set to be 50 steps long for Atari and DeepMind Lab and 100 for the DeepMind control suite. It is important to note that the environment was not reset after each trajectory, but rather the each new trajectory begins where the previous one ended. We train the agent and teacher jointly with RMSProp BID43 ) with a learning rate of 10 −4. We follow the preprocessing protocol of BID28, resizing to 84 × 84 pixels and scaling 8-bit pixel values to lie in the range. While originally designed for Atari, we apply this preprocessing pipeline across all environments used in this paper. In the point mass domain we use a control step equal to 5 times the task-specified default, i.e. the agent acts on every fifth environment step BID28. In all other Control Suite domains, we use the default. We use the "easy" version of the task where actuator semantics are fixed across environment episodes. Discrete action spaces admit function approximators which simultaneously compute the action values for all possible actions, as popularized in BID28. The action with maximal Q-value can thus be identified in time proportional to the cardinality of the action space. An enumeration of possible actions is no longer possible in the continuous setting. While approaches exist to enable continuous maximization in closed form BID14, they come at the cost of greatly restricting the functional form of Q.For ease of implementation, as well as comparison to other considered environments, we instead discretize the space of continuous actions. For all Control Suite environments considered except manipulator, we discretize an A-dimensional continuous action space into 3A discrete actions, consisting of the Cartesian product over action dimensions with values in {−1, 0, 1}. In the case of manipulator, we adopt a "diagonal" discretization where each action consists of setting one actuator to ±1, and all other actuators to 0, with an additional action consisting of every actuator being set to 0. This is a reasonable choice for manipulator because any position can be achieved by a concatenation of actuator actions, which may not be true of more complex Control Suite environments such as humanoid, where the agent's body is subject to gravity and successful trajectories may require multi-joint actuation in a single control time step. The subset of the Control Suite considered in this work was chosen primarily such that the discretized action space would be of a reasonable size. We leave extensions to continuous domains to future work. We ran two additional baselines on Seaquest and Montezuma's Revenge, ablating our use of hindsight experience replay in opposite ways. One involved training the goal-conditioned policy only in hindsight, without any learned goal achievement reward, i.e. p HER = 1. This approach achieved 12% of goals on Seaquest and 11.4% of goals on Montezuma's Revenge, making it comparable to a uniform random policy. This underscores the importance of learning a goal achievement reward. The second baseline consisted of DISCERN learning a goal achievement reward without hindsight experience replay, i.e. p HER = 0. This also performed poorly, achieving 11.4% of goals on Seaquest and 8% of goals on Montezuma's Revenge. Taken together, these preliminary suggest that the combination of hindsight experience replay and a learned goal achievement reward is important. For the sake of completeness, FIG3 reports goal achievement curves on Control Suite domains using the "diverse" goal selection scheme. Figure 4 for a description of the visualization. DISCERN learns to reliably control more dimensions of the underlying state than any of the baselines.

Represent the short summary for finding the corresponding scientific article (long)

Unsupervised reinforcement learning method for learning a policy to robustly achieve perceptually specified goals.

Represent the TLDR for finding the corresponding scientific article

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.