Question: Represent the scientific article to find a short TLDR

Question: This paper introduces a probabilistic framework for k-shot image classification. The goal is to generalise from an initial large-scale classification task to a separate task comprising new classes and small numbers of examples. The new approach not only leverages the feature-based representation learned by a neural network from the initial task (representational transfer), but also information about the classes (concept transfer). The concept information is encapsulated in a probabilistic model for the final layer weights of the neural network which acts as a prior for probabilistic k-shot learning. We show that even a simple probabilistic model achieves state-of-the-art on a standard k-shot learning dataset by a large margin. Moreover, it is able to accurately model uncertainty, leading to well calibrated classifiers, and is easily extensible and flexible, unlike many recent approaches to k-shot learning. A child encountering images of helicopters for the first time is able to generalize to instances with radically different appearance from only a handful of labelled examples. This remarkable feat is supported in part by a high-level feature-representation of images acquired from past experience. However, it is likely that information about previously learned concepts, such as aeroplanes and vehicles, is also leveraged (e.g. that sets of features like tails and rotors or objects like pilots/drivers are likely to appear in new images). The goal of this paper is to build machine systems for performing k-shot learning, which leverage both existing feature representations of the inputs and existing class information that have both been honed by learning from large amounts of labelled data. K-shot learning has enjoyed a recent resurgence in the academic community BID6 BID5 BID15 BID12. Current stateof-the-art methods use complex deep learning architectures and claim that learning good features for k-shot learning entails training for k-shot specifically via episodic training that simulates many k-shot tasks. In contrast, this paper proposes a general framework based upon the combination of a deep feature extractor, trained on batch classification, and traditional probabilistic modelling. It subsumes two existing approaches in this vein BID1, and is motivated by similar ideas from multi-task learning BID0. The intuition is that deep learning will learn powerful feature representations, whereas probabilistic inference will transfer top-down conceptual information from old classes. Representational learning is driven by the large number of training examples from the original classes making it amenable to standard deep learning. In contrast, the transfer of conceptual information to the new classes relies on a relatively small number of existing classes and k-shot data points, which means probabilistic inference is appropriate. While generalisation accuracy is often the key objective when training a classifier, calibration is also a fundamental concern in many applications such as decision making for autonomous driving and medicine. Here, calibration refers to the agreement between a classifier's uncertainty and the frequency of its mistakes, which has recently received increased attention. For example, show that the calibration of deep architectures deteriorates as depth and complexity increase. Calibration is closely related to catastrophic forgetting in continual learning. However, to our knowledge, uncertainty has so far been over-looked by the k-shot community even though it is high in this setting. Our basic setup mimics that of the motivating example above: a standard deep convolutional neural network (CNN) is trained on a large labelled training set. This learns a rich representation of images at the top hidden layer of the CNN. Accumulated knowledge about classes is embodied in the top layer softmax weights of the network. This information is extracted by training a probabilistic model on these weights. K-shot learning can then 1) use the representation of images provided by the CNN as input to a new softmax function, 2) learn the new softmax weights by combining prior information about their likely form derived from the original dataset with the k-shot likelihood. The main contributions of our paper are: 1) We propose a probabilistic framework for k-shot learning. It combines deep convolutional features with a probabilistic model that treats the top-level weights of a neural network as data, which can be used to regularize the weights at k-shot time in a principled Bayesian fashion. We show that the framework recovers L 2 -regularised logistic regression, with an automatically determined setting of the regularisation parameter, as a special case.2) We show that our approach achieves state-of-the-art on the miniImageNet dataset by a wide margin of roughly 6% for 1-and 5-shot learning. We further show that architectures with better batch classification accuracy also provide features which generalize better at k-shot time. This finding is contrary to the current belief that episodic training is necessary for good performance and puts the success of recent complex deep learning approaches to k-shot learning into context.3) We show on miniImageNet and CIFAR-100 that our framework achieves a good trade-off between classification accuracy and calibration, and it strikes a good balance between learning new classes and forgetting the old ones. K-shot learning task. We consider the following discriminative k-shot learning task: First, we receive a large dataset DISPLAYFORM0 of images u i and labels y i ∈ {1, . . ., C} that indicate which of the C classes each image belongs to. Second, we receive a small dataset D = {u i, y i} N i=1 of C new classes, y i ∈ {C + 1, C + C}, with k images from each new class. Our goal is to construct a model that can leverage the information in D and D to predict well on unseen images u * from the new classes; the performance is evaluated against ground truth labels y *. Summary. In contrast to several recent k-shot learning approaches that mimic the k-shot learning task by episodic training on simulated k-shot tasks, we propose to use the large dataset D to train a powerful feature extractor on batch classification, which can then be used in conjunction with a simple probabilistic model to perform k-shot learning. In 2003, Bakker & Heskes introduced a general probabilistic framework for multi-task learning with multi-head models, in which all parameters of a generic feature extractor are shared between a set of tasks, and only the weights of the top linear layer (the "heads") are task dependent. In the following, we frame k-shot learning in a similar setting and propose a probabilistic framework for k-shot learning in this vein. Our framework comprises four phases that we refer to as 1) representational learning, 2) concept learning, 3) k-shot learning, and 4) k-shot testing, cf. FIG0. We then show that, for certain modelling assumptions, the obtained method is equivalent/related to regularised logistic regression with a specific choice for the regularisation parameter. We provide a high-level description of the probabilistic framework and present a more detailed derivation in Appendix A. While it might appear overly formal, the ing scheme will be simple and practical, and the probabilistic phrasing will make it extensible and automatic (no free parameters).Feature extractor and representational learning. We first introduce a convolutional neural network (CNN) Φ ϕ as feature extractor whose last hidden layer activations are mapped to two sets of softmax output units corresponding to the C classes in the large dataset D and the C classes in the small dataset D, respectively. These separate mappings are parametrized by weight matrices W for the old classes and W for the new classes. Denoting the output of the final hidden layer as x = Φ ϕ (u), the first softmax units compute p(y n | x n, W) = softmax(W x n) and the second p(y n | x n, W) = softmax(Wx n), cf. FIG0.For representational learning (phase 1) the large dataset D is used to train the CNN Φ ϕ using standard deep learning optimisation approaches. This involves learning the parameters ϕ of the feature extractor up to the last hidden layer, as well as the softmax weights W. The network parameters ϕ are fixed from this point on and shared across later phases. Probabilistic modelling. The next goal is to build a probabilistic method for k-shot prediction that transfers structure from the trained softmax weights W to the new k-shot softmax weights W and combines it with the k-shot training examples. Thus, given a test image u * during k-shot testing (phase 4), we compute its feature representation x * = Φ(u *), and the prediction for the new label y * is found by averaging the softmax outputs over the posterior distribution of the softmax weights given the two datasets, DISPLAYFORM0 To this end, we consider a general class of probabilistic models in which the two sets of softmax weights are generated from shared hyperparameters θ, so that p(W, W, θ) = p(θ)p(W|θ)p(W|θ) as indicated in the graphical model in FIG0. In this way, the large dataset D contains information about θ that in turn constrains the new softmax weights W. We further assume that there is very little uncertainty in W once the large initial training set is observed and so a maximum a posteriori (MAP) estimate, as returned by standard deep learning, suffices. As a consequence of this approximation and the structure of the model, the original data D are not required for the k-shot learning phase. Instead, the weights learned from these data, W, can themselves be treated as observed data, which induce a predictive distribution over the k-shot weights p(W| W MAP) via Bayes' rule. This argument is fully explained in Appendix A. We refer to this step as concept learning (phase 2) and note that all probabilistic modelling happens in the definition of p(W, W, θ), (see Secs. 2.2 and 2.3).During k-shot learning (phase 3) we treat this predictive distribution as our new prior on the weights and again use Bayes' rule to combine it with the softmax likelihood of the k-shot training examples D to obtain a new posterior over the weights that now also incorporates D, DISPLAYFORM0 Finally, we approximate Eq. by its MAP estimate W, so that the integral in Eq. becomes DISPLAYFORM0 The probabilistic model over the weights is key: a good model will transfer useful knowledge that improves performance. However, the usual trade-off between model complexity and learnability is particularly egregious in our setting as the weights W are few and high-dimensional and the number of k-shot samples is small. With an eye on simplicity, we make two simplifying assumptions. First, treating the weights from the hidden layer to the softmax outputs as a vector, we assume independence. Second, we assume the distribution between the weights of old and new classes to be identical, DISPLAYFORM0 After extensive testing, we found that a Gaussian model for the weights strikes the best compromise in the trade-off between complexity and learnability, cf. Sec. 4.2 for a detailed model comparison. Our method. We use a simple Gaussian model p(w|θ) = N (w|µ, Σ) with its conjugate Normal- DISPLAYFORM0 DISPLAYFORM1, and the posterior at k-shot time becomes DISPLAYFORM2 For details see Appendix C.1. For k-shot testing we use the MAP estimates for the weights of the new classes. We found that restricting the covariance matrix to be isotropic, Σ = σ 2 I, performed best at k-shot learning, probably due to the small number of data points to learn from as mentioned above. Relation to logistic regression. Standard logistic regression corresponds to the maximum likelihood (MLE) solution of the softmax likelihood p(y n | x n, W) = softmax(Wx n). Often, L 2 -regularisation on the weights W with inverse regularisation strength 1/C reg is used; the solution to this regularised optimisation problem corresponds to the MAP solution of a model with isotropic Gaussian prior on the weights with zero mean: DISPLAYFORM3 This method is analogous to Eq.. However, the probabilistic framework has several advantages: i) modelling assumptions and approximations are made explicit, ii) it is strictly more general and can incorporate non-zero means µ MAP, whereas standard regularised logistic regression assumes zero mean, and iii) the probabilistic interpretation provides a principled way of choosing the regularisation constant using the trained weights W: C reg = 2σ Embedding methods map the k-shot training and test points into a non-linear space and perform classification by assessing which training points are closest, according to a metric, to the test points. Siamese Networks BID5 train the embedding using a same/different prediction task derived from the original dataset and use a weighted L 1 metric for classification. Matching Networks BID15 construct a set of k-shot learning tasks from the original dataset to train an embedding defined through an attention mechanism that linearly combines training labels weighted by their proximity to test points. More recently, Prototypical Networks BID12 are a streamlined version of Matching Networks in which embedded classes are summarised by their mean in the embedding space. These embedding methods learn representations for k-shot learning, but do not directly leverage concept transfer. Amortised optimisation methods ) also simulate related k-shot learning tasks from the initial dataset, but instead train a second network to initialise and optimise a CNN to perform accurate classification on these small datasets. This method can then be applied for new k-shot tasks. Importantly, both embedding and amortised inference methods improve when the system is trained for a specific k-shot task: to perform well in 5-shot learning, training is carried out with episodes containing 5 examples in each class. The general statement appears to be that training specifically for k-shot learning is essential for building features which generalise well at k-shot testing time. The approach proposed in this paper is more flexible; it is not tailored for a specific k and, thus, does not require retraining when switching, e.g., from 5-shot to 10-shot learning. Moreover, BID12 find that using a larger number of k-shot classes for the training episodes (e.g., train with 20 k-shot classes per episode when testing on only 5 new k-shot classes) can be beneficial, and they choose this number by cross-validation on a validation-set. This is in alignment with our finding that training with more data and more classes improves performance at k-shot time. Deep probabilistic methods include the approach developed in this paper. The methods in this family are not unique to deep learning, and the idea of treating weights as data from which to transfer has been widely applied in multi-task learning BID0. The work most closely related to our own is not an approach to k-shot learning per se, but rather a method for training CNNs with highly imbalanced classes. It is similar in that it trains a form of Gaussian mixture model over the final layer weights using MAP inference that regularises learning. BID1 propose an elegant approach to k-shot learning that is an instance of the framework described here: a Gaussian model is fit to the weights with MAP inference. The evaluation is promising, but preliminary. One of the goals of this paper is to provide a comprehensive evaluation. While not using a probabilistic approach, develop a method for k-shot learning that trains a recognition model to amortise MAP inference for the softmax weights which can then be used at k-shot learning time. While this method trains the mapping from activation to weights jointly with the classifier, and thus does not learn from the weights per se, it does exploit the structure in the weights for k-shot learning. The code used to produce the following experiments will be made available after review. Dataset. miniImageNet has become a standard testbed for k-shot learning and is derived from the ImageNet ILSVRC12 dataset BID9 by extracting 100 out of the 1000 classes. Each class contains 600 images downscaled to 84 × 84 pixels. We use the 100 classes (64 train, 16 validation, 20 test) proposed by. As our approach does not require a validation set, we use both the training and validation data for the representational learning. Representational learning. We employ standard CNNs that are inspired by ResNet-34 and VGG BID11 for the representational learning on the C base classes, cf. Phase 1 in Sec. 2.1. These trained networks provide both W MAP and the fixed feature representation Φ ϕ for the k-shot learning and testing. We employed standard data augmentation from ImageNet for the representational learning but highlight that no data augmentation was used during the k-shot training and testing. For details on the architecture, training, and data augmentation see Appendix D.4. t-SNE embeddings BID14 of the learned last layer weights show sensible clusters, which highlights the structure exploited by the probabilistic model, see Appendix E.1.Baselines and competing methods. We compare against several baselines as well as recent stateof-the-art methods mentioned in Sec. 3. The baselines are computed on the features x = Φ φ (u) from the last hidden layer of the trained CNN: (i) Nearest Neighbours with cosine distance and (ii) regularized logistic regression with regularisation constant set either by cross-validation or (iii) using the variance of the weights, C = 2σ Testing protocol. We evaluate the methods on 600 random k-shot tasks by randomly sampling 5 classes from the 20 test classes and perform 5-way k-shot learning. Following BID12, we use 15 randomly selected images per class for k-shot testing to compute accuracies and calibration. Overall k-shot performance. We report performance on the miniImageNet dataset in Tab. 1 and Figs. 2 and 3. The best method uses as feature extractor a modified ResNet-34 with 256 features, DISPLAYFORM0 ResNet-34 + Isotropic Gaussian (ours) 56.3 ± 0.4% 73.9 ± 0.3% 78.5 ± 0.3% Matching Networks (reimplemented, 1-shot) 46.8 ± 0.5% --Matching Networks (reimplemented, 5-shot) -62.7 ± 0.5% -Meta-Learner LSTM 43.4 ± 0.8% 60.6 ± 0.7% -Prototypical Nets (1-shot) BID12 49.4 ± 0.8% 65.4 ± 0.7% -Prototypical Nets (5-shot) BID12 45.1 ± 0.8% 68.2 ± 0.7% - Table 1: Accuracy on 5-way classification on miniImageNet. Our best method, an isotropic Gaussian model using ResNet-34 features consistently outperforms all competing methods by a wide margin. trained with all 600 examples per training class, and a simple isotropic Gaussian model on the weights for concept learning. Despite its simplicity, our method achieves state-of-the-art and beats prototypical networks by a wide margin of about 6%. The baseline methods using the same feature extractor are also state-of-the-art compared to prototypical networks and both logistic regressions show comparable accuracy to our methods except for on 1-shot learning. In terms of log-likelihoods, Log Reg (C = 2σ 2 W) fares slightly better, whereas Log Reg (cv) is much worse. Deeper features lead to better k-shot learning. We investigate the influence of different feature extractors of increasing complexity on performance in Fig. 3: i) a VGG style network (500 train images per class), ii) a ResNet-34 (500 examples per class), and iii) a ResNet-34 (all 600 examples per class). We find that the complexity of the feature extractor as well as training set size consistently correlate with the accuracy at k-shot time. For instance, on 5-shot, Gauss (iso) achieves 65% accuracy with a VGG network and 74% with a ResNet trained with all available data, a significant increase of almost 10%. Moreover, Gauss (iso) outperforms Log Reg (C = 2σ 2 W) on 1-shot learning across models, and performs similarly on 5-and 10-shot. We attribute the difference to the former's ability to also model the mean of the Gaussian, whereas logistic regression assumes a zero mean. Importantly, this implies that training specifically for k-shot learning is not necessary for achieving high generalisation performance on this k-shot problem. On the contrary, training a powerful deep feature extractor on batch classification using all of the available training data, then building a simple probabilistic model using the learned features and weights achieves state-of-the-art. Recent models that use episodic training cannot leverage such deep feature extractors as for them the depth of the model is limited by the nature of training itself. The reference baseline in the k-shot learning literature is nearest neighbours, which performs on par with Gauss (iso) on 1-shot learning but is outperformed by all methods on 5-and 10-shot. This is evidence that building a simple classifier on top of the learned features works significantly better for k-shot learning than nearest neighbours. Calibration. A classifier is said to be calibrated when the probability it predicts for belonging to a given class is on par with the probability of it being the correct prediction. In other words, when ) to bright (C = 10). ECE plots are provided in Appendix E.3. examples for which it predicts a probability p of belonging to a given class are correctly classified for a fraction p of the examples. A calibration curve visualises the proportion of examples correctly classified as a function of their predicted probability; a perfectly calibrated classifier should in a diagonal line. Following, we consider the log likelihood on the k-shot test examples as well as Expected Calibration Error (ECE) as summary measures of calibration. ECE can be interpreted as the weighted average of the distance of the calibration curve to the diagonal. We find that Log Reg (C = 2σ 2 W) and Gauss (iso) provide better accuracy and calibration than Log Reg (cross-validation), cf. FIG3. The difference in calibration quality for different regularisations of logistic regression highlights the importance of choosing the right constant, as we discuss now. Choice of the regularisation constant for logistic regression. The so far suggest that training a simple linear model such as regularised logistic regression might be sufficient to perform well in k-shot learning. However, while the accuracy at k-shot time does not vary dramatically as the regularisation constant changes, the calibration does, and jointly maximizing both quantities is not possible, cf. the first two plots of Fig. 4. The standard (frequentist) method to tune this constant is cross validation, which is not applicable in the 1-shot setting, and suffers from lack of data in 5-and 10-shot. Contrary, our probabilistic framework provides a principled way of selecting this regularisation parameter by transfer from the training weights: Log Reg (C = 2σ 2 W) strikes a good balance between accuracy and log-likelihood. The third plot in Fig. 4 reports log-likelihood as a function of accuracy and provides further visualisation of the achieved trade-off between accuracy and calibration for Log Reg (C = 2σ 2 W), as well as the failure of Log Reg (cross-validation) to achieve a good compromise in 5-and 10-shot. Evaluation in an online setting. We also briefly consider the online setting, in which we jointly test on 80 old and 5 new classes, for which catastrophic forgetting BID2 ) is a well known problem. During k-shot learning and testing we employ a softmax which includes both the new and the old weights ing in a total of 85 weight vectors. We utilise ResNet-34 trained on 500 images per class to retain 100 test images on the old classes. While the k-shot weights were modelled probabilistically, we use the MAP estimate W ) only lose a couple of percent on the accuracy of the old classes, and perform well on the new classes, striking a good trade-off between forgetting and learning at k-shot time. For unregularised (MLE) logistic regression, the new weights completely dominate the old ones, highlighting that the right regularisation is important. Yet, cross-validation in this setting is often very challenging. When training Logistic Regression without including the old weights ("only new"), the new weights are dominated by the old ones and fail to learn the new classes, making training in the presence of the old weights an essential component for online learning. We performed an extensive comparison between different probabilistic models of the weights using different inference procedures, which we present in Appendix E.4. We report on the CIFAR-100 dataset on (i) Gaussian, (ii) mixture of Gaussians, and (iii) Laplace, all with either MAP estimation or Hybrid Monte Carlo sampling. We found that the simple Gaussian model is on par with or outperforms other methods at k-shot time, which we attribute to it striking a good balance between choosing a complex model, which may better fit the weights, and statistical efficiency, as the number of weights C (80 in our case) is often smaller than the dimensionality of the feature representation (256 in our case), cf. Sec. 2. This finding is supported by computing the log-likelihood of held out training weights under such model, with the Gaussian model performing best. Experiments using Hybrid Monte Carlo sampling for k-shot learning returned very similar performance to MAP estimation and at a much higher computational cost, due to the difficulty of performing sampling in such a high dimensional parameter space. Our recommendation is that practitioners should use simple models and employ simple inference schemes to estimate all free parameters thereby avoiding expending valuable data on validation sets. We present a probabilistic framework for k-shot learning that exploits the powerful features and class information learned by a neural network on a large training dataset. Probabilistic models are then used to transfer information in the network weights to new classes. Experiments on miniImageNet using a simple Gaussian model within our framework achieve state-of-the-art for 1-shot and 5-shot learning by a wide margin, and at the same time return well calibrated predictions. This finding is contrary to the current belief that episodic training is necessary to learn good k-shot features and puts the success of recent complex deep learning approaches to k-shot learning into context. The new approach is flexible and extensible, being applicable to general discriminative models and kshot learning paradigms. For example, preliminary on online k-shot learning indicate that the probabilistic framework mitigates catastrophic forgetting by automatically balancing performance on the new and old classes. The Gaussian model is closely related to regularised logistic regression, but provides a principled and fully automatic way to regularise. This is particularly important in k-shot learning, as it is a low-data regime, in which cross-validation performs poorly and where it is important to train on all available data, rather than using validation sets. Appendix to "Discriminative k-shot learning using probabilistic models"A DETAILS ON THE DERIVATION AND APPROXIMATIONS FROM SEC. 2.1As stated in the main text, the probabilistic k-shot learning approach comprises four phases mirroring the dataflow:Phase 1: Representational learning. The large dataset D is used to train the CNN Φ ϕ using standard deep learning optimisation approaches. This involves learning both the parameters ϕ of the feature extractor up to the last hidden layer, as well as the softmax weights W. The network parameters ϕ are fixed from this point on and shared across phases. This is a standard setup for multitask learning and in the present case it ensures that the features derived from the representational learning can be leveraged for k-shot learning. Phase 2: Concept learning. The softmax weights W are effectively used as data for concept learning by training a probabilistic model that detects structure in these weights which can be transferred for k-shot learning. This approach will be justified in the next section. For the moment, we consider a general class of probabilistic models in which the two sets of weights are generated from shared hyperparameters θ, so that p(W, W, θ) = p(θ)p(W|θ)p(W|θ) (see FIG0).Phases 3 and 4: k-shot learning and testing. Probabilistic k-shot learning leverages the learned representation Φ ϕ from phase 1 and the probabilistic model p(W, W, θ) from phase 2 to build a (posterior) predictive model for unseen new examples using examples from the small dataset D. Given the dataflow and the assumed probabilistic model in FIG0, a completely probabilistic approach would involve the following steps. In the concept learning phase, the initial dataset would be used to form the posterior distribution over the concept hyperparameters p(θ | D). The k-shot learning phase combines the information about the new weights provided by D with the information in the k-shot dataset D to form the posterior distribution DISPLAYFORM0 To see this, notice that DISPLAYFORM1 The graphical model in FIG0 entails that D is conditionally independent from D given W, such that DISPLAYFORM2 We recover Eq. by adding p(D) to the constant of proportionality. Inference in this model is generally intractable and requires approximations. The main challenge is computing the posterior distribution over the hyper-parameters given the initial dataset. However, progress can be made if we assume that the posterior distribution over the weights can be well approximated by the MAP value p(W | D) ≈ δ(W − W MAP). This is an arguably justifiable assumption as the initial dataset is large and so the posterior will concentrate on narrow modes (with similar predictive performance). In this case p(θ | D) ≈ p(θ | W MAP) and, due to the structure of the probabilistic model, all instances of D in Eq. and Eq. can be replaced by the analogous expressions involving W. This greatly simplifies the learning pipeline as the probabilistic modelling only needs to have access to the weights returned by representational learning. Remaining intractabilities involve only a small number of data points D and can be handled using standard approximate inference tools. The following summarizes the approximations and computational steps for each phase of training. Phase 4: k-shot testing. Approximate inference is used to compute p(y DISPLAYFORM0 In this section we briefly discuss different inference methods for the probabilistic models. In the main text we only considered MAP inference as we found that other more complicated inference schemes do not yield a practical benefit. However, in Appendix E.4 we provide a detailed model comparison, in which we also consider other approximate inference methods. In all cases the gradients of the densities w.r.t. W can be computed, enabling MAP inference in the k-shot learning phase to be efficiently performed via gradient-based optimisation using L-BFGS BID24 . Alternatively, Markov Chain Monte Carlo (MCMC) sampling can be performed to approximate the associated integral, see Eq.. Due to the high dimensionality of the space and as gradients are available, we employ Hybrid Monte Carlo (HMC) BID27 sampling in the form of the recently proposed NUTS sampler that automatically tunes the HMC parameters (step size and number of leapfrog steps) BID20. For the GMMs we employed pymc3 BID29 to perform MAP inference. As discussed in Sec. 2.1, we specify our model through p(W, W, θ) thus defining p(W | W M AP) in Eq.. This section analyses different priors on the weights: (i) Gaussian models, (ii) Gaussian mixture models, and (iii) Laplace distribution. In the main paper, we only use a Gaussian model with MAP inference, as we saw no significant advantage in using other, more complex models. However, we provide an extensive comparison of the different models in Appendix E.4. Possibly the simplest approach consists of modelling p(W | W) as a Gaussian distribution: DISPLAYFORM0 Details for this section can be found in BID26. The normal-inverse-Wishart distribution for µ and Σ is a conjugate prior for the Gaussian, which allows for the posterior to be written in closed form. More precisely, DISPLAYFORM1 where Z is the normalising constant. The posterior p(µ, Σ | W) also follows a normal-inverse-Wishart distribution: DISPLAYFORM2 where DISPLAYFORM3 and S is the sample covariance of W.For this model, we can integrate in closed form, which in the following multivariate Student t-distribution: DISPLAYFORM4.As with other approaches, one can also compute the MAP solutions for the mean µ MAP and covariance DISPLAYFORM5 For both the analytic posterior and the MAP approximation, p(W | W) depends on the hyperparameters of the normal-inverse-Wishart distribution: µ 0, ν 0, κ 0 and Λ 0. There are different ways to choose these hyperparameters. One way would be by optimising the log probability of held out training weights, see Appendix E.4 for a brief discussion. In practise, it is common to choose uninformative or data dependent priors as discussed by Murphy (2012, Chapter 4). A Gaussian mixture model can potentially leverage cluster structure in the weights (animal classes might have similar weights, for example). This is related to the tree-based prior proposed in. MAP inference is performed because exact inference is intractable. Similarly to the Gaussian case, different structures for the covariance of each cluster were tested. In our experiments, we fit the parameters of the GMM via maximum likelihood using the EM algorithm. GMM consists on modelling p(W | W) as a mixture of Gaussians with S components: DISPLAYFORM0 where S s=1 π s = 1. In this work, we only compute the MAP mean and covariance for each of the clusters, as opposed to averaging over the parameters of the mixture. The ing posterior is DISPLAYFORM1 The components of the mixture are fit in two ways. For CIFAR-100, the classes are grouped into 20 superclasses, each containing 5 of the 100 classes. One option is therefore to initialize 20 components, each fit with the data points in the corresponding superclass. For each such individual Gaussian, the MAP inference method presented in the previous section can be used. In order to increase the number of weight examples in each superclass, we merge the original superclasses into 9 larger superclasses. The merging of the superclasses is the following:• Aquatic mammals + fish• flowers + fruit and vegetables + trees• insects + non-insect invertebrates + reptiles• medium-sized mammals + small mammals• large carnivores + large omnivores and herbivores• people• large man-made outdoor things + large natural outdoor things• food containers + household electrical devices + household furniture• Vehicles 1 + Vehicles 2.The parameters of the mixture can also be fit using maximum likelihood with EM. We use the implementation of EM in scikit-learn. Both 3 and 10 clusters are considered in CIFAR-100. Weight log-likelihoods under this model and k-shot performance can be found in Appendix E.4.Note that, similarly to the Gaussian model, we consider isotropic, diagonal or full covariance models for the covariance matrices. Sparsity is an attractive feature which could be helpful for modelling the weights. Indeed, it is reasonable to assume that each class uses a set of characteristic features which drive classification accuracy, while others are irrelevant. Sparse models would then provide sensible regularization. As such, we consider a product of independent Laplace distribution. Sec. 2.3 highlights the relation between a Gaussian prior on the weights and L 2 regularised logistic regression. One can similarly show that the Laplace prior is related to L 1 regularised logistic regression, which is well known for encouraging sparse weight vectors. We consider a prior which factors along the feature dimensions: DISPLAYFORM0 where the product over j is along the feature dimensions and the sum over i is across the classes. We fit the parameters µ and λ via maximum likelihood: DISPLAYFORM1 An isotropic Laplace model with mean µ and scale λ is also considered: DISPLAYFORM2 where DISPLAYFORM3 To construct miniImageNet we use the same classes as initially proposed by and used in BID12, which is split into 64 training classes (cf. Tab. 2), 16 validation classes (cf. Tab. 3), and 20 test classes (cf. Tab. 4). We will make a full list of image files available. As we do not require a validation set, we combine the training and validation set to form an extended training set. We extract 600 images per class from the ImageNet 2012 Challange dataset BID23, scale the shorter side to 84 pixels and then centrally crop to 84 × 84 pixels, that is, we preserve the original aspect ratio of the image content. We use these coloured 84 × 84 × 3 images as input for representational and k-shot learning and testing. In order to train very deep models, such as a ResNet, we need to perform data augmentation as is the case when training full ImageNet. We use the following standard data augmentation from ImageNet that we adapt to the size of the input images:• random horizontal flipping • randomly paste image into 100 × 100 frame and cut out central 84 × 84 pixels • randomly change brightness, contrast, saturation and lightingWe highlight that we do not perform any data augmentation for the k-shot learning and k-shot testing but use the original 84 × 84 colour images as input to the feature extractor. CIFAR-100 consists of 100 classes each with 500 training and 100 test images of size 32 × 32.The classes are grouped into 20 superclasses with 5 classes each. For example, the superclass "fish" contains the classes aquarium fish, flatfish, ray, shark, and trout. Unless otherwise stated, we used a random split into 80 base classes and 20 k-shot learning classes. For k-shot learning and testing, we split the 100 classes into 80 base classes used for network training and 20 k-shot learning classes., 2, 3, 4, 5, 6, 7, 9, 10, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 27, 28, 32, 34, 35, 36, 38, 40, 42, 43, 44, 45, 46, 48, 49, 50, 51, 52, 53, 54, 55, 56, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 69, 70, 73, 74, 75, 76, 77, 78, 79, 80, 82, 83, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99 ] classes_heldout = We provide an exhaustive comparison of different probabilistic models for this k-shot learning task in Appendix E.4. DISPLAYFORM0 The network architecture is inspired by the ResNet-34 architecture for ImageNet ) that uses convolution blocks, with two convolutions each, that are bridged by skip connections. As a base, we utilise the example code 2 provided by tensorpack (https://github.com/ppwwyyxx/tensorpack), a neural network training library built on top of tensorflow (Martín BID25 . We adapt the number of features as well as the size of the last fully connected layer to account for the smaller number of training samples and training classes. The final architecture is detailed in Tab. 5. n03400231 frying pan, frypan, skillet n02108551 Tibetan mastiff n02687172 aircraft carrier, carrier, flattop, attack aircraft carrier n04296562 stage n13133613 ear, spike, capitulum n02165456 ladybug, ladybeetle, lady beetle, ladybird, ladybird beetle n03337140 file, file cabinet, filing cabinet n02966193 carousel, carrousel, merry-go-round, roundabout, whirligig n02074367 dugong, Dugong dugon n02105505 komondor n04389033 tank, army tank, armored combat vehicle, armoured combat vehicle n09246464 cliff, drop, drop-off n03924679 photocopier n03527444 holster n04612504 yawl n01749939 green mamba n04251144 snorkel n03347037 fire screen, fireguard n04067472 reel n03998194 prayer rug, prayer mat n13054560 bolete n02747177 ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin, trash barrel, trash bin n04435653 tile roof n02108089 boxer n03908618 pencil box, pencil case n01770081 harvestman, daddy longlegs, Phalangium opilio n03676483 lipstick, lip rouge n03220513 dome n04515003 upright, upright piano n04258138 solar dish, solar collector, solar furnace n04509417 unicycle, monocycle n01704323 triceratops n04443257 tobacco shop, tobacconist shop, tobacconist n02089867 Walker hound, Walker foxhound n01910747 jellyfish n02111277 Newfoundland, Newfoundland dog n04243546 slot, one-armed bandit n01558993 robin, American robin, Turdus migratorius n03047690 clog, geta, patten, sabot n03854065 organ, pipe organ n03476684 hair slide n02113712 miniature poodle n07747607 orange n03838899 oboe, hautboy, hautbois n07584110 consomme n02795169 barrel, cask n03017168 chime, bell, gong n04275548 spider web, spider's web n04604644 worm fence, snake fence, snake-rail fence, Virginia fence n02606052 rock beauty, Holocanthus tricolor n01843383 toucan n02457408 three-toed sloth, ai, Bradypus tridactylus n03062245 cocktail shaker n03207743 dishrag, dishcloth n02108915 French bulldog n06794110 street sign n02823428 beer bottle n03888605 parallel bars, bars n04596742 wok n02091831 Saluki, gazelle hound n02101006 Gordon setter n02120079 Arctic fox, white fox, Alopex lagopus n01532829 house finch, linnet, Carpodacus mexicanus n07697537 hotdog, hot dog, red hot Table 2 : Training classes for miniImageNet as proposed by n03075370 combination lock n02971356 carton n03980874 poncho n02114548 white wolf, Arctic wolf, Canis lupus tundrarum n03535780 horizontal bar, high bar n03584254 iPod n02981792 catamaran n03417042 garbage truck, dustcart n03770439 miniskirt, mini n02091244 Ibizan hound, Ibizan Podenco n02174001 rhinoceros beetle n09256479 coral reef n02950826 cannon n01855672 goose n02138441 meerkat, mierkat n03773504 missiles Table 3 : Validation classes for miniImageNet as proposed by n02116738 African hunting dog, hyena dog, Cape hunting dog, Lycaon pictus n02110063 malamute, malemute, Alaskan malamute n02443484 black-footed ferret, ferret, Mustela nigripes n03146219 cuirass n03775546 mixing bowl n03544143 hourglass n04149813 scoreboard n03127925 crate n04418357 theater curtain, theatre curtain n02099601 golden retriever n02219486 ant, emmet, pismire n03272010 electric guitar n04146614 school bus n02129165 lion, king of beasts, Panthera leo n04522168 vase n07613480 trifle n02871525 bookshop, bookstore, bookstall n01981276 king crab, Alaska crab, Alaskan king crab, Alaska king crab, Paralithodes camtschatica n02110341 dalmatian, coach dog, carriage dog n01930112 nematode, nematode worm, roundworm Table 5 : Network architecture. All unnamed layers are 2D convolutions with stated kernel size and padding SAME; the output of the shaded layer corresponds to Φ ϕ (u), the feature space representation of the image u, which is used as input for probabilistic k-shot learning. The network is trained using a decaying learning rate schedule and momentum SGD and is implemented in tensorpack using tensorflow. VGG-style Network for CIFAR-100 Output size Layers DISPLAYFORM0 FullyConnected, ELU C FullyConnected, SoftMax VGG-style Network for miniImageNet Output size Layers DISPLAYFORM1 FullyConnected, ELU C FullyConnected, SoftMax Table 6: Network architectures. All 2D convolutions have kernel size 3 × 3 and padding SAME; max-pooling is performed with stride 2. The output of the shaded layer corresponds to Φ ϕ (u), the feature space representation of the image u, which is used as input for probabilistic k-shot learningThe network architecture was inspired by the VGG networks , but does not employ batch normalisation. To speed up training, we employ exponential linear units (ELUs), which have been reported to lead to faster convergence as compared to ordinary ReLUs BID17 To regularise the networks, we employ dropout BID32 and regularisation of the weights in the fully connected layers. The networks are trained with the ADAM optimiser BID22 with decaying learning rate. The network is implemented in tensorpack using tensorflow. Figure 6: t-SNE embedding of the CIFAR-100 weights W trained using a VGG style architecture. The points are coloured according to their respective superclass. The colouring by superclass makes the structure in the weights evident, as t-SNE overall recovers the structure in the dataset. For instance, oak tree, palm tree, willow tree and pine tree form a cluster on the bottom right. This structure motivates our approach, as the training weights contain information which may be useful at k-shot time, for instance given a few example from chestnut trees. Structure is still present and we observe meaningful patterns, even though the classes in miniImageNet are more unique than in CIFAR-100. For instance, goose, house finch, toucan, Arctic fox, green mamba and other animals are clustered on the top, with birds close to each other. Examples of other small clusters include poncho and miniskirt, or organ and oboe. For readability, not all class names are plotted. We provide t-SNE embeddings for the weights of a VGG network trained in CIFAR-100 and a ResNet-34 trained on miniImageNet. A structure in the weights is apparent and provides motivation for our framework. The can be seen in FIG6 E.2 EXTENDED ON miniIMAGENET Fig. 8 provides extended on k-shot learning for the miniImageNet dataset for different network architectures. We investigate the influence of different feature extractors of increasing complexity and training data size on performance on: i) a VGG style network trained on 500 images per class, ii) a ResNet-34 trained on 500 examples per class, and iii) a ResNet-34 trained on all 600 examples per class. ResNet-34 trained with 500 images per class; bottom: a VGG style network trained with 500 images per class. We highlight that for all three architectures the order of the different methods as well as the main messages are the same. However, the general performance in terms of accuracy and calibration differ between the architectures. The more complex architecture trained on most images performs best in terms of accuracy, indicating that it learns better features for k-shot learning. Both ResNets behave very similarly on calibration whereas the VGG-style network performs better (lower ECE and higher log likelihood as well as more diagonal calibration curve). This is in line with observations by that calibration of deep architectures gets worse as depth and complexity increase. ) to bright (C = 10). In addition to Fig. 4 we also provide for calibration in terms of ECE (lower is better), which are consistent with log likelihoods (higher is better): The Bayesian inspired choice of the regularisation parameter strikes a good balance between accuracy and calibration and consistently outperforms cross-validated choice of the parameter. Optimised value of mean negative log probability Gauss (iso) −175.9 ± 0.3 Gauss (MAP prior) −196.1 ± 0.5 Gauss (integr. prior) −200.6 ± 0.4 GMM 3-means (iso) −179.0 ± 0.3 GMM 3-means (diag) −181.2 ± 0.3 GMM 10-means iso −181.6 ± 0.4 GMM 10-means (diag) −181.6 ± 0.4 Laplace (iso) −173.8 ± 0.4 Laplace (diag) 0 −100 −200 −176.6 ± 0.5 Table 9: Held-out log probabilities on random 70/10-splits of the training weights for the different models on CIFAR-100. Values are averaged over 50 splits. Figure 10: Results on CIFAR-100 for VGG style architecture. We report accuracy, log-likelihood and calibration for the methods and inference procedures presented in Tab. 8. With the exception of GMM (10, iso) and Laplace, all methods are similar terms of accuracy and log-likelihood. Gauss (integr. prior) HMC and Gauss (MAP) HMC are slightly better calibrated than our proposed Gauss (MAP) iso, but require significantly more computation for the sampling procedure.behave very similar but that multivariate Gaussian models generally outperform other models. We attribute the good performance of the simpler models to the small number of data points (C − 10 = 70 training weights) and the high dimensionality of the space, which entail that fitting even simple models is difficult. Thus, more complicated models cannot improve over them.k-shot performance in CIFAR-100. Accuracies are measured on a 5-way classification task on the k-shot classes for k ∈ {1, 5, 10}. Results were averaged two-fold: (i) 20 random splits of the 5 k-shot classes; (ii) 10 repetitions of each split with different k-shot training examples. Among our models, no statistically significant difference in accuracy is observed, with the exception of Laplace MAP and GMM (iso), which consistently underperforms. These findings are consistent in terms of log-likelihoods, see the first and second plots in FIG0. Finally, our methods are generally well calibrated, with Gaussian models generally better than Laplace models. Moreover, all methods (with the exception of Laplace and GMM (10, iso) have low ECE and high accuracy, see the third and fourth plots of FIG0. While Gauss (integr. prior) HMC and Gauss (MAP) HMC are sightly better calibrated than our proposed method in the main paper, Gauss (MAP) iso, we believe the gain in calibration is not worth the significant increase in computational resources needed for the sampling procedure. Interestingly, both GMM approaches are not able to outperform the other, simpler models. This is in line with the previous observation that the simpler models are better able to explain the weights. Again, we attribute this inability of mixture models to use their larger expressivity/capacity to the small number of data points and the high-dimensionality of weight-space which means learning even simple models is difficult. These observations suggest that the use of mixture models in this type of k-shot learning framework is not beneficial and is in contrast to the approach of, who employ a tree-structured mixture model. The authors show compare a model in which the assignments to the superclasses in the tree are optimized over against a model with a naive initialisation of the superclass assignments, and show that the first outperforms the second. However, they do not compare against a simpler baseline, e.g., a single Gaussian model. Overall, we observe that there is no significant benefit of more complex methods over the simple isotropic Gaussian, either in terms of accuracy, log-likelihood or calibration. Thus, our recommendation is that practitioners should use simple models and employ simple inference schemes to estimate all free parameters thereby avoiding expending valuable data on validation sets

represent the short summary for finding the corresponding scientific article (long)

This paper introduces a probabilistic framework for k-shot image classification that achieves state-of-the-art results

Represent the short summary for finding the corresponding scientific article (long)!

We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.


Question: represent the text to find a very short summary (1-2 sentences max)

Question: Building on the recent successes of distributed training of RL agents, in this paper we investigate the training of RNN-based RL agents from distributed prioritized experience replay. We study the effects of parameter lag ing in representational drift and recurrent state staleness and empirically derive an improved training strategy. Using a single network architecture and fixed set of hyper-parameters, the ing agent, Recurrent Replay Distributed DQN, quadruples the previous state of the art on Atari-57, and matches the state of the art on DMLab-30. It is the first agent to exceed human-level performance in 52 of the 57 Atari games. Reinforcement Learning (RL) has seen a rejuvenation of research interest recently due to repeated successes in solving challenging problems such as reaching human-level play on Atari 2600 games BID15, beating the world champion in the game of Go BID21, and playing competitive 5-player DOTA BID18. The earliest of these successes leveraged experience replay for data efficiency and stacked a fixed number of consecutive frames to overcome the partial observability in Atari 2600 games. However, with progress towards increasingly difficult, partially observable domains, the need for more advanced memory-based representations increases, necessitating more principled solutions such as recurrent neural networks (RNNs). The use of LSTMs BID8 within RL has been widely adopted to overcome partial observability BID5 BID16 BID3 BID4.In this paper we investigate the training of RNNs with experience replay. We have three primary contributions. First, we demonstrate the effect of experience replay on parameter lag, leading to representational drift and recurrent state staleness. This is potentially exacerbated in the distributed training setting, and ultimately in diminished training stability and performance. Second, we perform an empirical study into the effects of several approaches to RNN training with experience replay, mitigating the aforementioned effects. Third, we present an agent that integrates these findings to achieve significant advances in the state of the art on Atari-57 BID1 and matches the state of the art on DMLab-30 BID0. To the best of our knowledge, our agent, Recurrent Replay Distributed DQN (R2D2), is the first to achieve this using a single network architecture and fixed set of hyper-parameters. Our work is set within the Reinforcement Learning (RL) framework BID23, in which an agent interacts with an environment to maximize the sum of discounted, γ ∈, rewards. We model the environment as a Partially Observable Markov Decision Process (POMDP) given by the tuple (S, A, T, R, Ω, O) BID17 BID10 BID12. The underlying Markov Decision Process (MDP) is defined by (S, A, T, R), where S is the set of states, A the set of actions, T a transition function mapping state-actions to probability distributions over next states, and R: S × A → R is the reward function. Finally, Ω gives the set of observations 2.2 DISTRIBUTED REINFORCEMENT LEARNING Recent advances in reinforcement learning have achieved significantly improved performance by leveraging distributed training architectures which separate learning from acting, collecting data from many actors running in parallel on separate environment instances BID3 BID4 BID18 a; BID11.Distributed replay allows the Ape-X agent to decouple learning from acting, with actors feeding experience into the distributed replay buffer and the learner receiving (randomized) training batches from it. In addition to distributed replay with prioritized sampling, Ape-X uses n-step return targets BID22, the double Q-learning algorithm, the dueling DQN network architecture BID26 and 4-framestacking. Ape-X achieved state-of-the-art performance on Atari-57, significantly out-performing the best single-actor algorithms. It has also been used in continuous control domains and again showed state-of-the-art , further demonstrating the performance benefits of distributed training in RL.IMPALA BID3 ) is a distributed reinforcement learning architecture which uses a first-in-first-out queue with a novel off-policy correction algorithm called V-trace, to learn sequentially from the stream of experience generated by a large number of independent actors. IMPALA stores sequences of transitions along with an initial recurrent state in the experience queue, and since experience is trained on exactly once, this data generally stays very close to the learner parameters. BID3 showed that IMPALA could achieve strong performance in the Atari-57 and DMLab-30 benchmark suites, and furthermore was able to use a single large network to learn all tasks in a benchmark simultaneously while maintaining human-level performance. We propose a new agent, the Recurrent Replay Distributed DQN (R2D2), and use it to study the interplay between recurrent state, experience replay, and distributed training. R2D2 is most similar to Ape-X, built upon prioritized distributed replay and n-step double Q-learning (with n = 5), generating experience by a large number of actors (typically 256) and learning from batches of replayed experience by a single learner. Like Ape-X, we use the dueling network architecture of BID26, but provide an LSTM layer after the convolutional stack, similarly to BID4.Instead of regular (s, a, r, s) transition tuples, we store fixed-length (m = 80) sequences of (s, a, r) in replay, with adjacent sequences overlapping each other by 40 time steps, and never crossing episode boundaries. When training, we unroll both online and target networks BID15 on the same sequence of states to generate value estimates and targets. We leave details of our exact treatment of recurrent states in replay for the next sections. Like Ape-X, we use 4-frame-stacks and the full 18-action set when training on Atari. On DMLab, we use single RGB frames as observations, and the same action set discretization as BID7. Following the modified Ape-X version in BID19, we do not clip rewards, but instead use an invertible value function rescaling of the form h(x) = sign(x)(|x| + 1 − 1) + x which in the following n-step targets for the Q-value function: DISPLAYFORM0 Here, θ − denotes the target network parameters which are copied from the online network parameters θ every 2500 learner steps. Our replay prioritization differs from that of Ape-X in that we use a mixture of max and mean absolute n-step TD-errors δ i over the sequence: p = η max i δ i + (1 − η)δ. We set η and the priority exponent to 0.9. This more aggressive scheme is motivated by our observation that averaging over long sequences tends to wash out large errors, thereby compressing the range of priorities and limiting the ability of prioritization to pick out useful experience. Finally, compared to Ape-X, we used the slightly higher discount of γ = 0.997, and disabled the loss-of-life-as-episode-end heuristic that has been used in Atari agents in some of the work since BID15. A full list of hyper-parameters is provided in the Appendix. We train the R2D2 agent with a single GPU-based learner, performing approximately 5 network updates per second (each update on a mini-batch of 64 length-80 sequences), and each actor performing ∼ 260 environment steps per second on Atari (∼ 130 per second on DMLab). In order to achieve good performance in a partially observed environment, an RL agent requires a state representation that encodes information about its state-action trajectory in addition to its current observation. The most common way to achieve this is by using an RNN, typically an LSTM BID8, as part of the agent's state encoding. To train an RNN from replay and enable it to learn meaningful long-term dependencies, whole state-action trajectories need to be stored in replay and used for training the network. BID5 compared two strategies of training an LSTM from replayed experience:• Using a zero start state to initialize the network at the beginning of sampled sequences.• Replaying whole episode trajectories. The zero start state strategy's appeal lies in its simplicity, and it allows independent decorrelated sampling of relatively short sequences, which is important for robust optimization of a neural network. On the other hand, it forces the RNN to learn to recover meaningful predictions from an atypical initial recurrent state ('initial recurrent state mismatch'), which may limit its ability to fully rely on its recurrent state and learn to exploit long temporal correlations. The second strategy on the other hand avoids the problem of finding a suitable initial state, but creates a number of practical, computational, and algorithmic issues due to varying and potentially environment-dependent sequence length, and higher variance of network updates because of the highly correlated nature of states in a trajectory when compared to training on randomly sampled batches of experience tuples. BID5 observed little difference between the two strategies for empirical agent performance on a set of Atari games, and therefore opted for the simpler zero start state strategy. One possible explanation for this is that in some cases, an RNN tends to converge to a more'typical' state if allowed a certain number of'burn-in' steps, and so recovers from a bad initial recurrent state on a sufficiently long sequence. We also hypothesize that while the zero start state strategy may suffice in the mostly fully observable Atari domain, it prevents a recurrent network from learning actual long-term dependencies in more memory-critical domains (e.g. on DMLab).To fix these issues, we propose and evaluate two strategies for training a recurrent neural network from randomly sampled replay sequences, that can be used individually or in combination: DISPLAYFORM0 Figure 1: Top row shows Q-value discrepancy ∆Q as a measure for recurrent state staleness. (a) Diagram of how ∆Q is computed, with green box indicating a whole sequence sampled from replay. For simplicity, l = 0 (no burn-in). (b) ∆Q measured at first state and last state of replay sequences, for agents training on a selection of DMLab levels (indicated by initials) with different training strategies. Bars are averages over seeds and through time indicated by bold line on x-axis in bottom row. (c) Learning curves on the same levels, varying the training strategy, and averaged over 3 seeds.• Stored state: Storing the recurrent state in replay and using it to initialize the network at training time. This partially remedies the weakness of the zero start state strategy, however it may suffer from the effect of'representational drift' leading to'recurrent state staleness', as the stored recurrent state generated by a sufficiently old network could differ significantly from a typical state produced by a more recent version.• Burn-in: Allow the network a'burn-in period' by using a portion of the replay sequence only for unrolling the network and producing a start state, and update the network only on the remaining part of the sequence. We hypothesize that this allows the network to partially recover from a poor start state (zero, or stored but stale) and find itself in a better initial state before being required to produce accurate outputs. In all our experiments we will be using the proposed agent architecture from Section 2.3 with replay sequences of length m = 80, with an optional burn-in prefix of l = 40 or 20 steps. Our aim is to assess the negative effects of representational drift and recurrent state staleness on network training and how they are mitigated by the different training strategies. For that, we will compare the Qvalues produced by the network on sampled replay sequences when unrolled using one of these strategies and the Q-values produced when using the true stored recurrent states at each step (see Figure 1a, showing different sources for the hidden state).More formally, let o t,..., o t+m and h t,..., h t+m denote the replay sequence of observations and stored recurrent states, and denote by h t+1 = h(o t, h t ; θ) and q(h t ; θ) the recurrent state and Qvalue vector output by the recurrent neural network with parameter vector θ, respectively. We writê h t for the hidden state, used during training and initialized under one of the above strategies (either DISPLAYFORM1 is computed by unrolling the network with parametersθ on the sequence o t, . . ., o t+l+m−1 . We estimate the impact of representational drift and recurrent state staleness by their effect on the Q-value estimates, by measuring Q-value discrepancy DISPLAYFORM2 for the first (i = l) and last (i = l + m − 1) states of the non-burn-in part of the replay sequence (see Figure 1a for an illustration). The normalization by the maximal Q-value helps comparability between different environments and training stages, as the Q-value range of an agent can vary dras-tically between these. Note that we are not directly comparing the Q-values produced at acting and training time, q(h t ; θ) and q(ĥ t ;θ), as these can naturally be expected to be distinct as the agent is being trained. Instead we focus on the difference that from applying the same network (parameterized byθ) to the distinct recurrent states. In Figure 1b, we are comparing agents trained with the different strategies on several DMLab environments in terms of this proposed metric. It can be seen that the zero start state heuristic in a significantly more severe effect of recurrent state staleness on the outputs of the network. As hypothesized above, this effect is greatly reduced for the last sequence states compared to the first ones, after the RNN has had time to recover from the atypical start state, but the effect of staleness is still substantially worse here for the zero state than the stored state strategy. Another potential downside of the pure zero state heuristic is that it prevents the agent from strongly relying on its recurrent state and exploit long-term temporal dependencies, see Section 5.We observe that the burn-in strategy on its own partially mitigates the staleness problem on the initial part of replayed sequences, while not showing a significant effect on the Q-value discrepancy for later sequence states. Empirically, this translates into noticeable performance improvements, as can be seen in Figure 1c. This itself is noteworthy, as the only difference between the pure zero state and the burn-in strategy lies in the fact that the latter unrolls the network over a prefix of states on which the network does not receive updates. In informal experiments (not shown here) we observed that this is not due to the different unroll lengths themselves (i.e., the zero state strategy without burn-in, on sequences of length l + m, performed worse overall). We hypothesize that the beneficial effect of burn-in lies in the fact that it prevents'destructive updates' to the RNN parameters ing from highly inaccurate initial outputs on the first few time steps after a zero state initialization. The stored state strategy, on the other hand, proves to be overall much more effective at mitigating state staleness in terms of the Q-value discrepancy, which also leads to clearer and more consistent improvements in empirical performance. Finally, the combination of both methods consistently yields the smallest discrepancy on the last sequence states and the most robust performance gains. We conclude the section with the observation that both stored state and burn-in strategy provide substantial advantages over the naive zero state training strategy, in terms of (indirect) measures of the effect of representation drift and recurrent state staleness, and empirical performance. Since they combine beneficially, we use both of these strategies (with burn-in length of l = 40) in the empirical evaluation of our proposed agent in Section 4. Additional on the effects of distributed training on representation drift and Q-value discrepancy are given in the Appendix. In this section we evaluate the empirical performance of R2D2 on two challenging benchmark suites for deep reinforcement learning: Atari-57 BID1 and DMLab-30 BID0. One of the fundamental contributions of Deep Q-Networks (DQN) BID15 was to set as standard practice the use of a single network architecture and set of hyper-parameters across the entire suite of 57 Atari games. Unfortunately, expanding past Atari this standard has not been maintained and, to the best of our knowledge, at present there is no algorithm applied to both Atari-57 and DMLab-30 under this standard. In particular, we will compare performance with Ape-X and IMPALA for which hyper-parameters are tuned separately for each benchmark. For R2D2, we use a single neural network architecture and a single set of hyper-parameters across all experiments. This demonstrates greater robustness and generality than has been previously observed in deep RL. It is also in pursuit of this generality, that we decided to disable the (Atari-specific) heuristic of treating life losses as episode ends, and did not apply reward clipping. Despite this, we observe state-of-the-art performance in both Atari and DMLab, validating the intuitions derived from our empirical study. A more detailed ablation study of the effects of these modifications is presented in the Appendix. The Atari-57 benchmark is built upon the Arcade Learning Environment (ALE) BID1, and consists of 57 classic Atari 2600 video games. Initial human-level performance was. Right: Example individual learning curves of R2D2, averaged over 3 seeds, and Ape-X, single seed.achieved by DQN BID15, and since then RL agents have improved significantly through both algorithmic and architectural advances. Currently, state of the art for a single actor is achieved by the recent distributional reinforcement learning algorithms IQN and Rainbow BID6, and for multi-actor , Ape-X.Figure 2 (left) shows the median human-normalized scores across all games for R2D2 and related methods (see Appendix for full Atari-57 scores and learning curves). R2D2 achieves an order of magnitude higher performance than all single-actor agents and quadruples the previous state-ofthe-art performance of Ape-X using fewer actors (256 instead of 360), ing in higher sampleand time-efficiency. Table 1 lists mean and median human-normalized scores for R2D2 and other algorithms, highlighting these improvements. In addition to achieving state-of-the-art on the entire task suite, R2D2 also achieves the highest ever reported agent scores on a large fraction of the individual Atari games, in many cases'solving' the respective games by achieving the highest attainable score. In FIG0 (right) we highlight some of these individual learning curves of R2D2. As an example, notice the performance on MS.PACMAN is even greater than that of the agent reported in BID25, which was engineered specifically for this game. Furthermore, we notice that Ape-X achieves super-human performance for the same number of games as Rainbow, and that its improvements came from improving already strong scores. R2D2 on the other hand is super-human on 52 out of 57 games. Of those remaining, we anecdotally observed that three (SKIING, SOLARIS, and PRIVATE EYE) can reach super-human performance with higher discount rates and faster target network updates. The other two (MONTEZUMA'S REVENGE and PITFALL) are known hard exploration problems, and solving these with a general-purpose algorithm will likely require new algorithmic insights. DMLab-30 is a suite of 30 problems set in a 3D first-person game engine, testing for a wide range of different challenges BID0. While Atari can largely be approached with only frame-stacking, DMLab-30 requires long-term memory to achieve reasonable performance. Perhaps because of this, and the difficulty of integrating recurrent state with experience replay, topperforming agents have, to date, always come in the form of actor-critic algorithms trained in (near) on-policy settings. For the first time we show state-of-the-art performance on DMLab-30 using a value-function-based agent. We stress that, different from the state-of-the-art IMPALA architecture BID3, the R2D2 agent uses the same set of hyper-parameters here as on Atari. Here we are mainly interested in comparing to the IMPALA'experts', not to its multi-task variant. Since the original IMPALA experts were trained on a smaller amount of data (approximately 333M Figure 3 : DMLab-30 comparison of R2D2 and R2D2+ with our re-run of IMPALA shallow and deep in terms of mean-capped human-normalized score BID3 . DMLab-30 Human-Normalized Score Median Mean Median Mean-Capped Ape-X 434.1% 1695.6% --Reactor BID4 187.0% ---IMPALA, deep BID3 191 Table 1 : Comparison of Atari-57 and DMLab-30 . R2D2 average final score over 3 seeds (1 seed for feed-forward variant), IMPALA final score over 1 seed, Ape-X best training score with 1 seed. Our re-run of IMPALA uses the same improved action set from BID7 as R2D2, and is trained for a comparable number of environment frames (10B frames; the original IMPALA experts in BID3 were only trained for approximately 333M frames). R2D2+ refers to the adapted R2D2 variant matching deep IMPALA's 15-layer ResNet architecture and asymmetric reward clipping, as well as using a shorter target update period of 400.environment frames) and since R2D2 uses the improved action set introduced in BID7, we decided to re-run the IMPALA agent with improved action set and for a comparable training time (10B environment frames) for a fairer comparison, ing in substantially improved scores for the IMPALA agent compared to the original in BID3, see Table 1.Figure 3 compares R2D2 with IMPALA. We note that R2D2 exceeds the performance of the (shallow) IMPALA version, despite using the exact same set of hyper-parameters and architecture as the variant trained on Atari, and in particular not using the'optimistic asymmetric reward clipping' used by all IMPALA agents 1.To demonstrate the potential of our agent, we also devise a somewhat adapted R2D2 version for DMLab only (R2D2+) by adding asymmetric reward clipping, using the 15-layer ResNet from IMPALA (deep), and reducing the target update frequency from 2500 to 400 for better sample efficiency. To fit the larger model in GPU memory, we reduced the batch size from 64 to 32 in these runs only. We observe that this modified version yields further substantial improvements over standard R2D2 and matches deep IMPALA in terms of sample efficiency as well as asymptotic performance. Both our re-run of deep IMPALA and R2D2+ are setting new state-of-the-art scores on the DMLab-30 benchmark. Atari-57 is a class of environments which are almost fully observable (given 4-frame-stack observations), and agents trained on it are not necessarily expected to strongly benefit from a memoryaugmented representation. The main algorithmic difference between R2D2 and its predecessor, Ape-X, is the use of a recurrent neural network, and it is therefore surprising by how large a margin R2D2 surpasses the previous state of the art on Atari. In this section we analyze the role of the LSTM network and other algorithmic choices for the high performance of the R2D2 agent. Since the performance of asynchronous or distributed RL agents can depend on subtle implementational details and even factors such as precise hardware setup, it is impractical to perform a direct comparison to the Ape-X agent as reported in. Instead, here we verify that the LSTM and its training strategy play a crucial role for the success of R2D2 by a comparison of the R2D2 agent with a purely feed-forward variant, all other parameters held fixed. Similarly, we consider the performance of R2D2 using reward clipping without value rescaling (Clipped) and using a smaller discount factor of γ = 0.99 (Discount). The ablation in Figure 4 show very clearly that the LSTM component is crucial for boosting the agent's peak performance as well as learning speed, explaining much of the performance difference to Ape-X. Other design choices have more mixed effects, improving in some games and hurting performance in others. Full ablation (in particular, an ablation over the full Atari-57 suite of the feed-forward agent variant, as well as an ablation of the use of the life-loss-as-episode-termination heuristic) are presented in the Appendix. In our next experiment we test to what extent the R2D2 agent relies on its memory, and how this is impacted by the different training strategies. For this we select the Atari game MS.PACMAN, on which R2D2 shows state-of-the-art performance despite the game being virtually fully observable, and the DMLab task EMSTM WATERMAZE, which strongly requires the use of memory. We train two agents on each game, using the zero and stored state strategies, respectively. We then evaluate these agents by restricting their policy to a fixed history length: at time step t, their policy uses an LSTM unrolled over time steps o t−k+1,..., o t, with the hidden state h t−k replaced by zero instead of the actual hidden state (note this is only done for evaluation, not at training time of the agents).In Figure 5 (left) we decrease the history length k from ∞ (full history) down to 0 and show the degradation of agent performance (measured as mean score over 10 episodes) as a function of k. We additionally show the difference of max-Q-values and the percentage of correct greedy actions (where the unconstrained variant is taken as ground truth).We observe that restricting the agent's memory gradually decreases its performance, indicating its nontrivial use of memory on both domains. Crucially, while the agent trained with stored state shows higher performance when using the full history, its performance decays much more rapidly than for the agent trained with zero start states. This is evidence that the zero start state strategy, used in past RNN-based agents with replay, limits the agent's ability to learn to make use of its memory. While this doesn't necessarily translate into a performance difference (like in MS.PACMAN), it does so whenever the task requires an effective use of memory (like EMSTM WATERMAZE). This advantage of the stored state compared to the zero state strategy may explain the large performance difference between R2D2 and its close cousin Reactor BID4, which trains its LSTM policy from replay with the zero state strategy. Finally, the right and middle columns of Figure 5 show a monotonic decrease of the quality of Qvalues and the ing greedy policy as the available history length k is decreased to 0, providing a simple causal link between the constraint and the empirical agent performance. For a qualitative comparison of different behaviours learned by R2D2 and its feed-forward variant, we provide several agent videos at https://bit.ly/r2d2600. Here we take a step back from evaluating performance and discuss our empirical findings in a broader context. There are two surprising findings in our . First, although zero state initialization was often used in previous works BID5 BID4, we have found that it leads to misestimated action-values, especially in the early states of replayed sequences. Moreover, without burn-in, updates through BPTT to these early time steps with poorly estimated outputs seem to give rise to destructive updates and hinder the network's ability to recover from sub-optimal initial recurrent states. This suggests that either the context-dependent recurrent state should be stored along with the trajectory in replay, or an initial part of replayed sequences should be reserved for burn-in, to allow the RNN to rely on its recurrent state and exploit long-term temporal dependencies, and the two techniques can also be combined beneficially. We have also observed that the underlying problems of representational drift and recurrent state staleness are potentially exacerbated in the distributed setting (see Appendix), highlighting the importance of robustness to these effects through an adequate training strategy of the RNN.Second, we found that the impact of RNN training goes beyond providing the agent with memory. Instead, RNN training also serves a role not previously studied in RL, potentially by enabling better representation learning, and thereby improves performance even on domains that are fully observable and do not obviously require memory (cf. BREAKOUT in the feed-forward ablation).Finally, taking a broader view on our empirical , we note that scaling up of RL agents through parallelization and distributed training allows them to benefit from huge experience throughput and achieve ever-increasing over broad simulated task suites such as Atari-57 and DMLab-30. Impressive as these are in terms of raw performance, they come at the price of high sample complexity, consuming billions of simulated time steps in hours or days of wall-clock time. One widely open avenue for future work lies in improving the sample efficiency of these agents, to allow applications to domains that do not easily allow fast simulation at similar scales. Another remaining challenge, very apparent in our on Atari-57, is exploration: Save for the hardest-exploration games from Atari-57, R2D2 surpasses human-level performance on this task suite significantly, essentially'solving' many of the games therein. Figure 6: Left: Parameter lag experienced with distributed prioritized replay with (top) 256 and (bottom) 64 actors on four DMLab levels: explore obstructed goals large (eogl), explore object rewards many (eorm), lasertag three opponents small (lots), rooms watermaze (rw). Center: initialstate and Right: final-state Q-value discrepancy for the same set of experiments. In this section, we investigate the effects of distributed training of an agent using a recurrent neural network, where a large number of actors feed their experience into a replay buffer for a single learner. On the one hand, the distributed setting typically presents a less severe problem of representational drift than the single-actor case, such as the one studied in BID5. This is because in relative terms, the large amount of generated experience is replayed less frequently (on average, an experience sample is replayed less than once in the Ape-X agent, compared to eight times in DQN), and so distributed agent training tends to give rise to a smaller degree of'parameter lag' (the mean age, in parameter updates, of the network parameters used to generate an experience, at the time it is being replayed).On the other hand, the distributed setting allows for easy scaling of computational resources according to hardware or time constraints. An ideal distributed agent should therefore be robust to changes in, e.g., the number of actors, without careful parameter re-tuning. As we have seen in the previous section, RNN training from replay is sensitive to the issue of representational drift, the severity of which can depend on exactly these parameters. To investigate these effects, we train the R2D2 agent with a substantially smaller number of actors. This has a direct (inversely proportional) effect on the parameter lag (see Figure 6 (left)). Specifically, in our experiments, as the number of actors is changed from 256 to 64, the mean parameter lag goes from 1500 to approximately 5500 parameter updates, which in turn impacts the magnitude of representation drift and recurrent state staleness, as measured by ∆Q in Section 3.The right two columns in Figure 6 show an overall increase of the average ∆Q for the smaller number of actors, both for first and last states of replayed sequences. This supports the above intuitions and highlights the increased importance of an improved training strategy (compared to the zero state strategy) in the distributed training setting, if a certain level of empirical agent performance is to be maintained across ranges of extrinsic and potentially hardware dependent parameters..'reset' refers to the agent variant using life losses as full episode terminations (preventing value function bootstrapping across life loss events, as well as resetting the LSTM state), whereas'roll' only prevents value function bootstrapping, but unrolls the LSTM for the duration of a full episode (potentially spanning multiple life losses). In this section we give additional experimental supporting our empirical study in the main text. FIG2 gives a more in-depth view of the ablation from Figure 4. We see that, with the exception of the feed-forward ablation, there are always games in which the ablated choice performs better. Our choice of architecture and configuration optimizes for overall performance and general (cross-domain) applicability, but for individual games there are different configurations that would yield improved performance. Additionally, in FIG4 we compare R2D2 with variants using the life loss signal as episode termination. Both ablation variants interrupt value function bootstrapping past the life loss events, but differ in that one ('reset') also resets the LSTM state at these events, whereas the other ('roll') only resets the LSTM state at actual episode boundaries, like regular R2D2. Despite the fact that the life loss heuristic is generally helpful to speed up learning in Atari, we did not use it in our main R2D2 agent for the sake of generality of the algorithm. Atari-57 -Human-normalized Median Ape-X R2D2, FF Rainbow Reactor Figure 9: Comparing sample efficiency between state-of-the-art agents on Atari-57. We observe a general trend of increasing final performance being negatively correlated with sample efficiency, which holds for all four algorithms compared. In Figure 9 we compare the sample efficiency of R2D2 with recent state-of-the-art agents on Atari-57 in terms of human-normalized median score. As expected, the more distributed agents have worse sample efficiency early on, but also much improved long-term performance. This is an interesting correlation on its own, but we add that R2D2 appears to achieve a qualitatively different performance curve than any of the other algorithms. Note that, while Ape-X has a larger number of actors than R2D2 (360 compared to 256), its learner processes approximately 20 batches of size 512 per second, whereas R2D2 performs updates on batches of 64 × 80 observations (batch size × sequence length), at a rate of approximately 5 per second. This in a reduced'replay ratio' (effective number of times each experienced observation is being replayed): On average, Ape-X replays each observation approximately 1.3 times, whereas this number is only about 0.8 for R2D2, which explains the initial sample efficiency advantage of Ape-X. R2D2 uses the same 3-layer convolutional network as DQN BID15, followed by an LSTM with 512 hidden units, which feeds into the advantage and value heads of a dueling network BID26, each with a hidden layer of size 512. Additionally, the LSTM receives as input the reward and one-hot action vector from the previous time step. On the four language tasks in the DMLab suite, we are using the same additional language-LSTM with 64 hidden units as IMPALA BID3 Target network update interval 2500 updates Value function rescaling h(x) = sign(x)(|x| + 1 − 1) + x, = 10 −3 Table 2: Hyper-parameters values used in R2D2. All missing parameters follow the ones in Ape-X.As is usual for agent training on Atari since BID15, we cap all (training and evaluation) episodes at 30 minutes (108, 000 environment frames Table 3 : Performance of R2D2 and R2D2+, averaged over 3 seeds, compared to our own singleseed re-run of IMPALA (shallow/deep) with improved action-set and trained on the same amount of data (10B environment frames). Compared to standard R2D2, the R2D2+ variant uses a shorter target network update frequency (400 compared to 2500), as well as the substantially larger 15-layer ResNet and the custom'optimistic asymmetric reward clipping' from BID3

Represent the TLDR for finding the corresponding scientific article!

Investigation on combining recurrent neural networks and experience replay leading to state-of-the-art agent on both Atari-57 and DMLab-30 using single set of hyper-parameters.

With a set of modifications, under 10 LOC, to A2C you get an off-policy actor-critic that outperforms A2C and performs similarly to ACER. The modifications are large batchsizes, aggressive clamping, and policy "forcing" with gumbel noise.