Question: Represent the article for finding a single-sentence TLDR

Question: Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks. In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and contextual bandits tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks. Ensembling is one of the oldest tricks in machine learning literature . By combining the outputs of several models, an ensemble can achieve better performance than any of its members. Many researchers demonstrate that a good ensemble is one where the ensemble's members are both accurate and make independent errors . In neural networks, SGD and its variants are the most common optimization algorithm. The random noise from sampling mini-batches of data in SGD-like algorithms and random initialization of the deep neural networks, combined with the fact that there is a wide variety of local minima solutions in high dimensional optimization problem , in the following observation: deep neural networks trained with different random seeds can converge to very different local minima although they share similar error rates. One of the consequence is that neural networks trained with different random seeds will usually not make all the same errors on the test set, i.e. they may disagree on a prediction given the same input even if the model has converged. Ensembles of neural networks benefit from the above observation to achieve better performance by averaging or majority voting on the output of each ensemble member . It is shown that ensembles of models perform at least as well as its individual members and diverse ensemble members lead to better performance . More recently, showed that deep ensembles give reliable predictive uncertainty estimates while remaining simple and scalable. A further study confirms that deep ensembles generally achieves the best performance on out-of-distribution uncertainty benchmarks compared to other methods such as MC-dropout . Despite their success on benchmarks, ensembles in practice are limited due to their expensive computational and memory costs, which increase linearly with the ensemble size in both training and testing. Computation-wise, each ensemble member requires a separate neural network forward pass of its inputs. Memory-wise, each ensemble member requires an independent copy of neural network weights, each up to millions (sometimes billions) of parameters. This memory requirement also makes many tasks beyond supervised learning prohibitive. For example, in lifelong learning, a natural idea is to use a separate ensemble member for each task, adaptively growing the total number of parameters by creating a new independent set of weights for each new task. No previous work achieves competitive performance on lifelong learning via ensemble methods, as memory is a major bottleneck. Our contribution: In this paper, we aim to address the computational and memory bottleneck by building a more parameter efficient ensemble model: BatchEnsemble. We achieve this goal by exploiting a novel ensemble weight generation mechanism: the weight of each ensemble member is generated by the Hadamard product between: a. one shared weight among all ensemble members. b. one rank-one matrix that varies among all members, which we refer to as fast weight in the following sections. Figure 1 compares testing and memory cost between BatchEnsemble and naive ensemble. Unlike typical ensembles, BatchEnsemble is mini-batch friendly, where it is not only parallelizable across devices like typical ensembles but also parallelizable within a device. Moreover, it incurs only minor memory overhead because a large number of weights are shared across ensemble members. Empirically, we show that BatchEnsemble has the best trade-off among accuracy, running time, and memory on several deep learning architectures and learning tasks: CIFAR-10/100 classification with ResNet32 and WMT14 EN-DE/EN-FR machine translation with Transformer . Additionally, we show that BatchEnsemble is also effective in uncertainty evaluation on contextual bandits. Finally, we show that BatchEnsemble can be successfully applied in lifelong learning and scale up to 100 sequential learning tasks without catastrophic forgetting and the need of memory buffer. In this section, we describe relevant about ensembles, uncertainty evaluation, and lifelong learning for our proposed method, BatchEnsemble. Bagging, also called boostrap aggregating, is an algorithm to improve the total generalization performance by combining several different models . The strategy to combine those models such as averaging and majority voting are known as ensemble methods. It is shown that ensembles of models perform at least as well as each of its ensemble member . Moreover, ensembles achieve the best performance when each of their members makes independent errors . Related work on ensembles: Ensembles have been studied extensively for improving model performance (; ; ;). One major direction in ensemble research is how to reduce their cost at test time. developed a method to compress large, complex ensembles into smaller and faster models which achieve faster test time prediction. developed the above approach further by distilling the knowledge in an ensemble of models into one single neural network. Another major direction in ensemble research is how to reduce their cost at training time. forms ensembles by combining the output of networks within a number of training checkpoints, named Horizontal Voting Vertical Voting and Horizontal Stacked Ensemble. Additionally, models trained with different regularization and augmentation can be used as ensemble to achieve better performance in semi-supervised learning . More recently, proposed Snapshot ensemble, in which a single model is trained by cyclic learning rates so that it is encouraged to visit multiple local minima. Those local minima solutions are then used as ensemble members. proposed fast geometric ensemble where it finds modes that can be connected by simple curves. Each mode can taken as one ensemble member. Explicit ensembles are expensive so another line of work lies on what so-called "implicit" ensembles. For example, Dropout can be interpreted as creating an exponential number of weight-sharing sub-networks, which are implicitly ensembled in test time prediction . MC-dropout can be used for uncertainty estimates . Although deep neural networks achieve state-of-the-art performance on a variety of benchmarks, their predictions are often poorly calibrated. Bayesian neural networks , which fit a distribution to the weights rather than a point estimate, are often used to model uncertainty. However, they requires modifications to the traditional neural network training scheme. Deep ensembles have been proposed as a simple and scalable alternative, and have been shown to make well-calibrated uncertainty estimates . Several metrics had been proposed to measure the quality of uncertainty estimates. In Section 4.4, we use the contextual bandits benchmark , where maximizing reward is of direct interest; this requires good uncertainty estimates in order to balance exploration and exploitation. Appendix D also uses Expected Calibrated Error (ECE) as an uncertainty metric. In lifelong learning, the model trains on a number of tasks in a sequential (online) order, without access to entire previous tasks' data . One core difficulty of lifelong learning is "catastrophic forgetting": neural networks tend to forget what it has learnt after training on the subsequent tasks . Previous work on alleviating catastrophic forgetting can be divided into two categories. In the first category, updates on the current task are regularized so that the neural network does not forget previous tasks. Elastic weight consolidation (EWC) applies a penalty on the parameter update based on the distance between the parameters for the new and the old task evaluated by Fisher information metric. Other methods maintain a memory buffer that stores a number of data points from previous tasks. For example, gradient episodic memory approach penalizes the gradient on the current task so that it does not increase the loss of examples in the memory buffer . Another approach focuses on combining existing experience replay algorithms with lifelong learning . In the second category, one increases model capacity as new tasks are added. For example, progressive neural networks (PNN) copy the entire network for the previous task and add new hidden units when adopting to a new task. This prevents forgetting on previous tasks by construction (the network on previous tasks remains the same). However, it leads to significant memory consumption when faced with a large number of lifelong learning tasks. Some following methods expand the model in a more parameter efficient way at the cost of introducing an extra learning task and not entirely preventing forgetting. applies group sparsity regularization to efficiently expand model capacity; Figure 2: An illustration on how to generate the ensemble weights for two ensemble members. For each training example in the mini-batch, it receives an ensemble weight W i by elementwise multiplying W, which we refer to as "slow weights", with a rank-one matrix F i, which we refer to as "fast weights." The subscript i represents the selection of ensemble member. Since W is shared across ensemble members, we term it as "shared weight" in the following paper. Vectorization: We show how to make the above ensemble weight generation mechanism parallelizable within a device, i.e., where one computes a forward pass with respect to multiple ensemble members in parallel. This is achieved by the fact that manipulating the matrix computations for a mini-batch. Let x denote the activations of the incoming neurons in a neural network layer. The next layer's activations are given by: where φ denotes the activation function and the subscript n represents the index in the mini-batch. The output represents next layer's activations from the i th ensemble member. To vectorize these computations, we define matrices R and S whose rows consist of the vectors r i and s i for all examples in the mini-batch. The above equation is vectorized as: where X is the mini-batch input. By computing Eqn. 5, we can obtain the next layer's activations for each ensemble member in a mini-batch friendly way. This allows us to take the full advantage of GPU parallelism to implement ensemble efficiently. To match the input and the ensemble weight, we can divide the input mini-batch into M sub-batches and each sub-batch receives ensemble weight Ensembling During Testing: In our experiments, we take the average of predictions of each ensemble member. Suppose the test batch size is B and there are M ensemble members. To achieve an efficient implementation, one repeats the input mini-batch M times, which leads to an effective batch size B · M. This enables all ensemble members to compute the output of the same B input data points in a single forward pass. It eliminates the need to calculate the output of each ensemble member sequentially and therefore reduces the ensemble's computational cost. The only extra computation in BatchEnsemble over a single neural network is the Hadamard product, which is cheap compared to matrix multiplication. Thus, BatchEnsemble incurs almost no additional computational overhead (Figure 1). 1 One limitation of BatchEnsemble is that if we keep the minibatch size the same as single model training, each ensemble member gets only a portion of input data. In practice, the above issue can be remedied by increasing the batch size so that each ensemble member receives the same amount of data as ordinary single model training. Since BatchEnsemble is parallelizable within a device, increasing the batch size incurs almost no computational overhead in both training and testing stages on the hardware that can fully utilize large batch size. Moreover, when increasing the batch size reaches its diminishing return regime, BatchEnsemble can still take advantage from even larger batch size by increasing the ensemble size. The only memory overhead in BatchEnsemble is the set of vectors, {r 1, . . ., r m} and {s 1, . . ., s m}, which are cheap to store compared to the weight matrices. By eliminating the need to store full weight matrices of each ensemble member, BatchEnsemble has almost no additional memory cost. For example, BatchEnsemble of ResNet-32 of size 4 incurs 10% more parameters while naive ensemble incurs 4X more. The significant memory cost of ensemble methods limits its application to many real world learning scenarios such as multi-task learning and lifelong learning, where one might apply an independent copy of the model for each task. This is not the case with BatchEnsemble. Specifically, consider a total of T tasks arriving in sequential order. Denote D t = (x i, y i, t) as the training data in task t where t ∈ {1, 2, . . ., T} and i is the index of the data point. Similarly, denote the test data set as T t = (x i, y i, t). At test time, we compute the average performance on T t across all tasks seen so far as the evaluation metric. To extend BatchEnsemble to lifelong learning, we compute the neural network prediction in task t with weight W t = W • (r t s t) in task t. In other words, each ensemble member is in charge of one lifelong learning task. For the training protocol, we train the shared weight W and two fast weights r 1, s 1 on the first task, min where L 1 is the objective function in the first task such as cross-entropy in image classification. On a subsequent task t, we only train the relevant fast weights r t, s t. min BatchEnsemble shares similar advantages as progressive neural networks (PNN): it entirely prevents catastrophic forgetting as the model for previously seen tasks remains the same. This removes the need of storing any data from previous task. In addition, BatchEnsemble has significantly less memory consumption than PNN as only fast weights are trained to adapt to a new task. Therefore, BatchEnsemble can easily scale to up to 100 tasks as we showed in Section 4.1 on split ImageNet. Another benefit of BatchEnsemble is that if future tasks arrive in parallel rather than sequential order, one can train on all the tasks at once (see Section 3.1). We are not aware of any other lifelong learning methods can achieve this. Limitations: BatchEnsemble is one step toward toward a full lifelong learning agent that is both immune to catastrophic forgetting and parameter-efficient. On existing benchmarks like split-CIFAR and split-ImageNet, Section 4.1 shows that BatchEnsemble's rank-1 perturbation per layer provides enough expressiveness for competitive state-of-the-art accuracies. However, one limitation of BatchEnsemble is that only rank-1 perturbations are fit to each lifelong learning task and thus the model's expressiveness is a valid concern when each task is significantly varied. Another limitation is that the shared weight is only trained on the first task. This implies that only information learnt for the first task can transfer to subsequent tasks. There is no explicit transfer, for example, between the second and third tasks. One solution is to enable lateral connections to features extracted by the weights of previously learned tasks, as done in PNN. However, we found that no lateral connections were needed for Split-CIFAR100 and Split-ImageNet. Therefore we leave the above solution to future work to further improve BatchEnsemble for lifelong learning. Section 4.1 firsts demonstrate the BatchEnsemble's effectiveness as an alternative approach to lifelong learning on Split-CIFAR and Split-ImageNet. We next evaluate BatchEnsemble on several benchmark datasets with common deep learning architectures in Section 4.2 and Section 4.3, including classification task with ResNet , neural machine translation with Transformer . Then, we demonstrate that BatchEnsemble can be used for uncertainty modelling in Section 4.4. Detailed description of datasets we used is in Appendix A.. PNN: Progressive neural network. BN-Tuned: Fine tuning Batch Norm layer per subsequent tasks. BatchE: BatchEnsemble. Upperbound: Individual ResNet-50 per task. We showcase BatchEnsemble for lifelong learning on Split-CIFAR100 and Split-ImageNet. Split-CIFAR100 proposed in is a harder lifelong learning task than MNIST permutations and MNIST rotations, where one introduces a new set of classes upon the arrival of a new task. Each task consists of examples from a disjoint set of 100/T classes assuming T tasks in total. To show that BatchEnsemble is able to scale to 100 sequential tasks, we also build our own Split-ImageNet dataset which shares the same property as Split-CIFAR100 except more classes (and thus more tasks) and higher image resolutions are involved. More details about these two lifelong learning datasets are provided in Appendix A. We consider T = 20 tasks on Split-CIFAR100, following the setup of. We used ResNet-18 with slightly fewer number of filters across all convolutional layers. Noted that for the purpose of making use of the task descriptor, we build a different final dense layer per task. We compare BatchEnsemle to progressive neural networks (PNN), vanilla neural networks, and elastic weight consolidation (EWC) on Split-CIFAR100. reported similar accuracies among DEN , RCL and PNN. Therefore we compare accuracy only to PNN which has an official implementation and only compare computational and memory costs to DEN and RCL in Appendix C. Figure 3b displays on Split-CIFAR100 over three metrics including accuracy, forgetting, and cost. The accuracy measures the average validation accuracy over total 20 tasks after lifelong learning ends. Average forgetting over all tasks is also presented in Figure 3b. Forgetting on task t is measured by the difference between accuracy of task t right after training on it and at the end of lifelong learning. It measures the degree of catastrophic forgetting. As showed in Figure 3b, BatchEnsemble achieves comparable accuracy as PNN while has 4X speed-up and 50X less memory consumption. It also preserves the no-forgetting property of PNN. Therefore BatchEnsemble has the best trade-off among all compared methods. For Split-ImageNet, we consider T = 100 tasks and apply ResNet-50 followed by a final linear classifier per task. The parameter overhead of BatchEnsemble on Split-ImageNet over 100 sequential tasks is 20%: the total number of parameters is 30M v.s. 25M (vanilla ResNet-50). PNN is not capable of learning 100 sequential tasks due to the significant memory consumption; other methods noted above have also not shown at ImageNet scale. Therefore we adopt two of our baselines. The first baseline is "BN-Tuned", which fine-tunes batch normalization parameters per task and which has previously shown strong performance for multi-task learning . To make a fair comparison, we augment the number of filters in BN-Tuned so that both methods have the same number of parameters. The second baseline is a naive ensemble which trains an individual ResNet-50 per task. This provides a rough upper bound on the BatchEnsemble's expressiveness per task. Note BatchEnsemble and both baselines are immune to catastrophic forgetting. So we consider validation accuracy on each subsequent task as evaluation metric. Figure 3a shows that In this section, we evaluate BatchEnsemble on the Transformer and the large-scale machine translation tasks WMT14 EN-DE/EN-FR. We apply BatchEnsemble to all self-attention layers with an ensemble size of 4. The ensemble in a selfattention layer can be interpreted as each ensemble member keeps their own attention mechanism and makes independent decisions. We conduct our experiments on WMT16 English-German dataset and WMT14 English-French dataset with Transformer base (65M parameters) and Transformer big (213M parameters). We maintain exactly the same training scheme and hyper-parameters between single Transformer model and BatchEnsemble Transformer model. As the shown in Figure 4, BatchEnsemble achieves a much faster convergence than a single model. Big BatchEnsemble Transformer is roughly 1.5X faster than single big Transformer on WMT16 English-German. In addition, the BatchEnsemble Transformer also gives a lower validation perplexity than big Transformer (Table 1). This suggests that BatchEnsemble is promising for even larger Transformers. We also compared BatchEnsemble to dropout ensemble (MC-drop in Table 1). Transformer single model itself has dropout layer. We run multiple forward passes with dropout mask during testing. The sample size is 16 which is already 16X more expensive than BatchEnsmeble. As Table 1 showed, dropout ensemble doesn't give better performance than single model. However, note Appendix B shows that while BatchEnemble's test BLEU score increases faster over the course of training, BatchEnsemble which gives lower validation loss does not necessarily improve BLEU score over a single model which is trained for long enough timesteps. We evaluate BatchEnsemble on classification tasks with CIFAR-10/100 dataset . We run our evaluation on ResNet32 . To achieve 100% training accuracy on CIFAR100, we use 4X more filters than the standard ResNet-32. In this section, we compare to MC-dropout which is also a memory efficient ensemble method. We add one more dense layer followed by dropout before the final linear classifier so that the number of parameters of MC-dropout are the same as BatchEnsemble. Most hyper-parameters are shared across the single model, BatchEnsemble, and MC-dropout. More details about hyper-parameters are in Appendix B. Note we increase the training iterations for BatchEnsemble to reach its best performance because each ensemble member gets only a portion of input data. We train both BatchEnsemble model and MC-dropout with 375 epochs on CIFAR-10/100, which is 50% more iterations than single model. Although the training duration is longer, BatchEnsemble is still significantly faster than training individual model sequentially. Another implementation that leads to the same performance is to increase the mini-batch size. For example, if we use 4X large minibatch size then there is no need to increase the training iterations. Table 2 shows that BatchEnsemble reaches better accuracy than single model and MC-dropout. We also calculate the accuracy of naive ensemble, whose members consist of individually trained single models. Its accuracy can be viewed as the upper bound of Ensemble methods. We also compare BatchEnsemble to naive ensemble of small models in Appendix F. In this section, we conduct analysis beyond accuracy, where we show that BatchEnsemble can be used for uncertainty modelling in contextual bandits. Appendix D evaluates the predictive uncertainty of BatchEnsemble on out-of-distribution tasks and ECE loss. We also show that BatchEnsemble preserves diversity among ensemble members in predictive distribution just like naive ensemble in Appendix E. For uncertainty modelling, we evaluate our BatchEnsemble method on the recently proposed bandits benchmark . Bandit data comes from different empirical problems that highlight several aspects of decision making. No single algorithm can outperform every other algorithm on every bandit problem. Thus, average performance of the algorithm over different problems is used to evaluate the quality of uncertainty estimation. The key factor to achieve good performance in contextual bandits is to learn a reliable uncertainty model. In our experiment, Thompson sampling samples from the policy given by one of the ensemble members. The fact that Dropout which is an implicit ensemble method achieves competitive performance on bandits problem suggests that ensemble can be used as uncertainty modelling. Indeed, Table 3 shows that BatchEnsemble with an ensemble size 8 achieves the best mean value on the bandits task. Both BatchEnsemble with ensemble size 4 and 8 outperform Dropout in terms of average performance. We also evaluate BatchEnsemble on CIFAR-10 corrupted dataset in Appendix D. Figure 7 shows that BatchEnsemble achieves promising accuracy, uncertainty and cost trade-off among all methods we compared. Moreover, combining BatchEnsemble and dropout ensemble leads to better uncertainty prediction. We introduced BatchEnsemble, an efficient method for ensembling and lifelong learning. BatchEnsemble can be used to improve the accuracy and uncertainty of any neural network like typical ensemble methods. More importantly, BatchEnsemble removes the computation and memory bottleneck of typical ensemble methods, enabling its successful application to not only faster ensembles but also lifelong learning on up to 100 tasks. We believe BatchEnsemble has great potential to improve in lifelong learning. Our work may serve as a starting point for a new research area. CIFAR: We consider two CIFAR datasets, CIFAR-10 and CIFAR-100 . Each consists of a training set of size 50K and a test set of size 10K. They are natural images with 32x32 pixels. In our experiments, we follow the standard data pre-processing schemes including zero-padding with 4 pixels on each sise, random crop and horizon flip (; ; . In machine translation tasks, we consider the standard training datasets WMT16 EnglishGerman and WMT14 English-French. WMT16 English-German dataset consists of roughly 4.5M sentence pairs. We follow the same pre-processing schemes in .Source and target tokens are processed into 37K shared sub-word units based on byte-pair encoding (BPE) . Newstest2013 and Newstest2014 are used as validation set and test set respectively. WMT14 English-French consists of a much larger dataset sized at 36M sentences pairs. We split the tokens into a 32K word-piece vocabulary . The dataset has the same set of images as CIFAR-100 dataset . It randomly splits the entire dataset into T tasks so each task consists of 100/T classes of images. To leverage the task descriptor in the data, different final linear classifier is trained on top of feature extractor per task. This simplifies the task to be a 100/T class classification problem in each task. i.e. random prediction has accuracy T /100. Notice that since we are not under the setting of single epoch training, standard data pre-processing including padding, random crop and random horizontal flip are applied to the training set. The dataset has the same set of images as ImageNet dataset . It randomly splits the entire dataset into T tasks so each task consists of 1000/T classes of images. Same as Split-CIFAR100, each task has its own final linear classifier. Data preprocessing is applied to the training data. In this section, we discuss some implementation details of BatchEnsemble. Weight Decay: In the BatchEnsemble, the weight of each ensemble member is never explicitly calculated because we obtain the activations directly by computing Eqn. 5. To maintain the goal of no additional computational cost, we can instead regularize the mean weight W over ensemble members, which can be efficiently calculated as where W is the shared weight among ensemble members, S and R are the matrices in Eqn. 5. We can also only regularize the shared weight and leave the fast weights unregularized because it only accounts for a small portion of model parameters. In practice, we find the above two schemes work equally. Diversity Encouragement: Additional loss term such as KL divergence among ensemble members can be added to encourage diversity. However, we find it sufficient for BatchEnsemble to have desired diversity by initializing the fast weight (s i and r i in Eqn. 1) to be random sign vectors. Also note that the scheme that each ensemble member is trained with different sub-batch of input can encourage diversity as well. The diversity analysis is provided in Appendix E. Machine Translation: The Transformer base is trained for 100K steps and the Transformer big is trained for 180K steps. The training steps of big model are shorter than because we terminate the training when it reaches the targeted perplexity on validation set. Experiments are run on 4 NVIDIA P100 GPUs. The BLEU score of Big Transformer on English-German task is in Figure 5. Although BatchEnsemble has lower perplexity as we showed in Section 4.2, we didn't observe a better BLEU score. Noted that the BLEU score in Figure 5 is lower than what reported. It is because in order to correctly evaluate model performance at a given timestep, we didn't use the averaging checkpoint trick. The dropout rate of Transformer base is 0.1 and 0.3 for Transformer big on English-German while remaining 0.1 on English-French. For dropout ensemble, we ran a grid search between 0.05 and 0.3 in the testing time and report the best validation perplexity. Classification: We train the model with mini-batch size 128. We also keep the standard learning rate schedule for ResNet. The learning rate decreases from 0.1 to 0.01, from 0.01 to 0.001 at halfway of training and 75% of training. The weight decay coefficient is set to be 10 −4. We use an ensemble size of 4, which means each ensemble member receives 32 training examples if we maintain the mini-batch size of 128. It is because Batch Normalization requires at least 32 examples to be effective on CIFAR dataset. As for the training budget, we train the single model for 250 epochs. Dynamically expandable networks and Reinforced continual learning are two recently proposed lifelong learning methods that achieve competitive performance. As discussed in Section 4.1, these two methods can be seen as an improved version progressive neural network (PNN) in terms of memory efficiency. As shown in , all three methods to similar accuracy measure in Split-CIFAR100 task. Therefore, among three evaluation metrics (accuracy, forgetting and cost), we only compare the accuracy of BatchEnsemble to PNN in Section 4.1 and compare the cost in this section. We first compute the cost relative to PNN on Split-CIFAR100 on LeNet and then compute the rest of the numbers base on what were reported in. Notice that PNN has no much computational overhead on Split-CIFAR100 because the number of total tasks is limited to 10. Even on the simple setup above, BatchEnsemble gives the best computational and memory efficiency. The advantage on large lifelong learning task such as Split-ImageNet would be even obvious. MC-drop BatchE NaiveE Single C10 2.89% 2.37% 2.32% 3.27% C100 8.99% 8.89% 6.82% 9.28% Similar to , we first evaluate BatchEnsemble on out-of-distribution examples from unseen classes. It is known that deep neural network tends to make over-confident predictions even if the prediction is wrong or the input comes from unseen classes. Ensembles of models can give better uncertainty prediction when the test data is out of the distribution of training data. To measure the uncertainty on the prediction, we calculate the predictive entropy of Single neural network, naive ensemble and BatchEnsemble. The is presented in Figure 6a. As we expected, single model produces over-confident predictions on unseen examples, whereas ensemble methods exhibit higher uncertainty on unseen classes, including both BatchEnsemble and naive ensemble. It suggests our ensemble weight generation mechanism doesn't degrade uncertainty modelling. Additionally, we calculate the Expected Calibration Error (ECE) of single model, naive ensemble and BatchEnsemble on both CIFAR-10 and CIFAR-100 in Table 6b. To calculate ECE, we group model predictions into M interval where n is the number of samples. ECE as a criteria of model calibration, measures the difference in expectation between confidence and accuracy . It shows that BatchEnsemble makes more calibrated prediction compared to single neural networks. Additionally, we evaluate the calibration of different mehtods on recently proposed CIFAR-10 corruption dataset . The dataset consists of over 30 types of corruptions to the images. It is commonly used to benchmark a wide range of methods on calibrated prediction . To the best of our knowledge, dropout ensemble is the state-of-the-art memory efficient ensemble method. Thus, in our paper, we compare BatchEnsemble to dropout ensemble in this section. Naive ensemble is also plotted as an upper bound of our method. As showed in Figure 7, BatchEnsemble achieves better calibration than dropout as the skew intensity increases. Moreover, dropout ensemble requires multiple forward passes to get the best performance. used sample size 128 while we found no significant difference between sample size 128 and 8. Note that even the sample size is 8, it is 8X more expensive than BatchEnsemble in the testing time cost. Finally, we showed that combining BatchEnsemble and dropout ensemble leads to better calibration. It is competitive to naive ensemble while keeping memory consumption efficient. It is also an evidence that BatchEnsemble is an orthogonal method to dropout ensemble. As we discussed in Section 2, ensemble benefits from the diversity among its members. We focus on the set of test examples on CIFAR-10 where single model makes confident incorrect predictions while ensemble model predicts correctly. We used the final models we reported in Section 4.3. In Figure 8, we randomly select examples from the above set and plot the prediction map of single model, each ensemble member and mean ensemble. As we can see, although some of the ensemble members make mistakes on thoes examples, the mean prediction takes the advantage of the model averaging and achieves better accuracy on CIFAR-10 classification task. We notice that BatchEnsemble preserves the diversity among ensemble members as naive ensemble. In this section, we compare BatchEnsemble to naive ensemble of small models on CIFAR-10/100 dataset. To maintain the same memory consumption as BatchEnsemble, we trained 4 independent ResNet14x4 models and evaluate the naive ensemble on these 4 models. This setup of naive ensemble still has roughly 10% memory overhead to BatchEnsemble. The are reported in Table 5. It shows that naive ensemble of small models achieves lower accuracy than BatchEnsemble. It illustrates that given the same memory budget, BatchEnsemble is a better choice over naive ensemble.

We introduced BatchEnsemble, an efficient method for ensembling and lifelong learning which can be used to improve the accuracy and uncertainty of any neural network like typical ensemble methods.

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question: Represent the text to find a very short summary (1-2 sentences max)!

Question: Reinforcement learning typically requires carefully designed reward functions in order to learn the desired behavior. We present a novel reward estimation method that is based on a finite sample of optimal state trajectories from expert demon- strations and can be used for guiding an agent to mimic the expert behavior. The optimal state trajectories are used to learn a generative or predictive model of the “good” states distribution. The reward signal is computed by a function of the difference between the actual next state acquired by the agent and the predicted next state given by the learned generative or predictive model. With this inferred reward function, we perform standard reinforcement learning in the inner loop to guide the agent to learn the given task. Experimental evaluations across a range of tasks demonstrate that the proposed method produces superior performance compared to standard reinforcement learning with both complete or sparse hand engineered rewards. Furthermore, we show that our method successfully enables an agent to learn good actions directly from expert player video of games such as the Super Mario Bros and Flappy Bird. Reinforcement learning (RL) deals with learning the desired behavior of an agent to accomplish a given task. Typically, a scalar reward signal is used to guide the agent's behavior and the agent learns a control policy that maximizes the cumulative reward over a trajectory, based on observations. This type of learning is referred to as "model-free" RL since the agent does not know apriori or learn the dynamics of the environment. Although the ideas of RL have been around for a long time BID24 ), great achievements were obtained recently by successfully incorporating deep models into them with the recent success of deep reinforcement learning. Some notable breakthroughs amongst many recent works are, the work from BID12 who approximated a Q-value function using as a deep neural network and trained agents to play Atari games with discrete control; who successfully applied deep RL for continuous control agents achieving state of the art; and BID22 who formulated a method for optimizing control policies with guaranteed monotonic improvement. In most RL methods, it is very critical to choose a well-designed reward function to successfully learn a good action policy for performing the task. However, there are cases where the reward function required for RL algorithms is not well-defined or is not available. Even for a task for which a reward function initially seems to be easily defined, it is often the case that painful hand-tuning of the reward function has to be done to make the agent converge on an optimal behavior. This problem of RL defeats the benefits of automated learning. In contrast, humans often can imitate instructor's behaviors, at least to some extent, when accomplishing a certain task in the real world, and can guess what actions or states are good for the eventual accomplishment, without being provided with the detailed reward at each step. For example, children can learn how to write letters by imitating demonstrations provided by their teachers or other adults (experts). Taking inspiration from such scenarios, various methods collectively referred to as imitation learning or learning from experts' demonstrations have been proposed BID21 ) as a relevant technical branch of RL. Using these methods, expert demonstrations can be given as input to the learning algorithm. Inverse reinforcement learning BID15; BID1; BID28 ), behavior cloning BID20 ), imitation learning BID6; BID5 ), and curiosity-based exploration ) are examples of research in this direction. While most of the prior work using expert demonstrations assumes that the demonstration trajectories contain both the state and action information (τ = {(s t)}) to solve the imitation learning problem, we, however, believe that there are many cases among real world environments where action information is not readily available. For example, a human teacher cannot tell the student what amount of force to put on each of the fingers when writing a letter. As such, in this work, we propose a reward estimation method that can estimate the underlying reward based only on the expert demonstrations of state trajectories for accomplishing a given task. The estimated reward function can be used in RL algorithms in order to learn a suitable policy for the task. The proposed method has the advantage of training agents based only on visual observations of experts performing the task. For this purpose, it uses a model of the distribution of the expert state trajectories and defines the reward function in a way that it penalizes the agent's behavior for actions that cause it to deviate from the modeled distribution. We present two methods with this motivation; a generative model and a temporal sequence prediction model. The latter defines the reward function by the similarity between the state predicted by the temporal sequence model trained based on the expert's demonstrations and the currently observed state. We present experimental of the methods on multiple environments and with varied settings of input and output. The primary contribution of this paper is in the estimation of the reward function based on state similarity to expert demonstrations, that can be measured even from raw video input. Model-free Reinforcement Learning (RL) methods learn a policy π(a t |s t) that produces an action from the current observation. BID12 showed that a q-value function q(s t, a t) can be approximated with a deep neural network, which is trained using hand-engineered scalar reward signals given to the agent based on its behavior. Similarly, actor-critic networks in Deep Deterministic Policy Gradients (DDPG) can enable state of the art continuous control, e.g. in robotic manipulation by minimizing the distance between the end effector and the target position. Since the success with DDPG, other methods such as Trust Region Policy Optimization (TRPO) BID22 ) and Proximal Policy Optimization (PPO) BID23 ) have been proposed as further improvements for model-free RL in continuous control problems. Although RL enables an agent to learn an optimal policy without supervised training data, in the standard case, it requires a difficult task of hand-tuning good reward functions for each environment. This has been pointed out previously in the literature BID1 ). Several kinds of approaches have been proposed to workaround or tackle this problem. An approach that does not require reward hand-tuning is behavior cloning based on supervised learning instead of RL. It learns the conditional distribution of actions given states in a supervised manner. Although it has an advantage of fast convergence BID5 ) (as behavior cloning learns a single action from states in each step), it typically in compounding of errors in the future states. An alternate approach is Inverse Reinforcement Learning (IRL) proposed in the seminal work by BID15. In this work, the authors try to recover the optimal reward function as a best description behind the given expert demonstrations from humans or other expert agents, using linear programming methods. It is based on the assumption that expert demonstrations are solutions to a Markov Decision Process (MDP) defined with a hidden reward function BID15 ). It demonstrated successful estimation of the reward function in case of relatively simple environments such as the grid world and the mountain car problem. Another use of the expert demonstrations is initializing the value function; this was described by BID27. Extending the work by BID15, entropy-based methods that compute the suitable reward function by maximizing the entropy of the expert demonstrations have been proposed by BID29. In the work by BID1, a method was proposed for recovering the cost function based on expected feature matching between observed policy and the agent behavior. Furthermore, they showed this to be the necessary and sufficient condition for the agent to imitate the expert behavior. More recently, there was some work that extended this framework using deep neural networks as non-linear function approximator for both policy and the reward functions BID28 ). In other relevant work by BID6, the imitation learning problem was formulated as a two-players competitive game where a discriminator network tries to distinguish between expert trajectories and agent-generated trajectories. The discriminator is used as a surrogate cost function which guides the agent's behavior in each step to imitate the expert behavior by updating policy parameters based on Trust Region Policy Optimization (TRPO) BID22 ). Related recent works also include model-based imitation learning BID2 ) and robust imitation learning BID26 ) using generative adversarial networks. All the above-mentioned methods, however, rely on both state and action information provided by expert demonstrations. Contrarily, we learn only from expert state trajectories in this work. A recent line of work aims at learning useful policies for agents even in the absence of expert demonstrations. In this regard, trained an agent with a combination of reward inferred with intrinsic curiosity and a hand-engineered, complete or even very sparse scalar reward signal. The curiosity-based reward is designed to have a high value when the agent encounters unseen states and a low value when it is in a state similar to the previously explored states. The work reported successful navigation in games like Mario and Doom without any expert demonstrations. In this paper, we compare our proposed methods with the curiosity-based approach and show the advantage over it in terms of the learned behaviors. However, our methods assumed state demonstrations are available as expert data while the curiosity-based method did not use any demonstration data. We consider an incomplete Markov Decision Process (MDP), consisting of states S and action space A, where the reward signal r: S × A → R, is unknown. An agent can act in an environment defined by this MDP following a policy π(a t |s t). Here, we assume that we have knowledge of a finite set of optimal or expert state trajectories DISPLAYFORM0., n}. These trajectories can represent joints angles, raw images or any other information depicting the state of the environment. Since the reward signal is unknown, our primary goal is to find a reward signal that enables the agent to learn a policy π, that can maximize the likelihood of these set of expert trajectories τ. In this paper, we assume that the reward signal can be inferred entirely based on the current state and next state information, r: S × S → R. More formally, we would like to find a reward function that maximizes the following objective: DISPLAYFORM1 where r(s t+1 |s t) is the reward function of the next state given the current state and p(s t+1 |s t) is the transition probability. We assume the performing to maximize the likelihood of next step prediction in equation 1 will be leading the maximizing the future reward when the task is deterministic. Because this likelihood is based on similarity with demonstrations which are obtained while an expert agent is performing by maximizing the future reward. Therefore we assume the agent will be maximizing future reward when it takes the action that gets the similar next step to expert demonstration trajectory data τ. Let, τ = {s i t} i=1:M,t=1:N be the optimal states visited by the expert agent, where M is the number of demonstration episodes, and N is the number of steps within each episode. We estimate an appropriate reward signal based on the expert state trajectories τ, which in turn is used to guide a reinforcement learning algorithm and learn a suitable policy. We evaluate two approaches to implement this idea. A straightforward approach is to first train a generative model using the expert trajectories τ. Rewards can then be estimated based on similarity measures between a reconstructed state value and the actual currently experienced state value of the agent. This method constrains exploration to the states that have been demonstrated by an expert and enables learning a policy that closely matches the expert. However, in this approach, the temporal order of states are ignored or not readily accounted for. This temporal order of the next state in the sequence is important for estimating the state transition probability function. As such, the next approach we take is to consider a temporal sequence prediction model that can be trained to predict the next state value given current state, based on the expert trajectories. Once again the reward value can be estimated as a function of the similarity measure between the predicted next state and the one actually visited by the agent. The following sub-sections describes both these approaches in detail. We train a deep generative model (three-layered fully connected auto-encoder) using the state values s i t for each step number t, sampled from the expert agent trajectories τ. The generative model is trained to minimize the following reconstruction loss (maximize the likelihood of the training data): DISPLAYFORM0 where θ * g represents the optimum parameters of the generative model. Following typical settings, we assume p(s i t ; θ g) to be a Gaussian distribution, such that equation FORMULA2 DISPLAYFORM1 The reward value is estimated as a function of the difference between the actual state value s t+1 and the generated output g(s DISPLAYFORM2 where s t is the current state value, and ψ can be a linear or nonlinear function, typically hyperbolic tangent or gaussian function. In this formulation, if the current state is similar to the reconstructed state value, i.e. g(s t ; θ g), the estimated reward value will be higher. However, if the current state is not similar to the generated state, the reward value will be estimated to be low. Moreover, as a reward value is estimated at each time step, this approach can be used even in problems which originally had a highly sparse engineered reward structure. In this approach, we learn a temporal sequence prediction model (the specific networks used are mentioned in the corresponding experiments sections) such that we can maximize the likelihood of the next state given the current state. As such the network is trained using the following objective function, DISPLAYFORM0 where θ * h represents the optimal parameters of the prediction model. We also assume the probability of the next state given the previous state value, p(s i t+1 |s i t ; θ h) to be a Gaussian distribution. As such the objective function can be seen to be minimizing the mean square error, DISPLAYFORM1 DISPLAYFORM2 The estimated reward here, can also be interpreted akin to the generative model case. Here, if the agent's policy takes an action that changes the environment towards states far away from the expert trajectories, the corresponding estimated reward value is low. If the actions of agent bring it close to the expert demonstrated trajectories, thereby making the predicted next state match with the actual visited state value, the reward is estimated to be high. This process of reward shaping or guidance can enable the agent to learn a policy that is optimized based on the expert demonstration trajectories. Algorithm 1 explains the step by step flow of the proposed methods. Given trajectories τ from expert agent 3: DISPLAYFORM3 end for 6: end procedure 7: procedure REINFORCEMENT LEARNING 8: DISPLAYFORM4 Observe state s t Select/execute action a t, and observe state s t+1 11: DISPLAYFORM0 Update the deep reinforcement learning network using the tuple (s t, a t, r t, s t+1) end for 14: end procedure In order to evaluate our reward estimation methods, we conducted experiments across a range of environments. We consider five different tasks, namely: robot arm reaching task (reacher) to a fixed target position, robot arm reaching task to a random target position, controlling a point agent for reaching a target while avoiding an obstacle, learning an agent for longest duration of flight in the Flappy Bird video game, and learning an agent for maximizing the traveling distance in Super Mario Bros video game. We consider a 2-DoF robot arm in the x-y plane that has to learn to reach with the end-effector a point target. The first arm of the robot is a rigidly linked point, with the second arm linked to its edge. It has two joint values θ = (θ 1, θ 2), θ 1 ∈ (−∞, +∞), θ 2 ∈ [−π, +π] and the lengths of arms are 0.1 and 0.11 units, respectively. The robot arm was initialized by random joint values at the initial step for each episode. In the following experiments, we have two settings: fixed point target, and a random target. The applied continuous action values a t is used to control the joint angles, such that,θ = θ t − θ t−1 = 0.05 a t. Each action value has been clipped the range [−1, 1]. The reacher task is enabled using the physics engine within the roboschool environment BID3 BID16 ). Figure 1 describes the roboshool environment. The robot arms are in blue, the blue-green point is the end-effector, and the pink dot is the desired target location. In this experiment, the target point p tgt is always fixed at (0.1, 0.1). The state vector s t consists of the following values: absolute end position of first arm (p 2), joint value of elbow (θ 2), velocities of the joints (θ 1,θ 2), absolute target position (p tgt), and the relative end-effector position from target (p ee − p tgt). We used DDPG ) for this task, with the number of steps for each episode being 500 in this experiment 1. The reward functions used in this task were as follows:Dense reward: DISPLAYFORM0 Sparse reward: r t = − tanh(α p ee − p tgt 2) + r env t, DISPLAYFORM1 DISPLAYFORM2 GM with r env t: rGM (1000 episodes): DISPLAYFORM3 GM with action: DISPLAYFORM4 where r env t is the environment specific reward, which is calculated based on the cost for current action, − a t 2. This regularization is required for finding the shortest path to reach the target. As this cost is critical for fast convergence, we use this in all cases. The dense reward is a distance between end-effector and the target, and the sparse reward is based on a bonus for reaching. The generative model parameters θ 2k is trained by τ 2k trajectories that contains only states of 2000 episodes from an agent trained during 1k episodes with dense reward. The generative network has 400, 300 and 400 units fully-connected layers, respectively. They also have ReLU activation function, with the batch size being 16, and number of epochs being 50. θ 1k is trained from τ 1k trajectories that is randomly picked 1000 episodes from τ 2k. The GM with action uses a generative model θ 2k,+a that is trained pairs of state and action for 2000 episodes for same agents as τ 2k. We use a tanh nonlinear function for the estimated reward in order to keep a bounded value. The α, β change sensitiveness of distance or reward, were both set to 100 2. Here, we also compare our with behavior cloning (BC) method BID20 where the trained actor networks directly use obtained pairs of states and actions. DISPLAYFORM5 Figure 1: The environment of reacher task. The reacher has two arms, and objective of agent is reaching endeffector (green) to target point (red). FIG3 shows the difference in performance by using the different reward functions 3. All methods are evaluated by a score (y-axis), which was normalized to a minimum and a maximum value. The proposed method, especially "GM with r env t ", manages to achieve a score nearing that of the dense reward, with the performance being much better as compared to the sparse reward setting. Moreover, the learning curves based on the rewards estimated with the generative model show a faster convergence rate. However, the without environment specific reward, i.e. with the additional action regularization term, takes a longer time to converge. This is primarily because of the fact that GM reward is reflective of the distance between target and end-effector, and cannot directly account for the action regularization. The GM reward based on τ 1k underperforms as compared with GM reward based on τ 2k because of the lack of demonstration data. FIG4 shows the reward value of each end-effector point. GM estimated reward using τ 2k has better reward map as compared to GM estimated reward using τ 1k. However, these demonstrations data FIG11 ) are biased by the robot trajectories. Thus, a method of generating demonstration data that normalizes or avoids such bias will further improve the reward structure. If the demonstrations contain the action information in addition to state information, behavior cloning achieves good performance. Surprisingly, however, when using both state and action information in the generative model, "GM [state, action]", the performance of the agent is comparatively poor. In this experiment, the target point p tgt is initialized by a random uniform distribution of [−0.27, +0.27], that includes points outside of the reaching range of the robot arms. Furthermore, we removed the relative position of the target from the input state information. This makes the task more difficult. The state vector s t has the following values: p 2, θ 2,θ 1,θ 2, p tgt. In the previous experiment, the distribution of states in expert trajectories is expected to be similar to the reward structure due to a fixed target location. However, when the target position changes randomly, this distribution is not fixed. We, therefore, evaluate with the temporal sequence prediction model h(s t ; θ h) in this experiment. The RL setting is same as the previous experiment, however we changed the total number of steps within each episode to 400. The reward functions used in this experiment were calculated as follows: DISPLAYFORM0 Sparse reward: r t = tanh(−α p ee − p tgt 2) + r env t, DISPLAYFORM1 LSTM reward: r t = tanh(−γ s t+1 − h(s t:t-n ; θ lstm) 2 ) + r DISPLAYFORM2 The expert demonstrations τ were obtained using the states of 2000 episodes running a trained agent with dense hand-engineered reward. The GM estimated reward uses the same setting as in the previous experiment. NS is a model that predicts the next state given current state, and was trained using demonstration data τ 4. The LSTM model uses Long short-term memory BID7 ) as a temporal sequence prediction model. The state in reacher task does not contain time sequence data, hence we use a finite state history as input. LSTM model has three layers 5 and one fully-connected layer with 40 ReLU activation units. The forward model based reward estimation is based on predicting the next state given both the current state and action 6. Here, we also compared with the baseline behavior cloning method. In this experiment, α is 100, β is 1, and γ is 10. FIG3 shows the performance of the trained agent using the different reward functions. In all cases using estimated rewards performances significantly better than the sparse reward case. The LSTM based prediction method gives the best , reaching close to the performance obtained with dense hand engineered reward function. As expected, the GM based reward estimation fails to work well in this relatively complex experimental setting. The NS model estimated reward, which predicts next state given only the current state information, has comparable performance with LSTM based prediction model during the initial episodes. The FM based reward function also performs poorly in this experiment. Comparatively, the direct BC works relatively well. This indicates that it is better to use behavior cloning than reward prediction when both state and action information are available from demonstration data. Starting with this experiment, we evaluate using only the temporal sequence prediction method. As such, here we use a finite history of the state values in order to predict the next state value. We assume that predicting a part of the state that is related to a given action allows the model to make a better estimate of the reward function. Former work by predicts a function of the next state, φ(s t+1) rather than predicting the raw value s t+1, as in this paper. In this experiment, we also changed the non-linear function ψ in the proposed prediction method to a Gaussian function (as compared to the hyperbolic tangent function used in previous experiments). This allows us to compare the robustness of our proposed method for reward estimation to different non-linear functions. We develop a new environment that adds an obstacle to the reaching task. This reacher is a twodimensional point (x, y) that uses position control. In FIG5 we show the modified environment setup. The agent's goal is to reach the target while avoiding the obstacle in this case. The initial position of agent, the target position, and an obstacle position were initialized randomly. The state value contains the agent absolute position (p t), current velocity of the agent (ṗ t), a target absolute position (p tgt), an obstacle absolute position (p obs), and the relative location of target and obstacle with respect to the agent (p t −p tgt, p t −p obs). Once again the RL algorithm used in this experiment was DDPG ) for continuous control 7. The number of steps for each episode set to 500. Here, we used the following reward functions: DISPLAYFORM0 LSTM reward: DISPLAYFORM1 DISPLAYFORM2 where h (s t:t-n ; θ lstm) is a network that predicts a selected part of state values given a finite history of state information. The dense reward is composed of both, the target distance cost and the obstacle distance bonus. The optimal state trajectories τ contains 800 "human guided" demonstration data. In this case, the LSTM network consisted of two layers, each with 256 units with ReLU activations. In this experiment, σ 1 is 0.005, and σ 2 is 0.002 8. FIG6 shows the performance with the different estimated or hand-engineered reward settings. The LSTM based prediction method learns to reach the target faster than the dense reward, while LSTM (s) has the best overall performance by learning with human-guided demonstration data. The agent (green) will move to reach the target (red), while avoiding the obstacle (pink). DISPLAYFORM3 We use a re-implementation BID10 ) of Android game, "Flappy Bird", in python (pygame). The objective of this game is to pass through the maximum number of pipes without collision. The control is a single discrete command of whether to flap the bird wings or not. The state has four consecutive gray frames (4 x 80 x 80). The RL is trained by DQN BID12 ) 9, and the update frequency of deep network is 100 steps. The used rewards are, DISPLAYFORM0 if pass through a pipe; −1 if collide to a pipe. LSTM reward: DISPLAYFORM1 which s t+1 is an absolute position of the bird, which can be given from simulator or it could be processed by pattern matching or CNN from raw images, h (s t ; θ lstm) is a predicted the absolute position. Hence, LSTM is trained for predicting absolute position of bird location given images. The τ of this experiment is 10 episodes data from a trained agent in the repository by BID10. Also, we also compared with the baseline behavior cloning method. In this experiment, σ is 0.02. FIG8 shows the of LSTM reward is better than normal "hand-crafted" reward. The reason for this situation is, the normal dense reward just describes the traveling distance, but our LSTM reward will teach which absolute transition of bird is good. Also, LSTM has better convergence than BC ; the reason is the number of demonstrations is not enough for behavior cloning method. In the final task, we consider a more complex environment in order to evaluate our proposed reward estimation method using only state information. Here we use the Super Mario Bros classic Nintendo video game environment . Our proposed method estimates reward values based on expert gameplay video data (using only the state information in the form of image frames).In this experiment, we also benchmarked against the recently proposed curiosity-based method ) using the implementation provided by the same authors BID17 ). This was used as the baseline reinforcement learning technique. Unlike in the actual game, here we always initialize Mario to the starting position rather than a previously saved checkpoint. This is a discrete control setup, where, Mario can make 14 types of actions 10. The state information consists of sequential input of four 42 x 42 gray image frames 11. Here we used the A3C RL algorithm BID13 ). We used gameplay of stage "1-1" for this experiment, with the objective of the agent being to travel as far as possible and achieve as high a score as possible. The rewards functions used in this experiment were as follows: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 Curiosity DISPLAYFORM3 DISPLAYFORM4 where position t is the current position of Mario at time t, score t is the current score value at time t, and s t are screen images from the Mario game at time t. The position and score information are obtained using the Mario game emulator. In this experiment, we use a three-dimensional convolutional neural network BID8 ) (3D-CNN) for our temporal sequence prediction method. In order to capture expert demonstration data, we took 15 game playing videos by five different people 12. In total, the demonstration data consisted of 25000 frames. The length of skipped frames in input to the temporal sequence prediction model was 36, as humans cannot play as fast as an RL agent; however, we did not change the skip frame rate for the RL agent. The 3D-CNN consisted of 4 layers 13 and a final layer to reconstruct the image. The agent was trained using 50 epochs with a batch size of 8. We implemented two prediction methods for reward estimation. In the naïve method the Mario agent will end up getting positive rewards if it sits in a fixed place without moving. This is because it can avoid dying by just not moving. However, this is clearly a trivial suboptimal policy. Hence, we implemented the alternate reward function based on the same temporal sequence prediction model, but we apply a threshold value that prevents the agent from converging onto such a trivial solution. Here, the value of ζ is 0.025, which was calculated based on the reward value obtained by just staying fixed at the initial position. FIG10 shows the performance with the different reward functions. Here, the graphs directly show the average over multiple trials. As observed, the agent was unable to reach large distances even while using "hand-crafted" dense rewards and did not converge to the goal every time 14; this behavior was also observed by for their reward case. As observed from the average curves of FIG10, the proposed 3D-CNN method learns relatively faster as compared to the curiosity-based agent ). As expected the 3D-CNN (naïve) method converged to a solution of remaining fixed at the initial state. As future work, we hope to improve the performance in this game setting using deeper RL networks, as well as large input image sizes. Overall estimating reward from φ(s t) without the need of action data, allows an agent to learn suitable policy directly from raw video data. The abundance of visual data creates ample opportunity for this type of reward estimation method to be explored further in different video game settings.10 A single action is repeated for six consecutive frames. Please refer to ) for details. 11 Every next six frames were skipped. 12 All videos consisted of games where the player succeeded in clearing the stage. 13 Two layers with (2 x 5 x 5), two layers with (2 x 3 x 3) kernels, all have 32 filters, and every two layers with stride.14 By our experiment, even if it trained long steps, such as 3.0M; it just reached around 600 -700 averagely. In this paper, we proposed two variations of a reward estimation method via state prediction by using state-only trajectories of the expert; one based on an autoencoder-based generative model and one based on temporal sequence prediction using LSTM. Both the models were for calculating similarities between actual states and predicted states. We compared the methods with conventional reinforcement learning methods in five various environments. As overall trends, we found that the proposed method converged faster than using hand-crafted reward in many cases, especially when the expert trajectories were given by humans, and also that the temporal sequence prediction model had better than the generative model. It was also shown that the method could be applied to the case where the demonstration was given by videos. However, detailed trends were different for the different environments depending on the complexity of the tasks. Neither model of our proposed method was versatile enough to be applicable to every environment without any changes of the reward definition. As we saw in the necessity of the energy term of the reward for the reacher task and in the necessity of special handling of the initial position of Mario, the proposed method has a room of improvements especially in modeling global temporal characteristics of trajectories. We would like to tackle these problems as future work. The DDPG's actor network has 400 and 300 unites fully-connected (fc) layers, the critic network has also 400 and 300 fully-connected layers, and each layer has a ReLU (Nair & Hinton FORMULA1) activation function. We put the tanh activation function at the final layer of actor network. Without this modification, the normal RL takes a long time to converge. Also, initial weights will be set from a uniform distribution U (−0.003, 0.003). The exploration policy is Ornstein-Uhlenbeck process BID25 ) (θ = 0.15, µ = 0, σ = 0.01), size of reply memory is 1M, and optimizer is Adam (Kingma & Ba FORMULA1). We implemented these experiments by Keras-rl , Keras BID4 ), and Tensorflow BID0 ) libraries.: These are scatter-plots of end-effector positions (blue) for each state of captured demonstration τ 500, τ 1k, τ 2k, each point is drawn by α is 0.01. And the fixed target position is also plotted (red). Notes that this is just plotting end-effector position, there is more variation in other state values. For example, even if the end-effector position were same, arms' pose (joint values) might be different. Note that τ 500 is not used in the experiment. The DDPG's actor network has 64 and 64 unites fully-connected layers, the critic network has also 64 and 64 fully-connected layers, and each layer has a ReLU activation function. Initial weights will be set from a uniform distribution U (−0.003, 0.003). The exploration policy is Ornstein-Uhlenbeck process BID25 ) (θ = 0.15, µ = 0, σ = 0.01), size of reply memory is 500k, and optimizer is Adam.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

represent the short summary for finding the corresponding scientific article (long)

The overall goal of this work is to enable sample-efficient imitation from expert demonstrations, both with and without the provision of expert action labels, through the use of f-divergences.