Question: Represent the article for finding a single-sentence TLDR!

Question: Network pruning is widely used for reducing the heavy computational cost of deep models. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. In this work, we make a rather surprising observation: fine-tuning a pruned model only gives comparable or even worse performance than training that model with randomly initialized weights. Our have several implications: 1) training a large, over-parameterized model is not necessary to obtain an efficient final model, 2) learned "important" weights of the large model are not necessarily useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited weights, is what leads to the efficiency benefit in the final model, which suggests that some pruning algorithms could be seen as performing network architecture search. Network pruning is a commonly used approach for obtaining an efficient neural network. A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model, 2) prune the unimportant weights according to a certain criterion, and 3) fine-tune the pruned model to regain accuracy. Generally, there are two common beliefs behind this pruning procedure. First, it is believed that training a large network first is important and the three-stage pipeline can outperform directly training the small model from scratch. Second, both the architectures and the weights of the pruned model are believed to be essential for the final efficient model, which is why most existing pruning methods choose to fine-tune the pruned model instead of training it from scratch. Also because of this, how to select the set of important weights is a very active research topic.Predefined: prune x% channels in each layer Automatic: prune a%, b%, c%, d% channels in each layer A 4-layer model Figure 1: Difference between predefined and non-predefined (automatically discovered) target architectures. The sparsity x is user-specified, while a, b, c, d are determined by the pruning algorithm. In this work, we show that both of the beliefs mentioned above are not necessarily true. We make a surprising observation that directly training the target pruned model from random initialization can achieve the same or better performance as the model obtained from the three-stage pipeline. This means that, for pruning methods with a predefined target architecture (Figure 1), starting with a large model is not necessary and one could instead directly train the target model from scratch. For pruning algorithms with automatically discovered target architectures, what brings the efficiency benefit is the obtained architecture, instead of the inherited weights. Our advocate a rethinking of existing network pruning algorithms: the preserved "important" weights from the large model are not necessary for obtaining an efficient final model; instead, the value of automatic network pruning methods may lie in identifying efficient architectures and performing implicit architecture search. Target Pruned Architectures. We divide pruning methods by whether the target pruned architecture is determined by either human or the pruning algorithm (see Figure 1). An example of designing predefined target architecture is specifying how many channels to prune in each layer. In contrast, when the target architecture is automatically discovered, the pruning algorithm determines how many channels to prune in each layer, by comparing the importance of channels across layers. Training from scratch. Because the pruned model requires less computation, it may be unfair to train the pruned model for the same number of epochs as the large model. In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denote training for the same amount of computation budget (measured by FLOPs). We use standard training hyper-parameters and data-augmentation schemes. The optimization method used is SGD with Nesterov momentum. In this section we present our experimental comparing training pruned models from scratch with fine-tuning. For method with predefined architectures, we evaluate L1-norm based channel pruning method. For method with automatically discovered target architectures, we use Network Slimming. The models, datasets and the number of epochs for fine-tuning are the same as those in the original paper. More for other pruning methods and transfer learning can be found in Appendix B. is one of the earliest work on channel pruning for convolutional networks. In each layer, a certain percentage of channels with smaller L 1 -norm of its filter weights will be pruned. Table 1 shows our . The Pruned Model column shows the list of predefined target models. We observe that in each row, scratch-trained models achieve at least the same level of accuracy as fine-tuned models, with Scratch-B slightly higher than Scratch-E in most cases. On ImageNet, both Scratch-B models are better than the fine-tuned ones. Network Slimming imposes L 1 -sparsity regularization on channel-wise scaling factors from Batch Normalization layers during training, and prunes channels with lower scaling factors afterward. Since the channel scaling factors are compared across layers, this method produces automatically discovered target architectures. As shown in TAB2, for all networks, the small models trained from scratch can reach the same accuracy as the fine-tuned models, where Scratch-B consistently outperforms the fine-tuned models. Morever, when we extend the standard training of large model, the above observation still holds. In this section, we demonstrate that the value of automatic network pruning methods actually lies in searching efficient architectures. We use Network Slimming as an example automatic method. Parameter Efficiency of Pruned Architectures. In FIG0 that uniformly prunes the same percentage of channels in all layer. All architectures are trained from random initialization for the same number of epochs without sparsity regularization. We see that the architectures obtained by Network Slimming are more parameter efficient, as they could achieve the same level of accuracy using 5× fewer parameters than uniformly pruning architectures. Generalizable Design Principles from Pruned Architectures. Given that the automatically discovered architectures tend to be parameter efficient, here we show a generalizable principle on designing better architectures. We show two design strategies: 1) "Guided Pruning": use the average number of channels in each layer stage (layers with the same feature map size) from pruned architectures to construct a new set of architectures. 2) "Transferred Guided Pruning": use these patterns from a different architecture on a different dataset. FIG0 (right) shows our . Here for "Transferred Guided Pruning", we use the patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100. We observe that both "Guided Pruning" (green) and "Transferred Guided Pruning" (brown) can both perform on par architectures directly pruned on the VGG-19 and CIFAR-100 (blue), and are significantly better than uniform pruning (red). This means we could distill generalizable design principles from pruned architectures. In practice, for methods with predefined target architectures, training from scratch is more computationally efficient and saves us from implementing the pruning procedure and tuning the additional hyper-parameters. Still, pruning methods are useful when a pretrained large model is already given, in this case fine-tuning is much faster. Also, obtaining multiple models of different sizes can be done quickly by pruning from a large model by different ratios. In summary, our experiments have shown that training the small pruned model from scratch can almost always achieve comparable or higher level of accuracy than the model obtained from the typical "training, pruning and fine-tuning" procedure. This changed our previous belief that over-parameterization is necessary for obtaining a final efficient model, and the understanding on effectiveness of inheriting weights that are considered important by the pruning criteria. We further demonstrated the value of automatic pruning algorithms could be regarded as finding efficient architectures and providing architecture design guidelines. Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression.. Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network.. Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets.. Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks.. Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming.. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015. Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database.. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.. Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks.. Zehao Huang and Naiyan Wang. Data-driven sparse structure selection for deep neural networks..[ Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks.. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks.. Zhiqiang Shen, Zhuang Liu, Jianguo Li, Yu-Gang Jiang, Yurong Chen, and Xiangyang Xue. Dsod: Learning deeply supervised object detectors from scratch.. Jianwei Yang, Jiasen Lu, Dhruv Batra, and Devi Parikh. A faster pytorch implementation of faster r-cnn. https://github.com/jwyang/faster-rcnn.pytorch, 2017. Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning.. Here we provide additional details on our experiment setups. Implementation. In order to keep our setup as close to the original paper as possible, we use the following protocols: 1) If a previous pruning method's training setup is publicly available, e.g. and, we adopt the original implementation; 2) Otherwise, for simpler pruning methods, e.g.,, we re-implement the three-stage pruning procedure and achieve similar to the original paper; 3) For the remaining two methods, the pruned models are publicly available but without the training setup, thus we choose to re-train both large and small target models from scratch. Interestingly, the accuracy of our re-trained large model is higher than what is reported in the original paper 3. In this case, to accommodate the effects of different frameworks and training setups, we report the relative accuracy drop from the unpruned large model. The of two pruning methods are already shown in the extended abstract, here in appendix we provide of the remaining four pruning methods. We also include an experiment on transfer learning from image classification to object detection. ThiNet greedily prunes the channel that has the smallest effect on the next layer's activation values. As shown in TAB7, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almost always achieve better performance than the fine-tuned model, often by a significant margin. The only exception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16 (FLOPs reduced by 15×), and as a , drastically reducing the training budget for Scratch-E. The training budget of Scratch-B for this model is also 7 times smaller than the original large model, yet it can achieve the same level of accuracy as the fine-tuned model. TAB8. Again, in terms of relative accuracy drop from the large models, scratch-trained models are better than the fine-tuned models. In summary, for pruning methods with predefined target architectures, training the small models for the same number of epochs as the large model (Scratch-E), is often enough to achieve the same accuracy as models output by the three-stage pipeline. Combined with the fact that the target architecture is predefined, in practice one would prefer to train the small model from scratch directly. Moreover, when provided with the same amount of computation budget (measured by FLOPs) as the large model, scratch-trained models can even lead to better performance than the fine-tuned models. Sparse Structure Selection also imposes sparsity regularization on the scaling factors during training to prune structures, and can be seen as a generalization of Network Slimming. Other than channels, pruning can be on residual blocks in ResNet or groups in ResNeXt. We examine residual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26. TAB13 shows our . On average Scratch-E outperforms pruned models, and for all models Scratch-B is better than both. Table 5: Results (accuracy) for residual block pruning using Sparse Structure Selection. In the original paper no fine-tuning is required so there is a "Pruned" column instead of "Fine-tuned" as before. Non-structured Weight Pruning prunes individual weights that have small magnitudes. This pruning granularity leaves the weight matrices sparse, hence it is commonly referred to as nonstructured weight pruning. Because all the network architectures we evaluated are fully-convolutional (except for the last fully-connected layer), for simplicity, we only prune weights in convolution layers here. Before training the pruned sparse model from scratch, we re-scale the standard deviation of the Gaussian distribution for weight initialization, based on how many non-zero weights remain in this layer. This is to keep a constant scale of backward gradient signal. As shown in TAB11, on CIFAR datasets, Scratch-E sometimes falls short of the fine-tuned , but Scratch-B is able to perform at least on par with the latter. On ImageNet, we note that sometimes even Scratch-B is slightly worse than fine-tuned . This is the only case where Scratch-B does not achieve comparable accuracy in our attempts. We hypothesize this could be due to the task complexity of ImageNet and the fine pruning granularity. Effects of Sparsity Regularization. Some methods use sparsity regularization during the training of large, over-parameterized models, to smooth the following pruning process, and in finetuning, no such sparsity is imposed. In all of our experiments presented above, no sparsity is used for training the pruned models from scratch. Here we analyze the effects of using sparsity regularization (or not) for Network Slimming. TAB13 shows the when all training procedures are with sparsity regularization. We can see that Scratch-B are still able to be on par with the fine-tuned models. TAB14 shows the when we do not use sparsity regularization in any training procedures, including large model training. We can see that when no sparsity is induced, Scratch-B can also consistently achieve comparable with the fine-tuned models. We have shown that the small pruned model can be trained from scratch to match the accuracy of the fine-tuned model in classification tasks. To see whether this phenomenon would also hold for transfer learning to other vision tasks, we evaluate the L 1 -norm based pruning method on the PASCAL VOC object detection task, using Faster-RCNN.Object detection frameworks usually require transferring model weights pre-trained on ImageNet classification, and one can perform pruning either before or after the weight transfer. More specifically, the former could be described as "train on classification, prune on classification, fine-tune on classification, transfer to detection", while the latter is "train on classification, transfer to detection, prune on detection, fine-tune on detection". We call these two approaches Prune-C (classification) and Prune-D (detection) respectively, and report the in TAB15 shows our , and we can see that the model trained from scratch can surpass the performance of fine-tuned models under the transfer setting. Another interesting observation from TAB15 is that Prune-C is able to outperform Prune-D, which is surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classification models. We hypothesize that this might be because pruning early in the classification stage makes the final model less prone to being trapped in a bad local minimum caused by inheriting weights from the large model. This is in line with our observation that Scratch-E/B, which trains the small models from scratch starting even earlier at the classification stage, can achieve further performance improvement. Here we provide that complements Section 4 for non-structured weight pruning. The networks and datasets used are the same as those used in Section 4. We also discuss the relation with conventional architecture search methods at the end. Figure 3(left) shows our on parameter efficiency of architectures obtained by non-structured pruning. Here "Uniform Sparsifying" means uniformly sparsifying individual weights in the network at a fixed probability. It can be seen that pruned architectures are more parameter-efficient than uniform sparsified architectures. Figure 3(right) shows our for architectures obtained by other different design strategies. For "Guided Sparsifying" and "Transferred Guided Sparsifying", we use the average sparsity patterns of 3 × 3 kernel in each layer stage to design new structures. Similar to Network Slimming, both "Guided Sparsifying" (green) and "Transferred Guided Sparsifying" (brown) are significantly more parameter-efficient than uniform sparsifying (red).Comparison with Traditional Architecture Search Methods. Conventional techniques for network architecture search include reinforcement learning [20, BID0 and evolutionary algorithms BID1 BID2 .In each iteration, a randomly initialized network is trained and evaluated to guide the search, and the search process usually requires thousands of iterations to find the goal architecture. In contrast, using network pruning as architecture search only requires a one-pass training, however the search space is restricted to the set of all "sub-networks" inside a large network, whereas traditional methods can search for more variations, e.g., activation functions or different layer orders. Recently, BID3 uses a similar pruning technique to Network Slimming to automate the design of network architectures; BID4 prune channels using reinforcement learning and automatically compresses the architecture. On the other hand, in the network architecture search literature, sharing/inheriting trained parameters BID5 BID6 has become a popular approach to accelerate the convergence and reduce the training budget during searching, though it would be interesting to investigate whether training from scratch would sometimes yield better as we observed in network pruning. We can see that these two areas, namely network pruning and architecture search, share many common traits and start to borrow wisdom from each other. We show the weight distribution of unpruned large models, fine-tuned pruned models and scratchtrained pruned models, for two pruning methods: Network Slimming and non-structured weight level pruning. We choose DenseNet-40 and CIFAR-10 for visualization and compare the weight distribution of unpruned models, fine-tuned models and scratch-trained models. For Network Slimming, the prune ratio is chosen to be 60%. For non-structured weight level pruning, the prune ratio is chosen to be 80%. FIG5 shows our . We can see that the weight distribution of fine-tuned models and scratch-trained pruned models are different from the unpruned large models -the weights that are close to zero are much fewer. This seems to imply that there are less redundant structures in the found pruned architecture, and support the effect of architecture search for automatic pruning methods.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

In network pruning, fine-tuning a pruned model only gives comparable or worse performance than training it from scratch. This advocate a rethinking of existing pruning algorithms.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Clustering is the central task in unsupervised learning and data mining. k-means is one of the most widely used clustering algorithms. Unfortunately, it is generally non-trivial to extend k-means to cluster data points beyond Gaussian distribution, particularly, the clusters with non-convex shapes . To this end, we, for the first time, introduce Extreme Value Theory (EVT) to improve the clustering ability of k-means. Particularly, the Euclidean space was transformed into a novel probability space denoted as extreme value space by EVT. We thus propose a novel algorithm called Extreme Value k-means (EV k-means), including GEV k-means and GPD k-means. In addition, we also introduce the tricks to accelerate Euclidean distance computation in improving the computational efficiency of classical k-means. Furthermore, our EV k-means is extended to an online version, i.e., online Extreme Value k-means, in utilizing the Mini Batch k-means to cluster streaming data. Extensive experiments are conducted to validate our EV k-means and online EV k-means on synthetic datasets and real datasets. Experimental show that our algorithms significantly outperform competitors in most cases. Clustering is a fundamental and important task in the unsupervised learning . It aims at clustering data samples of high similarity into the same cluster. The most well-known clustering algorithm is the k-means, whose objective is to minimize the sum of squared distances to their closest centroids. k-means has been extensively studied in the literature, and some heuristics have been proposed to approximate it . The most famous one is Lloyd's algorithm . The k-means algorithm is widely used due to its simplicity, ease of use, geometric intuition . Unfortunately, its bottleneck is that computational complexity reaches O(nkd) , since it requires computing the Euclidean distances between all samples and all centroids. The data is embedded in the Euclidean space , which causes the failure on clustering non-convex clusters . Even worse, k-means is highly sensitive to the initial centroids, which usually are randomly initialized. Thus, it is quite possible that the objective of k-means converges to a local minimum, which causes the instability of k-means, and is less desirable in practice. Despite a stable version -k-means++ gives a more stable initialization, fundamentally it is still non-trivial to extend k-means in clustering data samples of non-convex shape. To solve these problems, this paper improves the clustering ability of k-means by measuring the similarity between samples and centroids by EVT . In particular, we consider the generalized extreme value (GEV) distribution or generalized Pareto distribution (GPD) to transform the Euclidean space into a probability space defined as, extreme value space. GEV and GPD are employed to model the maximum distance and output the probability that a distance is an extreme value, which indicates the similarity of a sample to a centroid. Further, we adopt the Block Maxima Method (BMM) to choose the maximal distance for helping GEV fit the data. The Peaks-Over-Thresh (POT) method is utilized to model the excess of distance exceeding the threshold, and thus very useful in fitting the data for GPD. Formally, since both GEV and GPD can measure the similarity of samples and centroids, they can be directly utilized in k-means, i.e., GEV k-means and GPD k-means, which are uniformly called Extreme Value k-means (EV k-means) algorithm. In contrast to k-means, EV k-means is a probability-based clustering algorithm that clusters samples according to the probability output from GEV or GPD. Furthermore, to accelerate the computation of Euclidean distance, We expand the samples and the centroids into two tensors of the same shape, and then accelerate with the high performance parallel computing of GPU. For clustering steaming data, we propose online Extreme Value k-means based on Mini Batch kmeans . When fit the GEV distribution, we use mini batch data as a block. For the fitting of GPD, we dynamically update the threshold. The parameters of GEV or GPD are learned by stochastic gradient descent (SGD) . The main contributions are described as follows. This paper utilizes EVT to improve k-means in addressing the problem of clustering data of non-convex shape. We thus propose the novel Extreme Value k-means, including GEV k-means and GPD k-means. A method for accelerating Euclidean distance computation has also been proposed to solve the bottleneck of k-means. Under the strong theoretical support provided by EVT, we use GEV and GPD to transform Euclidean space into extreme value space, and measure the similarity between samples and centroids. Based on Mini Batch k-means, We propose online Extreme value k-means for clustering streaming data, which can learn the parameters of GEV and GPD online. We corroborate the effectiveness of EV k-means and online EV k-means by conducting experiments on synthetic datasets and real datasets. Experimental show that EV k-means and online EV k-means significantly outperform compared algorithms consistently across all experimented datasets. k-means and EVT have been extensively studied in the literature in many aspects (; ;). Previous work on k-means focused on the following aspects, such as determining the optimal k, initializing the centroids, and accelerating k-means.;; Van der; propose to select the optimal k value based on the genetic algorithm. Initializing the centroids is a hot issue in k-means . k-means++ is the most popular initialization scheme.;; proposed density-based initial centroid selection method, that is, selecting the initial cluster center according to the density distribution of the samples. propose using Markov chain Monte Carlo to accelerate k-means++ sampling. There is also a lot of work focused on solving the computational complexity of k-means. argued that using triangle inequality can accelerate k-means. showed that randomly sparse the original data matrix can significantly speed up the computation of Euclidean distance. EVT is widely used in many area, such as natural phenomena, finance, and traffic prediction. In recent years, EVT has many applications in the field of machine learning. However, far too little attention has been paid to the combination of k-means and EVT. proposes using generalized extreme value distribution for feature learning based on k-means. However, our method is significant different from this method. First, they compute the squared distance from a point to the nearest centroid and form a GEV regarding to each point, while we compute the distance from a centroid all data points and then fit the GEV or GPD regarding to each centroid. Second, their algorithm adds the likelihood function as a penalty term into the objective function of k-means, but our algorithm changes the objective function by fitting the GEV or GPD for each centroid and assign the data point to the one with the highest probabilities they belong to. Finally, this paper also presents GPD k-means and online Extreme Value k-means, which is not stated in this paper. The sum squared error is defined as Eq. indicates that the smaller J is, the higher degree of closeness between the samples and their centroids in the clusters, so the similarity of the samples in the clusters is higher. To find the global minimum of Eq. 1, we need to compute all possible cluster partitions, so k-means is an NP-hard problem . Lloyd's algorithm uses a greedy strategy to approximate the Eq. by iteratively optimizing between assigning cluster labels and updating centroids. Specifically, in assigning cluster labels, a cluster label is assigned to each sample according to the closest centroid. When the centroids are being updated, each centroid is updated to the mean of all samples in the cluster. These two steps loop iteratively until the centroids no longer change. In this subsection, we first introduce the statistical aspects of a sample maximum in Extreme Value Theory (EVT), which is a branch of statistics dealing with the stochastic behavior of extreme events found in the tail of probability distribution. Let X 1, X 2,..., X n be a sample of independent copy of X with distribution F. It is theoretically interesting to consider the asymptotic behavior of sample maximum and upper order statistics. More specifically, denote M n = max 1≤i≤n X i as the sample maximum, whose distribution is On the other hand, the upper order statistics of the sample is related to the survival function over a threshold u, which is EVT considers the non-degenerated limit when n → ∞ in Eq. and u ↑ x * in Eq. by re-scaling the objects, which is presented as the conditions of the maximum domain of attraction for F. Theorem 3.1 (Fisher-Tippett Theorem ) A distribution function F satisfis the condition of a maximum domain of attraction: if there exists a constant ξ ∈ R and sequences a n > 0, b n, n ∈ N such that The shape parameter ξ is called the extreme value index. Theorem 3.1 motivates the Block Maxima Method (BMM) : for the block size s ∈ {1, 2, . . ., n}, divide the sample into m = n/s blocks of length s. Since the data is independent, each block maxima has distribution F s and can be approximated by a three-parametric generalized extreme value distribution (GEV) G GEV (·; µ, σ, ξ) when the block size s is large enough and the number of blocks m is sufficient. The class of GEV distributions is defined as We treat the case of ξ = 0 as the limit of ξ → 0. An equivalent representation of the maximum domain of attraction condition is as follows: Theorem 3.2 (Pickands-Balkema-de Haan Theorem ) A distribution function F satisfies the condition of maximum domain of attraction: if there exists a constant ξ ∈ R and a positive function σ(t) such that where x * denotes the right end-point of the support of F. The clustering of GPD k-means in three isotropic Gaussian blobs. The color of the surface and contour in the figures represent the probability density of GPD. The closer to yellow, the greater the probability density. The closer to blue, the smaller the probability density. The upper order statistics of a sample usually provides useful information about the tail of the distribution F. Then Theorem 3.2 gives rise to an alternative peak-over-threshold (POT) approach : given sufficient large threshold u in Eq., we have that, for any X i > u, its conditional distribution can be approximated by a two-parametric generalized Pareto distribution (GPD) G GP D (·; σ, ξ), which is defined as Similarly, we treat the case of ξ = 0 as the limit of ξ → 0. The POT approach focuses on the excess over the threshold u to fit the GPD and asymptotically characterize the tail features of the distribution, while the BMM only approximates the GEV distribution when m is large enough. The BMM only uses a very small amount of dataset, and there may be cases where the submaximal value of one block is larger than the maximum value of the other block, which cannot be utilized. In contrast, POT method uses all data beyond the threshold to fit the GPD, making full use of the extreme data. However, there is no winner in theory. 4 THE EXTREME VALUE k-MEANS ALGORITHM 4.1 MEASURING SIMILARITY BY EXTREME VALUE THEORY Measuring similarity with Euclidean distance is the core step of k-means clustering. Similarly, for all clustering algorithms, how to measure the distance (dissimilarity) or similarity between the samples and the centroids is a critical issue as it determines the performance of the algorithm. However, due to the properties of Euclidean distance, k-means fails for clustering non-convex clusters. Therefore, this paper proposes to use the EVT to transform the Euclidean space into a probability space called the extreme value space. Fig. 1(a) demonstrates measuring similarity by GEV or GPD. The Euclidean distances from µ 1 and µ 3 to x t is much larger than the Euclidean distances from µ 1 and µ 3 to the most of surrounding samples, i.e. x t − µ 1 2 x i − µ 1 2, x i ∈ C 1 and x t − µ 3 2 x i − µ 3 2, x i ∈ C 3. Therefore, x t − µ 1 2 and x t − µ 3 2 are maximums concerning x i − µ 1 2, x i ∈ C 1 and x i − µ 3 2, x i ∈ C 3 with different degree. We want a distribution that can be used to model maximum distance and reflect the probability that a distance is belong to a cluster, which equivalent to the similarity between the sample and the centroid. Obviously, the EVT is a good choice. As described in Section 3.2, the BMM can be applied to fit the GEV distribution. In order to fit a GEV distribution for each cluster, we first compute the Euclidean distance d ij between Θ = {µ 1, µ 2, . . ., µ k} and sample x i ∈ X, i.e., d ij = x i − µ j 2, i ∈ {1, 2, . . ., n}, j ∈ {1, 2, . . ., k}. For the centroid µ j, its distances to all samples is denoted by d j = {d 1j, d 2j, . . ., d nj}. Then we divided them equally into m blocks of size s = n m (possibly the last block with no sufficient observations can be discarded), and then the maximum value of each block is taken to obtain the block maximum sequence M j. We use M j to estimate the parameters of GEV distributions for cluster C j. We assume the location parameter is zero for the reason that the position of centroids change small in the later stage of clustering. The most commonly used estimating method, maximum likelihood estimation (MLE), is implemented to estimate the two parameters of the GEV. The log likelihood function of GEV is derived from Eq., σj > 0 when ξ j = 0. We get the estimated valueσ j andξ j of σ j and ξ j by maximizing L GEV. Alternatively, we use the POT method to model the excess of Euclidean distance d j exceeding threshold u j for centroid µ j and fit the GPD. We first compute the excess that is defined as where k j is the total number of observations greater than the threshold u j. Then we also implement MLE to estimate the parameters of the GPD. The log likelihood function of GPD can be derived from Eq., 0 when ξ j > 0 and 0 y j i − σj ξj when ξ j < 0. We get the estimated valueσ j andξ j of σ j and ξ j by maximizing the L GP D. Finally, we can obtain the probability that x i belong to cluster C j through the GEV and GPD: The traditional k-means clusters samples in view of the closeness to the centroids of clusters. As described in Section 4.1, we can model the distribution classes of GEV and GPD to measure the similarity between the samples and the centroids. Thus we propose GEV k-means and GPD k-means, which are uniformly called the Extreme Value k-means (EV k-means) algorithm. In contrast to k-means, the proposed EV k-means is a probability-based clustering algorithm as it instead clusters samples by the probability output from GEV or GPD. The larger the block size s and the threshold u of BMM and POT, the smaller the deviation of MLE, but the larger the variance of MLE. Conversely, the smaller the block size s and the threshold u, the larger the deviation of the MLE, but the smaller the variance of the MLE. How to choose these two hyperparameters has not yet had a standard method, and it is necessary to comprehensively balance the relationship between deviation and variance in practical applications. Therefore, we set the block size by grid search and set threshold adaptively. Specifically, we first set the hyperparameter α to indicate the percentage of excess for all samples. Then we sort d j from big to small, and the u is set to the αn-th of sorted d j. The algorithm of GEV k-means has three steps: Given the dataset X, block size s and k initial centroids (obtained randomly or using k-means++ algorithm). During the step of fitting a GEV distribution, we firstly use BMM to select the maximal sample data M j for µ j. Then, we estimate the GEV parametersσ j andξ j by MLE using M j for µ j. So each cluster has its own independent GEV distribution. In the assigning labels step, each sample is assigned a cluster label based on the maximum probability, i.e., λ i = arg max j∈{1,2,...,k} P ij. In the updating centroid step, each centroid is updated to the mean of all samples in the cluster, i.e., µ i = 1 |Ci| x∈Ci x. There three steps are iterated until the centroids no longer change. The algorithm of GPD k-means is very similar to GEV k-means, except the fitting GPD distribution step. GPD k-means use the POT to model the excess y j of Euclidean distance d j exceeding threshold u j and fit the GPD. Fig. 1(b) and Fig. 1(c) show clustering of GPD k-means in three isotropic Gaussian blobs and show that the closer to the centroids, the greater the probability density. The main bottleneck of the k-means is the computation of Euclidean distances for the reason that the Euclidean distances between all samples and all centroids need to be computed. In naïve implementation, double-layer nested for loop is often used to perform operations on the CPU, which is very slow. This paper proposes an accelerated computation method to solve this bottleneck. Firstly, let matrix X ∈ R n×d represents samples consisting of n d-dimensional samples, and matrix C ∈ R k×d represents centroids consisting of k d-dimensional centroids. Secondly, insert a dimension between the two dimensions of matrix X and copy X along the new dimension to tensor X with shape of [n, k, d]. A similar operation for matrix C, adding a new dimension before the first dimension and copy C along the new dimension to tensor C with shape of [n, k, d]. Finally, the Euclidean distances between all samples and all centroids are D i,j = X − C 2, i ∈ {1, 2, . . ., n}, j ∈ {1, 2, . . ., k} that can be accelerate with the advantages of GPU parallel computing. The overall Extreme Value k-means algorithm is illustrated in Algorithm 1. In the era of Big Data, data is no longer stored in memory, but in the form of streams . Therefore, clustering streaming data is a significant and challenging problem. It is indispensable to design an Extreme Value k-menas algorithm that can learn online for clustering streaming data. This paper proposes the online Extreme Value k-means for clustering streaming data based on Mini Batch k-means . When fit the GEV distribution, we use mini batch data as a block and choose the maximum value for learning the parameters of GEV online. For the fitting of the GPD, the online EV k-means can dynamically update the threshold u and learn the parameters of GPD online. The Online Extreme Value k-means algorithm is illustrated in Algorithm 2. The algorithm randomly choose a mini batch contains b samples from the data stream each iteration. On the first iteration, it initializes the parameters of each GEV or GPD, and initializes centroid C on the mini batch. Then compute the Euclidean distances D using the accelerated computation method we proposed, update u j to tαn-th of sorted h, and compute the maximum M j and excess y j. Because the GEV and GPD parameters have not been updated at the first iteration, so P ij cannot be computed. Therefore, from the second iteration, the algorithm clusters the mini batch samples based on Mini Batch kmeans. Finally, the negative log-likelihood function of all GEVs or GPDs is summed to obtain L s, and the L s is minimized by SGD to update the parameters of GEV or GPD, which is equivalent to maximizing We evaluate the performance of the clustering algorithm by three widely used metrics, unsupervised clustering accuracy (ACC) , normalized mutual information (NMI) , and adjusted rand index (ARI) . Note that the values of ACC and NMI are Input: samples X ∈ R n×d, number of cluster k, block size s for GEV k-means, the percentage of excess α for GPD k-means Output: clusters C Initialize centroid C ∈ R k×d; repeat Cj = ∅, 1 j k; Perform transformation on X and C to obtain X and C, and then compute D = X − C 2; for j = 1, 2,..., k do // GEV k-means Obtain M j from d:.j by BMM; Estimate theσj,ξj by MLE on M j; // GPD k-means Obtain y j from D:,j by POT; Estimate theσj,ξj by MLE on y j; end for i = 1, 2,..., n do λi = arg max j∈{1,2,...,k} Pij; x; end until centroids no longer change; return clusters C; LGEV (M j ; σj, ξj) // online GEV k-means LGP D (y j ; σj, ξj) // online GPD k-means Compute the gradient Ls and then update the parameters of the GEV or GPD; end return centroid C; Figure 2: Visualization of six synthetic datasets shows the of our Extreme Value k-means compared to k-means, k-means++, k-medoid, bidecting k-means and spectral clustering. The from top to down are the clustering on the datasets D1, D2, D3, D4, D5 and D6, respectively. The nine algorithms from the first column to the ninth column are respectively k-means, k-means++, k-medoid, bidecting k-means, spectral clustering, GEV k-means (RM), GEV k-means (++), GPD k-means (RM), GPD k-means (++). in the range of 0 to 1, with 1 indicating the best clustering and 0 indicating the worst clustering. The value of ARI is in range of -1 to 1, -1 indicates the worst clustering, and 1 indicates the best clustering. We demonstrate our algorithm compared to other algorithms on six two-dimensional synthetic datasets. As illustrated in Fig. 2, there are the clustering of the datasets D1, D2, D3, D4, D5 and D6 from top to down. D1 consists of 5 isotropic Gaussian clusters, each of which has 100 samples. D2 consists of 15 isotropic Gaussian clusters, each of which has 4000 samples. D3 consists of two'C'-shaped clusters in the same direction, each of which has 250 samples. D4 consists of two clusters, each of which has 500 samples including a Gaussian blob and a'C'-shaped region. D5 consists of a Gaussian cluster having 500 samples and a'C'-shaped cluster having 250 samples. The difference between D5 and D4 is that the lower cluster in D5 has no'C'-shaped region, and Gaussian blobs has a larger variance. In D5, the upper cluster has 500 samples and the lower cluster has 250 samples. The centroids of GEV k-means and GPD k-means can be initialized randomly or using k-means++. Let'RM' and'++' denote randomly and using k-means++ initialize centroids, respectively. Therefore, there are nine algorithms in this experiment, k-means, k-means++, k-medoid , bisecting k-means , spectral clustering , GEV k-means (RM), GEV k-means (++), GPD k-means (RM), GPD k-means (++), respectively. From the clustering of the nine algorithms on six different synthetic datasets in Fig. 2, it can be seen that our four algorithms can successfully cluster convex and non-convex clusters, but the clustering of k-means and k-means++ on non-convex but visibly well-separated clusters are completely unsuccessful. In addition, the clustering k-medoid, bisecting k-means, spectral clustering on D3, D4, D5 is worse than our four algorithms. We evaluated the proposed EV k-means on nine real datasets: iris (n = 150, d = 4, k = 3), breast cancer (n = 683, d = 10, k = 2), live disorders (n = 145, d = 5, k = 2), heart (n = 270, d = 13, k = 2), diabetes (n = 768, d = 8, k = 2), glass (n = 214, d = 9, k = 6), vehicle (n = 846, d = 18, k = 4), MNIST and CIFAR10. The first seven datasets are available from LIBSVM Data website 1. MNIST is a dataset comprises 60,000 training gray-scale images and 10,000 gray-scale images of handwritten digits 0 to 9. Each of the training images is represented by an 84-dimensional vector obtained by LeNet . So the MNIST dataset we use has 60,000 samples with 84 features belonging to 10 classes, i.e., n = 60, 000, d = 84, k = 10. CIFAR10 is a dataset containing 50,000 taining and 10,000 test color images with 32 × 32 pixels, grouped into 10 different classes of equal size, representing 10 different objects. Each of the training images is represented by a 512-dimensional vector extracted by a ResNet-18 . Therefore, the CIFAR10 we use in the experiment has 50,000 samples with 512 features grouped in 10 classes, i.e., n = 50, 000, d = 84, k = 10. We repeat each experiment 10 times with different random seeds and took the mean of the of 10 times experiments as the final . In each of the experiments, all algorithms that initialize centroids randomly or by using k-means++ start from the same initial centroids. The of EV k-means on real datasets are shown in Tab. 1. As shown in Tab. 1, our proposed EV k-means on some datasets are comparable to other algorithms, and outperform other algorithms on MNIST and CIFAR10. We compare online EV k-means with k-means, k-means++, Mini Batch k-means (RM) and Mini Batch kmeans (++) on MNIST and CIFAR10. As illustrated in Tab. 2, the values of the three metrics of online EV k-means are slightly smaller than the values of EV k-means. However, the values of the three metrics of Mini Batch k-means are much smaller than the values of k-means. For example, the values of the three metrics of Mini Batch k-means on MNIST are 10%, 17%, 8% smaller than the values of k-means. However, the values

Represent the TLDR for finding the corresponding scientific article

This paper introduces Extreme Value Theory into k-means to measure similarity and proposes a novel algorithm called Extreme Value k-means for clustering.

Represent the short summary for finding the corresponding scientific article (long)

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the article for finding a single-sentence TLDR

Question: Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum. This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning. We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space. In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards. Reinforcement learning (RL) aims to learn the optimal policy of a certain task by maximizing the cumulative reward acquired from the environment. Recently, deep reinforcement learning has enjoyed great successes in many domains with short-term and dense reward feedback BID16 ) (e.g. Atari games, TORCS). However, many real world problems are inherently based on sparse, binary rewards, where the agent needs to travel through a large number of states before a success (or failure) signal can be received. For instance, in the grasping task of a robotic arm, the most accurate task reward would be a binary reward only if the agent successfully picked up the the object or not. This kind of goal-oriented tasks with sparse rewards are considered the most difficult challenges in reinforcement learning. That is, the environment only provides a final success signal when the agent has accomplished the whole task. The search space of these tasks may exponentially expand as state chain extends, which adds to the difficulty of reaching the final goal with conventional reinforcement learning algorithms. There have been multiple lines of approaches for tackling such sparse reward problems, including the ideas based on intrinsic motivation BID23 BID3 BID10, hierarchical reinforcement learning BID6 BID13, curriculum learning BID12 BID24 and experience-based off-policy learning BID2. Recently, a particularly promising approach is the idea of reverse curriculum generation BID7. By reversing the traditional reinforcement learning process and gradually expanding the set of start states from the goal to the starting point, this method does achieve substantial progress in some simulation tasks. However, the newly sampled start set mixes old and new states, which leads to inefficiency in training, as the agent spends a large amount of time reviewing rather than learning new skills. This problem aggravates in large state space tasks where the expansion could be really slow, and the storing of history data is also impracticable due to the memory limitation. To solve goal-oriented tasks and avoid the drawbacks mentioned above, we propose a Tendency Reinforcement Learning (TRL) architecture which shapes a reward function with former experience and use it to stabilize and accelerate training. To achieve this, a discriminative reward (the "tendency") is shaped to output a judgment on the success tendency of each state, and the reward function for the agent is hybrid, which combines the final goal reward with the tendency hints. We define a set of start states as a phase, and continue extending it from the final goal, until it reaches where the task starts. In each phase, the tendency reward influences the agent's training. After training each phase, the tendency reward is updated using collected experience. This mutual promotion contributes to a dramatic improvement in training efficiency and no longer requires keeping all history data. In this paper, we introduce three novel techniques: First, we propose a hybrid reward design that combines the final reward with tendency hints. Secondly, with the introduction of a phase administrator, the curriculum for each phase is automated, which increases the efficiency. Lastly, we develop an optional keyframe scheme, that is, our framework is also compatible with additional keyframes with no strict accuracy requirement, as long as they are effective at reducing irrelevant search space. The major contribution of this work is that we present a reliable tendency reinforcement learning method that is capable of training agents to solve large state space tasks with only the final reward, taking raw pixels as perception. In our experiments, we apply and test the model in diverse domains, including a maze game, Super Mario Bros and 3 simulated tasks (a robotic grasping task, a crane conveyance task and a pick-and-place challenge), where it proves more efficient and competitive than the prior works. Recently, great progress has been made in RL by approximating the policy and value functions through neural networks BID14 BID22 BID16. However, these RL methods often suffer from extremely slow training progress or fail to learn in sparse reward tasks. There have been several lines of approaches to tackling such problems, namely curriculum learning, hierarchical RL, intrinsic motivation and demonstration-augmented RL, each with its own constraints and assumptions. Curriculum learning works by deconstructing a difficult, long-horizon learning task into a sequence of easy to hard tasks BID12. While classic curriculum is often defined manually, recently in RL a number of work has shown promising automated curriculum generation BID20 BID24; BID7. Our work is closest to this line of work. In particular, reverse curriculum generation BID6 has proposed a very similar curriculum generation through exploration backwards from the goal states, and has shown promising in solving a variety of sparse reward problems. However, the critical difference of our approach from theirs is that we propose the use of the tendency reward trained discriminatively to summarize the past histories of successful and unsuccessful states, allowing our algorithm not to store the whole history of starting sets and further provide a dense shaping reward. Empirically our design choice is found to scale to larger state tasks with longer planning horizon better than this prior approach. Hierarchical RL and intrinsic motivations are two other approaches for tackling the sparse reward problems. Hierarchical RL (; BID13 learns to plan using a hierarchical of low-level policies, often directly provided to the high-level planner, to drastically reduce the search dimension of the problem, while intrinsic motivation often provides additional dense rewards in terms of novelty BID23 BID10 BID3 or some notion of empowerment BID17 to improve the exploration under no rewards. However, both approaches assume forward-based exploration to solve the sparse reward task, which becomes increasingly difficult as the dimensionality and the time horizon increase, unless strong structured prior is provided in HRL. Our method, with the similar assumptions as in reverse curriculum learning, allows much more efficient learning and exploration. Lastly, there are a number of work that utilizes demonstrations to tackle such problems. Inverse reinforcement learning infers the underlying reward function from expert demonstrations BID18 BID0), and recent work has used such reward function as potential-based shaping reward to accelerate learning. Another line of work uses demonstrations that consist of keyframes to do Keyframe-based Learning . These methods, however, make strong assumptions about the quality of demonstrations they receive. By contrast, our optional keyframe scheme only requires a small number of key states instead of several full demonstration trajectories, and is robust to noises. Specifically, for a challenging robotic manipulation task from perception, we only required 5 key states to solve the task. Train policy π beginning from P 1 with environmental reward; Collect trajectories & train Tendency Reward T (·); DISPLAYFORM0 Train policy π beginning from P i+1 with hybird Reward; Update Tendency Reward; i ← i + 1; return π; We revisit the conventional reinforcement learning setting where an agent interacts with the environment Env sequentially. At each time step t, the agent receives an observation O t and chooses an action a t from a set of action space A according to its policy π. Then the environment will give the agent the next state s t+1 and a reward r t. The goal of the agent is to maximize the expected return from each state s t through accumulating the reward by R t = ∞ k=0 γ k r t+k. The tendency reinforcement learning model is based on three assumptions that follows BID7 ) and hold true for a variety of practical learning problems. Assumption 1. We can arbitrarily reset the agent into any start state s ∈ P i.Assumption 2. At least one state s g is provided such that s g ∈ S g.Assumption 3. The Markov Chain induced by taking uniformly sampled random actions has a communicating class 1 including all start states in P task and the given goal state s g.Assumption 1 can be satisfied in many robotic manipulation problems by storing the start states in P i with low-dimensional data (e.g. angle of joints and velocity of motors). Acquiring these data and sending commands to let robotic apply these settings are widely provided by standard robotic APIs, and such functions are similar in game engines. This assumption enables the agent to train from recommended start states, which proves to increase learning efficiency BID11. Combining Assumption 1 with Assumption 2, we are able to reset the state to s g. Assumption 3 ensures that there exists a trajectory formed from a sequence of actions from any start state in P task to s g, and vice versa. This assumption can be satisfied by many robotic manipulation problems and games, as long as there are no major irreversibilities in the system. Symbols used in our paper are listed in table 1 for reference. Naïve applications of conventional RL tend to fail in goal-oriented tasks with a large state space and sparse rewards. However, this kind of tasks are common in real world. TRL is able to tackle them by reducing aimless searching with the application of 1) a discriminative tendency reward shaping that utilizes the agent's former experience; 2) a phase administrator that assigns customized curriculum to the agent; 3) a keyframe scheme that shrinks search space and accelerates learning in large state space multistage tasks (optional).Our key idea is to let the agent utilize previous experiences through a tendency classifier trained on history trajectories in a supervised manner (Sec. 4.1). We define each phase as a set of start states. In training, curriculum are automatically adjusted by a phase administrator (Sec. 4.2). In each phase, the agent learns from rewards produced by the tendency classifier, and trains the classifier with new experiences. The overview of training framework is shown in FIG0 and the corresponding algorithm is sketched in Alg. 1. The tendency reward in our model is a binary classifier (e.g. SVM, NN) that takes an observation O as input and outputs a judgment on whether that state leads to the final success. We name it tendency reward mainly because it provides slight guiding hints to speed up the training process in each phase and those large state space tasks thus become trainable. In training, each time when the agent steps onto a state, it would refer to the tendency reward for a judgment. If the state is evaluated as "favorable", the agent would receive a positive tendency hint as a reward. Otherwise, a negative reward would be issued. To prevent the agent from stalling at a favorable state for unlimited tendency rewards, we defined that such revisited states do not provide any reward, and applied a Gaussian kernel to detect them. The kernel is used with d most recent frames in a bid to detect repeated states, as is shown in Eq. FORMULA2: DISPLAYFORM0 We assume that s now is a revisited state if ρ is larger than 0.9. The parameter d is 20% of the max steps of each environment and σ = 1 & δ = 0 are constants in our experiments. The binary tendency classifier trains on both success and failure trajectories and we use logistic loss as the classifier loss. The hybrid reward is defined in Eq.. DISPLAYFORM1 where λ is a scale factor (10 −3 in our experiments) based on the following hypothesis: a positive reward received from the environment should be more important than the output of T (·). Since the tendency hints are very subtle compared to the final reward, the agent's desire to achieve the final goal won't be interfered with. Each time when the agent has made some progress in P current (being able to accomplish the task starting from 80% of start set), the x current for the generation of P current+1 will be adjusted by the phase administrator based on the agent's performance. During the i th phase generation process, the agent start from P i and randomly take x i steps. The newly reached states would then be included into P i+1 if the agent's success rate on that state is between a lower-bound α and an upper-bound β, which guarantee the new states to a proper difficulty (α is 0.1 and β is 0.9 in our experiments).In each phase generation, a reasonable x i is required. Large x i means less phases, while it might increase training iterations in P i+1. Therefore, in order to strike a balance, the phase administrator adjusts x i according to the number of training iterations N i in P i. This adjustment is achieved with a sigmoid function shown in equation. DISPLAYFORM0 whereN is the average of N 1, N 2,..., N i−1.The pseudo code of the phase administrator is sketched in Algorithm 2 in Appendix. B. In tasks with a vast state space, it is usually more difficult for P to reach any state s nearby P task. Those domains are difficult for both prior approaches and tendency RL. In those cases, we show that by providing just a few keyframes, our tendency RL algorithm can take advantage of them to shrink the search space. Ideally, the keyframes should be precise and indicate important states for the task, and in practice fine hand-engineering is often involved. However, for TRL, several rough key states would be sufficient. The agent still extends its phases with a regularly, while after each extension, it additionally checks whether the newly generated phase has covered some states nearby a certain key state (by a Gaussian kernel approximation function as is described in Eq. FORMULA5). If so, the next phase would be directly sampled from that key state rather than generated form P current. In this way, the search space can decrease remarkably. DISPLAYFORM0 If J is larger than 30% of K, P current is believed to cover some states around s ki. Then the next phase is sampled nearby s ki rather than extended from P current. Pseudo code is shown in Alg. 3 in Appendix. B.Note that our principle is distinct from LfD (learn from demonstration) methods in that the agent is not eager to follow the keyframes and doesn't get additional reward from them. In fact, these small number of keyframes are only used to shrink the search space, and our method also shows robustness to imperfect keyframe settings in experiment where we deliberately mix misleading states into the keyframes, and found that the influence is not significant (Sec. 5.3). To evaluate our model, we design five experiments 2 with different levels of difficulties FIG1. In all of these experiments, we apply A3C BID16 as the RL solver. The details of environments and network are elaborated in Appendix. C and D. We first test the efficiency and validity of our model. After testifying the correctness of the overall model, we visualize the distribution of tendency hints and the starting sets of phases. Then we focus on the optional keyframe scheme, verify its compatibility to our method and prove our method's robustness to keyframes of different qualities. Finally, we apply our model to long-term manipulation challenges with large state spaces to show our advantages. The training of the 40 × 40 maze task takes around 35000 training steps (1 hour and 40 minutes) with 6 threads on GPU. Then the agent can reach the exit from the start point of the maze with a 9 × 9 observation around it. To investigate the influence of tendency reward on training, we conduct an experiment comparing the training effect under three conditions: using tendency reward without history phases, using history phases without tendency reward and using neither tendency reward nor history phases. This is not only an ablation experiment but also a comparison with prior work, since the second condition is basically the reverse curriculum generation learning method BID7. The shows that our model performs well FIG2. We also show the improvement when using phase administrator which let the curriculum automated FIG3. observations. Phase administrator proves to improve training efficiency. The agent takes nearly 40000 training steps (around 21 hours) with 8 threads on GPU to learn how to complete the level 1-1 of Super Mario Bros. The policy learned by the agent is shown in Appendix. E.1.2. In order to find out the influence of the tendency reward, we traverse the whole states of some parts of the game and show both of the positive and negative tendency hints (Fig. 5). Figure 5: The distribution of tendency hints and generated phases in parts of Super Mario Bros game. The purple color represents positive hints while the blue ones indicate negative hints. The points in the figure indicate the starting points of Mario. Points with same color belong to a same phase. DISPLAYFORM0 As we can see in these figures, our agent has acquired the ability to find enemies nearby and dodge or even kill them by jumping up to their heads. Besides, it also recognizes the danger of the pits and can choose the right time to jump over them. These figures show the guiding effect of the tendency reward, which does a great help for the agent when it is far away from the final goal. We also visualize parts of the phases in Mario, which effectively lead the agent to constantly learn new states farther from goal state (Fig. 5). We then enlarge the Maze to 100 × 100 also with a 9 × 9 observation. With 8 key states as the keyframes, the training takes around 50000 training steps (7 hours and 14 minutes). We also test our agent's robustness to imperfect keyframes, related can be referred in Appendix. E.3.We also compare our model with two LfD methods: data-efficient RL BID21 and potential-based reward shaping (PBRS) from demonstration. The former resets the agent randomly to someplace on an expert trajectory while the latter uses an artificial potential reward shaped based on demonstrations. We test them with both reasonable and unreasonable keyframes / demonstrations. As is shown in FIG4, data-efficient RL is quite sensitive to the demonstration data and fluctuates constantly facing unreasonable demonstrations. PBRS from demonstration can achieve similar performance to TRL, but requires a lot more human efforts of hand-engineering its reward function (otherwise it just wouldn't work). Different from these methods, TRL does not need expert demonstration (10 labeled keyframes are sufficient) and requires no further human elaboration. Also, the training of TRL is the most stable of three all. The following experiments mainly focus on illustrating the practicality of our model on solving difficult, real-world robotic manipulation problems with minimal assumptions. When confronted with long-term challenges with large state space, to avoid wasting time on unnecessary states, our keyframe scheme is applied to shrink the search space. In the three experiments, we add less than 15 keyframes to assist training (5 in robotic arm, 5 in conveyance challenge of the crane and 14 in pick-and-place task), and the are quite satisfying. The grasping task on robotic arm simulation is a typical large state space problem, where the gripper can move anywhere within the range of the arm. In our design, a camera is fastened to the forepart of the gripper and provides a 64 × 64 depth image as the agent's observation. We intuitively design Learning curves of our model, the data-efficient RL and PBRS from demonstration with keyframes / demonstrations of different qualities. Tendency RL is able to achieve a quite satisfactory performance without much human labor.5 key states and train the whole task for 33360 iterations (4 hours 19 minutes) with 12 threads on GPU, and it turns out that our model can solve the task reliably. To further investigate the relationship between training efficiency and scale of keyframes, we conduct the same experiment with 5, 8, 11, 13, 15, 18 key points respectively, as is shown in Appendix. E.4. The conveyance challenge is also a 3D manipulation task with a larger state space. The agent should pick up the ball, take it to correct basket, and drop it right into that basket to receive a positive reward. The camera is also fixed on the gripper that produces 64 × 64 RGB images as input. Our model is trained for 99240 iterations (5 hours 2 minutes) with 12 threads on GPU. Since this task involves multiple operations and has a large state space, we roughly add five key states to help the agent learn different skills, as is shown in Appendix. E.5.With the intention of finding out the distribution of "favorable" and "adverse" states given by the tendency reward, we make the agent travel through the whole environment at an appropriate height under two conditions: 1) the gripper jaw is open with the target ball fixed on the center of plane; 2) the ball is grasped by the gripper and moves with it. Then we record the tendency reward at each position FIG5.As expected, areas close to the target basket are assigned the highest tendency hints, which is quite natural as these states are reached before of after a successful drop. On the other hand, locations above those incorrect baskets tend to incur negative tendency hints. More details can be found in Appendix. E.5. Of all the three experiments in robotic manipulation, this challenge has the largest state space due to endless combinations of the agent's location and orientation and enormous possible positions of plates. This time, the agent observes through a panorama camera that provides 160 × 32 first-person point of view. The whole task can be divided into several identical pick-and-place operations that resemble the conveyance challenge. We provide 14 intuitively designed keyframes to accelerate training (Appendix. E.6). However, these keyframes are not quite sufficient in such a huge state space, and the agent shows confusion from time to time. Our phase administrator plays a significant role in phase arrangement, and actually makes up for the lack of expert data. Since the agent performs well within each pick-and-place operation, new phases are more concentrated on the intervals between operations. We develop a tendency reinforcement learning algorithm that resolves complicated goal-oriented tasks, and evaluate it with multiple experiments based on raw perceptions. Our method consists of two core components: a binary tendency classifier that provides dense hints on whether a state leads to success and a phase administrator that generates a reverse curriculum. In five distinct environments, we find that tendency RL method is efficient and stable enough to converge on a reliable policy within hours even facing intractable problems, and does not require history data which is impossible to acquire in large space domains. One limitation of our current approach is that our model requires the environment to be capable of resetting to some arbitrary states, which is also a limitation of the reverse curriculum generation method BID7. A promising future direction is to additionally train reset controllers that automatically reset the system to the initial state following each episode, together with a corresponding forward controller BID8.In future work, our goal is to apply the tendency RL method to real world robotic manipulation tasks with perceptions. Furthermore, we plan to extend the binary tendency classifier into a predictor that outputs the expected success rate on current state, so that our model can also be applied to stochastic environments. Additionally, we intend to release a subset of our code and the trained models to facilitate further research and comparisons. Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. A DEFINITIONS AND FORMULAE • Final goal: States that indicate the task is successfully finished (e.g. the exit in maze).• Phase: A set of start states at a certain time period.• Start state: A state where the agent starts for the final goal in a certain phase.• Success state: The states leading to the final goal by current policy.• Task start phase: States where the entire task begins, which is the final aim of phases extensions (e.g. the start point in maze).• Keyframe: A small number of key states artificially designed to shrink the search space in large space tasks. Algorithm 2: Phase Administrator input: Current phase P i, number of training iterations in the i th phase N i. output: The next phase P i+1. Calculate x i from N i with Equation; DISPLAYFORM0 Algorithm 3: TendencyRL with optional keyframes input: Goal state s g, key states s k1,..., s kn, task start phase P task. output: Trained policy π for the extreme large space task. DISPLAYFORM0 {Q j}; DISPLAYFORM1 return π;C ENVIRONMENT DETAILS C.1 MAZE Our first environment is a maze task. The size of the whole maze is 100×100. There is only one exit (final goal) in the middle of the bottom line and one starting point at. The agent is required to find its own way to move from the starting point to exit with a limit observation covering a 9 × 9 area around it. At each time-step, the agent has four actions, namely walk north, walk south, walk east and walk west, expressed as (N, S, E, W), and transitions are made deterministically to an adjacent cell, unless there is a block, in which case no movement occurs. The environment only gives 1 point reward when the agent has accomplished the finial goal. On other conditions, the reward remains 0. The second environment is level 1-1 of Super Mario Bros BID4 koltafrickenfer, 2017; asfdfdfd, 2012). The agent should complete the level from start point via a 16×13 observation, which indicates the nearby situation. When the agent reaches its destination, it would receive 1 point reward, otherwise it receives 0 point. The agent has 5 actions: jump, right, left, jump to right and jump to left, expressed as (J, R, L, JR, JL). Mario could not achieve any reward by killing enemies or collecting coins. The third environment is a grasping task on a robotic arm simulation in MuJoCo. The end effector of the 6-DOF robotic arm is a two-finger gripper with a camera attached between the fingers, which produces 64 × 64 overlooking depth images as the agent's observations. In order to manipulate in a more intuitive fashion, poses of joints have been mapped into 3D coordinates, enabling the gripper to move in 6 different directions. There is also a special grasping action that closes the gripper jaws, by which the target cube can be grasped. The agent is initialized randomly above a plane, and 1 point reward would be received if and only if the target cube placed on the plane is captured by the gripper. The fourth task is a conveyance challenge of a crane simulated by the physics engine MuJoCo. The agent can interact with a ball placed on a plane using a 3-finger gripper connected to the crane, with 8 actions available (6 for free movements and 2 for gripper manipulations). Surrounding the ball from a distance are some target baskets, among which one basket is randomly selected and colored green. The agent is able to observe through a camera fixed on the gripper that produces 64 × 64 RGB images, and only by dropping the ball into the green basket would the agent receive 1 point reward. C.5 RECURRENT PICK-AND-PLACEThe fifth task involves pick-and-place operations simulated in MuJoCo, which requires the agent to stack some plates into a neat pile. This time, the agent observes through a panorama camera that provides 160 × 32 first-person point of view. Fixed before the camera is a electromagnet capable of attracting one plate at a time, which is used for plate transfer. Available actions include turning left / right, stepping forward / backward, charging / releasing the electromagnet. The final goal is to gather and pile up all the plates into a straight stack, which is the only approach for the agent to obtain 1 point reward. D NETWORK ARCHITECTURES The convolution neural network of the policy network has a first layer with 12 4 × 4 filters of stride 1, followed by a layer with 24 3 × 3 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1. Then we add a LSTM layer with 384 units which followed by a fully connected layer with 96 hidden units. The convolution neural network of the policy network has a first layer with 48 4 × 4 filters of stride 1, followed by a layer with 48 3 × 3 filters of stride 1, followed by a layer with 96 2 × 2 filters of stride 1. Then we add a LSTM layer with 960 units which followed by a fully connected layer with 240 hidden units. The convolution neural network of the policy network has a first layer with 12 4×4 filters of stride 1, followed by a layer with 24 3 × 3 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1, followed by a layer with 48 2 × 2 filters of stride 1. Then we add a LSTM layer with 1536 units which followed by a fully connected layer with 96 hidden units. Especially, in the pick-and-place challenge, there is an additional convolution layer with 96 2 × 2 filters of stride 1, and the LSTM layer contains 480 units. E.1 SUPER MARIO BROS DETAILS E.1.1 TRAINING The training of Super Mario Bros is shown in FIG6. In Super Mario Bros, we compare our model with another one trained based on curiosity-driven BID19. We let the agents randomly move several steps at the beginning before run- ning the models, then compare the completion degree 3 of two models. With the environment only providing a reward when Mario reaches the final goal, we find that our method obtains absolute advantage (Table 2). We also use the model trained in level 1-1 to run the level 1-2 and 1-3 of the game. Since the latter two levels are not trained and require new skills that the first level does not include FIG7, both of the methods may suffer more failures. We try the maximum entropy IRL in the 100 × 100 maze task with more than two days' training and about 44G virtual memory used, but it still failed to form a feasible reward function. Since there are 10,000 states in the maze, the traditional IRL methods seem to suffer from the bottleneck of computing ability. To test our model's robustness to imperfect keyframe settings, instead of arranging the keyframes onto the optimal trajectory, we set some of them onto a byway. We choose 8 special phases sampled from the misleading keyframes and show the policy the agent has learned in these phases. As is shown in FIG0, in the sixth phase, the agent refuses to detour from the left side and finds a shortcut direct to the exit. A possible explanation is that our tendency reward only provide slight guiding hints which do not bother the agent's exploration for new policies. DISPLAYFORM0 Negative Hint Positive Hint FIG0: The agent's policies in 8 special phases sampled from misleading key states. The purple points are collected by combining the testing for 30 times. In the end the detouring keyframes are completely neglected by the agent. We conduct the grasping experiment with 5, 8, 11, 13, 15 and 18 keyframes respectively. The influence of keyframe scale on training efficiency is demonstrated in FIG0. FIG0: Learning curve of the grasping task with different numbers of keyframes. The training efficiency is relatively higher when 13 keyframes are used. If the number of keyframes is too large or too small, training tends to become less efficient, but still stable. An interesting observation in conveyance challenge is, the areas with positive tendency hints are more concentrated and the boundary between "favorable" and "adverse" states is more explicit when the gripper jaw is open. This is because the states where the agent is not holding the ball are often related to grasping and dropping, which require more accuracy than just transferring the ball. This experiment also proves that our agent has superb robustness to imperfect keyframe settings. In our previous assumptions, the agent should rise to a proper height where all baskets were completely in view, before moving towards the correct basket FIG0. However, it turns out that the agent learns to determine the target basket location at a lower height, where only portions of the baskets are in view. The regular reverse curriculum algorithm usually fails if there exists an irreversible process in the system. The irreversible process is defined as:∃s, s ∈ S: (∃n > 0 : P (s n = s |s 0 = s) > 0) ∧ (∀n > 0 : P (s n = s|s 0 = s) = 0)In such cases, the states s and s are not connected, and an agent starting from s will never reach s since that probability is 0. We define a absorbing state s a as a state that satisfies P (s a |s a) = 1 ∧ ∀s ∈ S: s = s a −→ P (a|s) = 0To be more generalized, we define a set S a of states to be a absorbing set if it satisfies P (s |s) = 0 if s ∈ S a ∧ s / ∈ S aConsider a phase extension progress where P i+1 is generated from P i, if a large portion of states in P i+1 belong to some absorbing sets, it would be hard to for the new phase to include elements not in these absorbing sets. Therefore, the training is likely to be contained within these sets and no actual progress could be made since the phase extension makes no sense. However, with additional keyframes, this limitation could be avoided even with irreversible processes and absorbing sets. The mechanism is described as follows: When we sample states nearby a keyframe that is not in any absorbing set, the sampled states might happen to belong to some absorbing set. Although phase extensions are constrained within the absorbing sets, the generated phase might also cover some states sampled nearby a keyframe. Thus, according to the phase administrator algorithm (Alg. 2), that keyframe could still be reached.

Represent the short summary for finding the corresponding scientific article (long)

We propose Tendency RL to efficiently solve goal-oriented tasks with large state space using automated curriculum learning and discriminative shaping reward, which has the potential to tackle robot manipulation tasks with perception.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.