Question: represent the article for finding a single-sentence tldr

Question: The prohibitive energy cost of running high-performance Convolutional Neural Networks (CNNs) has been limiting their deployment on resource-constrained platforms including mobile and wearable devices. We propose a CNN for energy-aware dynamic routing, called the EnergyNet, that achieves adaptive-complexity inference based on the inputs, leading to an overall reduction of run time energy cost without noticeably losing (or even improving) accuracy. That is achieved by proposing an energy loss that captures both computational and data movement costs. We combine it with the accuracy-oriented loss, and learn a dynamic routing policy for skipping certain layers in the networks, that optimizes the hybrid loss. Our empirical demonstrate that, compared to the baseline CNNs, EnergyNetcan trim down the energy cost up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining the same testing accuracies. It is further encouraging to observe that the energy awareness might serve as a training regularization and can even improve prediction accuracy: our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively. While deep learning-powered Internet of Things (IoT) devices promise to dramatically revolutionize the way we live and work by enhancing our ability to recognize, analyze, and classify the world around us, this revolution has yet to be unleashed due to many fundamental challenges. Edge devices, such as smart phones, smart sensors, drones and robots, have limited energy and computation resources since they are battery-powered and have a small form factor. On the other hand, high-performance Convolutional Neural Networks (CNNs) come at a cost of prohibitive energy consumption BID0. The CNNs with the highest accuracy have hundreds of layers and tens of millions of parameters. When deployed in practice, such networks drain the battery very quickly BID1. Recently, there have been a number of methods proposed to reduce energy cost in CNNs, while not hampering their predictive power. Most of them aim to reduce the model size or the number of computations BID2 BID3 BID4 BID5 BID6 BID7 BID8 BID9 BID10 BID11. However, BID1 shows that a smaller model size and fewer operations might not necessarily lead to a lower energy cost. BID1 uses energy cost to guide the pruning process, where the layer with the highest energy cost is pruned first. BID12 formulates the CNN training process as an optimization problem under a certain energy budget constraint. While both methods BID1 BID12 show promising towards pursuing more energy-efficient CNN models, they do not incorporate energy costs into the training loss function to explicitly learn a more energy-efficient model. Furthermore, once their model structures are learned from training, it can only be fixed during the inference time, and there is no room for input-dependent adaptivity. This paper proposes a new CNN model that combines energy cost with a dynamic routing strategy to enable adaptive energy-efficient inference. Our proposed model, termed as EnergyNet, is a gated CNN architecture which employs conditional computing to route the input data through the network Figure 1: EnergyNet Structure: each green circle G indicates an RNN gate and each blue square under G indicates one block of layers in the base model. To reduce the energy cost, the RNN gates generate routing strategies dynamically for different input images. By sharing the parameters between all RNN gates, they will have only 0.04% of the energy cost of the base CNN model, which is negligible. In this specific example, only the first and third blocks get executed.in an efficient path. Built on a base network (such as ResNet-34 or ResNet-50 BID13), EnergyNet uses an additional gating network BID10 to decide whether the current input should skip certain layers in the network or not. It optimizes a weighted combination of an accuracy loss and an energy loss which captures both the computational and memory data movement costs, under which EnergyNet is trained to find the optimal routing policy to reduce the energy cost of the model without degrading the prediction accuracy. Our empirical demonstrate that, compared to the base network without gating nor dynamic inference, EnergyNet can trim down the energy cost up to 40% and 65%, during inference on the CIFAR10 and Tiny ImageNet testing sets, respectively, while maintaining almost the same testing accuracy. Interestingly enough, we find the energy-aware EnergyNet can even achieve win-win, by simultaneously improving the prediction accuracy and saving energy, potentially due to its equivalent effect as a training regularization to avoid overfitting. For example, our models achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 accuracy on Tiny ImageNet when saving up to 50% energy, respectively. Overview: EnergyNet implements an effective dynamic routing algorithm using a set of gating networks, which shares similar ideas with BID10, as depicted in Figure 1. Each gating network associates with a block of layers in the EnergyNet. Given an input image, the gating networks decide if the corresponding block should be skipped or not. The input to each block is first sent to the gating network G, whose output is either 0 or 1. If it is 0, the block will be skipped; otherwise, it will process the input normally as in the base model. If the input and output of the block have different dimensions, then we can perform a linear projection using a shortcut connection to match the dimensions as in BID13. The core innovation in EnergyNet is the adoption of a new energy-aware loss function for learning the gating (skipping) policy, whose details we defer to the next subsection. In our implementation, we adopt the recurrent gates (RNNGates) as in BID10 (see FIG1). Gate-I (b) FFGate-II It is composed of a global average pooling followed by a linear projection that reduces the features to a 10-dimensional vector. A Long Short Term Memory (LSTM) BID14 network that contains a single layer of dimension 10 is applied to generate a binary scalar. As mentioned in BID10, this RNNGate design incurs a negligible overhead compared to its feed-forward counterpart (0.04% vs. 12.5% of the computation of the residual blocks when the baseline architecture is a ResNet). In order to further reduce the energy cost due to loading parameters into the memory, all RNNGates in the EnergyNet share the same weights. Energy-aware Learning for Dynamic Routing: The dynamic routing in EnergyNet is learned by minimizing an energy cost together with the accuracy loss. In particular, the learning goal in the EnergyNet is defined as: min DISPLAYFORM0 Here, α is a weighting coefficient of the energy loss, and W and G denote the parameters of the base model and the gating network, respectively. Also, L(W, G) denotes the prediction loss, and E(W, G) denotes the energy cost of the CNN model associated with W and G, which is calculated by accumulating the energy cost of the layers that are not skipped. In order to compute the energy cost of each layer, we adopt the following energy model: DISPLAYFORM1 where e i and e M AC denote the energy costs of accessing the i-th memory hierarchy and one multiplyand-accumulate (MAC) operation BID15, respectively, while # M AC and # acci denote the total number of MAC operations and accesses to the i-th memory hierarchy, respectively. Note that state-of-the-art CNN accelerators commonly employ such a hierarchical memory architecture for minimizing the dominant memory access and data movement costs. In this work, we consider the most commonly used design of three memory hierarchies including the main memory, the cache memory, and local register files BID15, and employ a state-of-the-art simulation tool called "SCALE-Sim" BID16 to calculate the number of memory accesses # acci and the total number of MACs # M AC. Summary: We show that EnergyNet saves more energy than the baseline ResNet after training on CIFAR10 and Tiny ImageNet BID17. In particular, compared to the baseline ResNet, EnergyNet saves up to 40% and 65% energy cost without degrading the prediction accuracy, when processing CIFAR10 and TinyImageNet images, respectively. More encouragingly, our models can achieve 0.7% higher top-1 testing accuracy than the baseline on CIFAR-10 when saving up to 27% energy, and 1.0% higher top-5 testing accuracy on Tiny ImageNet when saving up to 50% energy, respectively. We use the ResNet-38 and ResNet-50 in BID13 as the baseline models for constructing and evaluating EnergyNet models on CIFAR-10 and Tiny ImageNet, with the ing models denoted as EnergyNet-38 and EnergyNet-50, respectively. The training process contains three steps. In step I, we set the weighting coefficient α to a small value (e.g., 0.1), which helps the model converge to the baseline accuracy first. In step II, we increase α to a larger value (e.g., 0.9) and retrain the model obtained from step I. For step III, it is only triggered if the model sees an accuracy loss larger than a threshold (default 0.1%) from step II: we then set α to a small value (e.g., 0.1) again to retrain the ing model from step II for restoring the accuracy. Such a three-step strategy proves to help stabilize training and gain better performance. Discussion: We use energy savings as a metric to quantify the ing energy efficiency improvement of EnergyNet. The energy savings is defined as E s /E total, where E total and E s are the energy cost of the baseline model and the skipped layers due to EnergyNet. From FIG2, we can conclude that EnergyNet achieves the goal of reducing energy cost while preserving or even improving the prediction accuracy. In particular, the accuracy of EnergyNet-38 and EnergyNet-50 will not drop when the energy savings is as high as 40% and 65%, respectively. To confirm that these experimental are not just a coincidence, we performed 20 trials of experiments using EnergyNet-38 and observed that the confidence interval with a 95% confidence level for the mean of the prediction accuracy and the energy savings are [92.47%, 92.58%] and [39.55%, 40.52%], respectively, verifying the reproducibility of EnergyNet's prediction accuracy and ing energy savings. We observe that EnergyNet can achieve a higher accuracy than the original ResNet model. We conjecture that this is because EnergyNet can overcome overfitting when performing the dynamic routing for energy savings. Further EnergyNet can aggressively reduce energy cost by about 4×, over both the ResNet-38 and ResNet-50 baselines, at the cost of 3% and 4% testing accuracy losses, respectively. 3

This paper proposes a new CNN model that combines energy cost with a dynamic routing strategy to enable adaptive energy-efficient inference.

Represent the short summary for finding the corresponding scientific article (long):

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: represent the article for finding a single-sentence tldr!

Question: Log-linear models models are widely used in machine learning, and in particular are ubiquitous in deep learning architectures in the form of the softmax. While exact inference and learning of these requires linear time, it can be done approximately in sub-linear time with strong concentrations guarantees. In this work, we present LSH Softmax, a method to perform sub-linear learning and inference of the softmax layer in the deep learning setting. Our method relies on the popular Locality-Sensitive Hashing to build a well-concentrated gradient estimator, using nearest neighbors and uniform samples. We also present an inference scheme in sub-linear time for LSH Softmax using the Gumbel distribution. On language modeling, we show that Recurrent Neural Networks trained with LSH Softmax perform on-par with computing the exact softmax while requiring sub-linear computations. Deep neural networks have achieved impressive successes in tasks spanning vision BID9 BID16, language BID3, speech BID6 BID27 and videos BID1. While these models can vastly differ in architecture, activation functions, and presence of recurrence, they (almost) all share a common trait: the softmax layer. The softmax layer, or log-linear model, is a widely used model in machine learning and statistics that transforms a feature vector into a distribution over the output space, modeling log-probabilities as a linear function of the feature vector. For example, in object classification, the softmax layer at the end of a deep convolutional network transforms a feature vector into a probability distribution over classes for the image; in language modeling using recurrent neural networks, it maps the hidden state to a distribution over next words. While parameterizing for logits offers modeling flexibility, inference and learning have linear runtime in the number of classes. Indeed, both of these require computing the un-normalized probability for every class to compute the partition function and retrieve an actual probability distribution. Problems with large output spaces arise naturally in many areas like natural language processing (NLP), where the output space is a language's vocabulary and can be on the order of hundreds of thousands of elements BID15; BID12. This can also occur in computer vision BID14 when attempting tag prediction on massive, weakly-labeled datasets such as Flickr100M BID31.Many solutions have been proposed to address this bottleneck, all revolving around two themes: approximation of the softmax probabilities or computation of exact probabilities for an approximate model. Canonical examples of the former are importance sampling (IS) or noise contrastive estimation (NCE; BID8). Instead of computing probabilities over the whole output space, these methods compute the softmax over a smaller, sampled vocabulary and re-weight the probabilities, providing an unbiased estimator. An illustration of the latter is Hierarchical Softmax BID24, where the output classes are first clustered such that you only need to compute the softmax over a smaller output space. While the former is an unbiased estimate, it comes with no concentration guarantees, and it is often more art than science to craft proposal distributions which will provide low-variance estimators. The latter, while efficient, requires carefully hand-crafted clustering of the output space, at the risk of making mistakes from which there is no recovery. More recently, estimators based on nearest neighbor search have been proposed for inference and learning in log-linear models BID25 BID26. These estimators hinge on Maximum Inner Product Search using Locality-Sensitive to retrieve the largest logits of the distribution and account for the tail with uniformly sampled classes. They boast strong theoretical guarantees and well-established concentration bounds. However, they were constrained to toy settings and not directly applicable to real-world, large-scale, machine learning. In this work, we build upon these estimators to make them amenable to deep learning practitioners, without losing any theoretical guarantees. We first show how they can be extended to be usable within training of deep learning models, then present our efficient implementation, adapted to deep learning hardware and frameworks. Finally, we show the applicability and efficiency of our method by evaluating on a real-world task: language modeling. We show significant perplexity gains against competing methods with significant speed-ups. Our contributions are as follows:• We present a new deep learning layer, LSH Softmax, an efficient replacement for the softmax layer based on Locality-Sensitive Hashing and the Gumbel distribution, for any deep learning architecture, with strong theoretical guarantees for sub-linear learning and inference.• We provide details for efficient implementation on deep learning hardware (GPUs) and modern deep learning frameworks BID0 BID19 ).• Empirically, we show, on several datasets, that training and sampling from LSH Softmax performs similarly to an exact softmax while requiring significantly less FLOPS. In this section, we first provide a quick overview of Neural Networks and the most popular classification layer, the softmax layer. We then present the Gumbel distribution BID7 and introduce Locality-Sensitive Hashing BID11, both of which our estimator is built upon for inference and learning. Notationally, X is the input space, e.g. X R d and Y is a discrete output space: Y {1, . . ., C}. Feedforward Networks Neural networks models are built hierarchically by applying linear and non-linear transformations in alternating fashion. Formally, given input x ∈ X, an m-layer neural network with σ(·) non-linearity transforms x into h defined as: DISPLAYFORM0 {W i} i≤m and {b i} i≤m are learned weights of the network. σ(·) denotes an element-wise nonlinearity such as ReLU (max(·, 0)) or sigmoid ((1 + exp(−·)) −1 ).Recurrent Networks Recurrent Neural Networks (RNN) are an extension of the previous setting to arbitrarily long sequences by keeping an internal state h t. Formally, given an input sequence (x 1, . . ., x T), it can be written as a dynamical system of the form: DISPLAYFORM1 where U and V are learnable weight matrices. In practice, this parametrization is not wellconditioned for optimization as it can be subject to vanishing or exploding gradients and in practice the Longer Short Term Memory (LSTM; BID10) is preferred. In both cases, these outputs are then given as input to a softmax layer which produces a distribution over the output space Y. In the rest of this work, we denote by φ the parameters of the neural network. The softmax layer is the common name given to a log-linear model for multi-classification at the end of a neural network. Let us consider the multi-classification setting with inputs in X and outputs in Y. Given a feature vector ψ(x) and C weight vectors {θ c} c≤C, the softmax layer parameterizes the following distribution: DISPLAYFORM0 In the particular case of neural networks, p(y|x; θ, φ) ∝ exp(h T θ i). {h T θ i} i≤C are called the logits. It is important to note that computing the distribution over the output space, for inference or learning, requires O(C) operations. For the rest of this work, θ denotes the parameters of the softmax whereas φ denotes the parameters of the neural network (producing the feature). First introduced by BID7, the Gumbel distribution is defined by the following cumulative distribution function: p(G < s) = exp(− exp(−s)). More practically, one can sample from the Gumbel distribution by first sampling U ∼ U and returning G = − log(− log(U)). This distribution is particularly useful as it casts sampling as optimization. Theorem 1 (Maddison et al. FORMULA0). Let {y i} i≤C be un-normalized probabilities (or logits) over Y and let {G i} i≤C be i.i.d Gumbel variables. Then: DISPLAYFORM0 Nearest neighbor search is a task that arises in many fields, such as information retrieval. Given a fixed set of vectors S and a distance, this task consists of, given any incoming query q, returning the vectors closest to the query according to the specified distance. In this work, we will be interested in the Maximum Inner Product Search (MIPS) task. Let S = {s 1, . . ., s N} be a subset of R d. Given a query q ∈ R d, MIPS aims at retrieving arg max s∈S q T s. This requires Θ(N) operations as one has to compute the dot-product of q with all elements of S.In the case where we assume that, for a given set S, it is needed to retrieve the nearest neighbor for a large numbers of queries, we can achieve amortized sub-linear time. This problem is commonly addressed with space partitioning techniques, such as Locality-Sensitive Hashing (LSH; BID11). LSH leverages hashing to reduce the number of candidate vectors to evaluate, based on the idea that similar vectors will hash in the same bucket. We have the following : Theorem 2 BID11 ). Given a set S of size N, a similarity measure d(·, ·) and a family of hash functions H s.t. for S > T and p > q: DISPLAYFORM0 we can construct a data structure s.t. given an incoming query q, a nearest neighbor can be retrieved, with high probability, in sub-linear time O(N ρ log N) with ρ log p log q < 1.Recent work builds on top of LSH to either reduce the number of tables BID18, or utilize more expressive hash functions BID2. A common family of hash is the hyperplane hash, i.e. for v ∼ N (0, I), h v (x) = sign v T x, also called Signed Random Projections BID4. For the rest of this work, we denote b the number of hashing bits (equivalently, the number of random vectors) per table, and L the number of tables. In this section, we show how we can apply Theorem 3.5 of BID26 to enable sublinear learning of softmax parameters in the context of deep models, i.e. where both weights and inputs can change. This is crucial for real-world use. Deep learning models for both classification BID16 and generation BID22 are often trained with a maximum-likelihood objective. Formally, given a training pair (x, y) ∈ X × Y, one aims at maximizing log p(y|x; θ, φ), where θ ∈ Θ and φ ∈ Φ are respectively the parameters of the softmax and of the neural network. To optimize this model, the usual method is to use back-propagation BID28 to differentiate and then perform stochastic gradient descent (SGD; BID17) on θ and φ. Let's denote by f (x; φ) h the feature vector given as input to the softmax. Given our notation, the objective is written as (x, y, θ, φ) DISPLAYFORM0 For backpropagation, we need to compute the gradient of w.r.t to both θ and h -the gradient w.r.t. h is then passed down to compute the gradient w.r.t. φ. DISPLAYFORM1 with DISPLAYFORM2 Computing these gradients clearly requires O(|Y|) operations. In practice, this constitutes a major bottleneck for large output spaces. BID26 shows how to compute expectation in in sub-linear time, with a well-concentrated estimator using an LSH structure. Intuitively, we can build a good estimate of the partition function by retrieving the largest logits (using LSH) and accounting for the tail with uniform samples. Applying this , we can compute the expectations necessary to compute the softmax gradients in sub-linear time. This is described in Theorem 3. Theorem 3 (LSH Softmax for Learning). Let h = f (x; φ) be input to a softmax layer with parameters {θ c} c≤C and define (x, y, θ, φ) as previously. Given S, the k-nearest neighbors of h in {θ c} c≤C and T, l uniform samples from {1, . . ., C} − S, let us define: DISPLAYFORM3 DISPLAYFORM4 These estimators are well concentrated: i.e. for, δ > 0, if k = l = O n 2 3 1 1 δ, then with probability greater than 1 − δ: DISPLAYFORM5 While Theorem 3 provides computation of the gradients in sub-linear time, it is only usable in a setting where the weights ({θ i} i≤C ) are not updated. Indeed, querying nearest neighbors in sublinear time assumes that an appropriate data structure (here LSH) was built in advance. However, when training deep models, we are required to update the weights at every training step. This necessitates online updating of the LSH structure. To maintain the sub-linear runtime, we perform these updates in a sparse manner. We describe in Algorithm 1 how this estimator can be used in a training loop, with weight updating and sparse LSH updates. Proposition 4. The softmax computations described in Algorithm 1 run in sub-linear time., n iters number of training iterations. Initialize θ and φ Initialize the MIPS structure with {θ i} i≤|V|. for j ≤ n iters doSample an example (x, y) from D. DISPLAYFORM0 Find S, k-nearest-neighbors of h using the MIPS. Define T as l indexes uniformly sampled from Y − S. DISPLAYFORM1 Pass downĝ h for back-propagation. Re-hash the updated vectors (at most (k + l)) into the right buckets. end forProof. The softmax computations can be split into three parts: retrieving nearest neighbors, computing the forward/backward passes, and rehashing updated vectors. With a sub-linear MIPS such as LSH, the first part is guaranteed to be sub-linear. For the second part, computing the partition function and the entire gradient estimator requires computing a finite number of sums over DISPLAYFORM2 3 ) terms, which is sub-linear. The third part consists of re-hashing updated vectors. Re-hashing a vector is a constant operation (consisting of b × L dot-products) and thus, given that only a sub-linear number of vectors are updated, re-hashing is sub-linear. In the last section, we presented a method to speed-up training time based on an LSH data structure. In addition to these training time gains, LSH Softmax can be utilized for computational gains at inference time as well. While MAP inference can be easily derived from the MIPS structure, sampling from the conditional distribution is often required (e.g. to generate diverse sentences in language modeling or machine translation). These gains can be crucial for large-scale deployment. This is a direct application of BID26 that once again leverages a MIPS structure and the Gumbel distribution. By lazily evaluating Gumbel noise, once can devise an inference scheme which allows to sample from log-linear models in sub-linear time. Theorem 5 (LSH Softmax for Inference). We reuse the same notations as the once in Theorem 3. We define t − log(− log(1 − l/C)). Let {G i} i≤k be k samples from the Gumbel distribution. We then proceed to sample m ∼ Binomial(C, l/C), and sample T, m points from Y − S with associated Gumbels {G i} i≤m s.t. each G i are larger than t. Let us define: DISPLAYFORM0 Let, δ > 0, we then have the two following :1. For k = l ≥ log 1 δ,ŷ is a sample from p(y|x; θ, φ) with probability greater than 1 − δ.2. This inference scheme runs in sub-linear time. Proof. BID26 We denote by p Gumbel (·|h; θ) the implicit distribution over Y provided by this inference scheme. While we can sample from p Gumbel, we note that the likelihood is intractable. We also emphasize that this scheme can be utilized for any softmax model, regardless of the training method. Recent successes of deep neural networks hinge on their efficient implementation on specialized hardware: Graphics Processor Units (GPU), which enables training of large models in reasonable time. Often, methods with theoretically faster runtime are dismissed by practitioners because of their incompatibility with the hardware, rendering them hard to implement efficiently and ultimately not widely used. In this section, we first detail how our method is indeed amenable to GPU implementation and can amount to wall-clock gains in practice, and explain why LSH Softmax is easy to implement in the context of modern deep learning frameworks who often provide a gradient computation API.GPU Implementation Standard LSH implementations consist of three steps: DISPLAYFORM0 b, retrieve candidates in each of the L tables. Let us denote C q the number of candidates retrieved.3. Distances: Given those candidates {x 1, . . ., x Cq} ⊂ R d, compute the distances {q T x i} i≤Cq and only return the closest one. It is also important to note that deep learning models are often trained using minibatch optimization; let us describe how each of these steps can be computed efficiently and in the minibatch setting. The first step is amenable to the GPU setting; a batch of queries {q i} i≤m ⊂ R d can be represented by Q ∈ R m×d. Given that the hyperplanes are similarly presented in matrix form i.e. H ∈ R d×(b×L), the hashing step is equivalent to sign (Q · H) ∈ {0, 1} m×(b×L). This is the type of operations that GPUs excel at: matrix-multiply followed by element-wise function. The second step, while not as compatible with GPU, is still massively parallelizable using multithreading on CPU. Given the computed signatures, one can run parallelism at the query level (i.e. each thread retrieves candidates for a given query), rendering that step efficient. It also allows for more memory-efficient look-up such as BID18.The last operation is, once again, very amenable to GPU. It simply consists of a gather (i.e. building a matrix with the appropriate indexes from the candidates) into a 3-d tensor. Indeed, after the previous step, the LSH structure returns m lists of s candidates, and the gather step returns the appropriate vectors from the vocabulary into a 3-d tensor of shape R m×s×d. As the batched queries can be also seen as a 3-d tensor R m×d×1, computing the exact distances then reduces to a batch matrix-multiply which is a very efficient operation on GPU.Software Implementation Another crucial point for practitioners is the ability to rely on frameworks automatically providing gradients, such as BID0 BID19, to implement deep learning models; this abstracts away the need to write down the exact gradients which can be both cumbersome and error-prone. An additional advantage of our estimator is that it can be effortlessly implemented in these frameworks. Indeed, given logits computed over the nearest-neighbors and the additional uniformly sampled indexes, one can compute the estimate of the partition function and thus an estimate of the loss. Computing the gradient estimators now reduces to differentiating this loss, which can be very simply done using the framework's differentiation API. After having presented our new layer LSH Softmax, we now proceed to show its applicability and efficiency in a real-world setting for deep learning practitioners, specifically towards language modeling. We first show that our method significantly outperforms approximate softmax baselines while performing within 20% of the performance of the exact softmax. We then provide a computational comparison. While we evaluate our method on NLP tasks, we want to emphasize that it is directly applicable to other domains, such as vision. However, public vision benchmark datasets with large output spaces require significantly more computational resources (e.g. 98 GPU nodes for 8 days for Flickr100M BID31) which is outside the scope of this paper. Language modeling is the task of, given a sequence of words (w 1, . . ., w T) in a vocabulary V, estimating p(w 1, . . ., w T) = t≤T p(w t |w <t). Substantial work has been done to model these distributions using non-parametric n-gram counts with additional smoothing techniques, but can fail to model long histories because of an exponential number of sequences. Recently, parametric models using RNNs have shown impressive success on these tasks BID22. In this setting, large output spaces arise naturally, as the vocabulary size can range from 10 4 to 10 6. We first describe our experimental protocol, and then report perplexity (ppl) of LSH Softmax against a set of baselines on this task for several datasets. Datasets We evaluate our method on three standard datasets for Language Modeling with varying number of characters and vocabulary size:• Penn TreeBank (PTB): We follow the pre-processing described by BID22, which in 929k training tokens, 73k validation and 82k test tokens with a 10k vocabulary size.• Text8 is a dataset consisting of the first 100 millions characters of Wikipedia, and has a vocabulary size of 44k. This dataset has been used recently in the context of language modeling BID33. We use the 90M first words for training and split the remaining between the validation and test set.• Wikitext-2. First introduced in BID21, this is a selected corpus of Wikipedia articles. It has a vocabulary size of 33k and contains 217k tokens. As previously, we split between a training, validation and testing set. Baselines We evaluate the performance of models trained with exact softmax i.e. computed over the entire output space, Biased Importance Sampled softmax (BIS), as presented in BID12, which consists of sub-sampling the vocabulary according to a proposal distribution based on unigram counts, and Negative Sampling (NS), proposed in, equivalent to (BIS) with a uniform distribution, standard Importance Sampling BID15 and (Our models are trained using SGD using gradient clipping, with an initial learning rate of 20. This learning rate is annealed when the validation perplexity plateaus. Our models are trained for 40 epochs for PTB, 3 epochs for Text8, 25 epochs for Wikitext-2. With the notations of Theorem 3, for LSH Softmax, we choose k = 10 |V| and l = |V|. For the IS and NS baselines, we choose to sample k + l classes from the output space for a fair comparison. We choose the number of bits per signature b log 2 |V| and choose L, number of tables, to have sufficient recall for the MIPS task. We report perplexity for a fixed architecture but comparing different softmax evaluations; we present both learning curves and perplexity on each set. We report the perplexity of all trained models using the exact probabilities i.e. the full softmax. Perplexities are reported in Table 1 and learning curves in FIG1 . We see that LSH Softmax consistently outperforms the approximate baselines by a fair margin while performing a similar number of operations, showcasing the strength of this estimator. We also observe from the training curves that approximate methods' performances tend to plateau, as IS and NS cannot target the proper classes to push down. In constrast, LSH Softmax does not. Having established that the proposed estimator performs very well on real-world tasks, we now proceed to evaluate the computation gains. It is important to note that for models with large output spaces, the softmax computation can amount to about 80% of the total computation (Joulin et 2016; BID13 ; we thus choose to only evaluate computational gains in the softmax layer. We evaluate our method in CPU, with a batch size of 1, to have an accurate estimation of the ratio of FLOPS. We report both speed-up and validation perplexity (ppl) relative difference with the exact softmax for LSH Softmax and NS. Note that NS requires the same number of operations as importance sampling (IS) but outperforms it in all tasks. Additionally, we show the speed-ups one can achieve on the One Billion Word dataset BID5, whose ppl was not evaluated due to computational constraints. We report the in Table 2. We observe that, while faster, NS performs significantly worse than LSH Softmax. Furthermore, its performance deteriorates significantly when increasing the size of the output space, contrary to LSH Softmax which always performs in the same relative range. Table 2: LSH Softmax performs closest to the exact softmax and handily outperforms importance sampling based methods with no concentration guarantees. In recent years, MIPS-based estimators for log-linear models have been explored in the literature. BID32 propose retrieving the largest logits using LSH and estimating the Softmax using only those classes. Their method is encompassed in ours by simply setting l to 0. However, we note that not accounting for the tail can lead to highly biased gradients. Indeed, BID26 show that, using only the top-k largest values leads to significantly worse performance. In a similar direction, BID30 propose using LSH at each layer and only retaining the largest activations which can be viewed as a form of adaptive dropout. This work differs with ours in two ways: first of all, their paper provides no theoretical guarantees and secondly, they focus on reducing memory footprint which is not the aim of our work. Finally, BID29 proposed using the LSH structure as a proposal distribution to evaluate the Softmax. While unbiased and efficient, their method does not offer any concentration guarantees and the estimator can have arbitrarily bad variance. In this work, we presented LSH Softmax, a softmax approximation layer for large output spaces with sub-linear learning and inference cost (in the number of states) and strong theoretical guarantees. We showcased both its applicability and efficiency by evaluating LSH on a common NLP task, language modeling. On several datasets for this task, we report perplexity closest to exact training among all baselines, as well as significant speed-ups. Our hope is that, for any architecture, this layer could be chosen in lieu of softmax, when the output space is sufficiently large to warrant the approximation. To that end, we plan to release source-code with the camera-ready version.

Represent the TLDR for finding the corresponding scientific article

we present LSH Softmax, a softmax approximation layer for sub-linear learning and inference with strong theoretical guarantees; we showcase both its applicability and efficiency by evaluating on a real-world task: language modeling.

represent the tldr for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR.

Question: Can the success of reinforcement learning methods for simple combinatorial optimization problems be extended to multi-robot sequential assignment planning? In addition to the challenge of achieving near-optimal performance in large problems, transferability to an unseen number of robots and tasks is another key challenge for real-world applications. In this paper, we suggest a method that achieves the first success in both challenges for robot/machine scheduling problems. Our method comprises of three components. First, we show any robot scheduling problem can be expressed as a random probabilistic graphical model (PGM). We develop a mean-field inference method for random PGM and use it for Q-function inference. Second, we show that transferability can be achieved by carefully designing two-step sequential encoding of problem state. Third, we resolve the computational scalability issue of fitted Q-iteration by suggesting a heuristic auction-based Q-iteration fitting method enabled by transferability we achieved. We apply our method to discrete-time, discrete space problems (Multi-Robot Reward Collection (MRRC)) and scalably achieve 97% optimality with transferability. This optimality is maintained under stochastic contexts. By extending our method to continuous time, continuous space formulation, we claim to be the first learning-based method with scalable performance in any type of multi-machine scheduling problems; our method scalability achieves comparable performance to popular metaheuristics in Identical parallel machine scheduling (IPMS) problems. Suppose that we are given a set of robots and seek to serve a set of spatially distributed tasks. A reward is given for serving each task promptly -ing in a time-decaying reward collection problem -or when completing the entire set of tasks -ing in a makespan minimization problem. As the capability to control and route individual robots has increased [], efficient orchestration of robots arises as an important remaining concern for such problems. Multi-robot planning problems. In this paper, we focus on orchestration problems that can be formulated as robot planning problems. A key assumption in such orchestration problems is that we are given information on the "duration of time required for an assigned robot to complete a task". This duration may be deterministic (e.g. as in a Traveling Salesman Problem (TSP) or Vehicle Routing Problem (VRP)) or random with given probability distribution (c.f., []). 1. We call this duration the task completion time. Due to their combinatorial nature, robot planning problems suffer from exponential computational complexity. Even in the context of single-robot scheduling problems (e.g., TSP) scalability is a concern. Planning for multiple robots exacerbates the scalability issue. While scalable heuristic methods have been developed for various deterministic multi-robot planning problems (c.f., [Rossi Proposed methods. In the seminal paper [], the authors observed that combinatorial optimization problems such as TSP can be formulated as sequential decision making problems. Decision making in such a sequential framework relies on an estimate of future costs Q(s, a) for an existing task sequence s and candidate next task a. With this estimate, given the prior decisions s at each decision step, they select the next task a to minimize the future cost estimate. []'s solution framework relies on the following three assumptions. 1) For each combinatorial optimization problem, one can heuristically choose how to induce a graph representation of (s, a). In the case of TSP, the paper induces a fully connected graph for every possible next task. 2) This induced graph representation can be considered as a probabilistic graphical model (PGM) []. This PGM can be used with a graph-based mean-field inference method called structure2vec [] to infer Q(s, a) for use in combinatorial optimization problems. 3) Inference of Q(s, a) can be learned by the reinforcement framework called fitted Q-iteration. We create a solution framework to achieve scalability and transferability for multi-robot planning that builds in numerous directions upon the foundation of [] as follows: 1. State representation and mean-field inference theory for random PGM. Instead of heuristically inducing a PGM, we show that a robot scheduling problem exactly induces a random PGM. Since there exists no mean-field inference theory for random PGM, we develop the theory and corresponding new structure2vec iteration. 2. Sequential encoding of information for transferability. To achieve transferability in terms of the number of robots and tasks, we carefully design a two-step hierarchical mean-field inference []. Each step is designed to infer certain information. The first step is designed to infer each task's relative graphical distance from the robots. The second step is designed to infer Q(s, a) (a here refers to a joint assignment of robots). While the first step is by its nature transferable to any number of tasks and robots, the transferability in inference of the second step is achieved by the scale-free characteristic of fitted Q-iteration [van]. That is, the relative magnitudes of Q(s, a) values are sufficient to select an action a. 3. Auction-based assignment. Even if we can infer Q(s, a) precisely, the computation time required to select an action a using the maximum Q(s, a) operation exponentially increases as robots and tasks increase. To resolve this issue, we suggest a heuristic auction that is enabled by the transferability of our Q(s, a) inference. Even though this heuristic auction selects a with only polynomial computational complexity, it provides surprisingly good choices for a. (In fact, this heuristic auction increases the performance empirically relative to using the max operation.) time τ i to complete -we call this the processsing time. This time is the same independent of which machine serves the task. We incorporate one popular extension and allow'sequence-dependent setup times'. In this case, a machine must conduct a setup prior to serving each task. The duration of this setup depends on the current task i and the task j that was previously served on that machine -we call this the setup time. The completion time for each task is thus the sum of the setup time and processing time. Under this setting, we solve the IPMS problem for make-span minimization as discussed in []. That is, we seek to minimize the total time spent from the start time to the completion of the last task. The IPMS formulation resembles our MRRC formulation in continuous-time and continuous-space and we relegate the detailed formulation to Appendix B. In Section 2, we formulated multi-robot/machine planning problems as sequential joint assignment decision problems. As in [], we will select a joint assignment using a Q-function based policy. Since we thus choose action a t with the largest inferred Q(s t, a t) value in state s t, the development of a Q(s t, a t) inference method is a key issue. Toward this end and motivated by these robot planning problems, we provide new in random PGM-based mean-field inference methods and a subsequent extension of the graph-neural network based inference method called structure2vec [] in Section 3.1. In Section 3.2, we discuss how a careful encoding of information using the extended structure2vec of Section 3.1 enables precise and transferable Q(s t, a t) inference. Since the computational complexity required to identify the best joint assignment is exponential with respect to the number of robots and tasks, Section 3.3 discusses how the transferability of our Q(s t, a t) inference method enables a good action choice heuristic with polynomial computational complexity. PGM. Given random variables {X k}, suppose that joint distribution of {X k} can be factored as PGM-based mean-field inference. One popular use of this PGM information is PGM-based mean-field inference. In mean-field inference, we find a surrogate distribution Q(X 1, . . ., X n) = i Q i (x i) that has smallest Kullback-Leibler distance to original joint distribution P (X 1, . . ., X n). We then use this surrogate distribution to solve the original inference problem. [] shows that when we are given PGM information, {Q i (x i)} can be analytically computed by a fixed point equation. Despite that this usefulness, in most inference problems it is unrealistic to assume we know or can infer probability distributions of a PGM. This limitation was addressed in [] using a method called structure2vec.. [] suggests that an inference problem with graph-structured data (e.g. a molecule classification problem) can be seen as a particular PGM structure that consists of two types of random variables. One type of random variables {X k} is one that serves as input of inference problem (e.g. X k denotes atomic number of atom k). Another type of random variables {H k} is latent random variable where H k is a latent random variable related to X k. Existence of probabilistic relationships among {H k} are assumed heuristically from graph structure of data. Then the particular PGM structure they assume is, where V denotes the set of vertex indexes. The goal of mean-field inference problem is to find a surrogate distribution Q k (h k) for posterior marginal P ({h k}|{x k}). However, we can't compute {Q k (h k)} since we are not given φ (H k |H i) nor φ (H k |X k). To overcome this limitation, [] develops a method called structure2vec that only requires the structure of the PGM for mean-field inference. structure2vec embeds the mean-field inference procedure, i.e. fixed point iteration on {Q k (h k)}, into fixed point iterations of neural networks on vectors {μ k}. Derivation of such fixed point iterations of neural networks can be found in and can be written asμ k = σ W 1 x k + W 2 j =kμ j where σ denotes Relu function and W denotes parameters of neural networks. Robot scheduling as random PGM-based mean-field inference. All applications of structure2vec in [Dai et al. (2016; 2017) ] heuristically decide the structure of PGM of each data point from its graph structure. The key observation we make is that inference problems in robot scheduling exactly induce a'random' PGM structure (to be precise, a 'random' Bayesian Network). Given that we start from state s t and action a t, consider a random experiment "sequential decision making using policy φ". In this experiment, we can define an event as'How robots serve all the remaining tasks in which sequence'. We call one such event a'scenario'. For each task t i ∈ T t, define a random variable X i as'a characteristic of task t i' (e.g. when task i is served). Given a scenario, the relationships among {X i} satisfy as a Bayesian Network. For details, see Appendix C) Note that we do not know which scenario will occur from time t and thus do not know which PGM will be realized. Besides, the inference of probability of each scenario is challenging. Putting aside this problem for a while, we first define a'random PGM' and'semi-cliques'. Denote the set of all random variables in the inference problem as X = {X i}. A random PGM is a probabilistic model of how a PGM is randomly chosen from a set of all possible PGMs on X 4. Next, denote the set of all possible probabilistic relationships on X as C X. We call them'semi-cliques'. In robot scheduling problem, a semi-clique D ij ∈ C X is a conditional dependence X i |X j. The semi-clique D ij presents as an actual clique if and only if the robot which finishes task t i chooses task t j as the next task. We will now prove that we don't have to infer the probability of each scenario, i.e. random PGM model itself. The following theorem for mean-field inference with random PGM is an extension of mean-field inference with PGM [] and suggests that only a simple inference task is required: inference of the presence probability of each semi-cliques. Theorem 1. Random PGM based mean field inference Suppose we are given a random PGM on X = {X k}. Also, assume that we know presence probability {p m} for all semi-cliques where Z k is a normalizing constant and φ m is the clique potential for clique m. From this new , we can develop the structure2vec inference method for random PGM. As in [], we restrict our discussion to when every semi-clique is between two random variables. In this case, a semi-clique can be written as D ij with its presence probability p ij. Lemma 1. Structure2vec for random PGM. Suppose we are given a random PGM model with X = {X k}. Also, assume that we know presence probability {p ij} for all semi-cliques C X = {D ij}. The fixed point iteration in Theorem 1 for posterior marginal P ({H k}|{x k}) can be embedded in a nonlinear function mapping with embedding vectorμ k as Proof of Thorem 1 and lemma 1. For brevity, proofs are relegated to the Appendix D and E. Corollary 1. For a robot scheduling problem with set of tasks t i ∈ T t, the random PGM representation for structure2vec in lemma 1 is ((T t, E {p ij} inference procedure employed in this paper is as follows. Denote ages of task i, j as age i, age j. Note that if we generate M samples of ij as {e is an unbiased and consistent estimator of E[f ( ij, age i, age j)]. For each sample k, for each task i and task j, we form a vector of u k ij = (e k ij, age i, age j) and compute We obtain {p ij} from {g ij} using softmax. Algorithm details are in Appendix F. In this section, we show how Q(s t, a t) can be precisely and transeferably inferred using a two-step structure2vec inference method (For theoretical justifications on hierarchical variational inference, see). We here assume that we are given (T t, E T T t) and inferred {p ij} so that Corollary 1 can be applied. For brevity, we illustrate the inference procedure for the special case when task completion time is deterministic (Appendix G illustrates how we can combine random sampling to inference procedure to deal with task completion times as a random variable). Step 1. Distance Embedding. The output vectors {μ 1 k} of structure2vec embeds a local graph information around that vector node []. We here focus on embedding information of robot locations around a task node and thus infer each task's'relative graphical distance' from robots around it. As the input of first structure2vec ({x k} in lemma 1), we only use robot assignment information (if t k is an assigned task, we set x k as 'task completion time of assignment'; if t k is not an assigned task:, we set x k = 0). This procedure is illustrated in Figure 1. According to [], the output vectors {μ 1 k} of structure2vec will include sufficient information about the relative graphical distance from all robots to each task. Step 2. Value Embedding. The second step is designed to infer'How much value is likely in the local graph around each task'. Remind that vectors {μ 1 k}, output vectors of the first step, carries information about the relative graphical distance from all robots to each task. We concatenate'age' of each tasks {age k} to each corresponding vector in {μ 1 k} and use the ing graph as an input ({x k} in lemma 1) of second structure2vec, as illustrated in Figure 1. Again, vectors {μ 2 k} of the output graph of second structure2vec operation embeds a local graph structure around each node. Our intuition is that {μ 2 k} includes sufficient information about'How much value is likely in the local graph around each task'. Step 3. Computing Q(s t, a t). To infer Q(s t, a t), we aggregate the embedding vectors for all nodes, i.e.,μ 2 = kμ 2 k to get one vectorμ 2 which embeds the'value likeliness' of the global graph. We then use a layer of neural network to mapμ 2 into Q(s t, a t). The detailed algorithm of above whole procedure (combined with random task completion times) is illustrated in Appendix G. Why are each inference steps transferable? For the first step, it is trivial; the inference problem is a scale-free task. In the second step, the'value likeliness' will be underestimated or overestimated according to the ratio of (number of robots/number of tasks) in a local graph: underestimated if the ratio in training environment is smaller than the ratio in the testing environment; overestimated otherwise. The key idea solving this problem is that this over/under-estimation does not matter in Q-function based action decision [van] as long as the order of Q-function value among actions are the same. While analytic justification of this order invariance is beyond this paper's scope, the fact that there is no over/underestimation issue in the first step inference problem helps this justification. In Q-function based action choice, at each time-step t, we find an action with largest Q(s t, a t). We call this action choice operation'max-operation'. The problem in max-operation in the multi-robot setting is that the number of computation exponentially increases as the number of robots and tasks increases. In this section, we show that transferability of Q-function inference enables designing an efficient heuristic auction that replaces max operation. We call it auction-based policy(ADP) and denote it as φ Q θ, where Q θ indicates that we compute φ Q θ using current Q θ estimator. At time-step t, a state s t is a graph G t = (R t ∪ T t, E t) as defined in section 2.1. Our ADP, φ Q θ, finds an action a t (which is a matching in bipartite graph ((R t ∪ T t), E RT t ) of graph G t ) through iterations between two phases: the bidding phase and the consensus phase. We start with a bidding phase. All robots initially know the matching determined in previous iterations. We denote this matching as Y, a bipartite subgraph of ((R t ∪T t), E RT t ). When making a bid, a robot r i ignores all other unassigned robots. For example, suppose robot r i considers t j for bidding. For r i, Y ∪ ij is a proper action (according to definition in section 2.1) in a'unassigned robot-ignored' problem. Robot r i thus can compute Q(s t, Y ∪ ritj) of'unassigned robot-ignored' problem for all unassigned task t j. If task t * is with the highest value, robot r i bids {rit *, Q(s t, Y ∪ rit *)} to auctioneer. Since number of robots ignored by r i is different at each iteration, transferability of Q-function inference plays key role. The consensus phase is simple. The auctioneer finds the bid with the best value, say {*, bid value with *}. Then auctioneer updates everyone's Y as Y ∪ {*}. These bidding and consensus phases are iterated until we can't add an edge to Y anymore. Then the central decision maker chooses Y as φ Q θ (s k). One can easily verify that the computational complexity of computing φ Q θ is O (|L R | |L T |), which is only polynomial. While theoretical performance guarantee of this heuristic auction is out of this paper's scope, in section 5 we show that empirically this heuristic achieves near-optimal performance. 4 LEARNING ALGORITHM In fitted Q-iteration, we fit θ of Q θ (s t, a t) with stored data using Bellman optimality equation. That is, chooses θ that makes small. Note that every update of θ needs at least one max-operation. To solve this issue, we suggest a learning framework we call auction-fitted Q-iteration. What we do is simple: when we update θ, we use auction-based policy(ADP) defined in section 3.3 instead of max-operation. That is, we seek the parameter θ that minimizes How can we conduct exploration in Auction-fitted Q-iteration framework? Unfortunately, we can't use -greedy method since such randomly altered assignment is very likely to cause a catastrophic in problems with combinatorial nature. In this paper, we suggest that parameter space exploration [] can be applied. Recall that we use Q θ (s k, a k) to get policy φ Q θ (s k). Note that θ denotes all neural network parameters used in the structure2vec iterations introduced in Section 5. Since Q θ (s k, a k) is parametrized by θ, exploration with φ Q θ (s k) can be performed by exploration with parameter θ. Such exploration in parameter space has been introduced in the policy gradient RL literature. While this method was originally developed for policy gradient based methods, exploration in parameter space can be particularly useful in auction-fitted Q-iteration. The detailed application is as follows. When conducting exploration, apply a random perturbation on the neural network parameters θ in structure2vec. The ing a perturbation in the Q-function used for decision making via the auction-based policy φ Q θ (s k) throughout that problem. Similarly, when conducting exploitation, the current surrogate Q-function is used throughout the problem. Updates for the surrogate Q-function may only occur after each problem is complete (and typically after a group of problems). For MRRC, we conduct a simulation experiment for a discrete time, discrete state environment. We use maze (see Figure 1) generator of UC Berkeley CS188 Pacman project [] to generate large size mazes. We generated a new maze for every training and testing experiments. Under the deterministic environment, the robot succeeds its movement 100%. Under stochastic environment, a robot succeeds its intended movement in 55% on the grid with dots and for every other direction 15% each; on the grid without dots, the rates are 70% and 10%. As described in section 2, routing problems are already solved. That is, each robot knows how to optimally (in expectation) reach a task. To find an optimal routing policy, we use Dijkstra's algorithm for deterministic environments and dynamic programming for stochastic environments. The central assignment decision maker has enough samples of task completion time for every possible route. We consider two reward rules: Linearly decaying rewards obey f (age) = 200 − age until reaching 0, where age is the task age when served; For nonlinearly decaying rewards, f (t) = λ t for λ = 0.99. Initial age of tasks were uniformly distributed in the interval. Performance test. We tested the performance under four environments: deterministic/linear rewards, deterministic/nonlinear rewards, stochastic/linear rewards, stochastic/nonlinear rewards. There are three baselines used for performance test: exact baseline, heuristic baseline, and indirect baseline. For the experiment with deterministic with linearly decaying rewards, an exact optimal solution for mixed-integer exists and can be used as a baseline. We solve this program using Gurobi with 60-min cut to get the baseline. We also implemented the most up-to-date heuristic for MRRC in []. For any other experiments with nonlinearly decaying rewards or stochastic environment, such an exact optimal solution or other heuristics methods does not exist. In these cases, we should be conservative when talking about performance. Our strategy is to construct a indirect baseline using a universally applicable algorithm called Sequential greedy algorithm (SGA) []. SGA is a polynomial-time task allocation algorithm that shows decent scalable performance to both linear and non-linear rewards. For stochastic environments, we use mean task completion time for task allocation and re-allocate the whole tasks at every timesteps. We construct our indirect baseline as'ratio between our method and SGA for experiments with deterministic-linearly decaying rewards'. Showing that this ratio is maintained for stochastic environments in both linear/nonlinear rewards suffices our purpose. Table 1 shows experiment for (# of robots, # of tasks) =,,,,,,; For linear/deterministic rewards, our proposed method achieves nearoptimality (all above 95% optimality). While there is no exact or comparable performance baseline for experiments under other environments, indirect baseline (%SGA) at least shows that our method does not lose %SGA for stochastic environments compared with %SGA for deterministic environments in both linear and nonlinear rewards. Scalability test. We count the training requirements for 93% optimality for seven problem sizes (# of robots N R, # of tasks N T) =,,,,, with deterministic/linearly decaying rewards (we can compare optimality only in this case). As we can see in Table 2, the training requirement shown not to scale as problem size increases. Transferability test. Suppose that we trained our learning algorithm with problems of three robots and 30 Tasks. We can claim transferability of our algorithm if our algorithm achieves similar performance for testing with problems of 8 robots and 50 tasks when compared with the algorithm specifically trained with problems of 8 robots and 50 tasks, the same size as testing. Table 3 shows our comprehensive experiment to test transeferability. The in the diagonals (where training size and testing size is the same) becomes a baseline, and we can compare how the networks trained with different problem size did well compare to those . We could see that lower-direction transfer tests (trained with larger size problem and tested with smaller size problems) shows only a small loss in performance. For upper-direction transfer tests (trained with smaller size problem and tested with larger size problem), the performance loss was up 4 percent. Ablation study. There are three components in our proposed method: 1) a careful encoding of information using two-layers of structure2vec, 2) new structure2vec equation with random PGM and 3) an auction-based assignment. Each component was removed from the full method and tested to check the necessity of the component. We test the performance in a deterministic/linearly decaying rewards (so that there is an optimal solution available for comparison). The experimental are shown in Figure 2. While the full method requires more training steps, only the full method achieves near-optimal performance. For IPMS, we test it with continuous time, continuous state environment. While there have been many learning-based methods proposed for (single) robot scheduling problems, to the best our knowledge our method is the first learning method to claim scalable performance among machinescheduling problems. Hence, in this case, we focus on showing comparable performance for large problems, instead of attempting to show the superiority of our method compared with heuristics specifically designed for IPMS (actually no heuristic was specifically designed to solve our exact problem (makespan minimization, sequence-dependent setup with no restriction on setup times)) For each task, processing times is determined using uniform. For every (task i, task j) ordered pair, a unique setup time is determined using uniform. As illustrated in section 2, we want to minimize make-span. As a benchmark for IPMS, we use Google OR-Tools library. This library provides metaheuristics such as Greedy Descent, Guided Local Search, Simulated Annealing, Tabu Search. We compare our algorithm's with the heuristic with the best for each experiment. We consider cases with 3, 5, 7, 10 machines and 50, 75, 100 jobs. The are provided in Table 4. Makespan obtained by our method divided by the makespan obtained in the baseline is provided. Although our method has limitations in problems with a small number of tasks, it shows comparable performance to a large number of tasks and shows its value as the first learning-based machine scheduling method that achieves scalable performance. We presented a learning-based method that achieves the first success for multi-robot/machine scheduling problems in both challenges: scalable performance and tranferability. We identified that robot scheduling problems have an exact representation as random PGM. We developed a meanfield inference theory for random PGM and extended structure2vec method of. To overcome the limitations of fitted Q-iteration, a heuristic auction that was enabled by transferability is suggested. Through experimental evaluation, we demonstrate our method's success for MRRC problems under a deterministic/stochastic environment. Our method also claims to be the first learning-based algorithm that achieves scalable performance among machine scheduling algorithms; our method achieves a comparable performance in a scalable manner. Our method for MRRC problems can be easily extended to ride-sharing problems or package delivery problems. Given a set of all user requests to serve, those problems can be formulated as a MRRC problem. For both ride-sharing and package delivery, it is reasonable to assume that the utility of a user depends on when she is completely serviced. We can model how the utility of a user decreases over time since when it appears and set the objective function of problems as maximizing total collected user utility. Now consider a task'deliver user (or package) from A to B'. This is actually a task "Move to location A and then move to location B". If we know the completion time distribution of each move (as we did for MRRC), the task completion time is simply the sum of two random variables corresponding to task completion time distribution of the moves in the task. Indeed, ride-sharing or package delivery problems are of such tasks (We can ignore charging moves for simplicity, and also we don't have to consider simple relocation of vehicles or robots since we don't consider random customer arrivals). Therefore, both ride-sharing problems and package delivery problems can be formulated as MRRC problems. A MRRC WITH CONTINUOUS STATE/CONTINUOUS TIME SPACE FORMULATION, OR WITH SETUP TIME AND PROCESSING TIME In continuous state/continuous time space formulation, the initial location and ending location of robots and tasks are arbitrary on R 2. At every moment at least a robot finishes a task, we make assignment decision for a free robot(s). We call this moments as'decision epochs' and express them as an ordered set (t 1, t 2, . . ., t k, . . .). Abusing this notation slightly, we use (·) t k = (·) k. Task completion time can consist of three components: travel time, setup time and processing time. While a robot in the travel phase or setup phase may be reassigned to other tasks, we can't reassign a robot in the processing phase. Under these assumptions, at each decision epoch robot r i is given a set of tasks it can assign itself: if it is in the traveling phase or setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unfinished task. This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are defined as follows: R k is the set of all robots and T k is the set of all tasks; The set of directed edges where a directed edge is a random variable which denotes task completion time of robot i in R k to service task j in T k and a directed edge titj ∈ E T T k denotes a task completion time of a robot which just finished serving task i in T k to service task j in T k. E RT k contains information about each robot's possible assignments:, where E ri t is a singleton set if robot i is in the processing phase and it must be assigned to its unfinished task, and otherwise it is the set of possible assignments from robot r i to remaining tasks that are not in the processing phase. Action. The action a k at decision epoch k is the joint assignment of robots given the current state s k = G k. The feasible action should satisfy the two constraints: No two robots can be assigned to a task; some robots may not be assigned when number of robots are more than remaining tasks. To best address those restrictions, we define an action a k at time t as a maximal bipartite matching in bipartite sub-graph ((R k ∪ T k), E RT k ) of graph G k. For example, robot i in R k is matched with task j in T k in an action a k if we assign robot i to task j at decision epoch t. We denote the set of all possible actions at epoch k as A k. Reward. In MRRC, Each task has an arbitrarily determined initial age. At each decision epoch, the age of each task increases by one. When a task is serviced, a reward is determined only by its age when serviced. Denote this reward rule as R(k). One can easily see that whether a task is served at epoch k is completely determined by s k, a k and s k+1. Therefore, we can denote the reward we get with s k, a k and s k+1 as R(s k, a k, s k+1). Objective. We can now define an assignment policy φ as a function that maps a state s k to action a k. Given s 0 initial state, an MRRC problem can be expressed as a problem of finding an optimal assignment policy φ * such that As written in 2.2, IPMS is a problem defined in continuous state/continuous time space. Machines are all identical, but processing times of tasks are all different. In this paper, we discuss IPMS with'sequence-dependent setup time'. A machine's setup time required for servicing a task i is determined by its previously served task j. In this case, the task completion time is the sum of setup time and processing time. Under this setting, we solve IPMS problem for make-span minimization objective discussed in [] (The constraints are different in this problem though); That is, minimizing total time spent from start to end to finish all tasks. Every time there is a finished task, we make assignment decision for a free machine. We call this times as'decision epochs' and express them as an ordered set (t 1, t 2, . . ., t k, . . .). Abusing this notation slightly, we use (·) t k = (·) k. Task completion time for a machine to a task consists of two components: processing time and setup time. While a machine in setup phase may be reassigned to another task, we can't reassign a machine in the processing phase. Under these assumptions, at each epoch, a machine r i is given a set of tasks it can assign: if it is in the setup phase, it can be assigned to any tasks or not assigned; if it is in the processing phase, it must be reassigned to its unfinished task. This problem can be cast as a Markov Decision Problem (MDP) whose state, action, and reward are defined as follows: State. State s k at decision epoch k is a directed graph G k = (R k ∪ T k, E k): R k is the set of all machines and T k is the set of all tasks; The set of directed edges where a directed edge ritj ∈ E RT k is a random variable which denotes task completion time of machine i in R k to service task j in T k and a directed edge titj ∈ E T T k denotes a task completion time of a machine which just finished serving task i in T k to service task j in T k. E RT k contains information about each robot's possible assignments: E RT k = ∪ i E ri k, where E ri k is a singleton set if machine i is in the processing phase and it must be assigned to its unfinished task, and otherwise it is the set of possible assignments from machine r i to remaining tasks that are not in the processing phase. Action. Defined the same as MRRC with continuous state/time space. Reward. In IPMS, time passes between decision epoch t and decision epoch t + 1. Denote this time as T t. One can easily see that T t is completely determined by s k, a k and s k+1. Therefore, we can denote the reward we get with s k, a k and s k+1 as T (s k, a k, s k+1). Objective. We can now define an assignment policy φ as a function that maps a state s k to action a k. Given s 0 initial state, an MRRC problem can be expressed as a problem of finding an optimal assignment policy φ * such that T (s k, a k, s k+1) |s 0. Here we analytically show that robot scheduling problem randomly induces a random Bayesian Network from state s t. Given starting state s t and action a t, a person can repeat a random experiment of "sequential decision making using policy φ". In this random experiment, we can define events' How robots serve all remaining tasks in which sequence'. We call such an event a'scenario'. For example, suppose that at time-step t we are given robots {A, B}, tasks {1, 2, 3, 4, 5}, and policy φ. One possible scenario S * can be {robot A serves task 3 → 1 → 2 and robot B serves task 5 → 4}. Define random variable X k a task characteristic, e.g.'The time when task k is serviced'. The question is,'Given a scenario S *, what is the relationship among random variables {X k}'? Recall that in our sequential decision making formulation we are given all the'task completion time' information in the s t description. Note that, task completion time is only dependent on the previous task and assigned task. In our example above, under scenario S *'when task 2 is served' is only dependent on'when task 1 is served'. That is, P (X 2 |X 1, X 3, S *) = P (X 2 |X 1, S *). This relationship is called'conditional independence'. Given a scenario S *, every relationship among {X i |S *} can be expressed using this kind of relationship among random variables. A graph with this special relationship is called'Bayesian Network' [We first define necessary definitions for our proof. In a random PGM, a PGM is chosen among all possible PGMs on {X k} and semi-cliques C. Denote the set of all possible factorization as F = {S 1, S 2, ..., S N} where a factorization with index k is denoted as S k ⊆ C. Suppose we are given P ({S = S m}).

Represent this text to find a much much longer article about it with say hundreds-thousands of words

RL can solve (stochastic) multi-robot/scheduling problems scalably and transferably using graph embedding

The paper designs two algorithms for the stochastic AUC maximization problem with state-of-the-art complexities when using deep neural network as predictive model, which are also verified by empirical studies.


Question: Represent the article for finding a single-sentence TLDR

Question: Curriculum learning consists in learning a difficult task by first training on an easy version of it, then on more and more difficult versions and finally on the difficult task. To make this learning efficient, given a curriculum and the current learning state of an agent, we need to find what are the good next tasks to train the agent on. Teacher-Student algorithms assume that the good next tasks are the ones on which the agent is making the fastest progress or digress. We first simplify and improve them. However, two problematic situations where the agent is mainly trained on tasks it can't learn yet or it already learnt may occur. Therefore, we introduce a new algorithm using min max ordered curriculums that assumes that the good next tasks are the ones that are learnable but not learnt yet. It outperforms Teacher-Student algorithms on small curriculums and significantly outperforms them on sophisticated ones with numerous tasks. Curriculum learning. An agent with no prior knowledge can learn a lot of tasks by reinforcement, i.e. by reinforcing (taking more often) actions that lead to higher reward. But, for some very hard tasks, it is impossible. Let's consider the following task:Figure 1: The agent (in red) receives a reward of 1 when it picks up the blue ball in the adjacent room. To do so, it has to first open the gray box, take the key inside and then open the locked door. This is an easy task for humans because we have prior knowledge: we know that a key can be picked up, that we can open a locked door with a key, etc... However, most of the time, the agent starts with no prior knowledge, i.e. it starts by acting randomly. Therefore, it has a probability near 0 of achieving the task in a decent number of time-steps, so it has a probability near 0 of getting reward, so it can't learn the task by reinforcement. One solution to still learn this task is to do curriculum learning BID0 ), i.e. to first train the agent on an easy version of the task, where it can get reward and learn, then train on more and more difficult versions using the previously learnt policy and finally, train on the difficult task. Learning by curriculum may be decomposed into two parts:1. Defining the curriculum, i.e. the set of tasks the agent may be trained on. 2. Defining the program, i.e. the sequence of curriculum's tasks it will be trained on. These two parts can be done online, during training. Curriculum learning algorithms. Defining a curriculum and a program can be done manually, e.g. by defining a hand-chosen performance threshold for advancement to the next task BID6; BID5 ).However, if an efficient algorithm is found, it may save us a huge amount of time in the future. Besides, efficient (and more efficient than humans) algorithms are likely to exist because they can easily mix in different tasks (what is hard for humans) and then:• avoid catastrophic forgetting by continuously retraining on easier tasks;• quickly detect learnable but not learnt yet tasks. Hence, it motivates the research of curriculum learning algorithms. Curriculum learning algorithms can be grouped into two categories:1. curriculum algorithms: algorithms that define the curriculum; 2. program algorithms: algorithms that define the program, i.e. that decide, given a curriculum and the learning state of the agent, what are the good next tasks to train the agent on. In this paper, we will focus on program algorithms, in the reinforcement learning context. Recently, several such algorithms emerged, focused on the notion of learning progress BID4; BID3 BID2 ). BID4 proposed four algorithms (called Teacher-Student) based on the assumption that the good next tasks are the ones on which the agent is making the fastest progress or digress. We first simplify and improve Teacher-Student algorithms (section 4). However, even improved, two problematic situations where the agent is mainly trained on tasks it can't learn or it already learnt may occur. Therefore, we introduce a new algorithm (section 5), focused on the notion of mastering rate, based on the assumption that the good next tasks are the ones that are learnable but not learnt yet. We show that this algorithm outperforms Teacher-Student algorithms on small curriculums and significantly outperforms them on sophisticated ones with numerous tasks. 2.1 CURRICULUM LEARNING First, let's recall some general curriculum learning notions defined in BID3. A curriculum C is a set of tasks {c 1, ..., c n}. A sample x is a drawn from one of the tasks of C. A distribution d over C is a family of non-negative summing to one numbers indexed by DISPLAYFORM0 Without loss of generality, we propose to perceive a distribution d over tasks C as coming from an attention a over tasks, i.e. d:= ∆(a). An attention a over C is a family of non-negative numbers indexed by C, i.e. a = (a c) c∈C with a c ≥ 0. Intuitively, a c represents the interest given to task c. Let A C be the set of attentions over C. BID4; BID3 BID2, several distribution converters are used (without using this terminology): DISPLAYFORM1 • the argmax distribution converter: ∆ Amax (a) c:= 1 if c = argmax c a c 0 otherwise.A greedy version of it is used in BID4 DISPLAYFORM2 and u the uniform distribution over C.• the exponential distribution converter: BID3 ). DISPLAYFORM3 • the Boltzmann distribution converter: BID4 ). DISPLAYFORM4 • The powered distribution converter: BID2 ). DISPLAYFORM5 An attention function a: N → A C is a time-varying sequence of attentions over C. A program d can be rewritten using this notion: d(t):= ∆(a(t)) for a given attention converter ∆.Finally, a program algorithm can be defined as follows: DISPLAYFORM6 An agent A; DISPLAYFORM7 The Teacher-Student paper BID4 ) presents four attention functions called Online, Naive, Window and Sampling. They are all based on the idea that the attention must be given by the absolute value of an estimate of the learning progress over tasks, i.e. A(t):= |β(t)| where β c (t) is an estimate of the learning progress of the agent A on task c. For the Window attention function, they first estimate the "instantaneous" learning progress of the agent A on task c by the slope β Linreg c (t) of the linear regression of the points (t 1, r t1),..., (t K, r t K) where t 1,..., t K are the K last time-steps when the agent was trained on a sample of c and where r ti is the return got by the agent at these time-steps. From this instantaneous learning progress, they define β c as the weighted moving average of β DISPLAYFORM0 For all the algorithms, a Boltzmann or greedy argmax distribution converter is used. For example, here is the GAmax Window program algorithm proposed in the paper: DISPLAYFORM1 An agent A; β:= 0; for t ← 1 to T do a:= |β|; d:= ∆ GAmax (a); Draw a task c from d and then a sample x from c; Train A on x and observe return r t; β Linreg c:= slope of lin. reg. of (t 1, r t1),..., (t K, r t K); β c:= αβ Three curriculums were used to evaluate the algorithms, called BlockedUnlockPickup, KeyCorridor and ObstructedMaze (see appendix A for screenshots of all tasks). They are all composed of Gym MiniGrid environments BID1 ).These environments are partially observable and at each time-step, the agent receives a 7 × 7 × 3 image (figure 1) along with a textual instruction. Some environments require language understanding and memory to be efficiently solved, but the ones chosen in the three curriculums don't. The agent gets a reward of 1 − n nmax when the instruction is executed in n steps with n ≤ n max. Otherwise, it gets a reward of 0. Before simplifying and improving Teacher-Student algorithms, here are some suggestions about which distribution converters and attention functions of the Teacher-Student paper to use and not to use. First, in this paper, two distribution converters are proposed: the greedy argmax and the Boltzmann ones. We don't recommend to use the Boltzmann distribution converter because τ is very hard to tune in order to get a distribution that is neither deterministic nor uniform. Second, four attention functions are proposed: the Online, Naive, Window and Sampling ones. We don't recommend to use:• the Naive attention function because it is a naive version of the Window one and performs worst (see figure 5 in BID4);• the Sampling attention function because it performs worst than the Window one (see figure 5 in BID4). Moreover, the reason it was introduced was to avoid hyperparameters but it still require to tune a ε to avoid deterministic distributions (see algorithm 8 in BID4)... It remains the Online and Window attention functions. But, the Online one is similar to the Window one when K = 1.Finally, among all what is proposed in this paper, we only recommend to use the Window attention function with the greedy argmax distribution converter, i.e. to use the GAmax Window algorithm (algorithm 2). This is the only one we will consider in the rest of this section. Now, let's see how we can simplify and improve the GAmax Window algorithm.. This corresponds to the powered distribution converter when p = 1. We then replace the greedy argmax distribution converter by a greedy proportional one, and improve performances (figures 2 and 3). Algorithm 3: GProp Linreg algorithm input: A curriculum C; An agent A; β Linreg:= 0; for t ← 1 to T do a:= |β Linreg |; d:= ∆ GP rop (a); Draw a task c from d and then a sample x from c; Train A on x and observe return r t; β Linreg c:= slope of lin. reg. FIG6,..., (t K, r t K);In the rest of this article, this algorithm will be referred as our "baseline". The two following figures show that:• the GAmax Linreg algorithm performs similarly to the GAmax Window algorithm, as asserted before. It even seems a bit more stable because the gap between the first and last quartile is smaller.• the GProp Linreg algorithm performs better than the GAmax Linreg and GAmax Window algorithm, as asserted before. Algorithms introduced in BID4; BID3 BID2, and in particular Teacher-Student algorithms and the baseline algorithm, are focused on the notion of learning progress, based on the assumption that the good next tasks are the ones on which the agent is making the fastest progress or digress. However, two problematic situations may occur:1. The agent may be mainly trained on tasks it already learnt. The frame B of the figure 4 shows that, around time-step 500k, the agent already learnt Unlock and UnlockPickup but is still trained 90% of the time on them, i.e. on tasks it already learnt. 2. It may be mainly trained on tasks it can't learn yet. The more the curriculum has tasks, the more it occurs:• The frame A of the figure 4 shows that, initially, the agent doesn't learn Unlock but is trained 66% of the time on UnlockPickup and BlockedUnlockPickup, i.e. on tasks it can't learn yet.• The figure 5 shows that agents spend most of the time training on the hardest task of the ObstructedMaze curriculum whereas they have not learnt yet the easy tasks.. The agent with seed 6 was trained on the BlockedUnlockPickup curriculum using the baseline algorithm. The return and probability during training are plotted. Two particular moments of the training are framed. Figure 5: 10 agents (seeds) were trained on the ObstructedMaze curriculum using the baseline algorithm. The mean return and mean probability during training are plotted. To overcome these issues, we introduce a new algorithm, focused on the notion of mastering rate, based on the assumption that the good next tasks to train on are the ones that are learnable but not learnt yet. Why this assumption? Because it can't be otherwise. Mainly training on learnt tasks or not learnable ones is a lost of time. This time must be spent training on respectively harder or easier tasks. In subsection 5.1, we first define what are learnt and learnable tasks and then, in subsection 5.2, we present this new algorithm. Learnt tasks. A min-max curriculum is a curriculum C = {c 1, ..., c n} along with:• a family (m 1 c) c∈C where m 1 c is an estimation of the minimum mean return the agent would get on task c. It should be higher than the true minimum mean return.• and a family (M 1 c) c∈C where M 1 c is an estimation of the maximum mean return the agent would get on task c. It should be lower than the true maximum mean return. On such a curriculum, we can define, for a task c:• the live mean returnr c (t) byr c:= m 1 c andr c (t) = (r t1 + ... + r t K)/K where t 1,..., t K are the K last time-steps when the agent was trained on a sample of c and where r ti is the return got by the agent at these time-steps;• the live minimum mean return by m c:= m From this, we can define, for a task c, the mastering rate M c (t):=r DISPLAYFORM0 Mc(t)−mc(t). Intuitively, a task c would be said "learnt" if M c (t) is near 1 and "not learnt" if M c (t) is near 0. FIG6,..., (t K, r t K); Finally, we can remark that the MR algorithm is just a more general version of Teacher-Student algorithms and the baseline algorithm. If we consider min-max ordered curriculums without edges (i.e. just curriculums), if δ = 0, and if we use the GProp dist converter instead of the Prop one, then the MR algorithm is exactly the GProp Linreg algorithm. The MR algorithm with δ = 0.6 (see appendix B for the min-max ordered curriculums given to this algorithm) outperforms the baseline algorithm on all the curriculums, especially on:• KeyCorridor where the median return of the baseline is near 0 after 10M time-steps on S4R3, S5R3 and S6R3 while the first quartile of the MR algorithm is higher than 0.8 after 6M time-steps (see FIG8).• ObstructedMaze where the last quartile of the baseline is near 0 after 10M time-steps on all the tasks while the last quartile of the MR algorithm is higher than 0.7 after 5M time-steps on 1Dl, 1Dlh, 1Dlhb, 2Dl, 2Dlhb (see FIG9).. A CURRICULUMS Three curriculums were used to evaluate the algorithms: BlockedUnlockPickup (3 tasks), KeyCorridor (6 tasks) and ObstructedMaze (9 tasks).(a) Unlock. nmax = 288 DISPLAYFORM0 Figure 8: BlockedUnlockPickup curriculum. In Unlock, the agent has to open the locked door. In the others, it has to pick up the box. In UnlockPickup, the door is locked and, in BlockedUnlockPickup, it is locked and blocked by a ball. The position and color of the door and the box are random. Here are the min-max ordered curriculums given to the MR algorithm in subsection 5.3.For every task c, we set m 1 c to 0 and M 1 c to 0.5. The real maximum mean return is around 0.9 but we preferred to take a much lower maximum estimation to show we don't need an accurate one to get the algorithm working. FIG6: GProp Linreg and MR with δ = 0.6 were each tested with 10 different agents (seeds) on the BlockedUnlockPickup curriculum. The median return during training, between the first and last quartile, are plotted.

We present a new algorithm for learning by curriculum based on the notion of mastering rate that outperforms previous algorithms.

Represent the TLDR for finding the corresponding scientific article.

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.