Question: Represent the article for finding a single-sentence TLDR

Question: Recent literature has demonstrated promising on the training of Generative Adversarial Networks by employing a set of discriminators, as opposed to the traditional game involving one generator against a single adversary. Those methods perform single-objective optimization on some simple consolidation of the losses, e.g. an average. In this work, we revisit the multiple-discriminator approach by framing the simultaneous minimization of losses provided by different models as a multi-objective optimization problem. Specifically, we evaluate the performance of multiple gradient descent and the hypervolume maximization algorithm on a number of different datasets. Moreover, we argue that the previously proposed methods and hypervolume maximization can all be seen as variations of multiple gradient descent in which the update direction computation can be done efficiently. Our indicate that hypervolume maximization presents a better compromise between sample quality and diversity, and computational cost than previous methods. Generative Adversarial Networks (GANs) BID13 offer a new approach to generative modeling, using game-theoretic training schemes to implicitly learn a given probability density. Prior to the emergence of GAN architectures, realistic generative modeling remained elusive. When offering unparalleled realism, GAN training remains fraught with stability issues. Commonly reported shortcomings involved in the GAN game are the lack of useful gradients provided by the discriminator, and mode collapse, i.e. lack of diversity in the generator's samples. Considerable research effort has been devoted in recent literature in order to overcome training instability 1 within the GAN framework. Some architectures such as BEGAN BID4 ) have applied auto-encoders as discriminators and proposed a new loss to help stabilize training. Methods such as TTUR BID16, in turn, have attempted to define schedules for updating the generator and discriminator differently. The PacGAN algorithm proposes to modify the discriminator's architecture which will receive m concatenated samples as input, while modifications to alternate updates in SGD were introduced in . These samples are jointly classified as either real or generated, and authors show that this enforces sample diversity. In SNGAN , authors introduce spectral normalization on the discriminator aiming to ensure Lipschitz continuity, which is empirically shown to consistently yield high quality samples when different sets of hyperparameters are used. Recent works have proposed to tackle GANs instability issues using multiple discriminators. propose a GAN variation in which one generator is trained against a set of discriminators, where each discriminator sees a fixed random projection of the inputs. Prior work, including GMAN BID9 has also explored training against multiple discriminators. In this paper, we build upon Neyshabur et al.'s introduced framework and propose reformulating the average loss minimization aiming to further stabilize GAN training. Specifically, we propose treating the loss signal provided by each discriminator as an independent objective function. To achieve this, we simultaneously minimize the losses using multi-objective optimization techniques. Namely, we exploit previously introduced methods in literature such as the multiple gradient descent algorithm (MGD) BID7. However, due to MGD's prohibitively high cost in the case of large neural networks, we propose the use of more efficient alternatives such as maximization of the hypervolume of the region defined between a fixed, shared upper bound on those losses, which we will refer to as the nadir point η *, and each of the component losses. In contrast to's approach, where the average loss is minimized when training the generator, hypervolume maximization (HV) optimizes a weighted loss, and the generator's training will adaptively assign greater importance to feedback from discriminators against which it performs poorly. Experiments performed on MNIST show that HV presents a good compromise in the computational cost-samples quality trade-off, when compared to average loss minimization or GMAN's approach (low quality and cost), and MGD (high quality and cost). Also, the sensitivity to introduced hyperparameters is studied and indicate that increasing the number of discriminators consequently increases the generator's robustness along with sample quality and diversity. Experiments on CIFAR-10 indicate the method described produces higher quality generator samples in terms of quantitative evaluation. Moreover, image quality and sample diversity are once more shown to consistently improve as we increase the number of discriminators. In summary, our main contributions are the following:1. We offer a new perspective on multiple-discriminator GAN training by framing it in the context of multi-objective optimization, and draw similarities between previous research in GANs variations and MGD, commonly employed as a general solver for multi-objective optimization. 2. We propose a new method for training multiple-discriminator GANs: Hypervolume maximization, which weighs the gradient contributions of each discriminator by its loss. The remainder of this document is organized as follows: Section 2 introduces definitions on multiobjective optimization and MGD. In Section 3 we describe prior relevant literature. Hypervolume maximization is detailed in Section 4, with experiments and presented in Section 5. Conclusions and directions for future work are drawn in Section 6. In this section we provide some definitions regarding multi-objective optimization literature which will be useful in the next sections. Henceforth, the boldface notation will be used to indicate vector-valued variables. Multi-objective optimization. A multi-objective optimization problem is defined as BID6: DISPLAYFORM0 where K is the number of objectives, Ω is the variables space and x = [x 1, x 2, ..., x n] T ∈ Ω is a decision vector or possible solution to the problem. F: Ω → R K is a set of K-objective functions that maps the n-dimensional variables space to the K-dimensional objective space. Pareto-dominance. Let x 1 and x 2 be two decision vectors. x 1 is said to dominate x 2 (denoted by x 1 ≺ x 2) if and only if f i (x 1) ≤ f i (x 2) for all i ∈ {1, 2, . . ., K} and f j (x 1) < f j (x 2) for some j ∈ {1, 2, . . ., K}. If a decision vector x is dominated by no other vector in Ω, x is said to be non-dominated. Pareto-optimality. A decision vector x * ∈ Ω is said to be Pareto-optimal if and only if there is no x ∈ Ω such that x ≺ x *, i.e. x * is a non-dominated solution. The Pareto-optimal Set (PS) is defined as the set of all Pareto-optimal solutions x ∈ Ω, i.e., P S = {x ∈ Ω|x is Pareto optimal}. The set of all objective vectors F(x) such that x is Pareto-optimal is called Pareto front (PF), that is P F = {F(x) ∈ R K |x ∈ P S}.Pareto-stationarity. Pareto-stationarity is a necessary condition for Pareto-optimality. For f k differentiable everywhere for all k, F is said to be Pareto-stationary at the point x if there exists a set of scalars α k, k ∈ {1, . . ., K}, such that: DISPLAYFORM1 Multiple Gradient Descent. Multiple gradient descent BID7 Schäffler et al., 2002; ) was proposed for the unconstrained case of multi-objective optimization of F(x) assuming a convex, continuously differentiable and smooth f k (x) for all k. MGD finds a common descent direction for all f k by defining the convex hull of all ∇f k (x) and finding the minimum norm element within it. Consider w * given by: DISPLAYFORM2 w * will be either 0 in which case x is a Pareto-stationary point, or w * = 0 and then w * is a descent direction for all f i (x). Similar to gradient descent, MGD consists in finding the common steepest descent direction w * t at each iteration t, and then updating parameters with a learning rate λ according to DISPLAYFORM3 3 RELATED WORK While we would prefer to always have strong gradients from the discriminator during training, the vanilla GAN makes this difficult to ensure, as the discriminator quickly learns to distinguish real and generated samples BID12, thus providing no meaningful error signal to improve the generator thereafter. BID9 proposed the Generative Multi-Adversarial Networks (GMAN) which consist in training the generator against a softmax weighted arithmetic average of K different discriminators, according to Eq. 4. DISPLAYFORM0 where DISPLAYFORM1, β ≥ 0, and L D k is the loss of discriminator k and defined as DISPLAYFORM2 where D k (x) and G(z) are the outputs of the k-th discriminator and the generator, respectively. The goal of using the proposed averaging scheme is to privilege worse discriminators and thus providing more useful gradients to the generator during training. Experiments were performed with β = 0 (equal weights), β → ∞ (only worst discriminator is taken into account), β = 1, and β learned by the generator. Models with K = {2, 5} were tested and evaluated using a proposed metric and the Inception score . However, showed that the simple average of discriminator's losses provided the best values for both metrics in most of the considered cases. Opposed to proposed training a GAN with K discriminators using the same architecture. Each discriminator D k sees a different randomly projected lower-dimensional version of the input image. Random projections are defined by a randomly initialized matrix W k, which remains fixed during training. Theoretical provided show that the distribution induced by the generator G will converge to the real data distribution p data, as long as there is a sufficient number of discriminators. Moreover, discriminative tasks in the projected space are harder, i.e. real and fake samples are more alike, thus avoiding early convergence of discriminators, which leads to common stability issues in GAN training such as mode-collapse BID12. Essentially, the authors trade one hard problem for K easier subproblems. The losses of each discriminator L D k are the same as shown in Eq. 5. However, the generator loss L G is defined as simply the sum of the losses provided by each discriminator, as shown in Eq. 6. This choice of L G does not exploit available information such as the performance of the generator with respect to each discriminator. DISPLAYFORM3 3.2 HYPERVOLUME MAXIMIZATION Consider a set of solutions S for a multi-objective optimization problem. The hypervolume H of S is defined as BID10: DISPLAYFORM4, where µ is the Lebesgue measure and η * is a point dominated by all x ∈ S (i.e. f i (x) is upper-bounded by η), referred to as nadir point. H(S) can be understood as the size of the space covered by {F(x)|x ∈ S} BID3.The hypervolume was originally introduced as a quantitative metric for coverage and convergence of Pareto-optimal fronts obtained through population based algorithms BID5. Methods based on direct maximization of H exhibit favorable convergence even in challenging scenarios, such as simultaneous minimization of 50 objectives BID3 We introduce a variation of the GAN game such that the generator solves the following multi-objective problem: DISPLAYFORM0 where each DISPLAYFORM1.., K}, is the loss provided by the k-th discriminator. Training proceeds as the usual formulation BID13, i.e. with alternate updates between the discriminators and the generator. Updates of each discriminator are performed to minimize the loss described in Eq. 5.A natural choice for generator's updates is the MGD algorithm, described in Section 2. However, computing the direction of steepest descent w * before every parameter update step, as required in MGD, can be prohibitively expensive for large neural networks. Therefore, we propose an alternative scheme for multi-objective optimization and argue that both our proposal and previously published methods can all be viewed as performing computationally more efficient versions of MGD update rule without the burden of having to solve a quadratric program, i.e. computing w *, every iteration. Fleischer BID10 has shown that maximizing H yields Pareto-optimal solutions. Since MGD converges to a set of Pareto-stationary points, i.e. a super-set of the Pareto-optimal solutions, hypervolume maximization yields a sub-set of the solutions obtained using MGD.We exploit the above mentioned property and define the generator loss as the negative loghypervolume, as defined in Eq. 8: DISPLAYFORM0 where the nadir point coordinate η is an upper bound for all l k. In Fig. 1 we provide an illustrative example for the case where K = 2. The highlighted region corresponds to e V. Since the nadir point η * is fixed, V will only be maximized, and consequently L G minimized, if each l k is minimized. Figure 1: 2D example of the objective space where the generator loss is being optimized. DISPLAYFORM1 Moreover, by adapting the shown in , the gradient of L G with respect to any generator's parameter θ is given by: DISPLAYFORM2 In other words, the gradient can be obtained by computing a weighted sum of the gradients of the losses provided by each discriminator, whose weights are defined as the inverse distance to the nadir point components. This formulation will naturally assign more importance to higher losses in the final gradient, which is another useful property of hypervolume maximization. Nadir point selection. It is evident from Eq. 9 that the selection of η directly affects the importance assignment of gradients provided by different discriminators. Particularly, as the quantity min k {η − l k} grows, the multi-objective GAN game approaches the one defined by the simple average of l k. Previous literature has discussed in depth the effects of the selection of η in the case of population-based methods BID1 BID7. However, those are not readily applicable for the single-solution case. As will be shown in Section 5, our experiments indicate that the choice of η plays an important role in the final quality of samples. Nevertheless, this effect becomes less relevant as the number of discriminators increases. Similarly to , we propose an adaptive scheme for η such that at iteration t: η t = δ max k {l k,t}, where δ > 1 is a user-defined parameter which will be referred to as slack. This enforces min k {η − l k} to be higher when max k {l k,t} is high and low otherwise, which induces a similar behavior as an average loss when training begins and automatically places more importance on the discriminators in which performance is worse as training progresses. Extra discussion and an illustrative example of the adaptation scheme adopted is presented in Appendix G.Comparison to average loss minimization. The upper bound proven by assumes that the marginals of the real and generated distributions are identical along all random projections. Average loss minimization does not ensure equally good approximation between the marginals along all directions. In case of a trade-off between discriminators, i.e. if decreasing the loss on a given projection increases the loss with respect to another one, the distribution of losses can be uneven. With HV on the other hand, especially when η is reduced throughout training, overall loss will be kept high as long as there are discriminators with high loss. This objective tends to prefer central regions of a trade-off, in which all discriminators present a roughly equally low loss. All methods described previously for the solution of GANs with multiple discriminators, i.e. average loss minimization , GMAN's weighted average BID9 and hypervolume maximization can be defined as MGD-like two-step algorithms consisting of:Step 1 -consolidating all gradients into a single update direction (compute the set α 1,...,K);Step 2 -updating parameters in the direction returned in step 1. Definition of Step 1 for the different methods studied here can be seen in the following: DISPLAYFORM0 Average loss minimization : BID9: DISPLAYFORM1 DISPLAYFORM2 We performed three sets of experiments aiming to analyze the following aspects: (i) How alternative methods for training GANs with multiple discriminators perform in comparison to MGD; (ii) How alternative methods perform in comparison to each other in terms of sample quality and coverage; and (iii) Whether the behavior induced by HV improves the with respect to the baseline methods. Firstly, we exploited the relatively low dimensionality of MNIST and used it as testbed for a comparison of MGD with the other approaches, i.e. average loss minimization (AVG), GMAN's weighted average loss, and HV, proposed in this work. Moreover, multiple initializations and slack combinations were evaluated in order to investigate how varying the number of discriminators affects robustness to those factors. Then, experiments were performed with CIFAR-10 while increasing the number of discriminators. We evaluated HV's performance compared to baseline methods, and the effect in samples quality. We also analyzed the impact on the diversity of generated samples by using the stacked MNIST dataset . Samples of generators trained on stacked MNIST, CIFAR-10, CelebA, and Cats dataset are shown in the Appendix. In all experiments performed, the same architecture, set of hyperparameters and initialization were used for both AVG, GMAN and our proposed method. The only different aspect is the generator loss. Unless stated otherwise, Adam was used to train all the models with learning rate, β 1 and β 2 set to 0.0002, 0.5 and 0.999, respectively. Mini-batch size was set to 64. The Fréchet Inception Distance (FID) BID16 was employed for comparison. Details on FID computation can be found in Appendix A. We employed MGD in our experiments with MNIST. In order to do so, a quadratic program has to be solved prior to every parameters update. For this, we used the Scipy's implementation of the Serial Least Square Quadratic Program solver 2.Three and four fully connected layers with LeakyReLU activations were used for the generator and discriminator, respectively. Dropout was also employed in the discriminator and the random projection layer was implemented as a randomly initialized norm-1 fully connected layer, reducing the vectorized dimensionality of MNIST from 784 to 512. A pretrained LeNet was used for FID computation. Experiments over 100 epochs with 8 discriminators are reported in Fig. 2 and Fig. 3. In Fig. 2, box-plots refer to 30 independent computations of FID over 10000 images sampled from the generator which achieved the minimum FID at train time. FID are measured at train time over 1000 images and the best values are reported in Fig. 3 along with the necessary time to achieve it. MGD outperforms all tested methods. However, its cost per iteration does not allow its use in more relevant datasets other than MNIST. Hypervolume maximization, on the other hand, performs closest to MGD than the considered baselines, while introducing no relevant extra cost. In Fig. 4, we analyze convergence in the Pareto-stationarity sense by plotting the norm of the update direction for each method, given by || K k=1 α k ∇l k ||. All methods converged to similar norms, leading to the that different Pareto-stationary solutions will perform differently in terms of quality of samples. FID as a function of wall-clock time is shown in Figure 22 (Appendix H).HV sensitivity to initialization and choice of δ. Analysis of the sensitivity of the performance with the choice of the slack parameter δ and initialization was performed under the following setting: models were trained for 50 epochs on MNIST with hypervolume maximization using 8, 16, 24 discriminators. Three independent runs (different initializations) were executed with each δ = {1.05, 1.5, 1.75, 2} and number of discriminators, totalizing 36 final models. FIG1 reports the box-plots obtained for 5 FID independent computations using 10000 images, for each of the 36 models obtained under the setting previously described. Results clearly indicate that increasing the number of discriminators yields much smaller variation in the FID obtained by the final model. We evaluate the performance of HV compared to baseline methods using the CIFAR-10 dataset. FID was computed with a pretrained ResNet BID15. ResNet was trained on the 10-class classification task of CIFAR-10 up to approximately 95% test accuracy. DCGAN and WGAN-GP BID14 were included in the experiments for FID reference. Same architectures as in were employed for all multi-discriminators settings. An increasing number of discriminators was used. Inception score as well as FID computed with other models are included in Appendix C.In Fig. 6, we report the box-plots of 15 independent evaluations of FID on 10000 images for the best model obtained with each method across 3 independent runs. Results once more indicate that HV outperforms other methods in terms of quality of the generated samples. Moreover, performance clearly improves as the number of discriminators grows. Fig. 7 shows the FID at train time, i.e. measured with 1000 generated samples after each epoch, for the best models across runs. Models trained against more discriminators clearly converge to smaller values. We report the norm of the update direction || K k=1 α k ∇l k || for each method in FIG4, Appendix C.: FID estimated over 1000 generated images at train time. Models trained against more discriminators achieve lower FID.Cost under the multiple discriminator setting. We highlight that even though training with multiple discriminators may be more computationally expensive when compared to conventional approaches, such framework supports fully parallel training of the discriminators, a feature which is not trivially possible in other GAN settings. For example in WGAN, the discriminator is serially updated multiple times for each generator update. In Fig. 10 at Appendix C, we provide a comparison between the wall-clock time per iteration between all methods evaluated. Serial implementations of discriminators updates with 8 and 16 discriminators were faster than WGAN-GP. We repeat the experiments in aiming to analyze how the number of discriminators impacts the sample diversity of the corresponding generator when trained using hypervolume maximization. The stacked MNIST dataset is employed and reported in are used for comparison. HV for 8, 16, and 24 discriminators were obtained with 10k and 26k generator images averaged over 10 runs. The number of covered modes along with the KL divergence between the generated mode distribution and test data are reported in Table 1 998.0 ± 1.8 0.120 ± 0.004 HV -24 disc.998.3 ± 1.1 0.116 ± 0.003 26k HV -8 disc.776.8 ± 6.4 1.115 ± 0.007 HV -16 disc.1000.0 ± 0.0 0.088 ± 0.002 HV -24 disc.1000.0 ± 0.0 0.084 ± 0.002 Table 1: Number of covered modes and reverse KL divergence for stacked MNIST.As in previous experiments, improved as we increased the number of discriminators. All evaluated models using HV outperformed DCGAN, ALI, Unrolled GAN and VEEGAN. Moreover, HV with 16 and 24 discriminators achieved state-of-the-art coverage values. Thus, the increase in models' capacity via using more discriminators directly ed in an improvement in generator's coverage. Training details as well as architectures information are presented in Appendix B. In this work we have shown that employing multiple discriminators is a practical approach allowing us to trade extra capacity, and thereby extra computational cost, for higher quality and diversity of generated samples. Such an approach is complimentary to other advances in GANs training and can be easily used together with other methods. We introduced a multi-objective optimization framework for studying multiple discriminator GANs, and showed strong similarities between previous work and the multiple gradient descent algorithm. The proposed approach was observed to consistently yield higher quality samples in terms of FID. Furthermore, increasing the number of discriminators was shown to increase sample diversity and generator robustness. Deeper analysis of the quantity || K k=1 α k ∇l k || is the subject of future investigation. We hypothesize that using it as a penalty term might reduce the necessity of a high number of discriminators. In BID16, authors proposed to use as a quality metric the squared Fréchet distance BID11 between Gaussians defined by estimates of the first and second order moments of the outputs obtained through a forward pass in a pretrained classifier of both real and generated data. They proposed the use of Inception V3 for computation of the data representation and called the metric Fréchet Inception Distance (FID), which is defined as: DISPLAYFORM0 where m d, Σ d and m g, Σ g are estimates of the first and second order moments from the representations of real data distributions and generated data, respectively. We employ FID throughout our experiments for comparison of different approaches. However, for each dataset in which FID was computed, the output layer of a pretrained classifier on that particular dataset was used instead of Inception. m d and Σ d were estimated on the complete test partitions, which are not used during training. Architectures of the generator and discriminator are detailed in TAB4, respectively. Batch normalization was used in all intermediate convolutional and fully connected layers of both models. We employed RMSprop to train all the models with learning rate and α set to 0.0001 and 0.9, respectively. Mini-batch size was set to 64. The setup in is employed and we build 128000 and 26000 samples for train and test sets, respectively. Table 4 presents the best FID (computed with a pretrained ResNet) achieved by each approach at train time, along with the epoch in which it was achieved, for each of 3 independent runs. Train time FIDs are computed using 1000 generated images. Table 4: Best FID obtained for each approach on 3 independent runs. FID is computed on 1000 generated images after every epoch. In FIG4, we report the norm of the update direction || K k=1 α k ∇l k || of the best model obtained for each method. Interestingly, different methods present similar behavior in terms of convergence in the Pareto-stationarity sense, i.e. the norm upon convergence is lower for models trained against more discriminators, regardless of the employed method. We computed extra scores using 10000 images generated by the best model reported in Table 4, i.e. the same models utilized to generate the shown in Fig. 6. Both Inception score and FID were computed with original implementations, while FID-VGG and FID-ResNet were computed using a VGG and a ResNet we pretrained. Results are reported with respect to DCGAN's scores. Table 5: Scores of different methods measure on generated CIFAR-10 samples. DCGAN scores are used as reference values, and report are the ratio between given model and DCGAN scores. Inception score is better when high, whereas FIDs are better when low. In TAB9 we present a comparison of minimum FID-ResNet obtained during training, along with computation cost in terms of time and space for different GANs, with both 1 and 24 discriminators. The computational cost of training GANs under a multiple discriminator setting is higher by design, in terms of both FLOPS and memory, if compared with single discriminators settings. However, a corresponding shift in performance is the of the additional cost. This effect was consistently observed considering 4 different well-known approaches, namely DCGAN , Least-square GAN (LSGAN) , and HingeGAN . The architectures of all single discriminator models follow the DCGAN, described in . For the 24 discriminators models, we used the architecture described in , which consists in removing the the normalization layers from DCGAN's discriminator and further adding the projection layer, inline with previous experiments reported for CIFAR-10 upscaled to 64x64. All models were trained with minibatch size of 64 during 150 epochs. Adam BID18 Furthermore, wall-clock time per iteration for different numbers of discriminators is shown in Fig. 10 for experiments with CIFAR-10 with serial updates of discriminators. Notice that while the increase in cost in terms of FLOPS and memory is unavoidable when multiple discriminators settings is employed, wall-clock time can be made close to single discriminators cases since training with respect to different discriminators can be implemented in parallel. On the other hand, extra cost in time introduced by other frameworks such as WGAN-GP or SNGAN cannot be trivially recovered. All reported in previous sections using CIFAR-10 were obtained with an upscaled version of the dataset. Here, we thus run experiments with the dataset in its original resolution aiming to contextualize our proposed approach with respect to previously introduced methods. To do so, we repeated similar experiments as reported in - TAB4, for the model referred to as standard CNN. The same architecture is employed and the spectral normalization is removed from the discriminators. Moreover, the same projection input is added in each of the discriminators. Results in terms of both FID and Inception score, evaluated on top of 5000 generated images as in as well as with 10000 images, are reported in TAB11 for our proposed approach and our implementation of , along with the FID measured using a ResNet classifier trained in advance. As can be seen, the addition of the multiple discriminators setting along with hypervolume maximization yields a relevant shift in performance for the DCGAN-like generator, taking all evaluated metrics to levels of recently proposed GANs. In this experiment, we verify whether the proposed multiple discriminators setting is capable of generating higher resolution images. For that, we employed the CelebA at a size of 128x128. We used a similar architecture for both generator and discriminators networks as described in the previous experiments. A convolutional layer with 2048 feature maps was added to both generator and discriminators architectures due to the increase in the image size. Adam optimizer with the same set of hyperparameters as for CIFAR-10 and CelebA 64x64 was employed. We trained models with 6, 8, and 10 discriminators during 24 epochs. Samples from each generator are shown in FIG7. We show the proposed multiple-discriminators setting scales to higher resolution even in the small dataset regime, by reproducing the experiments presented in BID17. We used the same architecture for the generator. For the discriminator, we removed batch normalization from all layers and used stride equal to 1 at the last convolutional layer, after adding the initial projection step. The Cats dataset 3 was employed, we followed the same pre-processing steps, which, in our case, yielded 1740 training samples with resolution of 256x256. Our model is trained using 24 discriminators and Adam optimizer with the same hyperparameters as for CIFAR-10 and CelebA previously described experiments. In FIG3 we show generator's samples after 288 training epochs. One epoch corresponds to updating over 27 minibatches of size 64.Figure 18: Cats generated using 24 discriminators after 288 training epochs. In this experiment we illustrate and confirm the introduced in , showing the effect of using an increasing number of random projections to train a GAN. We trained models using average loss minimization with 1 to 6 discriminators on the CelebA dataset for 15 epochs. Samples from the generator obtained in the last epoch are shown in FIG4. Generated samples are closer to real data as the number of random projections (and discriminators, consequently) increases.

We introduce hypervolume maximization for training GANs with multiple discriminators, showing performance improvements in terms of sample quality and diversity.

Represent the short summary for finding the corresponding scientific article (long)

It's important to consider optimization in function space, not just parameter space. We introduce a learning rule that reduces distance traveled in function space, just like SGD limits distance traveled in parameter space.


Question: Represent the scientific article to find a short TLDR.

Question: Designing of search space is a critical problem for neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit much smaller than the ones used in recent NAS algorithms. This search space facilitates direct selection of channel numbers and kernel sizes in convolutions. In addition, we propose a resource-aware architecture search algorithm which dynamically selects atomic blocks during training. The algorithm is further accelerated by a dynamic network shrinkage technique. Instead of a search-and-retrain two-stage paradigm, our method can simultaneously search and train the target architecture in an end-to-end manner. Our method achieves state-of-the-art performance under several FLOPS configurations on ImageNet with a negligible searching cost. We open our entire codebase at: https://github.com/meijieru/AtomNAS. Human-designed neural networks are already surpassed by machine-designed ones. Neural Architecture Search (NAS) has become the mainstream approach to discover efficient and powerful network structures (; ; ;). Although the tedious searching process is conducted by machines, humans still involve extensively in the design of the NAS algorithms. Designing of search spaces is critical for NAS algorithms and different choices have been explored. and utilize supernets with multiple choices in each layer to accommodate a sampled network on the GPU. Chen et al. (2019b) progressively grow the depth of the supernet and remove unnecessary blocks during the search. Tan & Le (2019a) propose to search the scaling factor of image resolution, channel multiplier and layer numbers in scenarios with different computation budgets. Stamoulis et al. (2019a) propose to use different kernel sizes in each layer of the supernet and reuse the weights of larger kernels for small kernels.; Tan & Le (2019b) adopts Inverted Residuals with Linear Bottlenecks (MobileNetV2 block) , a building block with light-weighted depth-wise convolutions for highly efficient networks in mobile scenarios. However, the proposed search spaces generally have only a small set of choices for each block. DARTS and related methods b; use around 10 different operations between two network nodes.;;; Stamoulis et al. (2019a) search the expansion ratios in the MobileNetV2 block but still limit them to a few discrete values. We argue that more fine-grained search space is essential to find optimal neural architectures. Specifically, the searched building block in a supernet should be as small as possible to generate the most diversified model structures. We revisit the architectures of state-of-the-art networks; Tan & Le (2019b); ) and find a commonly used building block: convolution -channel-wise operation -convolution. We reinterpret such structure as an ensemble of computationally independent blocks, which we call atomic blocks. This new formulation enables a much larger and more fine-grained search space. Starting from a supernet which is built upon atomic blocks, the search for exact channel numbers and various operations can be achieved by selecting a subset of the atomic blocks. For the efficient exploration of the new search space, we propose a NAS algorithm named AtomNAS to conduct architecture search and network training simultaneously. Specifically, an importance factor is introduced to each atomic block. A penalty term proportional to the computation cost of the atomic block is enforced on the network. By jointly learning the importance factors along with the weights of the network, AtomNAS selects the atomic blocks which contribute to the model capacity with relatively small computation cost. Training on large supernets is computationally demanding. We observe that the scaling factors of many atomic blocks permanently vanish at the early stage of model training. We propose a dynamic network shrinkage technique which removes the ineffective atomic blocks on the fly and greatly reduce the computation cost of AtomNAS. In our experiment, our method achieves 75.9% top-1 accuracy on ImageNet dataset around 360M FLOPs, which is 0.9% higher than state-of-the-art model (a). By further incorporating additional modules, our method achieves 77.6% top-1 accuracy. It outperforms MixNet by 0.6% using 363M FLOPs, which is a new state-of-the-art under the mobile scenario. In summary, the major contributions of our work are: 1. We propose a fine-grained search space which includes the exact number of channels and mixed operations (e.g., combination of different convolution kernels). 2. We propose an efficient end-to-end NAS algorithm named AtomNAS which can simultaneously search the network architecture and train the final model. No finetuning is needed after AtomNAS finishes. 3. With the proposed search space and AtomNAS, we achieve state-of-the-art performance on ImageNet dataset under mobile setting. Recently, there is a growing interest in automated neural architecture design. Reinforce learning based NAS methods (; b; a) are usually computational intensive, thus hampering its usage with limited computational budget. To accelerate the search procedure, ENAS represents the search space using a directed acyclic graph and aims to search the optimal subgraph within the large supergraph. A training strategy of parameter sharing among subgraphs is proposed to significantly increase the searching efficiency. The similar idea of optimizing optimal subgraphs within a supergraph is also adopted by;;;. A prominent disadvantage of the above methods is their coarse search spaces only include limited categories of properties, e.g. kernel size, expansion ratio, the number of layer, etc. Because of the restriction of search space, it is difficult to learn optimal architectures under computational resource constraints. On the contrary, our method proposes the fine-grained search space to enable searching more flexible network architectures under various resource constraints. Assuming that many parameters in the network are unnecessary, network pruning methods start from a computation-intensive model, identify the unimportant connections and remove them to get a compact and efficient network. Early method simultaneously learns the important connections and weights. However, non-regularly removing connections in these works makes it hard to achieve theoretical speedup ratio on realistic hardwares due to extra overhead in caching and indexing. To tackle this problem, structured network pruning methods (b; ; ; ;) are proposed to prune structured show that in structured network pruning, the learned weights are unimportant. This suggests structured network pruning is actually a neural architecture search focusing on channel numbers. Our method jointly searches the channel numbers and a mix of operations, which is a much larger search space. We formulate our neural architecture search method in a fine-grained search space with the atomic block used as the basic search unit. An atomic block is comprised of two convolutions connected by a channel-wise operation. By stacking atomic blocks, we obtain larger building blocks (e.g. residual block and MobileNetV2 block proposed in a variety of state-of-the-art models including ResNet, MobileNet V2/V3 . In Section 3.1, We first show larger network building blocks (e.g. MobileNetV2 block) can be represented by an ensembles of atomic blocks. Based on this view, we propose a fine-grained search space using atomic blocks. In Section 3.2, we propose a resource-aware atomic block selection method for end-to-end architecture search. Finally, we propose a dynamic network shrinkage technique in Section 3.3, which greatly reduces the search cost. Under the typical block-wise NAS paradigm b), the search space of each block in a neural network is represented as the Cartesian product C = i=1 P i, where each P i is the set of all choices of the i-th configuration such as kernel size, number of channels and type of operation. For example, C = {conv, depth-wise conv, dilated conv} × {3, 5} × {24, 32, 64, 128} represents a search space of three types of convolutions by two kernel sizes and four options of channel number. A block in the ing model can only pick one convolution type from the three and one output channel number from the four values. This paradigm greatly limits the search space due to the few choices of each configuration. Here we present a more fine-grained search space by decomposing the network into smaller and more basic building blocks. We denote f c,c (X) as a convolution operator, where X is the input tensor and c, c are the input and output channel numbers respectively. A wide range of manually-designed and NAS architectures share a structure that joins two convolutions by a channel-wise operation: where g is a channel-wise operator. For example, in VGG and a Residual Block , f 0 and f 1 are convolutions and g is one of Maxpool, ReLU and BN-ReLU; in a MobileNetV2 block , f 0 and f 1 are point-wise convolutions and g is depth-wise convolution with BN-ReLU in the MobileNetV2 block. Eq. can be reformulated as follows: where f ] is the operator of the i-th channel of g, and {f are obtained by splitting the kernel tensor of f 1 along the the input channel dimension. Each term in the summation can be seen as a computationally independent block, which is called atomic block. Fig. demonstrate this reformulation. By determining whether to keep each atomic block in the final model individually, the search of channel number c is enabled through channel selection, which greatly enlarges the search space. This formulation also naturally includes the selection of operators. To gain a better understanding, we first generalize Eq. as: Note the array indices i are moved to subscripts. In this formulation, we can use different types of operators for f 0i, f 1i and g i; in other words, f 0, f 1 and g can each be a combination of different operators and each atomic block can use different operators such as convolution with different kernel sizes. Formally, the search space is formulated as a supernet which is built based on the structure in Eq.; such structure satisfies Eq. and thus can be represented by atomic blocks; each of f 0, f 1 and g is a combination of operators. The new search space includes some state-of-the-art network architectures. For example, by allowing g to be a combination of convolutions with different kernel sizes, the MixConv block in MixNet (b) becomes a special case in our search space. In addition, our search space facilitates discarding any number of channels in g, ing in a more fine-grained channel configuration. In comparison, the channels numbers are determined heuristically in Tan & Le (2019b). In this work, we adopt a differentiable neural architecture search paradigm where the model structure is discovered in a full pass of model training. With the supernet defined above, the final model can be produced by discarding part of the atomic blocks during training. Following DARTS ), we introduce a scaling factor α to scale the output of each atomic block in the supernet. Eq. then becomes Here, each α i is tied with an atomic block comprised of three operators f c,1 1i,g i and f 1,c 0i. The scaling factors are learned jointly with the network weights. Once the training finishes, the atomic blocks with factors smaller than a threshold are discarded. We still need to address two issues related to the factor α. First, where should we put them in the supernet? The scaling parameters in the BN layers can be directly used as such scaling factors . In most cases, g contains at least one BN layer and we use the scaling parameters of the last BN layer in g as α. If g has no BN layers, which is rare, we can place α anywhere between f 0 and f 1, as long as we apply regularization terms to the weights of f 0 and f 1 (e.g., weight decays) in order to prevent weights in f 0 and f 1 from getting too large and canceling the effect of α. The second issue is how to avoid performance deterioration after discarding some of the atomic blocks. For example, DARTS discards operations with small scale factors after iterative training of model parameters and scale factors. Since the scale factors of the discarded operations are not small enough, the performance of the network will be affected which needs re-training to adjust the weights again. In order to maintain the performance of the supernet after dropping some atomics blocks, the scaling factors α of those atomic blocks should be sufficiently small. Inspired by the channel pruning work in , we add L1 norm penalty loss on α, which effectively Initialize the supernet and the exponential moving average; while epoch ≤ max epoch do Update network weights and scaling factors α by minimizing the loss function L; Update theα by Eq.; if Total FLOPs of dead blocks ≥ ∆ then Remove dead blocks from the supernet; end Recalculate BN's statistics by forwarding some training examples; Validate the performance of the current supernet; end Algorithm 1: Dynamic network shrinkage pushes many scaling factors to near-zero values. At the end of learning, atomic blocks with α close to zero are removed from the supernet. Note that since the BN scales change more dramatically during training due to the regularization term, the running statistics of BNs might be inaccurate and needs to be calculated again using the training set. With the added regularization term, the training loss is where λ is the coefficient of L1 penalty term, S is the index set of all atomic blocks, and E is the conventional training loss (e.g. cross-entropy loss combined with the weight decay term). |α i | is weighted by coefficient c i which is proportional to the computation cost of i-th atomic block, i.e. c i. By using computation costs aware regularization, we encourage the model to learn network structures that strike a good balance between accuracy and efficiency. In this paper, we use FLOPs as the criteria of computation cost. Other metrics such as latency and energy consumption can be used similarly. As a , the whole loss function L trades off between accuracy and FLOPs. Usually, the supernet is much larger than the final search . We observe that many atomic blocks become "dead" starting from the early stage of the search, i.e., their scaling factors α are close to zero till the end of the search. To utilize computational resources more efficiently and speed up the search process, we propose a dynamic network shrinkage algorithm which cuts down the network architecture by removing atomic blocks once they are deemed "dead". We adopt a conservative strategy to decide whether an atomic block is "dead": for scaling factors α, we maintain its momentumα which is updated aŝ where α t is the scaling factors at t-th iteration and β is the decay term. An atomic block is considered "dead" if bothα and α t are smaller than a threshold, which is set to 1e-3 throughout experiments. Once the total FLOPs of "dead" blocks reach a predefined threshold, we remove those blocks from the supernet. As discussed above, we recalculate BN's running statistics before deploying the network. The whole training process is presented in Algorithm 1. We show the FLOPs of a sample network during the search process in Fig. 2. We start from a supernet with 1521M FLOPs and dynamically discard "dead" atomic blocks to reduce search cost. The overall search and train cost only increases by 17.2% compared to that of training the searched model from scratch. We first describe the implementation details in Section 4.1 and then compare AtomNAS with previous state-of-the-art methods under various FLOPs constraints in Section 4.2. Finally, we provide more analysis about AtomNAS in Section 4.3. The picture on the left of Fig. 3 illustrates a search block in the supernet. Within this search block, f 0 is a 1 × 1 pointwise convolutions that expands the input channel number from C to 3 × 6C; g is a mix of three depth-wise convolutions with kernel sizes of 3 × 3, 5 × 5 and 7 × 7, and f 1 is another 1×1 pointwise convolutions that projects the channel number to the output channel number. Similar to , if the output dimension stays the same as the input dimension, we use a skip connection to add the input to the output. In total, there are 3 × 6C atomic blocks in the search block. The overall architecture of the supernet is shown in the table on the right of Fig. 3. The supernet has 21 search blocks. We use the same training configuration (e.g., RMSProp optimizer, EMA on weights and exponential learning rate decay) as ). We find that using this configuration is sufficient for our method to achieve good performance. Our are shown in Table 1 and Table 3. When training the supernet, we use a total batch size of 2048 on 32 Tesla V100 GPUs and train for 350 epochs. For our dynamic network shrinkage algorithm, we set the momentum factor β in Eq. to 0.9999. At the beginning of the training, all of the weights are randomly initialized. To avoid removing atomic blocks with high penalties (i.e., FLOPs) prematurely, the weight of the penalty term in Eq. is increased from 0 to the target λ by a linear scheduler during the first 25 epochs. By setting the weight of the L1 penalty term λ to be 1.8×10 −4, 1.2×10 −4 and 1.0×10 −4 respectively, we obtain networks with three different sizes: AtomNAS-A, AtomNAS-B, and AtomNAS-C. They have the similar FLOPs as previous state-of-the-art networks under 400M: MixNet-S (b), MixNet-M (b) and SinglePath (a). We apply AtomNAS to search high performance light-weight model on ImageNet 2012 classification task . Table 1 compares our methods with previous state-of-the-art models, either manually designed or searched. With models directly produced by AtomNAS, our method achieves the new state-of-the-art under all FLOPs constraints. Especially, AtomNAS-C achieves 75.9% top-1 accuracy with only 360M FLOPs, and surpasses all other models, including models like PDARTS and DenseNAS which have much higher FLOPs. Techniques like Swish activation function and Squeeze-and-Excitation (SE) module † means methods use extra techniques like Swish activation and Squeeze-and-Excitation module. fair comparison with methods that use these techniques, we directly modify the searched network by replacing all ReLU activation with Swish and add SE module with ratio 0.5 to every block and then retrain the network from scratch. Note that unlike other methods, we do not search the configuration of Swish and SE, and therefore the performance might not be optimal. Extra data augmentations such as MixUp and AutoAugment are still not used. We train the models from scratch with a total batch size of 4096 on 32 Tesla V100 GPUs for 250 epochs. Simply adding these techniques improves the further. AtomNAS-A+ achieves 76.3% top-1 accuracy with 260M FLOPs, which outperforms many heavier models including MnasNet-A2. It performs as well as Efficient-B0 (a) by using 130M less FLOPs and without extra data augmentations. It also outperforms the previous state-of-the-art MixNet-S by 0.5%. In addition, AtomNAS-C+ improves the top-1 accuracy on ImageNet to 77.6%, surpassing previous state-of-the-art MixNet-M by 0.6% and becomes the overall best performing model under 400M FLOPs. Fig. 4 visualizes the top-1 accuracy on ImageNet for different models. It's clear that our fine-grained search space and the end-to-end resource-aware search method boost the performance significantly. † denotes methods using extra network modules such as Swish activation and Squeeze-and-Excitation module. ‡ denotes using extra data augmentation such as MixUp and AutoAugment. * denotes models searched and trained simultaneously. Parameters FLOPs Top-1(%) Top-5(%) MobileNetV1 4.2M 575M 70.6 89.5 MobileNetV2 3.4M 300M 72.0 91.0 MobileNetV2 (our impl.) 3.4M 301M 73.6 91.5 MobileNetV2 (1.4) 6.9M 585M 74.7 92.5 ShuffleNetV2 3.5M 299M 72.6 -ShuffleNetV2 2× 7.4M 591M 74.9 -FBNet-A 4.3M 249M 73.0 -FBNet-C 5.5M 375M 74.9 -Proxyless (mobile) 4.1M 320M 74.6 92.2 SinglePath (a) 4.4M 334M 75.0 92.2 NASNet-A 5.3M 564M 74.0 91.6 DARTS (second order) 4.9M 595M 73.1 -PDARTS (cifar 10) (b) 4.9M 557M 75.6 92.6 DenseNAS-A 7.9M 501M 75.9 92.6 FairNAS-A (b) 4 3x3 32x112x112 16x112x112 24x56x56 24x56x56 24x56x56 24x56x56 40x28x28 40x28x28 40x28x28 40x28x28 80x14x14 80x14x14 80x14x14 80x14x14 96x14x14 96x14x14 96x14x14 96x14x14 192x7x7 192x7x7 192x7x7 192x7x7 Pooling FC 320x7x7 3 5 7 Figure 5: The architecture of AtomNAS-C. Blue, orange, cyan blocks denote atomic blocks with kernel size 3, 5 and 7 respectively; the heights of these blocks are proportional to their expand ratios. We plot the structure of the searched architecture AtomNAS-C in Fig. 5, from which we see more flexibility of channel number selection, not only among different operators within each block, but also across the network. In Fig. 6a, we visualize the ratio between atomic blocks with different kernel sizes in all 21 search blocks. First, we notice that all search blocks have convolutions of all three kernel sizes, showing that AtomNAS learns the importance of using multiple kernel sizes in network architecture. Another observation is that AtomNAS tends to keep more atomic blocks at the later stage of the network. This is because in earlier stage, convolutions of the same kernel size costs more FLOPs; AtomNAS is aware of this (thanks to its resource-aware regularization) and try to keep as less as possible computationally costly atomic blocks. To demonstrate the effectiveness of the resource-aware regularization in Section 3.2, we compare it with a baseline without FLOPs-related coefficients c i, which is widely used in network pruning (; b). Table 2 shows the . First, by using the same L1 penalty coefficient λ = 1.0 × 10 −4, the baseline achieves a network with similar performance but using much more FLOPs; then by increasing λ to 1.5 × 10 −4, the baseline obtain a network which has similar FLOPs but inferior performance (i.e., about 1.0% lower). In Fig. 6b we visualized the ratio of different types of atomic blocks of the baseline network obtained by λ = 1.5×10 −4. The baseline network keeps more atomic blocks in the earlier blocks, which have higher computation cost due to higher input resolution. On the contrary, AtomNAS is aware of the resource constraint, thus keeping more atomic blocks in the later blocks and achieving much better performance. As the BN's running statistics might be inaccurate as explained in Section 3.2 and Section 3.3, we re-calculate the running statistics of BN before inference, by forwarding 131k randomly sampled training images through the network. Table 3 shows the impact of the BN recalibration. The top-1 accuracies of AtomNAS-A, AtomNAS-B, and AtomNAS-C on ImageNet improve by 1.4%, 1.7%, and 1.2% respectively, which clearly shows the benefit of BN recalibration. Our dynamic network shrinkage algorithm speedups the search and train process significantly. For AtomNAS-C, the total time for search-and-training is 25.5 hours. For reference, training the final architecture from scratch takes 22 hours. Note that as the supernet shrinks, both the GPU memory consumption and forward-backward time are significantly reduced. Thus it's possible to dynamically change the batch size once having sufficient GPU memory, which would further speed up the whole procedure. In this paper, we revisit the common structure, i.e., two convolutions joined by a channel-wise operation, and reformulate it as an ensemble of atomic blocks. This perspective enables a much larger and more fine-grained search space. For efficiently exploring the huge fine-grained search space, we propose an end-to-end algorithm named AtomNAS, which conducts architecture search and network training jointly. The searched networks achieve significantly better accuracy than previous state-of-the-art methods while using small extra cost. Table 4: Comparision with baseline backbones on COCO object detection and instance segmentation. Cls denotes the ImageNet top-1 accuracy; detect-mAP and seg-mAP denotes mean average precision for detection and instance segmentation on COCO dataset. The detection of baseline models are from Stamoulis et al. (2019b). SinglePath+ (b) In this section, we assess the performance of AtomNAS models as feature extractors for object detection and instance segmentation on COCO dataset . We first pretrain AtomNAS models (without Swish activation function and Squeeze-and-Excitation (SE) module ) on ImageNet, use them as drop-in replacements for the backbone in the Mask-RCNN model (a) by building the detection head on top of the last feature map, and finetune the model on COCO dataset. We use the open-source code MMDetection (a). All the models are trained on COCO train2017 with batch size 16 and evaluated on COCO val2017. Following the schedule used in the open-source implementation of TPU-trained Mask-RCNN, the learning rate starts at 0.02 and decreases by a scale of 10 at 15-th and 20th epoch respectively. The models are trained for 23 epochs in total. Table 4 compares the with other baseline backbone models. The detection of baseline models are from Stamoulis et al. (2019b). We can see that all three AtomNAS models outperform the baselines on object detection task. The demonstrate that our models have better transferability than the baselines, which may due to mixed operations, a.k.a multi-scale are here, are more important to object detection and instance segmentation. https://github.com/tensorflow/tpu/tree/master/models/official/mask_ rcnn

Represent the TLDR for finding the corresponding scientific article

A new state-of-the-art on Imagenet for mobile setting

Represent the short summary for finding the corresponding scientific article (long).

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.


Question: Represent the scientific article to find a short TLDR

Question: We introduce Lyceum, a high-performance computational ecosystem for robotlearning. Lyceum is built on top of the Julia programming language and theMuJoCo physics simulator, combining the ease-of-use of a high-level program-ming language with the performance of native C. Lyceum is up to 10-20Xfaster compared to other popular abstractions like OpenAI’sGymand Deep-Mind’sdm-control. This substantially reduces training time for various re-inforcement learning algorithms; and is also fast enough to support real-timemodel predictive control with physics simulators. Lyceum has a straightfor-ward API and supports parallel computation across multiple cores or machines. The code base, tutorials, and demonstration videos can be found at: https://sites.google.com/view/lyceum-anon. Progress in deep learning and artificial intelligence has exploded in recent years, due in large part to growing computational infrastructure. The advent of massively parallel GPU computing, combined with powerful automatic-differentiation tools like TensorFlow and PyTorch (b), have lead to new classes of algorithms by enabling what was once computational intractable. These tools, alongside fast and accurate physics simulators like MuJoCo and associated frameworks like OpenAI's Gym and DeepMind's dm_control , have similarly transformed various aspects of robotic control like Reinforcement Learning (RL), Model-Predictive Control (MPC), and motion planning. These platforms enable researchers to give their ideas computational form, share with collaborators, and deploy their successes on real systems. From these advances, simulation to real-world (sim2real) transfer has emerged as a promising paradigm for robotic control. A growing body of recent work suggests that robust control policies trained in simulation can successfully transfer to the real world (a; ; ; ;). Despite these advances, many algorithms are computationally inefficient and have been unable to scale to complex problem domains. Training control policies with state-of-the-art RL algorithms often takes hours to days of compute time. For example, OpenAI's extremely impressive Dactyl work required 50 hours of training time across 6144 CPU cores and 8 powerful NVIDIA V100 GPUs. Such computational budgets are available only to a select few labs. Furthermore, such experiments are seldom run only once in deep learning and especially in deep RL. Indeed, RL algorithms are notoriously sensitive to choices of hyper-parameters and require reward shaping (; ; 2018;). Thus, many iterations of the learning process may be required, with humans in the loop, to improve reward and hyperparameters, before deploying solutions in real world. This computational bottleneck often leads to a scarcity of hardware , relative to the number of papers that propose new algorithms on highly simplified and well tuned benchmark tasks . Exploring avenues to reduce experiment turn around time is thus crucial to scaling up to harder tasks and making resource-intensive algorithms and environments accessible to research labs without massive cloud computing budgets. In a similar vein, computational considerations have also limited progress in model-based control algorithms. For real-time model predictive control, the computational restrictions manifest as the requirement to compute controls in bounded time with limited local resources. As we will show, existing frameworks such as Gym and dm_control, while providing a convenient abstraction in Python, are too slow to meet this real-time computation requirement. As a , most planning algorithms are run offline and deployed in open-loop mode on hardware. This is unfortunate, since it does not take feedback into account which is well known to be critical for stochastic control. Our Contributions: Our goal in this work is to overcome the aforementioned computational restrictions to enable faster training of policies with RL algorithms, facilitate real-time MPC with a detailed physics simulator, and ultimately enable researchers to engage more complex robotic tasks. To this end, we develop Lyceum, a computational ecosystem that uses the Julia programming language and the MuJoCo physics engine. Lyceum ships with the main OpenAI gym continuous control tasks, along with other environments representative of challenges in robotics. Julia's unique features allow us to wrap MuJoCo with zero-cost abstractions, providing the flexibility of a high-level programming language to enable easy creation of environments, tasks, and algorithms while retaining the performance of native C/C++. This allows RL and MPC algorithms implemented in Lyceum to be 10-100X faster compared to Gym and dm_control. We hope that this speedup will enable RL researchers to scale up to harder problems without increased computational costs, as well as enable real-time MPC that looks ahead through a simulator. Recently, various physics simulators and computational ecosystems around them have transformed robot learning research. They allow for exercising creativity to quickly generate new and interesting robotic scenes, as well as quickly prototype RL solutions. We summarize the main threads of related work below. Physics simulators: MuJoCo has quickly emerged as a leading physics simulators for robot learning research. It is fast and efficienct, and particularly well suited for contact rich tasks. Numerous recent works have also demonstrated simulation to reality transfer with MuJoCo through physically consistent system identification (a) or domain randomization (; ;). Our framework wraps MuJoCo in Julia and enables programming and research with a high level language yet retaining the speed of a low level language like C. While we primarily focus on MuJoCo, we believe that similar design principles can also be extended to other simulators such as Bullet and DART . Computational ecosystems: Even though MuJoCo has been around for at least 6-7 years, it's mainstream adoption and popularity grew more recently in the past 3-4 years after the introduction of computational ecosystems around the simulator. Specifically, OpenAI's gym and DeepMind's dm_control sparked a wave of interest by providing python bindings for MuJoCo (which itself is written in C) as well as easy-to-use environments and a highlevel python API. This has enabled the RL community to quickly access physics-based environments and prototype algorithms. Unfortunately, this flexibility comes at the price of computational efficiency. Existing ecosystems are slow due to inefficiencies and poor parallelization capabilities of Python. Prior works have tried to address some of the shortcomings of Python-based frameworks by attempting to add JIT compilation to the language (; a;) but only support a subset of the language, and do not achieve the same performance as Julia. developed a framework similar to Gym that supports distributed computing, but it still suffers the same performance issues of Python and multi-processing. Perhaps closest to our motivation is the work of , which demonstrates the usefulness of Julia as a language for robotics. However, it uses a custom and minimalist rigid body simulator with limited contact support. In contrast, our work attempts to address the inefficiencies of existing computational ecosystems through use of Julia, and directly wraps a high-quality and extensively supported simulator like MuJoCo with zero overhead. A number of algorithmic toolkits like OpenAI Baselines , MJRL , Soft-Learning , and RL-lab ; as well as environments like Hand Manipulation Suite , Door Gym , and Surreal Robosuite have been developed around existing computational ecosystems. Our framework supports all the underlying functionality needed to transfer these advances into our ecosystem (e.g. simulator wrappers and automatic differentiation through Flux). Lyceum comes with a few popular robotics algorithms out of the box like PPO and Natural Policy Gradient for RL as well as several MPC algorithms like "Model Predictive Path Integral" . In the future, we plan to port a number of additional algorithms and advances into our ecosystem. Robotic control with RL and MPC requires unique computational considerations when designing infrastructure and ecosystems. RL algorithms are typically inherently parallel. Consider the class of policy gradient RL algorithms which have demonstrated very impressive in a variety of tasks (; OpenAI;) or evolutionary methods for policy learning. They require rolling out many trajectories to form a dataset, on which the policy is updated. These rollouts can be parallelized and so we would like a computational infrastructure that yields performance which scales linearly with increasing number of cores. RL algorithms are also notoriously sensitive to many hyperparameter details, as well as reward shaping (; ; 2018). These invariably need to be done sequentially with a human in the loop, especially to refine and design the reward functions. Thus, experiment turn around times are critical, which are largely limited by the speed of serial operations within each thread. Similar considerations also apply to real-time MPC, where sampling based algorithms like MPPI , POLO (b), or iLQR (fitted LQR) (; ;) can be parallelized and would benefit from a framework that facilitates this. In addition, real-time MPC also poses the challenge of requiring computation of action to happen within the control loop time period, to incorporate correct real-world feedback. In robotic hardware, this typically must be done with locally available (often on-board) compute, since cloud computing invariably has significantly larger latency than hardware control loop period. This places an even stronger emphasis on extremely efficient serial operations within threads to match the strict bounded time computation requirements. We now describe the structure of our ecosystem and the advantages it provides. Lyceum consists of the following Julia packages which together empower robotics researchers with the ease of use of a high-level, dynamic programming language with all performance of a compiled, low-level language. 1. Lyceum.jl, a "base" package which defines a set of abstract environment and controller interfaces along with several utilities. 2. LyceumAI.jl, a collection of various algorithms for robotic control. 3. MuJoCo.jl, a low-level Julia wrapper for the MuJoCo physics simulator. 4. LyceumMuJoCo.jl, a high-level "environment" abstraction similar to Gym and dm_control. Julia is a general-purpose programming language developed in 2012 at MIT with a focus on technical computing . While a full description of Julia is beyond the scope of this paper, we highlight a few key aspects that we leverage in Lyceum and believe make Julia an excellent tool for robotics and RL researchers. Julia feels like a dynamic, interpreted scripting language. Its high-level syntax, garbage-collected memory management, and feature-rich "Read-Eval-Print-Loop" (REPL) command-line interface makes programming in Julia quick and interactive. Under the hood, however, Julia leverages the LLVM backend to "just-in-time" (JIT) compile native machine code that is as fast as C for a variety of hardware platforms. This allows researchers in robotics and RL to rapidly express their ideas in code while maintaining the high-performance at runtime that is critical to many problem domains. A core philosophy of Julia is that users' code should be just as powerful and fast as the core language. Indeed, most of Julia is itself implemented in Julia! This empowers users to create powerful tools and packages like Lyceum without requiring low-level knowledge of the language or waiting for a library to implement a desired feature. This is highly beneficial for algorithmic development since the kinds of operations available to the researcher are often restricted by what functionality is afforded by the computational platform. For example, use of optimization methods based on Newton's method were out of favor in machine learning until the feature of Hessian-vector products were implemented in low-level languages like C and wrapped into packages like TensorFlow and PyTorch. While many languages and libraries provide the notion of "broadcasting" or vectorizing certain operations, Julia extends this capability to all functions and can combine multiple operations into a single in-place, fused loop. Consider the following example: which compiles down to a single loop that computes 2y 2 + exp(z) for every element in Y and Z, assigning it to it's appropriate location in X. Critically, this allows researchers to flexibly and efficiently apply several operations to a data structure without worrying about which specific vectorized function to use (e.g. BLAS's gemm function) or waiting for a library to implement it if it does not exist. One the greatest benefits of Python is the massive ecosystem of packages it provides. While Julia has many powerful tools like Flux.jl for deep-learning and Optim.jl for optimization, Julia users are also able to easily interact with Python through PyCall.jl in just a few lines of code: using PyCall so = pyimport("scipy.optimize") so.newton(x -> cos(x) -x, 1) Similarly, Julia comes with a straightforward and zero-overhead interface for calling C as shown in this snippet from MuJoCo.jl: Similar tools also exist for other languages like R and C++. This crucially allows Julia researchers to tap into a deep well of tools and libraries and allows roboticists to interact with low-level hardware drivers. Julia also comes with extensive support for distributed and shared-memory multi-threading that allows users to trivially parallelize their code. The following example splits the indices of X across all the available cores and does an in-place multiplication of X by 2 in parallel across an internal thread pool: This equips researchers with the power to parallelize their implementations and make maximal use of high-core count CPUs without worrying about about over-scheduling their machines with too many threads. Lyceum is designed with this feature in mind and supports parallel computation out-of-the-box. All these features and more make writing both performant and generic code easy for the programmer, contributing to a rich ecosystem of open-source Julia packages like Lyceum. To handle the 3000+ packages available, Julia comes with a built-in package manager that prevents having to separately manage a build system and helps to avoid "dependency hell" by allowing each package to provide its own binaries and maintain separate versions of its dependencies. This means less time is spent getting things to run and more time for focusing on the task at hand. At the highest level we provide Lyceum.jl, This base package contains several convenience utilities used throughout the Lyceum ecosystem for data logging, multi-threading, and controller benchmarking (i.e. measuring throughput, jitter, etc.), and more. Lyceum.jl also contains interface definitions, such as AbstractEnv which LyceumMuJoCo.jl, discussed below, implements. This interface is similar to the popular Python frameworks Gym and dm_control with a few differences: 1. The ability to arbitrarily get/set the state of the simulator, a necessary feature for modelbased methods like MPC or motion planning. Note that an important component of this is defining a proper notion of a state, which is often missing from existing frameworks. 2. Optional, in-place versions for all functions (e.g. getstate! (·) which store the return value in a user-allocated data structure. This eliminates unnecessary memory allocations and garbage collection, enabling environments to be used in tight, real-time loops. 3. An optional "evaluation" metric. Often times reward functions are heavily "shaped" and hard to interpret. The evaluation metric serves as a measure of the true task-completion reward that you want to optimize for. We expect most users will be interested in implementing their own environments, which forms a crucial part of robotics research. Indeed, different researchers may be interested in different robots performing different tasks, ranging from whole arm manipulators to legged locomotion to dexterous anthropomorphic hands. To aid this process, we provide sensible defaults for most of the API. Additionally, only a subset of these functions need to be overridden should something other than the default behavior be desired. For example, an environment can only override getobs! (·) and observationspace(·), while our framework uses the information provided by observationspace(·) to pre-allocate the required data structure and pass it to getobs! (·). Having both in-place and out-of-place operations allows users to choose convenience vs greater performance as desired. This separation of interface and implementation allows for other simulators and back-ends (e.g. RigidBodySim.jl or DART) to be used in lieu of the MuJoCo-based environments we provide should the user desire. The next package we provide is MuJoCo.jl, a low-level Julia wrapper for MuJoCo which has a one-to-one correspondence to MuJoCo's C interface. We then build LyceumMuJoCo.jl, the MuJoCo implementation of our AbstractEnv API, on top of MuJoCo.jl. Along with this interface we provide ports of CartPole, Swimmer, Ant, HalfCheetah, and Humanoid from the popular OpenAI Gym framework, along with two new environments, each with their own task-specific reward and evaluation functions. The first is the problem domain of reconfiguration planning (Figure 2), which has been studied earlier in motion planning literature (kin; nie). The task is defined by a set of N total movable objects and M ⊂ N target objects. Each target object has a specified goal configuration while the remaining N \ M objects represent movable obstacles, possibly with their own set of constraints (e.g. on the right side of Figure 2 the objects must stay on the tabletop). The reconfiguration planning domain explores interesting challenges related to: (a) reward under-specification -where only desired goal configurations of a subset of objects are specified; thereby having a multiplicity of equally good and valid solutions; (b) sequential manipulation -in certain cases, some objects may have to be moved out of the way to reach and manipulate other objects; (c) variable number of objects -most current work in RL is concerned with manipulating a fixed number of known objects, whereas in reconfiguration planning there can be a wide variety in number of objects and scenes. We also provide a locomotion environment (Figure 2) in which an agent must navigate robustly through rough terrain, hills, and terraces, represented as height maps. These environments are procedurally-generated, allowing for the environment to be modified by changing a few parameters, support multi-threading out-of-the-box, are non-allocating, and have zero-overhead over using MuJoCo in C directly. Note that our goal is to not solve these task domains in this work, but rather provide an interesting set of environments the research community can work on. Coupled with these environments is LyceumAI.jl, a collection of algorithms for robotic control that similarly leverage Julia's performance and multi-threading abilities. Currently we provide implementations of "Model Predictive Path Integral Control" (MPPI), a stochastic shooting method for model-predictive control and Natural Policy Gradient. We designed our experiments and benchmarks to address the following questions: (a) Does Julia faciliate high-performance programming in the robotics and RL domains (as discussed in Section 3)? (b) Does our computational ecosystem and wrapper lead to faster implementation and experiment time than Gym and dm_control, and in particular is Lyceum faster than real-time for MPC? To answer the above question, we will consider different models of increasing complexity: CartPole, Ant, our reconfiguration environment using HERB, and Humanoid. All experiments are performed on a 16-core Intel i9-7960X with the CPU governor pinned at 1.2GHz so as to prevent dynamic scaling or thermal throttling from affecting . As Gym and dm_control do not come with support for parallel computing, the authors implement this functionality using Python's multiprocessing library as recommended in several GitHub Issues by the respective library authors. In the first task, we explore the parallel scaling performance of LyceumMuJoCo.jl against Gym, dm_control, and a native C implementation using a OpenMP thread pool. We collect 1024 samples in parallel with 1 through 16 threads using the humanoid.xml model that ships with MuJoCo. In the second task, we compare the relative sampling performance for each of the models listed above by again collecting 1024 samples in parallel but with a fixed 16 threads. The third task examines the real time factor using our Julia MPPI implementation and a similar implementation in Python using Gym for each of the models listed above. The real-time factor is defined as: ∆t Simulator ∆t Controller where ∆t Simulator is the timestep that our simulator uses and ∆t Controller is the timestep per iteration of MPPI measured by wall-clock time. A real-time factor of 1 indicates that the controller runs at the same speed as the simulator, while a real-time factor of 2 indicates that the controller runs twice as fast as the simulator. While we collect the data at the same, fixed 1.2GHz, we scale all timings up to what we would expect at 3.3GHz, the sustained clock-speed of our processor, for a more realistic presentation. 6 BENCHMARK Figure 3 illustrates how sampling throughput scales with increasing core counts for Gym, DMC, Lyceum, and a native C implementation used OpenMP. While Lyceum and C scale 98% and 95% of linear between 1 and 16 threads, Gym and DMC only achieve 29% and 23% scaling, respectively. Additionally, the authors noted that this gap grew when experiments were run without the CPU frequency was unlocked and allowed to scale to 3.3GHz. Upon profiling the Python implementation it was discovered that a significant component of the slowdown is due to inter-process communication overhead and waiting on locks. Thus it appears that the distributed Python implementations are largely IO-bound. Figure 3 compares sampling throughput across environments of different complexity. We provide this data in two forms: as a fraction of native C's throughput and as samples per second on a log scale. As seen, Lyceum and native C significantly outperform Gym and dm_control in all cases. The relative performance between the best-performing and worst-performing implementations for Ant, CartPole, HERB, and Humanoid are 44x, 286x, 116x, and 14x, respectively. The smaller differences in the more complex environments is likely due to the fact that each process in the Gym/dm_control environments is performing more work, amortizing the overhead of inter-process communication. It should also be noted that Lyceum and C scale linearly through the entire range of threads, while Gym and dm_control appear to flatline around 10 threads. This suggests that these frameworks may be unable to benefit from higher core count CPUs. While increased sampling throughput is critical for many algorithms, end-to-end performance matters most. Figure 1 compares an implementation of Proximal Policy Optimzation (PPO) in our framework against OpenAI Baseline's implementation for 1 million timesteps in the OpenAI Gym environments Swimmer, Hopper, and Humanoid. Each experiment was averaged over three random seeds and performed on a single core with the default OpenAI Baseline hyper-parameters. For Hopper and Swimmer, our implementation yields similar training curves to OpenAI, while lagging behind for the Humanoid experiment. This difference is likely due to the number of additions made to the core PPO algorithm in OpenAI Baseline's including reward and observation scaling, value function gradient clipping, orthonormal paramter initialization, and more. The important difference, however, is the far shorter training time as measured by wall clock time. This speed up comes from the increased sampling efficieny provided by Lyceum, as well as the high performance auto-differentiation framework Flux.jl. We intruced Lyceum, a new computational ecosystem for robot learning in Julia that provides the rapid prototyping and ease-of-use benefits of a high-level programming language, yet retaining the performance of a low-level language like C. We demonstrated that this ecosystem can obtain 10-20X speedups compared to existing ecosystems like OpenAI gym and dm_control. We also demonstrated that this speed up enables faster experimental times for RL algorithms, as well as real-time model predictive control. In the future, we hope to port over algorithmic infrastructures like OpenAI's baselines as well as environments like hand manipulation suite and DoorGym .

represent this text to find a much much longer article about it with say hundreds-thousands of words

A high performance robotics simulation and algorithm development framework.

Represent the TLDR for finding the corresponding scientific article

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.