Question: Represent the article for finding a single-sentence TLDR

Question: Neural architecture search (NAS) has achieved breakthrough success in a great number of applications in the past few years. It could be time to take a step back and analyze the good and bad aspects in the field of NAS. A variety of algorithms search architectures under different search space. These searched architectures are trained using different setups, e.g., hyper-parameters, data augmentation, regularization. This raises a comparability problem when comparing the performance of various NAS algorithms. NAS-Bench-101 has shown success to alleviate this problem. In this work, we propose an extension to NAS-Bench-101: NAS-Bench-201 with a different search space, on multiple datasets, and more diagnostic information. NAS-Bench-201 has a fixed search space and provides a unified benchmark for almost any up-to-date NAS algorithms. The design of our search space is inspired by the one used in the most popular cell-based searching algorithms, where a cell is represented as a directed acyclic graph. Each edge here is associated with an operation selected from a predefined operation set. For it to be applicable for all NAS algorithms, the search space defined in NAS-Bench-201 includes all possible architectures generated by 4 nodes and 5 associated operation options, which in 15,625 neural cell candidates in total. The training log using the same setup and the performance for each architecture candidate are provided for three datasets. This allows researchers to avoid unnecessary repetitive training for selected architecture and focus solely on the search algorithm itself. The training time saved for every architecture also largely improves the efficiency of most NAS algorithms and presents a more computational cost friendly NAS community for a broader range of researchers. We provide additional diagnostic information such as fine-grained loss and accuracy, which can give inspirations to new designs of NAS algorithms. In further support of the proposed NAS-Bench-102, we have analyzed it from many aspects and benchmarked 10 recent NAS algorithms, which verify its applicability. The deep learning community is undergoing a transition from hand-designed neural architecture (; ; to automatically designed neural architecture (; ; b;). In its early era, the great success of deep learning was promoted by novel neural architectures, such as ResNet , Inception, VGGNet , and Transformer . However, manually designing one architecture requires human experts to try numerous different operation and connection choices . In contrast to architectures that are manually designed, those automatically found by neural architecture search (NAS) algorithms require much less human interaction and expert effort. These NAS-generated architectures have shown promising in many domains, such as image recognition (; ;, sequence modeling (; b;), etc. Recently, a variety of NAS algorithms have been increasingly proposed. While these NAS methods are methodically designed and show promising improvements, many setups in their algorithms are different. Different search space is utilized, e.g., different macro skeletons of the whole architecture ) and a different operation set for the micro cell within the skeleton , etc. After a good architecture is selected, various strategies can be employed to train this architecture and report the performance, e.g., different data augmentation (;, different regularization, different scheduler, and different selections of hyper-parameters (; a). The validation set for testing the performance of the selected architecture is not split in the same way . These discrepancies raise a comparability problem when comparing the performance of various NAS algorithms, making it difficult to conclude their contributions. In response to this problem, NAS-Bench-101 and NAS-HPO-Bench are proposed. However, some NAS algorithms can not be applied directly on NASBench-101, and NAS-HPO-Bench only has 144 candidate architectures, which maybe insufficient to evaluate NAS algorithms. To extend these two benchmarks and towards better reproducibility of NAS methods 1, we propose NAS-Bench-201 with a fixed cell search space, inspired from the search space used in the most popular neural cell-based searching algorithms ). As shown in Figure 1, each architecture consists of a predefined skeleton with a stack of the searched cell. In this way, architecture search is transformed into the problem of searching a good cell. Each cell is represented as a densely-connected directed acyclic graph (DAG) as shown in the bottom section of Figure 1. Here the node represents the sum of the feature maps and each edge is associated with an operation transforming the feature maps from the source node to the target node. The size of the search space is related to the number of nodes defined for the DAG and the size of the operation set. In NAS-Bench-201, we choose 4 nodes and 5 representative operation candidates for the operation set, which generates a total search space of 15,625 cells/architectures. Each architecture is trained multiple times on three different datasets. The training log and performance of each architecture are provided for each run. The training accuracy/test accuracy/training loss/test loss after every training epoch for each architecture plus the number of parameters and floating point operations (FLOPs) are accessible. Hopefully, NAS-Bench-201 will show its value in the field of NAS research. It provides a unified benchmark for most up-to-date NAS algorithms including all cell-based NAS methods. With NASBench-201, researchers can focus on designing robust searching algorithm while avoiding tedious hyper-parameter tuning of the searched architecture. Thus, NAS-Bench-201 provides a relatively fair benchmark for the comparison of different NAS algorithms. It provides the full training log of each architecture. Unnecessary repetitive training procedure of each selected architecture can be avoided so that researchers can target on the essence of NAS, i.e., search algorithm. Another benefit is that the validation time for NAS largely decreases when testing in NAS-Bench-201, which provides a computational power friendly environment for more participations in NAS. It provides of each architecture on multiple datasets. The model transferability can be thoroughly evaluated for most NAS algorithms. In NAS-Bench-201, we provide systematic analysis of the proposed search space. We also evaluate 10 recent advanced NAS algorithms including reinforcement learning (RL)-based methods, evolutionary strategy (ES)-based methods, differentiable-based methods, etc. We hope our empirical analysis can bring some insights to the future designs of NAS algorithms. Our NAS-Bench-201 is algorithm-agnostic. Put simply, it is applicable to almost any up-to-date NAS algorithms. In this section, we will briefly introduce our NAS-Bench-201. The search space of NASBench-201 is inspired by cell-based NAS algorithms (Section 2.1). NAS-Bench-201 evaluates each architecture on three different datasets (Section 2.2). All implementation details of NAS-Bench-201 are introduced in Section 2.3. NAS-Bench-201 also provides some diagnostic information which can be used for potentially better designs of future NAS algorithms (discussed in Section 2.4). Macro Skeleton. Our search space follows the design of its counterpart as used in the recent neural cell-based NAS algorithms . As shown in the top of Figure 1, the skeleton is initiated with one 3-by-3 convolution with 16 output channels and a batch normalization layer . The main body of the skeleton includes three stacks of cells, connected by a residual block. Each cell is stacked N = 5 times, with the number of output channels as 16, 32 and 64 for the first, second and third stages, respectively. The intermediate residual block is the basic residual block with a stride of 2 , which serves to downsample the spatial size and double the channels of an input feature map. The shortcut path in this residual block consists of a 2-by-2 average pooling layer with stride of 2 and a 1-by-1 convolution. The skeleton ends up with a global average pooling layer to flatten the feature map into a feature vector. Classification uses a fully connected layer with a softmax layer to transform the feature vector into the final prediction. Searched Cell. Each cell in the search space is represented as a densely connected DAG. The densely connected DAG is obtained by assigning a direction from the i-th node to the j-th node (i < j) for each edge in an undirected complete graph. Each edge in this DAG is associated with an operation transforming the feature map from the source node to the target node. All possible operations are selected from a predefined operation set, as shown in Figure 1 (bottom-right). In our NAS-Bench-201, the predefined operation set O has L = 5 representative operations: zeroize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling layer. The convolution in this operation set is an abbreviation of an operation sequence of ReLU, convolution, and batch normalization. The DAG has V = 4 nodes, where each node represents the sum of all feature maps transformed through the associated operations of the edges pointing to this node. We choose V = 4 to allow the search space to contain basic residual block-like cells, which requires 4 nodes. Densely connected DAG does not restrict the searched topology of the cell to be densely connected, since we include zeroize in the operation set, which is an operation of dropping the associated edge. Besides, since we do not impose the constraint on the maximum number of edges , our search space is applicable to most NAS algorithms, including all cell-based NAS algorithms. We train and evaluate each architecture on CIFAR-10, CIFAR-100 , and ImageNet-16-120 . We choose these three datasets because CIFAR and ImageNet are the most popular image classification datasets. We split each dataset into training, validation and test sets to provide a consistent training and evaluation settings for previous NAS algorithms . Most NAS methods use the validation set to evaluate architectures after the architecture is optimized on the training set. The validation performance of the architectures serves as supervision signals to update the searching algorithm. The test set is to evaluate the performance of each searching algorithm by comparing the indicators (e.g., accuracy, model size, speed) of their selected architectures. Previous methods use different splitting strategies, which may in various searching costs and unfair comparisons. We hope to use the proposed splits to unify the training, validation and test sets for a fairer comparison. It is a standard image classification dataset and consists of 60K 32×32 colour images in 10 classes. The original training set contains 50K images, with 5K images per class. The original test set contains 10K images, with 1K images per class. Due to the need of validation set, we split all 50K training images in CIFAR-10 into two groups. Each group contains 25K images with 10 classes. We regard the first group as the new training set and the second group as the validation set. This dataset is just like CIFAR-10. It has the same images as CIFAR-10 but categorizes each image into 100 fine-grained classes. The original training set on CIFAR-100 has 50K images, and the original test set has 10K images. We randomly split the original test set into two group of equal size -5K images per group. One group is regarded as the validation set, and another one is regarded as the new test set. ImageNet-16-120: We build ImageNet-16-120 from the down-sampled variant of ImageNet (ImageNet16×16). As indicated in , down-sampling images in ImageNet can largely reduce the computation costs for optimal hyper-parameters of some classical models while maintaining similar searching . down-sampled the original ImageNet to 16×16 pixels to form ImageNet16×16, from which we select all images with label ∈ to construct ImageNet-16-120. In sum, ImageNet-16-120 contains 151.7K training images, 3K validation images, and 3K test images with 120 classes. By default, in this paper, "the training set", "the validation set", "the test set" indicate the new training, validation, and test sets, respectively. Training Architectures. In order to unify the performance of every architecture, we give the performance of every architecture in our search space. In our NAS-Bench-201, we follow previous ). We train each architecture with the same strategy, which is shown in Table 1. For simplification, we denote all hyperparameters for training a model as a set H, and we use H † to denote the values of hyper-parameter that we use. Specifically, we train each architecture via Nesterov momentum SGD, using the cross-entropy loss for 200 epochs in total. We set the weight decay as 0.0005 and decay the learning rate from 0.1 to 0 with a cosine annealing ). We use the same H † on different datasets, except for the data augmentation which is slightly different due to the image resolution. On CIFAR, we use the random flip with probability of 0.5, the random crop 32×32 patch with 4 pixels padding on each border, and the normalization over RGB channels. On ImageNet-16-120, we use a similar strategy but random crop 16×16 patch with 2 pixels padding on each border. Apart from using H † for all datasets, we also use a different hyper-parameter set H ‡ for CIFAR-10. It is similar to H † but its total number of training epochs is 12. In this way, we could provide bandit-based algorithms more options for the usage of short training budget (see more details in appendix). Metrics. We train each architecture with different random seeds on different datasets. We evaluate each architecture A after every training epoch. NAS-Bench-201 provides the training, validation, Table 2. Users can easily use our API to query the of each trial of A, which has negligible computational costs. In this way, researchers could significantly speed up their searching algorithm on these datasets and focus solely on the essence of NAS. We list the training/test loss/accuracies over different split sets on four datasets in Table 2. On CIFAR-10, we train the model on the training set and evaluate it on the validation set. We also train the model on the training and validation set and Table 3: We summarize some characteristics of NAS-Bench-101 and NAS-Bench-201. Our NASBench-201 can directly be applicable to almost any up-to-date NAS algorithms. In contrast, as pointed in , NAS algorithms based on parameter sharing or network morphisms cannot be directly evaluated on NAS-Bench-101. Besides, NAS-Bench-201 provides train/validation/test performance on three (one for NAS-Bench-101) different datasets so that the generality of NAS algorithms can be evaluated. It also provides some diagnostic information that may provide insights to design better NAS algorithms. evaluate it on the test set. These two paradigm follow the typical experimental setup on CIFAR-10 in previous literature (; ;). On CIFAR-100 and ImageNet-16-120, we train the model on the training set and evaluate it on both validation and test sets. Validation accuracy is a commonly used supervision signal for NAS. However, considering the expensive computational costs for evaluating the architecture, the signal is too sparse. In our NASBench-201, we also provide some diagnostic information which is some extra statistics obtained during training each architecture. Collecting these statistics almost involves no extra computation cost but may provide insights for better designs and training strategies of different NAS algorithms, such as platform-aware NAS , accuracy prediction , mutationbased NAS , etc. Architecture Computational Costs: NAS-Bench-201 provides three computation metrics for each architecture -the number of parameters, FLOPs, and latency. Algorithms that target on searching architectures with computational constraints, such as models on edge devices, can use these metrics directly in their algorithm designs without extra calculations. Fine-grained training and evaluation information. NAS-Bench-201 tracks the changes in loss and accuracy of every architecture after every training epochs. These fine-grained training and evaluation information shows the tendency of the architecture performance and could indicate some attributes of the model, such as the speed of convergence, the stability, the over-fitting or under-fitting levels, etc. These attributes may benefit the designs of NAS algorithms. Besides, some methods learn to predict the final accuracy of an architecture based on the of few early training epochs . These algorithm can be trained faster and the performance of the accuracy prediction can be evaluated using the fine-grained evaluation information. Parameters of optimized architecture. Our NAS-Bench-201 releases the trained parameters for each architecture. This can provide ground truth label for hypernetwork-based NAS methods , which learn to generate parameters of an architecture. Other methods mutate an architecture to become another one ). With NAS-Bench-201, researchers could directly use the off-the-shelf parameters instead of training from scratch and analyze how to transfer parameters from one architecture to another. To the best of our knowledge, NAS-Bench-101 is the only existing large-scale architecture dataset. Similar to NAS-Bench-201, NAS-Bench-101 also transforms the problem of architecture search into the problem of searching neural cells, represented as a DAG. Differently, NAS-Bench-101 defines operation candidates on the node, whereas we associate operations on the edge as inspired from (; b; . We summarize characteristics of our NAS-Bench-201 and NAS-Bench-101 in Table 3 . The main highlights of our NAS-Bench-201 are as follows. NAS-Bench-201 is algorithm-agnostic while NAS-Bench- 101 without any modification is only applicable to selected algorithms . The original complete search space, based on the nodes in NAS-Bench-101, is extremely huge. So, it is exceedingly difficult to efficiently traverse the training of all architectures. To trade off the computational cost and the size of the search space, they constrain the maximum number of edges in the DAG. However, it is difficult to incorporate this constraint in all NAS algorithms, such as NAS algorithms based on parameter-sharing . Therefore, many NAS algorithms cannot be directly evaluated on NAS-Bench-101. Our NAS-Bench-201 solves this problem by sacrificing the number of nodes and including all possible edges so that our search space is algorithm-agnostic. We provide extra diagnostic information, such as architecture computational cost, fine-grained training and evaluation time, etc., which give inspirations to better and efficient designs of NAS algorithms utilizing these diagnostic information. NAS-HPO-Bench hyper-parameter space for a simple 2-layer feed-forward network. Since NAS-HPO-Bench has only 144 architectures, it could be insufficient to evaluate different NAS algorithms. An overview of architecture performance. The performance of each architecture is shown in Figure 2. We show the test accuracy of every architecture in our search space in the left column of Figure 2. The training, validation and test accuracy with respect to the number of parameters are shown in the rest three columns, respectively. Results show that a different number of parameters will affect the performance of the architectures, which indicates that the choices of operations are essential in NAS. We also observe that the performance of the architecture can vary even when the number of parameters stays the same. This observation indicates the importance of how the operations/cells are connected. We compare the architectures with a clas-sical human-designed architecture (ResNet) in all cases, which is indicated by an orange star mark. ResNet shows competitive performance in three datasets, however, it still has room to improve, i.e., about 2% compared to the best architecture in CIFAR-100 and ImageNet-16-120, about 1% compared to the best one with the same amount of parameters in CIFAR-100 and ImageNet-16-120. Architecture ranking on three datasets. The ranking of every architecture in our search space is shown in Figure 3, where the architecture ranked in CIFAR-10 (x-axis) is ranked as in y-axis in CIFAR-100 and ImageNet-16-120, indicated by green and red markers respectively. The performance of the architectures shows a generally consistent ranking over the three datasets with slightly different variance, which serves to test the generality of the searching algorithm. Correlations of validation and test accuracies. We visualize the correlation between the validation and test accuracy within one dataset and across datasets in Figure 4. The correlation within one dataset is high compared to cross-dataset correlation. The correlation dramatically decreases as we only pick the top performing architectures. When we directly transfer the best architecture in one dataset to another (a vanilla strategy), it can not 100% secure a good performance. This phenomena is a call for better transferable NAS algorithms instead of vanilla strategy. We show the ranking of the performance of all architectures in different time stamps in Figure 5. The ranking based on the validation set (y axis) gradually converges to the ranking based on the final test accuracy (x axis). In this section, we evaluate 10 recent searching methods on our NAS-Bench-201, which can serve as baselines for future NAS algorithms in our dataset. Specifically, we evaluate some typical NAS algorithms: (I) Random Search algorithms, e.g., random search (RS) , random search with parameter sharing (RSPS) . (II) ES methods, e.g., REA. (III) RL algorithms, e.g., REINFORCE , ENAS . (IV) Differentiable algorithms. e.g., first order DARTS (DARTS-V1) , second order DARTS (DARTS-V2), GDAS (b), and SETN (a). (V) HPO methods, e.g., BOHB . We experimented all NAS algorithms on a single GeForce GTX 1080 Ti GPU. training on the CIFAR-10 train set and evaluating on its validation set; training on the CIFAR-10 train+validation sets and evaluating on its test set; training on the CIFAR-10 or ImageNet-16-120 train set and evaluating on their validation or test sets. "optimal" indicates the highest mean accuracy for each set. We report the mean and std of 500 runs for RS, REA, REINFORCE, and BOHB and of 3 runs for RSPS, DARTS, GDAS, SETN, and ENAS. Figure 6: We show of 500 runs for RS, REA, REINFORCE, and BOHB on CIFAR-10. The architecture is searched on CIFAR-10 and we report its validation accuracy (solid line) and test accuracy (dashed line) on three datasets. Each individual run is sorted by the validation accuracy of the searched architecture. We show the benefits for speed using our NAS-Bench-201 for different NAS algorithms in Table 4. For each NAS algorithm, once the searching procedure finished and the final architecture is found, our NAS-Bench-201 can directly return the performance of this architecture. With NAS-Bench-201, NAS algorithms without parameter sharing can significantly reduce the searching time into seconds. Notably, it still requires several GPU hours for NAS algorithms with parameter sharing to complete the searching. All algorithms use the training and validation set of CIFAR-10 to search architectures. In Table 5, Figure 6, Figure 7, and Figure 8, we report the performance of the searched architectures plus the optimal architecture on three datasets. We make the following observations: NAS methods without parameter sharing (REA, RS, REINFORCE, and BOHB) outperform others. This be because training a model for a few epochs with the converged LR scheduler (H ‡) can provide a good relative In Figure 7 and Figure 8, we show the performance of the architecture derived from each algorithm per searching epoch. DARTS-V1 will gradually over-fit to an architecture with all skip-connection operations. DARTS-V2 can alleviate this problem to some extent but will still over-fit after more epochs. It can further alleviate this problem by using batch statistics for BN layers. We train RSPS, GDAS, SETN, and ENAS five times longer than DARTS (250 epochs vs. 50 epochs). This is because at every iteration, RSPS, GDAS, SETN, and ENAS only optimize 1 |O|=5 parameters of the shared parameters, whereas DARTS optimize all shared parameters. The searched architecture performs similar for GDAS after 50 searching epochs. RSPS and SETN show a higher variance of the searched architecture compared to GDAS. Clarification. We have tried our best to implement each method. However, still, some algorithms might obtain non-optimal since their hyper-parameters might not fit our NAS-Bench-201. We empirically found that some NAS algorithms are sensitive to some hyper-parameters, whereas we try to compare them in a fair way as we can (Please see more explanation in Appendix). If researchers can provide better with different hyper-parameters, we are happy to update according to the new experimental . We also welcome more NAS algorithms to test on our dataset and would include them accordingly. How to avoid over-fitting on NAS-Bench-201? Our NAS-Bench-201 provides a benchmark for NAS algorithms, aiming to provide a fair and computational cost-friendly environment to the NAS community. The trained architecture and the easy-to-access performance of each architecture might provide some insidious ways for designing algorithms to over-fit the best architecture in our NASBench-201. Thus, we propose some rules which we wish the users will follow to achieve the original intention of NAS-Bench-201, a fair and efficient benchmark. 1. No regularization for a specific operation. Since the best architecture is known in our benchmark, specific designs to fit the structural attributes of the best performed architecture are insidious ways to fit our NAS-Bench-201. For example, as mentioned in Section 5, we found that the best architecture with the same amount of parameters for CIFAR10 on NAS-Bench-201 is ResNet. Restrictions on the number of residual connections is a way to over-fit the CIFAR10 benchmark. While this can give a good on this benchmark, the searching algorithm might not generalize to other benchmarks. 2. Use the provided performance. The training strategy affects the performance of the architecture. We suggest the users stick to the performance provided in our benchmark even if it is feasible to use other H to get a better performance. This provides a fair comparison with other algorithms. 3. Report of multiple searching runs. Since our benchmark can help to largely decrease the computational cost for a number of algorithms. Multiple searching runs give stable of the searching algorithm with acceptable time cost. Limitation regarding to hyper-parameter optimization (HPO). The performance of an architecture depends on the hyper-parameters H for its training and the optimal configuration of H may vary for different architectures. In NAS-Bench-201, we use the same configuration for all architectures, which may bring biases to the performance of some architectures. One related solution is HPO, which aims to search the optimal hyper-parameter configuration. However, searching the optimal hyper-parameter configurations and the architecture in one shot is too computationally expensive and still is an open problem. Potential designs using diagnostic information in NAS-Bench-201. As pointed in Section 2.4, different kinds of diagnostic information are provided. We hope that more insights about NAS could be found by analyzing these diagnostic information and further motivate potential solutions for NAS. For example, parameter sharing is the crucial technique to improve the searching efficiency, but the shared parameter would sacrifice the accuracy of each architecture. Could we find a better way to share parameters of each architecture from the learned 15,625 models' parameters? Generalization ability of the search space. It is important to test the generalization of observations on this dataset. An idea strategy is to do all benchmark experiments on a much larger search space. Unfortunately, it is prohibitive regarding the expensive computational cost. We bring some from and to provide some preliminary evidence of generalization. In Figure 2, we show the rankings of RS, REA, and REINFORCE is (REA > REINFORCE > RS). This is consistent with in NAS-Bench-101, which contains more architecture candidates. For NAS methods with parameter sharing, we find that GDAS ≥ DARTS ≥ ENAS, which is also consistent with in NAS-Bench-1SHOT1. Therefore, observations from our NAS-Bench-201 may generalize to other search spaces. In this paper, we introduce NAS-Bench-201 that extends the scope of reproducible NAS. In NASBench-201, almost any NAS algorithms can be directly evaluated. We train and evaluate 15,625 architecture on three different datasets, and we provide regarding different metrics. We comprehensively analyze our dataset and test some recent NAS algorithms on NAS-Bench-201 to serve as baselines for future works. In future, we will consider HPO and NAS together and much larger search space. We welcome researchers to try their NAS algorithms on our NAS-Bench-201 and would update the paper to include their . Table 6: We compare the correlation of different training strategies. The correlation coefficient between the validation accuracy after several training epochs on CIFAR-10 and the validation accuracy of full trained models on the CIFAR-10 training set, the test accuracy on CIFAR-10 trained with the training and validation sets, the validation/test accuracy on CIFAR-100 trained with the CIFAR-100 training set, the validation/test accuracy on ImageNet-16-120 trained with the ImageNet-16-120 training set. We use the validation accuracy after "EPOCHS" training epochs, where the the cosine annealing converged after "TOTAL" epochs. Number of unique architectures. In our NAS-Bench-201, we encode each architecture by a 6-dimensional vector. The i-th value in this vector indicates the operation in the i-th edge in a cell. Since we have 5 possible operations, there are 5 6 =15625 total unique models in this encoding. If we identify the isomorphic cell caused by the "skip-connect" operation, there are 12751 unique topology structures. If we identify the isomorphic cell caused by both "skip-connect" and "zeroize" operations, there are only 6466 unique topology structures. Note that, due to the numerical error, when given the same inputs, two architectures with the isomorphic cell might have different outputs. Note that, when we build our NAS-Bench-201, we train and evaluate every architecture without considering isomorphism. NAS-Bench-201 with bandit-based algorithms. Bandit-based algorithms, such as Hyperband and BOHB , usually train models with a short time budget. In our NAS-Bench-201, on CIFAR-10, we provide two options if you want to obtain the performance of a model trained with a short time budget: Results from H ‡, where the cosine annealing converged at the 12-th epoch. Results from H †, where the cosine annealing converged at the 200-th epoch. As shown in Table 6, the performance of these converged networks is much more likely to correlate highly with the performance after a larger number of iterations than just taking an earlier point of a single cosine annealing trajectory. Therefore, we choose the first option for all NAS algorithms that do not use parameter sharing. Based on the publicly available codes, we re-implement 10 NAS algorithms by ourselves to search architectures on our NAS-Bench-201. We provide the implementation details of each searching algorithm below. We consider the searching time of the first order DARTS as a baseline (about 12000 seconds on CIFAR-10). When evaluating RS, REINFORCE, ENAS, and BOHB, we set the total time budget as 12000 seconds for them. By default, for NAS algorithms with parameter sharing, we follow most hyper-parameters from DARTS and do not learn the scale and shift parameters for BN layers in each searching cell. We setup the searching procedure of RSPS, GDAS, SETN, ENAS five times longer than DARTS, because they optimize 1 5 of parameters but DARTS optimize all parameters per iteration. Most configurations can be found at https://github.com/D-X-Y/ AutoDL-Projects/tree/master/configs/nas-benchmark/algos. Random search (RS) . We randomly select architectures until the total 0 100 200 300 400 The index of runs The accuracy (%) sample-size= 3 sample-size= 5 sample-size=10 Figure 9: The effect of different sample sizes for REA on the CIFAR-10 validation set. training time plus the time of one evaluation procedure reaches the total budget. We use the validation accuracy after 12 training epochs (H ‡), which can be obtained directly in our NAS-Bench-201 as discussed in Section 2.4. The architecture with the highest validation accuracy is selected as the final searched architecture. Regularized evolution for image classifier architecture search (REA). We set the initial population size as 10, the number of cycles as infinity. The sample size is chosen as 10 from, according to Figure 9. We finish the algorithm once the simulated training time of the traversed architecture reaches the time budgets (12000 seconds). We use the validation accuracy after 12 training epochs (H ‡) as the fitness. ). We follow to use the REINFORCE algorithm as Figure 10: We evaluate the effect of different learning rates for REINFORCE, and report the CIFAR-10 validation accuracy of the searched architecture. a baseline RL method. We use an architecture encoding to parameterize each candidate in our search space as (; b . According to Figure 10, the learning date is set as. The momentum for exponential moving average of 0.9. We finish the training once the simulated training time reaches the time budgets (12000 seconds). The first order and second order DARTS (DARTS-V1 and DARTS-V2) . We train the shared parameters via Nesterov momentum SGD, using the cross-entropy loss for 50 epochs in total. We set weight decay as 0.0005 and momentum of 0.9. We decay the learning rate from 0.025 to 0.001 via cosine learning rate scheduler and clip the gradient by 5. We train the architecture encoding via Adam with the learning rate of 0.0003 and the weight decay of 0.001. We use the batch size of 64. The random horizontal flipping, random cropping with padding, and normalization are used for data augmentation. We choose these hyper-parameters following . Random search with parameter sharing (RSPS) . We train RSPS with the similar hyper-parameters as that of DARTS. Differently, we train the algorithm in 250 epochs in total. During each searching iteration, we randomly sample one architecture in each batch training. Each architecture uses the training mode for BN during training and the evaluation mode during evaluation . After training the shared parameters, we evaluate 100 randomly selected architectures with the shared parameters. For each architecture, we randomly choose one mini-batch with 256 validation samples to estimate the validation accuracy instead of using the whole validation set to calculate the precise validation accuracy. The one with the highest estimated validation accuracy will be selected. With the size of this mini-batch increasing, the more precise validation accuracy would be obtained and the better architecture would be selected. However, the searching costs will also be increased. We use the size of 256 to trade-off the accuracy and cost. Gradient-based search using differentiable architecture sampler (GDAS) (b). We use the most hyper-parameters as that of DARTS but train it for 250 epochs in total. The Gumbel-Softmax temperature is linearly decayed from 10 to 0.1. (a). We use the most hyperparameters as that of DARTS but train it for 250 epochs in total. After training the shared parameters, we select 100 architectures with the highest probabilities (encoded by the learned architecture en- Table 7 : The correlation between the probability or the one-shot validation accuracy (OSVA) and the ground truth accuracy on the CIFAR-10 validation set. "BN with Train" indicates that, during evaluation, the mean and variance of BN layers are calculated within each mini-batch. "BN with Eval" indicates that we accumulate mean and variance of BN layers in the training set and use these accumulated mean and variance for evaluation. We report the correlation as the average of 3 runs. coding). We evaluate these 100 selected architectures with the shared parameters. The evaluation procedure for these 100 architectures are the same as RSPS. ENAS . We use a two layer LSTM as the controller with the hidden size of 32. We use the temperature of 5 and the tanh constant of 2.5 for the sampling logits Following , we also add the the controller's sample entropy to the reward, weighted by 0.0001. We optimize the controller with Adam using the constant learning rate of 0.001. We optimize the network weights with SGD following the learning rate scheduler as the original paper and the batch size of 128. We did not impose any penalty to a specific operation. BOHB . We choose to use BOHB as an HPO algorithm on our NAS-Bench-201. We follow to set up the hyper-parameters for BOHB. We set the number of samples for the acquisition function to 4, the random fraction to 0%, the minimum-bandwidth to 0.3, the bandwidth factor to 3. We finish the algorithm once the simulated training time reaches the time budgets (12000 seconds). Parameter sharing becomes a common technique to improve the efficiency of differentiable neural architecture search methods (; b; a). The shared parameters are shared over millions of architecture candidates. It is almost impossible for the shared parameters to be optimal for all candidates. We hope to evaluate the trained shared parameters quantitatively. Specially, we use DARTS, GDAS, and SETN to optimize the shared parameters and the architecture encoding on CIFAR-10. For each architecture candidate, we can calculate its probability of being a good architecture from the architecture encoding following SETN (a). In addition, we can also evaluate a candidate using the shared parameters on the validation set to obtain "the one-shot validation accuracy". It is computationally expensive to evaluate all candidates on the whole validation set. To accelerate this procedure, we evaluate each architecture on a mini-batch with the size of 2048, and use the accuracy on this mini-batch to approximate "the one-shot validation accuracy". Ideally, the architecture ranking sorted by the probability or the one-shot validation accuracy should be similar to the ground truth ranking. We show the correlation between the proxy metric and the ground truth validation accuracy in Table 7. There are several observations: The correlation between the probability (encoded by the architecture encoding) and the ground truth accuracy is low. It suggests that the argmax-based deriving strategy can not secure a good architecture. It remains open on how to derive a good architecture after optimizing the shared parameters. The behavior of BN layers is important to one-shot validation accuracy. The accumulated mean and variance from the training set are harmful to one-shot accuracy. Instead, each architecture candidate should re-calculate the mean and variance of the BN layers. GDAS introduced Gumbel-softmax sampling when optimizing the architecture encoding. This strategy leads to a high correlation for the learned probability than that of DARTS. The uniform sampling strategy for training the shared parameters (a) can increase the correlation for one-shot accuracy compared to the strategy of the joint optimizing strategy (b;). In NAS-Bench-201 (version 1.0), every architecture is trained at least once. To be specific, 6219 architectures are trained once, 1621 architectures are trained twice, 7785 architectures are trained three times with different random seeds. Moreover, we are actively training all architectures with more seeds and will continue updating our NAS-Bench-201.

A NAS benchmark applicable to almost any NAS algorithms.

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Generative Adversarial Networks (GANs) are one of the most popular tools for learning complex high dimensional distributions. However, generalization properties of GANs have not been well understood. In this paper, we analyze the generalization of GANs in practical settings. We show that discriminators trained on discrete datasets with the original GAN loss have poor generalization capability and do not approximate the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization of the discriminator by pushing it toward the optimal discriminator. The penalty guarantees the generalization and convergence of GANs. Experiments on synthetic and large scale datasets verify our theoretical analysis. GANs BID6 are one of the most popular tools for modeling high dimensional data. The original GAN is, however, highly unstable and often suffers from mode collapse. Much of recent researches has focused on improving the stability of GANs BID21 BID8 BID14 BID10. On the theoretical aspect, BID17 proved that gradient based training of the original GAN is locally stable. BID8 further proved that GANs trained with Two Timescale Update Rule (TTUR) converge to local equilibria. However, the generalization of GANs at local equilibria is not discussed in depth in these papers. BID2 showed that the generator can win by remembering a polynomial number of training examples. The implies that a low capacity discriminator cannot detect the lack of diversity. Therefore, it cannot teach the generator to approximate the target distribution. In section 4, we discuss the generalization capability of high capacity discriminators. We show that high capacity discriminators trained with the original GAN loss tends to overfit to the mislabeled samples in training dataset, guiding the generator toward collapsed equilibria (i.e. equilibria where the generator has mode collapse). BID3 proposed to measure the generalization capability of GAN by estimating the number of modes in the model distribution using the birthday paradox. Experiments on several datasets showed that the number of modes in the model distribution is several times greater than the number of training examples. The author concluded that although GANs might not be able to learn distributions, they do exhibit some level of generalization. Our analysis shows that poor generalization comes from the mismatch between discriminators trained on discrete finite datasets and the theoretically optimal discriminator. We propose a zero-centered gradient penalty for improving the generalization capability of (high capacity) discriminators. Our zero-centered gradient penalty pushes the discriminator toward the optimal one, making GAN to converge to equilibrium with good generalization capability. Our contributions are as follow:1. We show that discriminators trained with the original GAN loss have poor generalization capability. Poor generalization in the discriminator prevents the generator from learning the target distribution. TAB0 compares the key properties of our 0-GP with one centered GP (1-GP) BID7 and zero centered GP on real/fake samples only (0-GP-sample) BID13. p r the target distribution p g the model distribution p z the noise distribution d x the dimensionality of a data sample (real or fake) d z the dimensionality of a noise sample supp(p) the support of distribution p x ∼ p r a real sample z ∼ p z a noise vector drawn from the noise distribution p z y = G(z) a generated sample D r = {x 1, ..., x n} the set of n real samples D (t) g = y (t) 1,..., y (t) m the set of m generated samples at step t DISPLAYFORM0 the training dataset at step t Gradient penalties are widely used in GANs literature. There are a plethora of works on using gradient penalty to improve the stability of GANs BID13 BID7 BID19 BID22 BID20. However, these works mostly focused on making the training of GANs stable and convergent. Our work aims to improve the generalization capability of GANs via gradient regularization. BID3 showed that the number of modes in the model distribution grows linearly with the size of the discriminator. The implies that higher capacity discriminators are needed for better approximation of the target distribution. BID3 studied the tradeoff between generalization and discrimination in GANs. The authors showed that generalization is guaranteed if the discriminator set is small enough. In practice, rich discriminators are usually used for better discriminative power. Our GP makes rich discriminators generalizable while remaining discriminative. Although less mode collapse is not exactly the same as generalization, the ability to produce more diverse samples implies better generalization. There are a large number of papers on preventing mode collapse in GANs. BID21; BID23 introduced a number of empirical tricks to help stabilizing GANs. showed the importance of divergences in GAN training, leading to the introduction of Wasserstein GAN. The use of weak divergence is further explored by BID15; BID16. BID12 advocated the use of mixed-batches, mini-batches of real and fake data, GP Formula Improve generalization Convergence guarantee to smooth out the loss surface. The method exploits the distributional information in a mini-batch to prevent mode collapse. VEEGAN BID24 uses an inverse of the generator to map the data to the prior distribution. The mismatch between the inverse mapping and the prior is used to detect mode collapse. If the generator can remember the entire training set, then the inverse mapping can be arbitrarily close the the prior distribution. It suggests that VEEGAN might not be able to help GAN to generalize outside of the training dataset. Our method helps GANs to discover unseen regions of the target distribution, significantly improve the diversity of generated samples. DISPLAYFORM0 In the original GAN, the discriminator D maximizes the following objective BID6 showed that if the density functions p g and p r are known, then for a fixed generator G the optimal discriminator is DISPLAYFORM0 DISPLAYFORM1 In the beginning of the training, p g is very different from p r so we have p r (x) p g (x), for x ∈ D r and p g (y)p r (y), for y ∈ D g. Therefore, in the beginning of the training D * (x) ≈ 1, for x ∈ D r and D * (y) ≈ 0, for y ∈ D g. As the training progresses, the generator will bring p g closer to p r. The game reaches the global equilibrium when p r = p g. At the global equilibrium, DISPLAYFORM2 One important of the original paper is that, if the discriminator is optimal at every step of the GAN algorithm, then p g converges to p r.In practice, density functions are not known and the optimal discriminator is approximated by optimizing the classification performance of a parametric discriminator DISPLAYFORM3 We call a discriminator trained on a discrete finite dataset an empirical discriminator. The empirically optimal discriminator is denoted byD *. BID2 DISPLAYFORM4 A discriminator D defines a divergence between two distributions. The performance of a discriminator with good generalization capability on the training dataset should be similar to that on the entire data space. In practice, generalization capability of D can be estimated by measuring the difference between its performance on the training dataset and a held-out dataset. It has been observed that if the discriminator is too good at discriminating real and fake samples, the generator cannot learn effectively BID6. The phenomenon suggests thatD * does not well approximate D *, and does not guarantee the convergence of p g to p r. In the following, we clarify the mismatch betweenD * and D *, and its implications. g are disjoint with probability 1 even when p g and p r are exactly the same. D * perfectly classifies the real and the fake datasets, andD DISPLAYFORM0 g. The value ofD * on D (t) does not depend on the distance between the two distributions and does not reflect the learning progress. The value ofD * on the training dataset approximates that of D * in the beginning of the learning process but not when the two distributions are close. When trained using gradient descent on a discrete finite dataset with the loss in Eqn. 1, the discriminator D is pushed towardD *, not D *. This behavior does not depend on the size of training set (see FIG0, 1b), implying that the original GAN is not guaranteed to converge to the target distribution even when given enough data. When the generator gets better, generated samples are more similar to samples from the target distribution. However, regardless of their quality, generated samples are still labeled as fake in Eqn. 1. The training dataset D is a bad dataset as it contains many mislabeled examples. A discriminator trained on such dataset will overfit to the mislabeled examples and has poor generalization capability. It will misclassify unseen samples and cannot teach the generator to generate these samples. FIG0 and 1b demonstrate the problem on a synthetic dataset consisting of samples from two Gaussian distributions. The discriminator in FIG0 overfits to the small dataset and does not generalize to new samples in FIG0. Although the discriminator in FIG0 was trained on a larger dataset which is sufficient to characterize the two distributions, it still overfits to the data and its value surface is very different from that of the theoretically optimal discriminator in FIG0.An overfitted discriminator does not guide the model distribution toward target distribution but toward the real samples in the dataset. This explains why the original GAN usually exhibits mode collapse behavior. Finding the empirically optimal discriminator using gradient descent usually requires many iterations. Heuristically, overfitting can be alleviated by limiting the number of discriminator updates per generator update. BID6 recommended to update the discriminator once every generator update. In the next subsection, we show that limiting the number of discriminator updates per generator update prevents the discriminator from overfitting. * is costly to find and maintain. We consider here a weaker notion of optimality which can be achieved in practical settings. Definition 1 (-optimal discriminator). Given two disjoint datasets D r and D g, and a number > 0, a discriminator D is -optimal if DISPLAYFORM0 As observed in BID6,D * does not generate usable gradient for the generator. Goodfellow et al. proposed the non-saturating loss for the generator to circumvent this vanishing gradient problem. For an -optimal discriminator, if is relatively small, then the gradient of the discriminator w.r.t. fake datapoints might not vanish and can be used to guide the model distribution toward the target distribution. Proposition 2. Given two disjoint datasets D r and D g, and a number > 0, an -optimal discriminator D exists and can be constructed as a one hidden layer MLP with O(d x (m + n)) parameters. Proof. See appendix B.Because deep networks are more powerful than shallow ones, the size of a deep -optimal discriminator can be much smaller than O(d x (m + n)). From the formula, the size of a shallow -optimal discriminator for real world datasets ranges from a few to hundreds of millions parameters. That is comparable to the size of discriminators used in practice. showed that even when the generator can generate realistic samples, a discriminator that can perfectly classify real and fake samples can be found easily using gradient descent. The experiment verified that -optimal discriminator can be found using gradient descent in practical settings. We observe that the norm of the gradient w.r.t. the discriminator's parameters decreases as fakes samples approach real samples. If the discriminator's learning rate is fixed, then the number of gradient descent steps that the discriminator has to take to reach -optimal state should increase. Proposition 3. Alternating gradient descent with the same learning rate for discriminator and generator, and fixed number of discriminator updates per generator update (Fixed-Alt-GD) cannot maintain the (empirical) optimality of the discriminator. Fixed-Alt-GD decreases the discriminative power of the discriminator to improve its generalization capability. The proof for linear case is given in appendix C.In GANs trained with Two Timescale Update Rule (TTUR) BID8, the ratio between the learning rate of the discriminator and that of the generator goes to infinity as the iteration number goes to infinity. Therefore, the discriminator can learn much faster than the generator and might be able to maintain its optimality throughout the learning process. Let's consider a simplified scenario where the real and the fake datasets each contains a single datapoint: DISPLAYFORM0. Updating the generator according to the gradient from the discriminator will push y (t) toward x. The absolute value of directional derivative of DISPLAYFORM1 DISPLAYFORM2 The directional derivate of the -optimal discriminator explodes as the fake datapoint approaches the real datapoint. Directional derivative exploding implies gradient exploding at datapoints on the line segment connecting x and y (t). If in the next iteration, the generator produces a sample in a region where the gradient explodes, then the gradient w.r.t. the generator's parameters explodes. Let's consider the following line integral DISPLAYFORM3 where C is the line segment from y (t) to x. As the model distribution gets closer to the target distribution, the length of C should be non increasing. Therefore, maximizing D(x) − D(y (t) ), or the discriminative power of D, leads to the maximization of the directional derivative of D in the direction ds. The original GAN loss makes D to maximize its discriminative power, encouraging gradient exploding to occur. Gradient exploding happens in the discriminator trained with TTUR in FIG2. Because TTUR can help the discriminator to maintain its optimality, gradient exploding happens and persists throughout the training process. Without TTUR, the discriminator cannot maintain its optimality so gradient exploding can happen sometimes during the training but does not persist (FIG2 . Because of the saturated regions in the sigmoid function used in neural network based discriminators, the gradient w.r.t. datapoints in the training set could vanishes. However, gradient exploding must happen at some datapoints on the path between a pair of samples, where the sigmoid function does not saturate. In FIG0, gradient exploding happens near the decision boundary. In practice, D r and D g contain many datapoints and the generator is updated using the average of gradients of the discriminator w.r.t. fake datapoints in the mini-batch. If a fake datapoint y 0 is very close to a real datapoint x 0, the gradient (∇D) y0 might explode. When the average gradient is computed over the mini-batch, (∇D) y0 outweighs other gradients. The generator updated with this average gradient will move many fake datapoints in the direction of (∇D) y0, toward x 0, making mode collapse visible. Although the theoretically optimal discriminator D * is generalizable, the original GAN loss does not push empirical discriminators toward D *. We aim to improve the generalization capability of empirical discriminators by pushing them toward D *. For any input v ∈ supp(p r) ∪ supp(p g), the value of D * (v) goes to 1 2 and the gradient (∇D) v goes to 0 as p g approaches p r. Consider again the line integral in Eqn. 4. As D * (x) and D * (y) approach 1 2 for all x ∈ supp(p r) and y ∈ supp(p g), we have DISPLAYFORM0 for all pairs of x and y and all paths C from y to x. That means, the discriminative power of D * must decrease as the two distributions become more similar. To push an empirical discriminator D toward D *, we force D to satisfy two requirements: DISPLAYFORM1 The first requirement can be implemented by sampling some datapoints v ∈ supp(p r) ∪ supp(p g) and force (∇D) v to be 0. The second requirement can be implemented by sampling pairs of real and fake datapoints (x, y) and force D(x) − D(y) to be 0. The two requirements can be added to the discriminator's objective as followŝ DISPLAYFORM0 where L is the objective in Eqn. 1. However, as discussed in section 4.2.2, an -optimal discriminator can have zero gradient on the training dataset and have gradient exploding outside of the training dataset. The gradient norm could go to infinity even when D(x) − D(y) is small. Regulating the difference between D(x) and D(y) is not an efficient way to prevent gradient exploding. We want to prevent gradient exploding on every path in supp(p r) ∪ supp(p g). Because (∇D *) v → 0 for all v ∈ supp(p r) ∪ supp(p g) as p g approach p r, we could push the gradient w.r.t. every datapoint on every path C ∈ supp(p r) ∪ supp(p g) toward 0. We note that, if (∇D) v → 0, ∀ v ∈ C then C (∇D) v · ds → 0. Therefore, the two requirements can be enforced by a single zero-centered gradient penalty of the form DISPLAYFORM1 The remaining problem is how to find the path C from a fake to a real sample which lies inside supp(p r) ∪ supp(p g). Because we do not have access to the full supports of p r and p g, and the supports of two distributions could be disjoint in the beginning of the training process, finding a path which lies completely inside the support is infeasible. In the current implementation, we approximate C with the straight line connecting a pair of samples, although there is no guarantee that all datapoints on that straight line are in supp(p r) ∪ supp(p g).That in the following objective DISPLAYFORM2 wherex = αx + (1 − α)y, x ∼ p r, y ∼ p g, and α ∼ U 1. We describe a more sophisticated way of finding a better path in appendix F.The larger λ is, the stronger (∇D)x is pushed toward 0. If λ is 0, then the discriminator will only focus on maximizing its discriminative power. If λ approaches infinity, then the discriminator has maximum generalization capability and no discriminative power. λ controls the tradeoff between discrimination and generalization in the discriminator. BID13 proposed to force the gradient w.r.t. datapoints in the real and/or fake dataset(s) to be 0 to make the training of GANs convergent. In section 4, we showed that for discrete training dataset, an empirically optimal discriminatorD * always exists and could be found by gradient descent. Although (∇D *) v = 0, ∀ v ∈ D,D * does not satisfy the requirement in Eqn. 5 and have gradient exploding when some fake datapoints approach a real datapoint. The discriminators in FIG0, 2c and 2d have vanishingly small gradients on datapoints in the training dataset and very large gradients outside. They have poor generalization capability and cannot teach the generator to generate unseen real datapoints. Therefore, zero-centered gradient penalty on samples from p r and p g only cannot help improving the generalization of the discriminator. Non-zero centered GPs do not push an empirical discriminator toward D * because the gradient does not converge to 0. A commonly used non-zero centered GP is the one-centered GP (1-GP) BID7 which has the following form DISPLAYFORM0 wherex = αx + (1 − α)y, x ∼ p r, y ∼ p g, and α ∼ U. Although the initial goal of 1-GP was to enforce Lipschitz constraint on the discriminator 2, BID5 found that 1-GP prevents gradient exploding, making the original GAN more stable. 1-GP forces the norm of gradients w.r.t. datapoints on the line segment connecting x and y to be 1. If all gradients on the line segment have norm 1, then the line integral in Eqn. 4 could be as large as x − y. Because the distance between random samples grows with the dimensionality, in high dimensional space x − y is greater than 1 with high probability. The discriminator could maximize the value of the line integral without violating the Lipschitz constraint. The discriminator trained with 1-GP, therefore, can overfit to the training data and have poor generalization capability. BID13 showed that zero-centered GP on real and/or fake samples (0-GP-sample) makes GANs convergent. The penalty is based on the convergence analysis for the Dirac GAN, an 1-dimensional linear GAN which learns the Dirac distribution. The intuition is that when p g is the same as p r, the gradient of the discriminator w.r.t. the fake datapoints (which are also real datapoints) should be 0 so that generator will not move away when being updated using this gradient. If the gradient from the discriminator is not 0, then the generator will oscillate around the equilibrium. Our GP forces the gradient w.r.t. all datapoints on the line segment between a pair of samples (including the two endpoints) to be 0. As a , our GP also prevents the generator from oscillating. Therefore, our GP has the same convergence guarantee as the 0-GP-sample. Discriminators trained with the original GAN loss tends to focus on the region of the where fake samples are close to real samples, ignoring other regions. The phenomenon can be seen in FIG2, 2c, 2d, 2h and 2i. Gradients in the region where fake samples are concentrated are large while gradients in other regions, including regions where real samples are located, are very small. The generator cannot discover and generate real datapoints in regions where the gradient vanishes. When trained with the objective in Eqn. 6, the discriminator will have to balance between maximizing L and minimizing the GP. For finite λ, the GP term will not be exactly 0. Let DISPLAYFORM0. Among discriminators with the same value of γ, gradient descent will find the discriminator that maximizes L. As discussed in section 4.2.2, maximizing L leads to the maximization of norms of gradients on the path from y to x. The discriminator should maximize the value η = Ex[(∇D)x ]. If γ is fixed then η is maximized when ∇Dx(i) = ∇Dx(j), ∀ i, j (Cauchy-Schwarz inequality). Therefore, our zero-centered GP encourages the gradients at different regions of the real data space to have the same norm. The capacity of D is distributed more equally between regions of the real data space, effectively reduce mode collapse. The effect can be seen in FIG2 1-GP encourages | ∇Dx(i) − 1| = | ∇Dx(j) − 1|, ∀ i, j. That allows gradient norms to be smaller than 1 in some regions and larger than 1 in some other regions. The problem can be seen in FIG2. The code is made available at https://github.com/htt210/ GeneralizationAndStabilityInGANs. To test the effectiveness of gradient penalties in preventing overfitting, we designed a dataset with real and fake samples coming from two Gaussian distributions and trained a MLP based discriminator on that dataset. The is shown in FIG0. As predicted in section 5.3, 0-GP-sample does not help to improve generalization. 1-GP helps to improve generalization. The value surface in FIG0 is smoother than that in FIG0. However, as discussed in section 5.3, 1-GP cannot help much in higher dimensional space where the pair-wise distances are large. The discriminator trained with our 0-GP has the best generalization capability, with a value surface which is the most similar to that of the theoretically optimal one. We increased the number of discriminator updates per generator update to 5 to see the effect of GPs in preventing overfitting. On the MNIST dataset, GAN without GP and with other GPs cannot learn anything after 10,000 iterations. GAN with our 0-GP can still learn normally and start produce recognizable digits after only 1,000 iterations. The confirms that our GP is effective in preventing overfitting in the discriminator. OF GANS SYNTHETIC DATAWe tested different gradient penalties on a number of synthetic datasets to compare their effectiveness. The first dataset is a mixture of 8 Gaussians. The dataset is scaled up by a factor of 10 to simulate the situation in high dimensional space where random samples are far from each other. The is shown in FIG2. GANs with other gradient penalties all fail to learn the distribution and exhibit mode collapse problem to different extents. GAN with our 0-GP (GAN-0-GP) can successfully learn the distribution. Furthermore, GAN-0-GP can generate datapoints on the circle, demonstrating good generalization capability. The original GAN collapses to some disconnected modes and cannot perform smooth interpolation between modes: small change in the input in large, unpredictable change in the output. GAN with zero-centered GP on real/fake samples only also exhibits the same "mode jumping" behavior. The behavior suggests that these GANs tend to remember the training dataset and have poor generalization capability. Fig. 9 in appendix D demonstrates the problem on MNIST dataset. We observe that GAN-0-GP behaves similar to Wasserstein GAN as it first learns the overall structure of the distribution and then focuses on the modes. An evolution sequence of GAN-0-GP is shown in FIG4 in appendix D. Results on other synthetic datasets are shown in appendix D. The on MNIST dataset is shown in Fig. 3. After 1,000 iterations, all other GANs exhibit mode collapse or cannot learn anything. GAN-0-GP is robust to changes in hyper parameters such BID23 on ImageNet of GAN-0-GP, GAN-0-GP-sample, and WGAN-GP. The code for this experiment is adapted from BID13. We used λ = 10 for all GANs as recommended by Mescheder et al. The critic in WGAN-GP was updated 5 times per generator update. To improve convergence, we used TTUR with learning rates of 0.0001 and 0.0003 for the generator and discriminator, respectively. as learning rate and optimizers. When Adam is initialized with large β 1, e.g. 0.9, GANs with other GPs cannot learn anything after many iterations. More samples are given in appendix D. DISPLAYFORM0 We observe that higher value of λ improves the diversity of generated samples. For λ = 50, we observe some similar looking samples in the generated data. This is consistent with our conjecture that larger λ leads to better generalization. When trained on ImangeNet BID4 ), GAN-0-GP can produce high quality samples from all 1,000 classes. We compared our method with GAN with 0-GP-sample and WGAN-GP. GAN-0-GP-sample is able to produce samples of state of the art quality without using progressive growing trick BID10. The in Fig. 4 shows that our method consistently outperforms GAN-0-GP-sample. GAN-0-GP and GAN-0-GP-sample outperform WGAN-GP by a large margin. Image samples are given in appendix D. In this paper, we clarify the reason behind the poor generalization capability of GAN. We show that the original GAN loss does not guide the discriminator and the generator toward a generalizable equilibrium. We propose a zero-centered gradient penalty which pushes empirical discriminators toward the optimal discriminator with good generalization capability. Our gradient penalty provides better generalization and convergence guarantee than other gradient penalties. Experiments on diverse datasets verify that our method significantly improves the generalization and stability of GANs. Pengchuan Zhang, Qiang Liu, Dengyong Zhou, Tao Xu, and Xiaodong He. On the discriminationgeneralization tradeoff in GANs. In International Conference on.A PROOF FOR PROPOSITION 1For continuous random variable V, P(V = v) = 0 for any v. The probability of finding a noise vector z such that G(z) is exactly equal to a real datapoint x ∈ D r via random sampling is 0. Therefore, the probability of a real datapoint x i being in the fake dataset D g is 0. Similarly, the probability of any fake datapoint being in the real dataset is 0. DISPLAYFORM0 Furthermore, due to the curse of dimensionality, the probability of sampling a datapoint which is close to another datapoint in high dimensional space also decrease exponentially. The distances between datapoints are larger in higher dimensional space. That suggests that it is easier to separate D r and D (t) g in higher dimensional space. To make the construction process simpler, let's assume that samples are normalized: DISPLAYFORM0 Let's use the following new notations for real and fake samples: DISPLAYFORM1 We construct the -optimal discriminator D as a MLP with 1 hidden layer. Let W 1 ∈ R (m+n)×dx and W 2 ∈ R m+n be the weight matrices of D. The total number of parameters in DISPLAYFORM2 We set the value of W 1 as DISPLAYFORM3 and W 2 as DISPLAYFORM4 Given an input v ∈ D, the output is computed as: DISPLAYFORM5 where σ is the softmax function. Let a = W 1 v, we have DISPLAYFORM6 As k → ∞, σ(W 1 v i) becomes a one-hot vector with the i-th element being 1, all other elements being 0. Thus, for large enough k, for any v j ∈ D, the output of the network is DISPLAYFORM7 C FIXED-ALT-GD CANNOT MAINTAIN THE OPTIMALITY OF -DISCRIMINATORS Let's consider the case where the real and the fake dataset each contain a single datapoint D r = {x}, D DISPLAYFORM8, and the discriminator and the generator are linear: DISPLAYFORM9 and the objective is also linear (Wasserstein GAN's objective): DISPLAYFORM10 The same learning rate α is used for D and G.At step t, the discriminator is -optimal DISPLAYFORM11 The gradients w.r.t. θ D and θ G are DISPLAYFORM12 DISPLAYFORM13 If the learning rate α is small enough, x − y (t) should decrease as t increases. As the empirical fake distribution converges to the empirical real distribution, x − y (t) → 0. The norm of gradient w.r.t. θ D, therefore, decreases as t increases and vanishes when the two empirical distributions are the same. From Eqn. 10, we see that, in order to maintain D's -optimality when x − y (t) decreases, θ D has to increase. From Eqn. 10 and 12, we see that the gradient w.r.t. θ G grows as the two empirical distributions are more similar. As x − y (t) → 0, DISPLAYFORM14 Because the same learning rate α is used for both G and D, G will learn much faster than D. Furthermore, because x − y (t) decreases as t increases, the difference DISPLAYFORM15 increases with t. The number of gradient steps that D has to take to reach the next -optimal state increases, and goes to infinity as x − y (t) → 0. Therefore, gradient descent with fixed number of updates to θ D cannot maintain the optimality of D.The derivation for the objective in Eqn. 1 is similar.: GANs trained with different gradient penalty on swissroll dataset. Although GAN-1-GP is able to learn the distribution, the gradient field has bad pattern. GAN-1-GP is more sensitive to change in hyper parameters and optimizers. GAN-1-GP fails to learn the scaled up version of the distribution. The entire ImageNet dataset with all 1000 classes was used in the experiment. Because of our hardware limits, we used images of size 64 × 64. We used the code from BID13, available at https://github.com/LMescheder/GAN stability, for our experiment. Generator and Discriminator are ResNets, each contains 5 residual blocks. All GANs in our experiment have the same architectures and hyper parameters. The configuration for WGAN-GP5 is as follows.

We propose a zero-centered gradient penalty for improving generalization and stability of GANs

represent the short summary for finding the corresponding scientific article (long).

We show that posterior collapse in linear VAEs is caused entirely by marginal log-likelihood (not ELBO). Experiments on deep VAEs suggest a similar phenomenon is at play.