Question: Represent the article for finding a single-sentence TLDR

Question: Class labels have been empirically shown useful in improving the sample quality of generative adversarial nets (GANs). In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information. With class aware gradient and cross-entropy decomposition, we reveal how class labels and associated losses influence GAN's training. Based on that, we propose Activation Maximization Generative Adversarial Networks (AM-GAN) as an advanced solution. Comprehensive experiments have been conducted to validate our analysis and evaluate the effectiveness of our solution, where AM-GAN outperforms other strong baselines and achieves state-of-the-art Inception Score (8.91) on CIFAR-10. In addition, we demonstrate that, with the Inception ImageNet classifier, Inception Score mainly tracks the diversity of the generator, and there is, however, no reliable evidence that it can reflect the true sample quality. We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality. Our proposed model also outperforms the baseline methods in the new metric. Generative adversarial nets (GANs) BID7 as a new way for learning generative models, has recently shown promising in various challenging tasks, such as realistic image generation BID17 BID26 BID9, conditional image generation BID12 BID2 BID13, image manipulation ) and text generation BID25.Despite the great success, it is still challenging for the current GAN models to produce convincing samples when trained on datasets with high variability, even for image generation with low resolution, e.g., CIFAR-10. Meanwhile, people have empirically found taking advantages of class labels can significantly improve the sample quality. There are three typical GAN models that make use of the label information: CatGAN BID20 builds the discriminator as a multi-class classifier; LabelGAN BID19 extends the discriminator with one extra class for the generated samples; AC-GAN BID18 jointly trains the real-fake discriminator and an auxiliary classifier for the specific real classes. By taking the class labels into account, these GAN models show improved generation quality and stability. However, the mechanisms behind them have not been fully explored BID6.In this paper, we mathematically study GAN models with the consideration of class labels. We derive the gradient of the generator's loss w.r.t. class logits in the discriminator, named as class-aware gradient, for LabelGAN BID19 and further show its gradient tends to guide each generated sample towards being one of the specific real classes. Moreover, we show that AC-GAN BID18 can be viewed as a GAN model with hierarchical class discriminator. Based on the analysis, we reveal some potential issues in the previous methods and accordingly propose a new method to resolve these issues. Specifically, we argue that a model with explicit target class would provide clearer gradient guidance to the generator than an implicit target class model like that in BID19. Comparing with BID18, we show that introducing the specific real class logits by replacing the overall real class logit in the discriminator usually works better than simply training an auxiliary classifier. We argue that, in BID18, adversarial training is missing in the auxiliary classifier, which would make the model more likely to suffer mode collapse and produce low quality samples. We also experimentally find that predefined label tends to in intra-class mode collapse and correspondingly propose dynamic labeling as a solution. The proposed model is named as Activation Maximization Generative Adversarial Networks (AM-GAN). We empirically study the effectiveness of AM-GAN with a set of controlled experiments and the are consistent with our analysis and, note that, AM-GAN achieves the state-of-the-art Inception Score (8.91) on CIFAR-10.In addition, through the experiments, we find the commonly used metric needs further investigation. In our paper, we conduct a further study on the widely-used evaluation metric Inception Score BID19 and its extended metrics. We show that, with the Inception Model, Inception Score mainly tracks the diversity of generator, while there is no reliable evidence that it can measure the true sample quality. We thus propose a new metric, called AM Score, to provide more accurate estimation on the sample quality as its compensation. In terms of AM Score, our proposed method also outperforms other strong baseline methods. The rest of this paper is organized as follows. In Section 2, we introduce the notations and formulate the LabelGAN BID19 and AC-GAN * BID18 ) as our baselines. We then derive the class-aware gradient for LabelGAN, in Section 3, to reveal how class labels help its training. In Section 4, we reveal the overlaid-gradient problem of LabelGAN and propose AM-GAN as a new solution, where we also analyze the properties of AM-GAN and build its connections to related work. In Section 5, we introduce several important extensions, including the dynamic labeling as an alternative of predefined labeling (i.e., class condition), the activation maximization view and a technique for enhancing the AC-GAN *. We study Inception Score in Section 6 and accordingly propose a new metric AM Score. In Section 7, we empirically study AM-GAN and compare it to the baseline models with different metrics. Finally we conclude the paper and discuss the future work in Section 8. In the original GAN formulation BID7, the loss functions of the generator G and the discriminator D are given as: DISPLAYFORM0 where D performs binary classification between the real and the generated samples and D r (x) represents the probability of the sample x coming from the real data. The framework (see Eq.) has been generalized to multi-class case where each sample x has its associated class label y ∈ {1, . . ., K, K+1}, and the K+1 th label corresponds to the generated samples BID19. Its loss functions are defined as: DISPLAYFORM0 DISPLAYFORM1 where D i (x) denotes the probability of the sample x being class i. The loss can be written in the form of cross-entropy, which will simplify our later analysis: DISPLAYFORM2 DISPLAYFORM3 where DISPLAYFORM4 H is the cross-entropy, defined as H(p, q)=− i p i log q i. We would refer the above model as LabelGAN (using class labels) throughout this paper. Besides extending the original two-class discriminator as discussed in the above section, BID18 proposed an alternative approach, i.e., AC-GAN, to incorporate class label information, which introduces an auxiliary classifier C for real classes in the original GAN framework. With the core idea unchanged, we define a variant of AC-GAN as the following, and refer it as AC-GAN *: DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 where D r (x) and D f (x) = 1 − D r (x) are outputs of the binary discriminator which are the same as vanilla GAN, u(·) is the vectorizing operator that is similar to v(·) but defined with K classes, and C(x) is the probability distribution over K real classes given by the auxiliary classifier. In AC-GAN, each sample has a coupled target class y, and a loss on the auxiliary classifier w.r.t. y is added to the generator to leverage the class label information. We refer the losses on the auxiliary classifier, i.e., Eq. FORMULA7 and FORMULA9, as the auxiliary classifier losses. The above formulation is a modified version of the original AC-GAN. Specifically, we omit the auxiliary classifier loss E (x,y)∼G [H(u(y), C(x))] which encourages the auxiliary classifier C to classify the fake sample x to its target class y. Further discussions are provided in Section 5.3. Note that we also adopt the − log(D r (x)) loss in generator. In this section, we introduce the class-aware gradient, i.e., the gradient of the generator's loss w.r.t. class logits in the discriminator. By analyzing the class-aware gradient of LabelGAN, we find that the gradient tends to refine each sample towards being one of the classes, which sheds some light on how the class label information helps the generator to improve the generation quality. Before delving into the details, we first introduce the following lemma on the gradient properties of the cross-entropy loss to make our analysis clearer. Lemma 1. With l being the logits vector and σ being the softmax function, let σ(l) be the current softmax probability distribution andp denote the target probability distribution, then DISPLAYFORM0 For a generated sample x, the loss in LabelGAN is L lab DISPLAYFORM1, as defined in Eq.. With Lemma 1, the gradient of L lab G (x) w.r.t. the logits vector l(x) is given as: DISPLAYFORM2 With the above equations, the gradient of L lab G (x) w.r.t. x is: DISPLAYFORM3 where Figure 1: An illustration of the overlaid-gradient problem. When two or more classes are encouraged at the same time, the combined gradient may direct to none of these classes. It could be addressed by assigning each generated sample a specific target class instead of the overall real class. DISPLAYFORM4 From the formulation, we find that the overall gradient w.r.t. a generated example x is 1−D r (x), which is the same as that in vanilla GAN BID7. And the gradient on real classes is further distributed to each specific real class logit l k (x) according to its current probability ratio DISPLAYFORM5 Dr(x). As such, the gradient naturally takes the label information into consideration: for a generated sample, higher probability of a certain class will lead to a larger step towards the direction of increasing the corresponding confidence for the class. Hence, individually, the gradient from the discriminator for each sample tends to refine it towards being one of the classes in a probabilistic sense. That is, each sample in LabelGAN is optimized to be one of the real classes, rather than simply to be real as in the vanilla GAN. We thus regard LabelGAN as an implicit target class model. Refining each generated sample towards one of the specific classes would help improve the sample quality. Recall that there are similar inspirations in related work. BID4 showed that the could be significantly better if GAN is trained with separated classes. And AC-GAN BID18 introduces an extra loss that forces each sample to fit one class and achieves a better . In LabelGAN, the generator gets its gradients from the K specific real class logits in discriminator and tends to refine each sample towards being one of the classes. However, LabelGAN actually suffers from the overlaid-gradient problem: all real class logits are encouraged at the same time. Though it tends to make each sample be one of these classes during the training, the gradient of each sample is a weighted averaging over multiple label predictors. As illustrated in Figure 1, the averaged gradient may be towards none of these classes. In multi-exclusive classes setting, each valid sample should only be classified to one of classes by the discriminator with high confidence. One way to resolve the above problem is to explicitly assign each generated sample a single specific class as its target. Assigning each sample a specific target class y, the loss functions of the revised-version LabelGAN can be formulated as: DISPLAYFORM0 DISPLAYFORM1 where v(y) is with the same definition as in Section 2.1. The model with aforementioned formulation is named as Activation Maximization Generative Adversarial Networks (AM-GAN) in our paper. And the further interpretation towards naming will be in Section 5.2. The only difference between AM-GAN and LabelGAN lies in the generator's loss function. Each sample in AM-GAN has a specific target class, which resolves the overlaid-gradient problem. AC-GAN BID18 ) also assigns each sample a specific target class, but we will show that the AM-GAN and AC-GAN are substantially different in the following part of this section. * is a combination of vanilla GAN and auxiliary classifier. AM-GAN can naturally conduct adversarial training among all the classes, while in AC-GAN *, adversarial training is only conducted at the real-fake level and missing in the auxiliary classifier. Both LabelGAN and AM-GAN are GAN models with K+1 classes. We introduce the following cross-entropy decomposition lemma to build their connections to GAN models with two classes and the K-classes models (i.e., the auxiliary classifiers). DISPLAYFORM0 With Lemma 2, the loss function of the generator in AM-GAN can be decomposed as follows: DISPLAYFORM1 The second term of Eq. actually equals to the loss function of the generator in LabelGAN: DISPLAYFORM2 Similar analysis can be adapted to the first term and the discriminator. Note that v r (x) equals to one. Interestingly, we find by decomposing the AM-GAN losses, AM-GAN can be viewed as a combination of LabelGAN and auxiliary classifier (defined in Section 2.2). From the decomposition perspective, disparate to AM-GAN, AC-GAN is a combination of vanilla GAN and the auxiliary classifier. The auxiliary classifier loss in Eq. FORMULA0 can also be viewed as the cross-entropy version of generator loss in CatGAN: the generator of CatGAN directly optimizes entropy H(R(D(x))) to make each sample have a high confidence of being one of the classes, while AM-GAN achieves this by the first term of its decomposed loss H(R(v(x)), R(D(x))) in terms of cross-entropy with given target distribution. That is, the AM-GAN is the combination of the cross-entropy version of CatGAN and LabelGAN. We extend the discussion between AM-GAN and CatGAN in the Appendix B. With the Lemma 2, we can also reformulate the AC-GAN * as a K+1 classes model. Take the generator's loss function as an example: DISPLAYFORM0 In the K+1 classes model, the K+1 classes distribution is formulated as DISPLAYFORM1 AC-GAN introduces the auxiliary classifier in the consideration of leveraging the side information of class label, it turns out that the formulation of AC-GAN * can be viewed as a hierarchical K+1 classes model consists of a two-class discriminator and a K-class auxiliary classifier, as illustrated in FIG1. Conversely, AM-GAN is a non-hierarchical model. All K+1 classes stay in the same level of the discriminator in AM-GAN.In the hierarchical model AC-GAN *, adversarial training is only conducted at the real-fake twoclass level, while misses in the auxiliary classifier. Adversarial training is the key to the theoretical guarantee of global convergence p G = p data. Taking the original GAN formulation as an instance, if generated samples collapse to a certain point x, i.e., p G (x) > p data (x), then there must exit another point x with p G (x) < p data (x). Given the optimal D(x) = pdata (x) pG(x)+pdata(x), the collapsed point x will get a relatively lower score. And with the existence of higher score points (e.g. x), maximizing the generator's expected score, in theory, has the strength to recover from the mode-collapsed state. In practice, the p G and p data are usually disjoint, nevertheless, the general behaviors stay the same: when samples collapse to a certain point, they are more likely to get a relatively lower score from the adversarial network. Without adversarial training in the auxiliary classifier, a mode-collapsed generator would not get any penalties from the auxiliary classifier loss. In our experiments, we find AC-GAN is more likely to get mode-collapsed, and it was empirically found reducing the weight (such as 0.1 used in BID9) of the auxiliary classifier losses would help. In Section 5.3, we introduce an extra adversarial training in the auxiliary classifier with which we improve AC-GAN *'s training stability and sample-quality in experiments. On the contrary, AM-GAN, as a non-hierarchical model, can naturally conduct adversarial training among all the class logits. In the above section, we simply assume each generated sample has a target class. One possible solution is like AC-GAN BID18, predefining each sample a class label, which substantially in a conditional GAN. Actually, we could assign each sample a target class according to its current probability estimated by the discriminator. A natural choice could be the class which is of the maximal probability currently: y(x) argmax i∈{1,...,K} D i (x) for each generated sample x. We name this dynamic labeling. According to our experiments, dynamic labeling brings important improvements to AM-GAN, and is applicable to other models that require target class for each generated sample, e.g. AC-GAN, as an alternative to predefined labeling. We experimentally find GAN models with pre-assigned class label tend to encounter intra-class mode collapse. In addition, with dynamic labeling, the GAN model remains generating from pure random noises, which has potential benefits, e.g. making smooth interpolation across classes in the latent space practicable. Activation maximization is a technique which is traditionally applied to visualize the neuron(s) of pretrained neural networks BID16 b; BID5 ).The GAN training can be viewed as an Adversarial Activation Maximization Process. To be more specific, the generator is trained to perform activation maximization for each generated sample on the neuron that represents the log probability of its target class, while the discriminator is trained to distinguish generated samples and prevents them from getting their desired high activation. It is worth mentioning that the sample that maximizes the activation of one neuron is not necessarily of high quality. Traditionally people introduce various priors to counter the phenomenon BID16 b). In GAN, the adversarial process of GAN training can detect unrealistic samples and thus ensures the high-activation is achieved by high-quality samples that strongly confuse the discriminator. We thus name our model the Activation Maximization Generative Adversarial Network (AM-GAN). Experimentally we find AC-GAN easily get mode collapsed and a relatively low weight for the auxiliary classifier term in the generator's loss function would help. In the Section 4.3, we attribute mode collapse to the miss of adversarial training in the auxiliary classifier. From the adversarial activation maximization view: without adversarial training, the auxiliary classifier loss that requires high activation on a certain class, cannot ensure the sample quality. That is, in AC-GAN, the vanilla GAN loss plays the role for ensuring sample quality and avoiding mode collapse. Here we introduce an extra loss to the auxiliary classifier in AC-GAN * to enforce adversarial training and experimentally find it consistently improve the performance: DISPLAYFORM0 where u(·) represents the uniform distribution, which in spirit is the same as CatGAN BID20.Recall that we omit the auxiliary classifier loss E (x,y)∼G H u(y)] in AC-GAN *. According to our experiments, E (x,y)∼G [H(u(y)] does improve AC-GAN *'s stability and make it less likely to get mode collapse, but it also leads to a worse Inception Score. We will report the detailed in Section 7. Our understanding on this phenomenon is that: by encouraging the auxiliary classifier also to classify fake samples to their target classes, it actually reduces the auxiliary classifier's ability on providing gradient guidance towards the real classes, and thus also alleviates the conflict between the GAN loss and the auxiliary classifier loss. One of the difficulties in generative models is the evaluation methodology BID22. In this section, we conduct both the mathematical and the empirical analysis on the widely-used evaluation metric Inception Score BID19 and other relevant metrics. We will show that Inception Score mainly works as a diversity measurement and we propose the AM Score as a compensation to Inception Score for estimating the generated sample quality. As a recently proposed metric for evaluating the performance of generative models, Inception Score has been found well correlated with human evaluation BID19, where a publiclyavailable Inception model C pre-trained on ImageNet is introduced. By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C(x), Inception Score is calculated via DISPLAYFORM0 where E x is short of E x∼G andC DISPLAYFORM1 is the overall probability distribution of the generated samples over classes, which is judged by C, and KL denotes the Kullback-Leibler divergence. As proved in Appendix D, E x KL C(x) C G can be decomposed into two terms in entropy: DISPLAYFORM2 A common understanding of how Inception Score works lies in that a high score in the first term H(C G) indicates the generated samples have high diversity (the overall class probability distribution evenly distributed), and a high score in the second term −E x [H(C(x))] indicates that each individual sample has high quality (each generated sample's class probability distribution is sharp, i.e., it can be classified into one of the real classes with high confidence) BID19.However, taking CIFAR-10 as an illustration, the data are not evenly distributed over the classes under the Inception model trained on ImageNet, which is presented in FIG3. It makes Inception Score problematic in the view of the decomposed scores, i.e., H(C G) and −E x [H(C(x))]. Such as that one would ask whether a higher H(C G) indicates a better mode coverage and whether a smaller H(C(x)) indicates a better sample quality. DISPLAYFORM0. A common understanding of Inception Score is that: the value of H(C G) measures the diversity of generated samples and is expected to increase in the training process. However, it usually tends to decrease in practice as illustrated in (c). (C(x) ) score of CIFAR-10 training data is variant, which means, even in real data, it would still strongly prefer some samples than some others. H(C(x)) on a classifier that pre-trained on CIFAR-10 has low values for all CIFAR-10 training data and thus can be used as an indicator of sample quality. We experimentally find that, as in FIG2, the value of H(C G) is usually going down during the training process, however, which is expected to increase. And when we delve into the detail of H(C(x)) for each specific sample in the training data, we find the value of H(C(x)) score is also variant, as illustrated in FIG3, which means, even in real data, it would still strongly prefer some samples than some others. The exp operator in Inception Score and the large variance of the value of H(C(x)) aggravate the phenomenon. We also observe the preference on the class level in FIG3, e.g., E x [H(C(x))]=2.14 for trucks, while E x [H(C(x))]=3.80 for birds. It seems, for an ImageNet Classifier, both the two indicators of Inception Score cannot work correctly. Next we will show that Inception Score actually works as a diversity measurement. Since the two individual indicators are strongly correlated, here we go back to Inception Score's original formulation E x [KL(C(x) C G )]. In this form, we could interpret Inception Score as that it requires each sample's distribution C(x) highly different from the overall distribution of the generator C G, which indicates a good diversity over the generated samples. As is empirically observed, a mode-collapsed generator usually gets a low Inception Score. In an extreme case, assuming all the generated samples collapse to a single point, then C(x)=C G and we would get the minimal Inception Score 1.0, which is the exp of zero. To simulate mode collapse in a more complicated case, we design synthetic experiments as following: given a set of N points {x 0, x 1, x 2, ..., x N −1}, with each point x i adopting the distribution C(x i) = v(i) and representing class i, where v(i) is the vectorization operator of length N, as defined in Section 2.1, we randomly drop m points, evaluate E x [KL(C(x) C G )] and draw the curve. As is showed in Figure 5, when N − m increases, the value of E x [KL(C(x) C G )] monotonically increases in general, which means that it can well capture the mode dropping and the diversity of the generated distributions. DISPLAYFORM0 All of them works properly (going down) in the training process. One remaining question is that whether good mode coverage and sample diversity mean high quality of the generated samples. From the above analysis, we do not find any evidence. A possible explanation is that, in practice, sample diversity is usually well correlated with the sample quality. 1 is generated, cannot be detected by E x [KL(C(x) C G )] score. It means that with an accordingly pretrained classifier, E x [KL(C(x) C G )] score cannot detect intra-class level mode collapse. This also explains why the Inception Network on ImageNet could be a good candidate C for CIFAR-10. Exploring the optimal C is a challenge problem and we shall leave it as a future work. However, there is no evidence that using an Inception Network trained on ImageNet can accurately measure the sample quality, as shown in Section 6.2. To compensate Inception Score, we propose to introduce an extra assessment using an accordingly pretrained classifier. In the accordingly pretrained classifier, most real samples share similar H(C(x)) and 99.6% samples hold scores less than 0.05 as showed in FIG3, which demonstrates that H(C(x)) of the classifier can be used as an indicator of sample quality. G is actually problematic when training data is not evenly distributed over classes, for that argmin H(C G) is a uniform distribution. To take theC train into account, we replace H(C G) with a KL divergence betweenC train andC G. So that DISPLAYFORM0 which requiresC G close toC train and each sample x has a low entropy C(x). The minimal value of AM Score is zero, and the smaller value, the better. A sample training curve of AM Score is showed in Figure 6, where all indicators in AM Score work as expected. 1 1 Inception Score and AM Score measure the diversity and quality of generated samples, while FID BID10 measures the distance between the generated distribution and the real distribution. Inception Score AM Score TAB0 7.04 ± 0.06 7.27 ± 0.07 --0.45 ± 0.00 0.43 ± 0.00 --GAN * 7.25 ± 0.07 7.31 ± 0.10 --0.40 ± 0.00 0.41 ± 0.00 --AC-GAN * 7.41 ± 0.09 7.79 ± 0.08 7.28 ± 0.07 7.89 ± 0.11 0.17 ± 0.00 0.16 ± 0.00 1.64 ± 0.02 1.01 ± 0.01 AC-GAN * + 8.56 ± 0.11 8.01 ± 0.09 10.25 ± 0.14 8.23 ± 0.10 0.10 ± 0.00 0.14 ± 0.00 1.04 ± 0.01 1.20 ± 0.01 LabelGAN 8.63 ± 0.08 7.88 ± 0.07 10.82 ± 0.16 8.62 ± 0.11 0.13 ± 0.00 0.25 ± 0.00 1.11 ± 0.01 1.37 ± 0.01 AM-GAN 8.83 ± 0.09 8.35 ± 0.12 11.45 ± 0.15 9.55 ± 0.11 0.08 ± 0.00 0.05 ± 0.00 0.88 ± 0.01 0.61 ± 0.01 To empirically validate our analysis and the effectiveness of the proposed method, we conduct experiments on the image benchmark datasets including CIFAR-10 and Tiny-ImageNet 2 which comprises 200 classes with 500 training images per class. For evaluation, several metrics are used throughout our experiments, including Inception Score with the ImageNet classifier, AM Score with a corresponding pretrained classifier for each dataset, which is a DenseNet BID11 model. We also follow BID18 and use the mean MS-SSIM BID23 of randomly chosen pairs of images within a given class, as a coarse detector of intra-class mode collapse. A modified DCGAN structure, as listed in the Appendix F, is used in experiments. Visual of various models are provided in the Appendix considering the page limit, such as Figure 9, etc. The repeatable experiment code is published for further research 3. The first question is whether training an auxiliary classifier without introducing correlated losses to the generator would help improve the sample quality. In other words, with the generator only with the GAN loss in the AC-GAN * setting. (referring as GAN *)As is shown in TAB0, it improves GAN's sample quality, but the improvement is limited comparing to the other methods. It indicates that introduction of correlated loss plays an essential role in the remarkable improvement of GAN training. The usage of the predefined label would make the GAN model transform to its conditional version, which is substantially disparate with generating samples from pure random noises. In this experiment, we use dynamic labeling for AC-GAN *, AC-GAN * + and AM-GAN to seek for a fair comparison among different discriminator models, including LabelGAN and GAN. We keep the network structure and hyper-parameters the same for different models, only difference lies in the output layer of the discriminator, i.e., the number of class logits, which is necessarily different across models. As is shown in TAB0, AC-GAN * achieves improved sample quality over vanilla GAN, but sustains mode collapse indicated by the value 0.61 in MS-SSIM as in TAB1. By introducing adversarial Model Score ± Std. DFM BID24 7.72 ± 0.13 Improved GAN BID19 8.09 ± 0.07 AC-GAN BID18 8.25 ± 0.07 WGAN-GP + AC BID9 8.42 ± 0.10 SGAN BID12 8.59 ± 0.12 AM-GAN (our work)8.91 ± 0.11 Splitting GAN BID8 8.87 ± 0.09 Real data 11.24 ± 0.12 Table 3: Inception Score comparison on CIFAR-10. Splitting GAN uses the class splitting technique to enhance the class label information, which is orthogonal to AM-GAN.training in the auxiliary classifier, AC-GAN * + outperforms AC-GAN *. As an implicit target class model, LabelGAN suffers from the overlaid-gradient problem and achieves a relatively higher per sample entropy (0.124) in the AM Score, comparing to explicit target class model AM-GAN (0.079) and AC-GAN * + (0.102). In the table, our proposed AM-GAN model reaches the best scores against these baselines. We also test AC-GAN * with decreased weight on auxiliary classifier losses in the generator FORMULA0 relative to the GAN loss). It achieves 7.19 in Inception Score, 0.23 in AM Score and 0.35 in MS-SSIM. The 0.35 in MS-SSIM indicates there is no obvious mode collapse, which also conform with our above analysis. AM-GAN achieves Inception Score 8.83 in the previous experiments, which significantly outperforms the baseline models in both our implementation and their reported scores as in Table 3. By further enhancing the discriminator with more filters in each layer, AM-GAN also outperforms the orthogonal work BID8 ) that enhances the class label information via class splitting. As the , AM-GAN achieves the state-of-the-art Inception Score 8.91 on CIFAR-10. It's found in our experiments that GAN models with class condition (predefined labeling) tend to encounter intra-class mode collapse (ignoring the noise), which is obvious at the very beginning of GAN training and gets exasperated during the process. In the training process of GAN, it is important to ensure a balance between the generator and the discriminator. With the same generator's network structures and switching from dynamic labeling to class condition, we find it hard to hold a good balance between the generator and the discriminator: to avoid the initial intra-class mode collapse, the discriminator need to be very powerful; however, it usually turns out the discriminator is too powerful to provide suitable gradients for the generator and in poor sample quality. Nevertheless, we find a suitable discriminator and conduct a set of comparisons with it. The can be found in TAB0. The general is similar to the above, AC-GAN * + still outperforms AC-GAN * and our AM-GAN reaches the best performance. It's worth noticing that the AC-GAN * does not suffer from mode collapse in this setting. In the class conditional version, although with fine-tuned parameters, Inception Score is still relatively low. The explanation could be that, in the class conditional version, the sample diversity still tends to decrease, even with a relatively powerful discriminator. With slight intra-class mode collapse, the per-sample-quality tends to improve, which in a lower AM Score. A supplementary evidence, not very strict, of partial mode collapse in the experiments is that: the | ∂z | is around 45.0 in dynamic labeling setting, while it is 25.0 in the conditional version. The LabelGAN does not need explicit labels and the model is the same in the two experiment settings. But please note that both Inception Score and the AM Score get worse in the conditional version. The only difference is that the discriminator becomes more powerful with an extended layer, which attests that the balance between the generator and discriminator is crucial. We find that, without the concern of intra-class mode collapse, using the dynamic labeling makes the balance between generator and discriminator much easier. DISPLAYFORM0 Note that we report of the modified version of AC-GAN, i.e., AC-GAN * in TAB0. If we take the omitted loss E (x,y)∼G [H(u(y), C(x))] back to AC-GAN *, which leads to the original AC-GAN (see Section 2.2), it turns out to achieve worse on both Inception Score and AM Score on CIFAR-10, though dismisses mode collapse. Specifically, in dynamic labeling setting, Inception Score decreases from 7.41 to 6.48 and the AM Score increases from 0.17 to 0.43, while in predefined class setting, Inception Score decreases from 7.79 to 7.66 and the AM Score increases from 0.16 to 0.20.This performance drop might be because we use different network architectures and hyper-parameters from AC-GAN BID18. But we still fail to achieve its report Inception Score, i.e., 8.25, on CIFAR-10 when using the reported hyper-parameters in the original paper. Since they do not publicize the code, we suppose there might be some unreported details that in the performance gap. We would leave further studies in future work. We plot the training curve in terms of Inception Score and AM Score in FIG6. Inception Score and AM Score are evaluated with the same number of samples 50k, which is the same as BID19. Comparing with Inception Score, AM Score is more stable in general. With more samples, Inception Score would be more stable, however the evaluation of Inception Score is relatively costly. A better alternative of the Inception Model could help solve this problem. The AC-GAN *'s curves appear stronger jitter relative to the others. It might relate to the counteract between the auxiliary classifier loss and the GAN loss in the generator. Another observation is that the AM-GAN in terms of Inception Score is comparable with LabelGAN and AC-GAN * + at the beginning, while in terms of AM Score, they are quite distinguishable from each other. In the CIFAR-10 experiments, the are consistent with our analysis and the proposed method outperforms these strong baselines. We demonstrate that the can be generalized with experiments in another dataset Tiny-ImageNet. The Tiny-ImageNet consists with more classes and fewer samples for each class than CIFAR-10, which should be more challenging. We downsize Tiny-ImageNet samples from 64×64 to 32×32 and simply leverage the same network structure that used in CIFAR-10, and the experiment is showed also in TAB0. From the comparison, AM-GAN still outperforms other methods remarkably. And the AC-GAN * + gains better performance than AC-GAN *. In this paper, we analyze current GAN models that incorporate class label information. Our analysis shows that: LabelGAN works as an implicit target class model, however it suffers from the overlaidgradient problem at the meantime, and explicit target class would solve this problem. We demonstrate that introducing the class logits in a non-hierarchical way, i.e., replacing the overall real class logit in the discriminator with the specific real class logits, usually works better than simply supplementing an auxiliary classifier, where we provide an activation maximization view for GAN training and highlight the importance of adversarial training. In addition, according to our experiments, predefined labeling tends to lead to intra-class mode collapsed, and we propose dynamic labeling as an alternative. Our extensive experiments on benchmarking datasets validate our analysis and demonstrate our proposed AM-GAN's superior performance against strong baselines. Moreover, we delve deep into the widelyused evaluation metric Inception Score, reveal that it mainly works as a diversity measurement. And we also propose AM Score as a compensation to more accurately estimate the sample quality. In this paper, we focus on the generator and its sample quality, while some related work focuses on the discriminator and semi-supervised learning. For future work, we would like to conduct empirical studies on discriminator learning and semi-supervised learning. We extend AM-GAN to unlabeled data in the Appendix C, where unsupervised and semi-supervised is accessible in the framework of AM-GAN. The classifier-based evaluation metric might encounter the problem related to adversarial samples, which requires further study. Combining AM-GAN with Integral Probability Metric based GAN models such as Wasserstein GAN could also be a promising direction since it is orthogonal to our work. DISPLAYFORM0 Label smoothing that avoiding extreme logits value was showed to be a good regularization BID21. A general version of label smoothing could be: modifying the target probability of discriminator) BID19 proposed to use only one-side label smoothing. That is, to only apply label smoothing for real samples: λ 1 = 0 and λ 2 > 0. The reasoning of one-side label smoothing is applying label smoothing on fake samples will lead to fake mode on data distribution, which is too obscure. DISPLAYFORM1 We will next show the exact problems when applying label smoothing to fake samples along with the log(1−D r (x)) generator loss, in the view of gradient w.r.t. class logit, i.e., the class-aware gradient, and we will also show that the problem does not exist when using the − log(D r (x)) generator loss. DISPLAYFORM2 The log(1−D r (x)) generator loss with label smoothing in terms of cross-entropy is DISPLAYFORM3 with lemma 1, its negative gradient is DISPLAYFORM4 DISPLAYFORM5 Gradient vanishing is a well know training problem of GAN. Optimizing D r (x) towards 0 or 1 is also not what desired, because the discriminator is mapping real samples to the distribution with DISPLAYFORM6 The − log(D r (x)) generator loss with target [1−λ, λ] in terms of cross-entropy is DISPLAYFORM7 the negative gradient of which is DISPLAYFORM8 DISPLAYFORM9 Without label smooth λ, the − log(D r (x)) always * preserves the same gradient direction as log(1−D r (x)) though giving a difference gradient scale. We must note that non-zero gradient does not mean that the gradient is efficient or valid. The both-side label smoothed version has a strong connection to Least-Square GAN BID15: with the fake logit fixed to zero, the discriminator maps real to α on the real logit and maps fake to β on the real logit, the generator in contrast tries to map fake sample to α. Their gradient on the logit are also similar. The auxiliary classifier loss of AM-GAN can also be viewed as the cross-entropy version of CatGAN: generator of CatGAN directly optimizes entropy H(R(D(x))) to make each sample be one class, while AM-GAN achieves this by the first term of its decomposed loss H(R(v(x)), R(D(x))) in terms of cross-entropy with given target distribution. That is, the AM-GAN is the cross-entropy version of CatGAN that is combined with LabelGAN by introducing an additional fake class. The discriminator of CatGAN maximizes the prediction entropy of each fake sample: DISPLAYFORM0 In AM-GAN, as we have an extra class on fake, we can achieve this in a simpler manner by minimizing the probability on real logits. DISPLAYFORM1 If v r (K+1) is not zero, that is, when we did negative label smoothing BID19, we could define R(v(K+1)) to be a uniform distribution. DISPLAYFORM2 As a , the label smoothing part probability will be required to be uniformly distributed, similar to CatGAN. In this section, we extend AM-GAN to unlabeled data. Our solution is analogous to. Under semi-supervised setting, we can add the following loss to the original solution to integrate the unlabeled data (with the distribution denoted as p unl (x)): DISPLAYFORM0 C.2 UNSUPERVISED SETTING Under unsupervised setting, we need to introduce one extra loss, analogy to categorical: DISPLAYFORM1 where the p ref is a reference label distribution for the prediction on unsupervised data. For example, p ref could be set as a uniform distribution, which requires the unlabeled data to make use of all the candidate class logits. This loss can be optionally added to semi-supervised setting, where the p ref could be defined as the predicted label distribution on the labeled training data E x∼pdata [D(x)]. As a recently proposed metric for evaluating the performance of the generative models, the InceptionScore has been found well correlated with human evaluation BID19, where a pre-trained publicly-available Inception model C is introduced. By applying the Inception model to each generated sample x and getting the corresponding class probability distribution C(x), Inception Score is calculated via Inception Score = exp E x KL C(x) C G,where E x is short of E x∼G andC G = E x [C(x)] is the overall probability distribution of the generated samples over classes, which is judged by C, and KL denotes the Kullback-Leibler divergence which is defined as KL(p q) = i p i log pi qi = i p i log p i − i p i log q i = −H(p) + H(p, q).An extended metric, the Mode Score, is proposed in BID3 to take the prior distribution of the labels into account, which is calculated via DISPLAYFORM0 where the overall class distribution from the training dataC train has been added as a reference. We show in the following that, in fact, Mode Score and Inception Score are equivalent. Lemma 3. Let p(x) be the class probability distribution of the sample x, andp denote another probability distribution, then Ex H p(x),p = H Ex p(x),p.With Lemma 3, we have log(Inception Score) DISPLAYFORM1 log(Mode Score) DISPLAYFORM2

Represent the TLDR for finding the corresponding scientific article.

Understand how class labels help GAN training. Propose a new evaluation metric for generative models.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the article for finding a single-sentence TLDR

Question: Modern neural networks are over-parametrized. In particular, each rectified linear hidden unit can be modified by a multiplicative factor by adjusting input and out- put weights, without changing the rest of the network. Inspired by the Sinkhorn-Knopp algorithm, we introduce a fast iterative method for minimizing the l2 norm of the weights, equivalently the weight decay regularizer. It provably converges to a unique solution. Interleaving our algorithm with SGD during training improves the test accuracy. For small batches, our approach offers an alternative to batch- and group- normalization on CIFAR-10 and ImageNet with a ResNet-18. Deep Neural Networks (DNNs) have achieved outstanding performance across a wide range of empirical tasks such as image classification BID1, image segmentation , speech recognition (a), natural language processing or playing the game of Go BID16. These successes have been driven by the availability of large labeled datasets such as ImageNet BID13, increasing computational power and the use of deeper models (b).Although the expressivity of the function computed by a neural network grows exponentially with depth BID12 ), in practice deep networks are vulnerable to both over-and underfitting (; BID1 b). Widely used techniques to prevent DNNs from overfitting include regularization methods such as weight decay BID2, Dropout (b) and various data augmentation schemes BID1 BID17 BID19 b). Underfitting can occur if the network gets stuck in a local minima, which can be avoided by using stochastic gradient descent algorithms (; ; BID18 BID0, sometimes along with carefully tuned learning rate schedules (b;).Training deep networks is particularly challenging due to the vanishing/exploding gradient problem. It has been studied for Recurrent Neural networks (RNNs) as well as standard feedforward networks (a; BID7 . After a few iterations, the gradients computed during backpropagation become either too small or too large, preventing the optimization scheme from converging. This is alleviated by using non-saturating activation functions such as rectified linear units (ReLUs) BID1 or better initialization schemes preserving the variance of the input across layers (; BID7 a). Failure modes that prevent the training from starting have been theoretically studied by.Two techniques in particular have allowed vision models to achieve "super-human" accuracy. Batch Normalization (BN) was developed to train Inception networks . It introduces intermediate layers that normalize the features by the mean and variance computed within the current batch. BN is effective in reducing training time, provides better generalization capabilities after training and diminishes the need for a careful initialization. Network architectures such as ResNet (b) and DenseNet use skip connections along with BN to improve the information flow during both the forward and backward passes. DISPLAYFORM0 Figure 1: Matrices W k and W k+1 are updated by multiplying the columns of the first matrix with rescaling coefficients. The rows of the second matrix are inversely rescaled to ensure that the product of the two matrices is unchanged. The rescaling coefficients are strictly positive to ensure functional equivalence when the matrices are interleaved with ReLUs. This rescaling is applied iteratively to each pair of adjacent matrices. In this paper, we address the more complex cases of biases, convolutions, max-pooling or skip-connections to be able to balance modern CNN architectures. However, BN has some limitations. In particular, BN only works well with sufficiently large batch sizes . For sizes below 16 or 32, the batch statistics have a high variance and the test error increases significantly. This prevents the investigation of highercapacity models because large, memory-consuming batches are needed in order for BN to work in its optimal range. In many use cases, including video recognition and image segmentation , the batch size restriction is even more challenging because the size of the models allows for only a few samples per batch. Another restriction of BN is that it is computationally intensive, typically consuming 20% to 30% of the training time. Variants such as Group Normalization (GN) cover some of the failure modes of BN.In this paper, we introduce a novel algorithm to improve both the training speed and generalization accuracy of networks by using their over-parameterization to regularize them. In particular, we focus on neural networks that are positive-rescaling equivalent BID8, i.e. whose weights are identical up to positive scalings and matching inverse scalings. The main principle of our method, referred to as Equi-normalization (ENorm), is illustrated in Figure 1 for the fullyconnected case. We scale two consecutive matrices with rescaling coefficients that minimize the joint p norm of those two matrices. This amounts to re-parameterizing the network under the constraint of implementing the same function. We conjecture that this particular choice of rescaling coefficients ensures a smooth propagation of the gradients during training. A limitation is that our current proposal, in its current form, can only handle learned skipconnections like those proposed in type-C ResNet. For this reason, we focus on architectures, in particular ResNet18, for which the learning converges with learned skip-connection, as opposed to architectures like ResNet-50 for which identity skip-connections are required for convergence. In summary,• We introduce an iterative, batch-independent algorithm that re-parametrizes the network within the space of rescaling equivalent networks, thus preserving the function implemented by the network; • We prove that the proposed Equi-normalization algorithm converges to a unique canonical parameterization of the network that minimizes the global p norm of the weights, or equivalently, when p = 2, the weight decay regularizer; • We extend ENorm to modern convolutional architectures, including the widely used ResNets, and show that the theoretical computational overhead is lower compared to BN (×50) and even compared to GN (×3); • We show that applying one ENorm step after each SGD step outperforms both BN and GN on the CIFAR-10 (fully connected) and ImageNet (ResNet-18) datasets.• Our code is available at https://github.com/facebookresearch/enorm. The paper is organized as follows. Section 2 reviews related work. Section 3 defines our Equinormalization algorithm for fully-connected networks and proves the convergence. Section 4 shows how to adapt ENorm to convolutional neural networks (CNNs). Section 5 details how to employ ENorm for training neural networks and Section 6 presents our experimental . This section reviews methods improving neural network training and compares them with ENorm. Since there is a large body of literature in this research area, we focus on the works closest to the proposed approach. From early works, researchers have noticed the importance of normalizing the input of a learning system, and by extension the input of any layer in a DNN BID4. Such normalization is applied either to the weights or to the activations. On the other hand, several strategies aim at better controlling the geometry of the weight space with respect to the loss function. Note that these research directions are not orthogonal. For example, explicitly normalizing the activations using BN has smoothing effects on the optimization landscape BID15.Normalizing activations. Batch Normalization normalizes the activations by using statistics computed along the batch dimension. As stated in the introduction, the dependency on the batch size leads BN to underperform when small batches are used. Batch Renormalization (BR) is a follow-up that reduces the sensitivity to the batch size, yet does not completely alleviate the negative effect of small batches. Several batch-independent methods operate on other dimensions, such as Layer Normalization (channel dimension) and Instance-Normalization (sample dimension) . Parametric data-independent estimation of the mean and variance in every layer is investigated by. However, these methods are inferior to BN in standard classification tasks. More recently, Group Normalization (GN) , which divides the channels into groups and normalizes independently each group, was shown to effectively replace BN for small batch sizes on computer vision tasks. Normalizing weights. Early weight normalization techniques only served to initialize the weights before training (; a). These methods aim at keeping the variance of the output activations close to one along the whole network, but the assumptions made to derive these initialization schemes may not hold as training evolves. More recently, BID14 propose a polar-like re-parametrization of the weights to disentangle the direction from the norm of the weight vectors. Note that Weight Norm (WN) does require mean-only BN to get competitive , as well as a greedy layer-wise initialization as mentioned in the paper. Optimization landscape. Generally, in the parameter space, the loss function moves quickly along some directions and slowly along others. To account for this anisotropic relation between the parameters of the model and the loss function, natural gradient methods have been introduced . They require storing and inverting the N × N curvature matrix, where N is the number of network parameters. Several works approximate the inverse of the curvature matrix to circumvent this problem BID5 BID6. Another method called Diagonal Rescaling BID3 proposes to tune a particular reparametrization of the weights with a block-diagonal approximation of the inverse curvature matrix. Finally, BID8 propose a rescaling invariant path-wise regularizer and use it to derive Path-SGD, an approximate steepest descent with respect to the path-wise regularizer. Positioning. Unlike BN, Equi-normalization focuses on the weights and is independent of the concept of batch. Like Path-SGD, our goal is to obtain a balanced network ensuring a good backpropagation of the gradients, but our method explicitly re-balances the network using an iterative algorithm instead of using an implicit regularizer. Moreover, ENorm can be readily adapted to the convolutional case whereas BID8 restrict themselves to the fully-connected case. In addition, the theoretical computational complexity of our method is much lower than the overhead introduced by BN or GN (see Section 5). Besides, WN parametrizes the weights in a polar-like manner, w = g × v/|v|, where g is a scalar and v are the weights, thus it does not balance the network but individually scales each layer. Finally, Sinkhorn's algorithm aims at making a single matrix doubly stochastic, while we balance a product of matrices to minimize their global norm. We first define Equi-normalization in the context of simple feed forward networks that consist of two operators: linear layers and ReLUs. The algorithm is inspired by Sinkhorn-Knopp and is designed to balance the energy of a network, i.e., the p -norm of its weights, while preserving its function. When not ambiguous, we may denote by network a weight parametrization of a given network architecture. We consider a network with q linear layers, whose input is a row vector x ∈ R n0. We denote by σ the point-wise ReLU activation. For the sake of exposition, we omit a bias term at this stage. We recursively define a simple fully connected feedforward neural network with L layers by y 0 = x, DISPLAYFORM0 and y q = y q−1 W q. Each linear layer k is parametrized by a matrix W k ∈ R n k−1 ×n k. We denote by f θ (x) = y q the function implemented by the network, where θ is the concatenation of all the network parameters. We denote by D(n) the set of diagonal matrices of size n × n for which all diagonal elements are strictly positive and by I n the identity matrix of size n × n. DISPLAYFORM1 Definition 2. θ andθ are rescaling equivalent if, for all k ∈ 1, q − 1, there exists a rescaling matrix DISPLAYFORM2 with the conventions that D 0 = I n0 and D q = I nq. This amounts to positively scaling all the incoming weights and inversely scaling all the outgoing weights for every hidden neuron. Two weights vectors θ andθ that are rescaling equivalent are also functionally equivalent (see Section 3.5 for a detailed derivation). Note that a functional equivalence class is not entirely described by rescaling operations. For example, permutations of neurons inside a layer also preserve functional equivalence, but do not change the gradient. In what follows our objective is to seek a canonical parameter vector that is rescaling equivalent to a given parameter vector. The same objective under a functional equivalence constraint is beyond the scope of this paper, as there exist degenerate cases where functional equivalence does not imply rescaling equivalence, even up to permutations. Given a network f θ and p > 0, we define the p norm of its weights as p (θ) = q k=1 W k p p. We are interested in minimizing p inside an equivalence class of neural networks in order to exhibit a unique canonical element per equivalence class. We denote the rescaling coefficients within the network as DISPLAYFORM0 n, where n is the number of hidden neurons. Fixing the weights {W k}, we refer to {D −1 k−1 W k D k} as the rescaled weights, and seek to minimize their p norm as a function of the rescaling coefficients: DISPLAYFORM1 We formalize the ENorm algorithm using the framework of block coordinate descent. We denote by DISPLAYFORM0 In what follows we assume that each hidden neuron is connected to at least one input and one output neuron. ENorm generates a sequence of rescaling coefficients δ (r) obtained by the following steps. Initialization. Define δ = (1, . . ., 1). Iteration. At iteration r, consider layer ∈ 1, q − 1 such that − 1 ≡ r mod q − 1 and define DISPLAYFORM1 Denoting uv the coordinate-wise product of two vectors and u/v for the division, we have DISPLAYFORM2 DISPLAYFORM3 Algorithm and pseudo-code. Algorithm 1 gives the pseudo-code of ENorm. By convention, one ENorm cycle balances the entire network once from = 1 to = q − 1. See Appendix A for illustrations showing the effect of ENorm on network weights. We now state our main convergence for Equi-normalization. The proof relies on a coordinate descent Theorem by Tseng FORMULA1 and can be found in Appendix B.1. The main difficulty is to prove the uniqueness of the minimum of ϕ. Theorem 1. Let p > 0 and (δ (r) ) r∈N be the sequence of rescaling coefficients generated by ENorm from the starting point δ as described in Section 3.3. We assume that each hidden neuron is connected to at least one input and one output neuron. Then, Convergence. The sequence of rescaling coefficients δ (r) converges to δ * as r → +∞. As a consequence, the sequence of rescaled weights also converges; Minimum global p norm. The rescaled weights after convergence minimize the global p norm among all rescaling equivalent weights; Uniqueness. The minimum is unique, i.e. δ * does not depend on the starting point δ. In the presence of biases, the network is defined as DISPLAYFORM0 For rescaling-equivalent weights satisfying, in order to preserve the inputoutput function, we define matched rescaling equivalent biases DISPLAYFORM1 show by recurrence that for every layer k, DISPLAYFORM2 where y k (resp. y k) is the intermediary network function associated with the weights W (resp. W). In particular, y q = y q, i.e. rescaling equivalent networks are functionally equivalent. We also compute the effect of applying ENorm on the gradients in the same Appendix. Equi-normalization is easily adapted to introduce a depth-wise penalty on each layer. We consider the weighted loss p,(c1,...,cq) (θ) = q k=1 c k W k p. This amounts to modifiying the rescaling coefficients asd DISPLAYFORM0 In Section 6, we explore two natural ways of defining c k: c k = c p(q−k) (uniform) and c k = 1/(n k−1 n k) (adaptive). In the uniform setup, we penalize layers exponentially according to their depth: for instance, values of c larger than 1 increase the magnitude of the weights at the end of the network. In the adaptive setup, the loss is weighted by the size of the matrices. We now extend ENorm to CNNs, by focusing on the typical ResNet architecture. We briefly detail how we adapt ENorm to convolutional or max-pooling layers, and then how to update an elementary block with a skip-connection. We refer the reader to Appendix C for a more extensive discussion. Sanity checks of our implementation are provided in Appendix E.1. Figure 2 explains how to rescale two consecutive convolutional layers. As detailed in Appendix C, this is done by first properly reshaping the filters to 2D matrices, then performing the previously described rescaling on those matrices, and then reshaping the matrices back to convolutional filters. This matched rescaling does preserve the function implemented by the composition of the two layers, whether they are interleaved with a ReLU or not. It can be applied to any two consecutive convolutional layers with various stride and padding parameters. Note that when the kernel size is 1 in both layers, we recover the fully-connected case of Figure 1. The MaxPool layer operates per channel by computing the maximum within a fixed-size kernel. We adapt Equation FORMULA12 to the convolutional case where the rescaling matrix D k is applied to the channel dimension of the activations y k. Then, DISPLAYFORM0 Thus, the activations before and after the MaxPool layer have the same scaling and the functional equivalence is preserved when interleaving convolutional layers with MaxPool layers. We now consider an elementary block of a ResNet-18 architecture as depicted in FIG0. In order to maintain functional equivalence, we only consider ResNet architectures of type C as defined in (b), where all shortcuts are learned 1 × 1 convolutions. As detailed in Appendix C, rescaling two consecutive blocks requires (a) to define the structure of the rescaling process, i.e. where to insert the rescaling coefficients and (b) a formula for computing those rescaling coefficients. ENorm & SGD. As detailed in Algorithm 2, we balance the network periodically after updating the gradients. By design, this does not change the function implemented by the network but will yield different gradients in the next SGD iteration. Because this re-parameterization performs a jump in the parameter space, we update the momentum using Equation FORMULA1 and the same matrices D k as those used for the weights. The number of ENorm cycles after each SGD step is an hyperparameter and by default we perform one ENorm cycle after each SGD step. In Appendix D, we also explore a method to jointly learn the rescaling coefficients and the weights with SGD, and report corresponding . Computational advantage over BN and GN. TAB2 provides the number of elements (weights or activations) accessed when normalizing using various techniques. Assuming that the complexity (number of operations) of normalizing is proportional to the number of elements and assuming all techniques are equally parallelizable, we deduce that our method (ENorm) is theoretically 50 times faster than BN and 3 times faster than GN for a ResNet-18. In terms of memory, ENorm requires no extra-learnt parameters, but the number of parameters learnt by BN and GN is negligible (4800 for a ResNet-18 and 26,650 for a ResNet-50). We implemented ENorm using a tensor library; to take full advantage of the theoretical reduction in compute would require an optimized CUDA kernel....... DISPLAYFORM0 Rescaling the weights of two consecutive convolutional layers that preserves the function implemented by the CNN. Layer k scales channel number i of the input activations by γ i and layer k+1 cancels this scaling with the inverse scalar so that the activations after layer k+1 are unchanged. Block k DISPLAYFORM0 We analyze our approach by carrying out experiments on the MNIST and CIFAR-10 datasets and on the more challenging ImageNet dataset. ENorm will refer to Equi-normalization with p = 2. Training. We follow the setup of. Input data is normalized by subtracting the mean and dividing by standard deviation. The encoder has the structure FC-ReLU-FC-ReLU-FC-ReLU-FC and the decoder has the symmetric structure. We use He's initialization for the weights. We select the learning rate in {0.001, 0.01, 0.1} and decay it linearly to zero. We use a batch size of 256 and SGD with no momentum and a weight decay of 0.001. For path-SGD, our implementation closely follows the original paper BID8 and we set the weight decay to zero. For GN, we cross-validate the number of groups among {5, 10, 20, 50}. For WN, we use BN as well as a greedy layer-wise initialization as described in the original paper. Results. While ENorm alone obtains competitive compared to BN and GN, ENorm + BN outperforms all other methods, including WN + BN. Note that here ENorm refers to Enorm using the adaptive c parameter as described in Subsection 3.6, whereas for ENorm + BN we use the uniform setup with c = 1. We perform a parameter study for different values and setups of the asymmetric scaling (uniform and adaptive) in Appendix E.2. Without BN, the adaptive setup outperforms all other setups, which may be due to the strong bottleneck structure of the network. With BN, the dynamic is different and the are much less sensitive to the values of c. Results without any normalization and with Path-SGD are not displayed because of their poor performance. Training. We first experiment with a basic fully-connected architecture that takes as input the flattened image of size 3072. Input data is normalized by subtracting mean and dividing by standard deviation independently for each channel. The first linear layer is of size 3072 × 500. We then consider p layers 500 × 500, p being an architecture parameter for the sake of the analysis. The last classification is of size 500 × 10. The weights are initialized with He's scheme. We train for 60 epochs using SGD with no momentum, a batch size of 256 and weight decay of 10 −3. Cross validation is used to pick an initial learning rate in {0.0005, 0.001, 0.005, 0.01, 0.05, 0.1}. Path-SGD, GN and WN are learned as detailed in Section 6.1. All are the average test accuracies over 5 training runs. Results. ENorm alone outperforms both BN and GN for any depth of the network. ENorm + BN outperforms all other methods, including WN + BN, by a good margin for more than p = 11 intermediate layers. Note that here ENorm as well as ENorm + BN refers to ENorm in the uniform setup with c = 1.2. The of the parameter study for different values and setups of the asymmetric scaling are similar to those of the MNIST setup, see Appendix E.2. Training. We use the CIFAR-NV architecture as described by. Images are normalized by subtracting mean and dividing by standard deviation independently for each channel. During training, we use 28 × 28 random crops and randomly flip the image horizontally. At test time, we use 28 × 28 center crops. We split the train set into one training set (40,000 images) and one validation set (10,000 images). We train for 128 epochs using SGD and an initial learning rate cross-validated on a held-out set among {0.01, 0.05, 0.1}, along with a weight decay of 0.001. The learning rate is then decayed quadratically to 10 −4. We compare various batch sizes together with the use of momentum (0.9) or not. The weights are initialized with He's scheme. In order to stabilize the training, we employ a BatchNorm layer at the end of the network after the FC layer for the Baseline and ENorm cases. For GN we cross-validate the number of groups among {4, 8, 16, 32, 64}. All are the average test accuracies over 5 training runs. Table 3: CIFAR-10 fully convolutional (higher is better).Results. See Table 3. ENorm + BN outperforms all other methods, including WN + BN, by a good margin. Note that here ENorm refers to ENorm in the uniform setup with the parameter c = 1.2 whereas ENorm + BN refers to the uniform setup with c = 1. A parameter study for different values and setups of the asymmetric scaling can be found in Appendix E.2. This dataset contains 1.3M training images and 50,000 validation images split into 1000 classes. We use the ResNet-18 model with type-C learnt skip connections as described in Section 4.Training. Our experimental setup closely follows that of GN . We train on 8 GPUs and compute the batch mean and standard deviation per GPU when evaluating BN. We use the Kaiming initialization for the weights (a) and the standard data augmentation scheme of BID19. We train our models for 90 epochs using SGD with a momentum of 0.9. We adopt the linear scaling rule for the learning rate and set the initial learning rate to 0.1B/256 where the batch size B is set to 32, 64, 128, or 256. As smaller batches lead to more iterations per epoch, we adopt a similar rule and adopt a weight decay of w = 10 DISPLAYFORM0 for B = 128 and 256, w = 10 −4.5 for B = 64 and w = 10 −5 for B = 32. We decay the learning rate quadratically to 10 −5 and report the median error rate on the final 5 epochs. When using GN, we set the number of groups G to 32 and did not cross-validate this value as prior work reports little impact when varying G from 2 to 64. In order for the training to be stable and faster, we added a BatchNorm at the end of the network after the FC layer for the Baseline and ENorm cases. The batch mean and variance for this additional BN are shared across GPUs. Note that the activation size at this stage of the network is B × 1000, which is a negligible overhead (see TAB2).Results. We compare the Top 1 accuracy on a ResNet-18 when using no normalization scheme, (Baseline), when using BN, GN and ENorm (our method). In both the Baseline and ENorm settings, we add a BN at the end of the network as described in 6.3. The are reported in Table 4. The performance of BN decreases with small batches, which concurs with prior observations . Our method outperforms GN and BN for batch sizes ranging from 32 to 128. GN presents stable across batch sizes. Note that values of c different from 1 did not yield better . The training curves for a batch size of 64 are presented in FIG2. While BN and GN are faster to converge than ENorm, our method achieves better after convergence in this case. Note also that ENorm overfits the training set less than BN and GN, but more than the Baseline case. Table 4: ResNet-18 on the ImageNet dataset (test accuracy). We applied ENorm to a deeper (ResNet-50), but obtained unsatisfactory . We observed that learnt skip-connections, even initialized to identity, make it harder to train without BN, even with careful layer-wise initialization or learning rate warmup. This would require further investigation. We presented Equi-normalization, an iterative method that balances the energy of the weights of a network while preserving the function it implements. ENorm provably converges towards a unique equivalent network that minimizes the p norm of its weights and it can be applied to modern CNN architectures. Using ENorm during training adds a much smaller computational overhead than BN or GN and leads to competitive performances in the FC case as well as in the convolutional case. Discussion. While optimizing an unbalanced network is hard BID8, the criterion we optimize to derive ENorm is likely not optimal regarding convergence or training properties. These limitations suggest that further research is required in this direction. We first apply ENorm to one randomly initialized fully connected network comprising 20 intermediary layers. All the layers have a size 500 × 500 and are initialized following the Xavier scheme. The network has been artificially unbalanced as follows: all the weights of layer 6 are multiplied by a factor 1.2 and all the weights of layer 12 are multiplied by 0.8, see FIG3. We then iterate our ENorm algorithm on the network, without training it, to see that it naturally re-balances the network, see FIG4. DISPLAYFORM0. We make the following assumptions: f is differentiable on D; X is compact; for each x ∈ D, each block coordinate function f: t → f (x 1, . . ., x −1, t, x +1, . . ., x B), where 2 ≤ ≤ B − 1, has at most one minimum. Under these assumptions, the sequence (x (r) ) r∈N generated by the coordinate descent algorithm is defined and bounded. Moreover, every cluster point of (x (r) ) r∈N is a local minimizer of f. STEP 1. We apply Theorem 2 to the function ϕ. This is possible because all the assumptions are verified as shown below. Recall that DISPLAYFORM1 DISPLAYFORM2 k, where C = max(C 0, 1) and DISPLAYFORM3 • For the first hidden layer, index k = 1. By assumption, every hidden neuron is connected at least to one neuron in the input layer. Thus, for every j, there exists i such that DISPLAYFORM4 Thus d 1 ∞ < CM.• For some hidden layer, index k. By assumption, every hidden neuron is connected at least to one neuron in the previous layer. Thus, for every j, there exists i such that DISPLAYFORM5 Using the induction hypothesis, we get DISPLAYFORM6 Thus, there exists a ball B such that δ / ∈ B implies ϕ(δ) > ϕ(δ 0). Thus, δ ∈ X implies that x ∈ B and X ⊂ B is bounded. Moreover, X is closed because ϕ is continuous thus X is compact and Assumption is satisfied. has a unique minimum as shown in Section 3.3, see Equation. The existence and uniqueness of the minimum comes from the fact that each hidden neuron is connected to at least one input and one output neuron, thus all the row and column norms of the hidden weight matrices W k are non-zero, as well as the column (resp. row) norms or W 1 (resp. W q). STEP 2. We prove that ϕ has at most one stationary point on D under the assumption that each hidden neuron is connected either to an input neuron or to an output neuron, which is weaker than the general assumption of Theorem 1.We first introduce some definitions. We denote the set of all neurons in the network by V. Each neuron ν ∈ V belongs to a layer k ∈ 0, q and has an index i ∈ 1, n k in this layer. Any edge e connects some neuron i at layer k − 1 to some neuron j at layer k, e = (k, i, j). We further denote by H the set of hidden neurons ν belonging to layers q ∈ 1, q − 1. We define E as the set of edges whose weights are non-zero, i.e. DISPLAYFORM0 For each neuron ν, we define prev(ν) as the neurons connected to ν that belong to the previous layer. We now show that ϕ has at most one stationary point on D. Directly computing the gradient of ϕ and solving for zeros happens to be painful or even intractable. Thus, we define a change of variables as follows. We define h as h: (0, +∞) DISPLAYFORM1 We next define the shift operator S: DISPLAYFORM2 and the padding operator P as DISPLAYFORM3 We define the extended shift operator S H = S • P. We are now ready to define our change of variables. We define χ = ψ • S H where DISPLAYFORM4 Since h is a C ∞ diffeomorphism, its differential [Dh](δ) is invertible for any δ. It follows that [Dϕ](δ) = 0 if, and only if, [Dχ](h(δ)) = 0. As χ is the composition of a strictly convex function, ψ, and a linear injective function, S H (proof after Step 3), it is strictly convex. Thus χ has at most one stationary point, which concludes this step. STEP 3. We prove that the sequence δ (r) converges. Step 1 implies that the sequence δ (r) is bounded and has at least one cluster point, as f is continuous on the compact X.Step 2 implies that the sequence δ (r) has at most one cluster point. We then use the fact that any bounded sequence with exactly one cluster point converges to conclude the proof. S IS INJECTIVE. Let x ∈ ker S H. Let us show by induction on the hidden layer index k that for every neuron ν at layer k, x ν = 0.• k = 1. Let ν be a neuron at layer 1. Then, there exists a path coming from an input neuron to ν 0 through edge e 1. By definition, P (x) ν0 = 0 and P (x) ν = x ν, hence S H (x) e1 = x ν − 0. Since S H (x) = 0 it follows that x ν = 0.• k → k + 1. Same reasoning using the fact that x ν k = 0 by the induction hypothesis. The case where the path goes from neuron ν to some output neuron is similar. We show by induction that for every layer k, i.e., DISPLAYFORM0 where y k (resp. y k) is the intermediary network function associated with weights W (resp. W). This holds for k = 0 since D 0 = I n0 by convention. If the property holds for some k < q − 1, then by we have DISPLAYFORM1 The same equations hold for k = q − 1 without the non-linearity σ. Using the chain rule and denoting by the loss of the network, for every layer k, using, we have DISPLAYFORM2 Similarly, we obtain DISPLAYFORM3 Equation FORMULA1 will be used to update the momentum (see Section 5) and Equation for the weights. Let us consider two consecutive convolutional layers k and k + 1, without bias. Layer k has C k filters of size C k−1 × S k × S k, where C k−1 is the number of input features and S k is the kernel size. This in a weight tensor T k of size C k × C k−1 × S k × S k. Similarly, layer k + 1 has a weight matrix T k+1 of size C k+1 × C k × S k+1 × S k+1. We then perform axis-permutation and reshaping operations to obtain the following 2D matrices: DISPLAYFORM0 For example, we first reshape T k as a 2D matrix by collapsing its last 3 dimensions, then transpose it to obtain M k. We then jointly rescale those 2D matrices using rescaling matrices D k ∈ D(k) as detailed in Section 3 and perform the inverse axis permutation and reshaping operations to obtain a right-rescaled weight tensor T k and a left-rescaled weight tensor T k+1. See Figure 2 for an illustration of the procedure. This matched rescaling does preserve the function implemented by the composition of the two layers, whether they are interleaved with a ReLU or not. It can be applied to any two consecutive convolutional layers with various stride and padding parameters. Note that when the kernel size is 1 in both layers, we recover the fully-connected case of Figure 1. We now consider an elementary block of a ResNet-18 architecture as depicted in FIG0. In order to maintain functional equivalence, we only consider ResNet architectures of type C as defined in (b), where all shortcuts are learned 1 × 1 convolutions. Structure of the rescaling process. Let us consider a ResNet block k. We first left-rescale the Conv1 and ConvSkip weights using the rescaling coefficients calculated between blocks k − 1 and k. We then rescale the two consecutive layers Conv1 and Conv2 with their own specific rescaling coefficients, and finally right-rescale the Conv2 and ConvSkip weights using the rescaling coefficients calculated between blocks k and k + 1.Computation of the rescaling coefficients. Two types of rescaling coefficients are involved, namely those between two convolution layers inside the same block and those between two blocks. The rescaling coefficients between the Conv1 and Conv2 layers are calculated as explained in Section 4.1. Then, in order to calculate the rescaling coefficients between two blocks, we compute equivalent block weights to deduce rescaling coefficients. We empirically explored some methods to compute the equivalent weight of a block using electrical network analogies. The most accurate method we found is to compute the equivalent weight of the Conv1 and Conv2 layers, i.e., to express the succession of two convolution layers as only one convolution layer denoted as ConvEquiv (series equivalent weight), and in turn to express the two remaining parallel layers ConvEquiv and ConvSkip again as a single convolution layer (parallel equivalent weight). It is not possible to obtain series of equivalent weights, in particular when the convolution layers are interleaved with ReLUs. Therefore, we approximate the equivalent weight as the parallel equivalent weight of the Conv1 and ConvSkip layers. In Section 3, we defined an iterative algorithm that minimizes the global p norm of the network DISPLAYFORM0 As detailed in Algorithm 2, we perform alternative SGD and ENorm steps during training. We now derive an implicit formulation of this algorithm that we call Implicit Equi-normalization. Let us fix p = 2. We denote by C(f θ (x), y) the cross-entropy loss for the training sample (x, y) and by 2 (θ, δ) the weight decay regularizer. The loss function of the network writes DISPLAYFORM1 where λ is a regularization parameter. We now consider both the weights and the rescaling coefficients as learnable parameters and we rely on automatic differentiation packages to compute the derivatives of L with respect to the weights and to the rescaling coefficients. We then simply train the network by performing iterative SGD steps and updating all the learnt parameters. Note that by design, the derivative of C with respect to any rescaling coefficient is zero. Although the additional overhead of implicit ENorm is theoretically negligible, we observed an increase of the training time of a ResNet-18 by roughly 30% using PyTorch 4.0 BID11. We refer to Implicit Equi-normalization as ENorm-Impl and to Explicit Equi-normalization as ENorm. We performed early experiments for the CIFAR10 fully-connected case. ENorm-Impl performs generally better than the baseline but does not outperform explicit ENorm, in particular when the network is deep. We follow the same experimental setup than previously, except that we additionally cross-validated λ. We also initialize all the rescaling coefficients to one.. Recall that ENorm or ENorm denotes explicit Equi-normalization while ENorm-Impl denotes Implicit Equinormalization. We did not investigate learning the weights and the rescaling coefficients at different speeds (e.g. with different learning rates or momentum). This may explain in part why ENorm-Impl generally underperforms ENorm in those early experiments. We perform sanity checks to verify our implementation and give additional . We apply our Equi-normalization algorithm to a ResNet architecture by integrating all the methods exposed in Section 4. We perform three sanity checks before proceeding to experiments. First, we randomly initialize a ResNet-18 and verify that it outputs the same probabilities before and after balancing. Second, we randomly initialize a ResNet-18 and perform successive ENorm cycles (without any training) and observe that the 2 norm of the weights in the network is decreasing and then converging, as theoretically proven in Section 3, see FIG5. We finally compare the evolution of the total 2 norm of the network when training it, with or without ENorm. We use the setup described in Subsection 6.2 and use p = 3 intermediary layers. The are presented in FIG6. ENorm consistently leads to a lower energy level in the network. MNIST auto-encoder. For the uniform setup, we test for three different values of c, without BN: c = 1 (uniform setup), c = 0.8 (uniform setup), c = 1.2 (uniform setup). We also test the adaptive setup. The adaptive setup outperforms all other choices, which may be due to the strong bottleneck structure of the network. With BN, the dynamics are different and the are much less sensitive to the values of c (see FIG7). For the uniform setup, we test for three different values of c, without BN: c = 1 (uniform setup), c = 0.8 (uniform setup), c = 1.2 (uniform setup). We also test the adaptive setup (see TAB8). Once again, the dynamics with or without BN are quite different. With or without BN, c = 1.2 performs the best, which may be linked to the fact that the ReLUs are cutting energy during each forward pass. With BN, the are less sensitive to the values of c.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Fast iterative algorithm to balance the energy of a network while staying in the same functional equivalence class

We introduce a type of neural network that is structurally resistant to adversarial attacks, even when trained on unaugmented training sets.  The resistance is due to the stability of network units wrt input perturbations.