Question: Represent the article for finding a single-sentence TLDR

Question: Federated learning, where a global model is trained by iterative parameter averaging of locally-computed updates, is a promising approach for distributed training of deep networks; it provides high communication-efficiency and privacy-preservability, which allows to fit well into decentralized data environments, e.g., mobile-cloud ecosystems. However, despite the advantages, the federated learning-based methods still have a challenge in dealing with non-IID training data of local devices (i.e., learners). In this regard, we study the effects of a variety of hyperparametric conditions under the non-IID environments, to answer important concerns in practical implementations: (i) We first investigate parameter divergence of local updates to explain performance degradation from non-IID data. The origin of the parameter divergence is also found both empirically and theoretically. (ii) We then revisit the effects of optimizers, network depth/width, and regularization techniques; our observations show that the well-known advantages of the hyperparameter optimization strategies could rather yield diminishing returns with non-IID data. (iii) We finally provide the reasons of the failure cases in a categorized way, mainly based on metrics of the parameter divergence. Over the recent years, federated learning has been a huge success to reduce the communication overhead in distributed training of deep networks. Guaranteeing competitive performance, the federated learning permits each learner to compute their local updates of each round for relatively many iterations (e.g., 1 epoch, 10 epochs, etc.), which provides much higher communication-efficiency compared to the conventional data parallelism approaches (for intra-datacenter environments, e.g., ;) that generally require very frequent gradient aggregation. Furthermore, the federated learning can also significantly reduce data privacy and security risks by enabling to conceal on-device data of each learner from the server or other learners; thus the approach can be applied well to environments with highly private data (e.g., personal medical data), it is now emerging as a promising methodology for privacypreserving distributed learning along with differential privacy-based methods (; ; ;). On this wise, the federated learning takes a simple approach that performs iterative parameter averaging of local updates computed from each learners' own dataset, which suggests an efficient way to learn a shared model without centralizing training data from multiple sources; but hereby, since the local data of each device is created based on their usage pattern, the heterogeneity of training data distributions across the learners might be naturally assumed in real-world cases. Hence, each local dataset would not follow the population distribution, and handling the decentralized non-IID data still remains a statistical challenge in the field of federated learning . For instance, observed severe performance degradation in multi-class classification accuracy under highly skewed non-IID data; it was reported that more diminishing returns could be yielded as the probabilistic distance of learners' local data from the population distribution increases. LearnerUpdate(k, w): // Run on learner k B ←(split P k into batches of size B) for each local epoch ε from 1 to E do for each batch b ∈ B do w ← w − η∇ (w; b) end for end for return w to server Contributions. To address the non-IID issue under federated learning, there have been a variety of recent works 1; nevertheless, in this paper we explore more fundamental factors, the effects of various hyperparameters. The optimization for the number of local iterations per round or learning rates has been handled in several literatures (e.g., ; Li et al. (2019c); ); by extension we discuss, for the first time to the best of our knowledge, the effects of optimizers, network depth/width, and regularization techniques. Our contributions are summarized as follows: First, as a root cause of performance degradation from non-IID data, we investigate parameter divergence of local updates at each round. The parameter divergence can be regarded as a direct response to learners' local data being non-IID sampled from the population distribution, of which the excessive magnitude could disturb the performance of the consequent parameter averaging. We also investigate the origin of the parameter divergence in both empirical and theoretical ways. Second, we observe the effects of well-known hyperparameter optimization methods 2 under the non-IID data environments; interestingly, some of our findings show highly conflicted aspects with their positive outcomes under "vanilla" training 3 or the IID data setting. Third, we analyze the internal reasons of our observations in a unified way, mainly using the parameter divergence metrics; it is identified that the rationale of the failures under non-IID data lies in some or all of (i) inordinate magnitude of parameter divergence, (ii) its steep fall phenomenon (described in Section 4.2), and (iii) excessively high training loss of local updates. In this study, Algorithm 1 is considered as a federated learning method, and it is written based on FedAvg . 4 We note that this kind of parameter averaging-based approach has been widely discussed in the literature, under various names, e.g., parallel (restarted) SGD and local SGD . In our experiments with Tensorflow , 5 we consider the multi-class classification tasks on CIFAR-10 and SVHN datasets. 2 we use the term hyperparameter optimization methods and hyperparametric methods interchangeably. 3 This term refers to the non-distributed training with a single machine, using the whole training examples. 4 Regarding the significance of the algorithm, we additionally note that Google is currently employing it on their mobile keyboard application (Gboard) (; ; ;). In this study we deal with image classification, which is also considered as the main applications of federated learning along with the language models . 5 Our source code is available at https://github.com/fl-noniid/fl-noniid Baseline network model: For the baseline deep network, we consider a CNN model that has three 3 × 3 convolutional layers with 64, 128, 256 output channels, respectively; and then three fully-connected layers with 512, 256, 10 output sizes, respectively (see Appendix A.1 for more detailed description). We use the term NetA-Baseline to denote this baseline model throughout this paper. Regularization configuration: For weight decay, we apply the method of decoupled weight decay regularization based on the fact that weight decay is equivalent to L 2 regularization only for pure SGD . The baseline value of the weight decay factor is set to 0.00005. As our regularization baseline, we consider not to apply any other regularization techniques additionally. We importantly note that if without any particular comments, the described in the following sections are ones obtained using the above baseline configurations of the network model and regularization. Environmental configuration: We consider 10 learners to have each 5000 nonoverlapping training examples; Table 1 summarizes our configuration of data settings; Non-IID(N) denotes a data setting that lets each learner to have training examples only for N class(es). The data settings in the Table 1 deal with data balanced cases where learners have the same amount of local data, and they are mainly considered in the following sections; we additionally note that one can refer to Appendix C.8 for the experiments with data unbalanced cases. For the IID and the non-IID data settings, T = 200 and 300 are used respectively, while E = 1 and minibatch size of 50 are considered commonly for the both. 6 One can find the remaining configurations for the experiments in Appendix A. Parameter divergence is recently being regarded as a strong cause of diminishing returns from decentralized non-IID data in federated learning (it is sometimes expressed in another way, gradient/loss divergence (b; c; ;) ). For the divergence metrics, many of the literatures usually handle the difference of each learner's local model parameters from one computed with the population distribution; it eventually also causes parameter diversity between the local updates as the data distributions become heterogeneous across learners. A pleasant level of parameter divergence could rather imply exploiting rich decentralized data (IID cases); however, if the local datasets are far from the population distribution, the consequent parameter averaging of the highly diverged local updates could lead to bad solutions away from the global optimum (non-IID cases). and Non-IID from the population distribution are 1.0 and 1.6, respectively. The origin of parameter divergence. In relation, it has been theoretically proven that the parameter divergence (between the global model parameters under FedAvg and those computed by vanilla SGD training) is directly related to the probabilistic distance of local datasets from the population distribution (see Proposition 3.1 in). In addition to it, for multi-class classification tasks, we here identify in lower level, that if data distributions in each local dataset are highly skewed and heterogeneous over classes, subsets of neurons, which have especially big magnitudes of the gradients in back propagation, become significantly different across learners; this leads to inordinate parameter divergence between them. As illustrated in Figure 1, under the IID data setting, the weight values in the output layer are evenly distributed relatively evenly across classes if the neurons of the model are initialized uniformly. However, we can observe under the non-IID data settings that the magnitudes of the gradients are distributed depending on each learner's data distribution. We also provide the corresponding theoretical analysis in Appendix B. Metrics. To capture parameter divergence under federated learning, we define the following two metrics using the notations in Algorithm 1. Since in our analysis we compare different network architectures or training settings together in a set, the number of neurons in the probed layers can become different, and values of model parameters can highly depend on the experimental manipulations; thus instead of Euclidean distance, in the two divergence metrics we use cosine distance that enables normalized (qualitative) measures. We also note that PD-VL is defined assuming the balancedness of data amount between learners, i.e., the same numbers of local iterations per round. The reason of probing parameter divergence being important is that the federated learning are performed based on iterative parameter averaging. That is, investigating how local updates are diverged can give a clue whether the subsequent parameter averaging yields positive returns; the proposed divergence metrics provide two ways for it. k is a subset (or the universal set) of w t k, we define parameter divergence between local updates as In addition, assume that P k is identical ∀k ∈ K, and let w t −1 be the vanilla-updated parameters, that is, the model parameters updated on the global parameters (i.e., w t−1) using IID training data during the same number of iterations with the actual learners (i.e., P k /B). Then, for z Relationship among probabilistic distance, parameter divergence, and learning performance. We consider Non-IID and Non-IID for non-IID data settings. Here we use earth mover's distance (EMD), also known as Wasserstein distance, to measure probabilistic distance of each data settings from the population distribution; the value becomes 1.0 and 1.6 for Non-IID and Non-IID, respectively. From the middle and right panels of Figure 2, it is seen that greater EMDs lead to bigger parameter divergence (refer to also Figure 9 in the appendix). Also, together with the left panel, we can observe the positive correlation between parameter divergence and learning performance. Therefore, we believe the parameter divergence metrics can help to reveal the missing link between data non-IIDness and the consequent learning performance. Note that one can also refer to the similar analysis with more various EMD in. From now on we describe our findings for the effects of various hyperparameter optimization methods with non-IID data on the federated learning algorithm. The considered hyperparametric methods have been a huge success to improve performance in deep learning; however, here we newly identify that under non-IID data settings, they could give negative/diminishing effects on performance of the federated learning algorithm. The following is the summary of our findings; we provide the complete experimental and further discussion in the next subsection and the appendix. Effects of optimizers. Unlike non-adaptive optimizers such as pure SGD and momentum SGD , Adam could give poor performance from non-IID data if the parameter averaging is performed only for weights and biases, compared to all the model variables (including the first and second moment) being averaged. Here we importantly note that both momentum SGD and Adam require the additional variables related to momentum as well as weights and biases; throughout the rest of the paper, the terms (optimizer name)-A and (optimizer name)-WB are used to refer to the parameter averaging being performed for all the variables 7 and only for weights & biases, respectively. Effects of network depth/width. It is also known that deepening "plain" networks (which simply stacks layers, without techniques such as information highways and shortcut connection ) yields performance degradation at a certain depth, even under vanilla training; however this phenomenon gets much worse under non-IID data environments. On the contrary, widening networks could help to achieve better outcomes; in that sense, the global average pooling could fail in this case since it significantly reduces the channel dimension of the (last) fully-connected layer, compared to using the max pooling. Effects of Batch Normalization. The well-known strength of Batch Normalization , the dependence of hidden activations in the minibatch , could become a severe drawback in non-IID data environments. Batch Renormalization helps to mitigate this, but it also does not resolve the problem completely. Effects of regularization techniques. With non-IID data, regularizations techniques such as weight decay and data augmentation could give excessively high training loss of local updates even in a modest level, which offsets the generalization gain. We now explain the internal reasons of the observations in the previous subsection. Through the experimental , we were able to classify the causes of the failures under non-IID data into three categories; the following discussions are described based on this. 8 Note that our discussion in this subsection is mostly made from the under Nesterov momentum SGD and on CIFAR-10; the complete including other optimizers (e.g., pure SGD, Polyak momentum SGD, and Adam) and datasets (e.g., SVHN) are given in Appendix C. Inordinate magnitude of parameter divergence. As mentioned before, bigger parameter divergence is the root cause of diminishing returns under federated learning methods with non-IID data. By extension, here we observe that even under the same non-IID data setting, some of the considered hyperparametric methods yield greater parameter divergence than when they are not applied. For example, from the left plot of Figure 3, we see that under the Non-IID setting, the parameter divergence values (in the last fully-connected layer) become greater as the network depth increases (note that NetA-Baseline, NetA-Deeper, and NetA-Deepest have 3, 6, and 9 convolutional layers, respectively; see also Appendix A.1 for their detailed architecture). The corresponding final test accuracy was found to be 74.11%, 73.67%, and 68.98%, respectively, in order of the degree of shallowness; this fits well into the parameter divergence . Since the NetA-Deeper and NetA-Deepest have twice and three times as many model parameters as NetA-Baseline, it can be expected enough that the deeper models yield bigger parameter divergence in the whole model; but our also show its qualitative increase in a layer level. In relation, we also provide the using the modern network architecture (e.g., ResNet ) in Table 8 of the appendix. From the middle plot of the figure, we can also observe bigger parameter divergence in a high level of weight decay under the Non-IID setting. Under the non-IID data setting, the test accuracy of about 72 ∼ 74% was achieved in the low levels (≤ 0.0001), but weight decay factor of 0.0005 yielded only that of 54.11%. Hence, this suggests that with non-IID data we should apply much smaller weight decay to federated learning-based methods. Here we note that if a single iteration is considered for each learner's local update per round, the corresponding parameter divergence will be of course the same without regard to degree of weight decay. However, in our experiments, the great number of local iterations per round (i.e., 100) made a big difference of the divergence values under the non-IID data setting; this eventually yielded the accuracy gap. We additionally observe for the non-IID cases that even with weight decay factor of 0.0005, the parameter divergence values are similar to those with the smaller factors at very early rounds in which the norms of the weights are relatively very small. In addition, it is observed from the right plot of the figure that Dropout also yields bigger parameter divergence under the non-IID data setting. The corresponding test accuracy was seen to be a diminishing return with Nesterov momentum SGD (i.e., using Dropout we can achieve +2.85% under IID, but only +1.69% is obtained under non-IID, compared to when it is not applied; see Table 2 ); however, it was observed that the generalization effect of the Dropout is still valid in test accuracy for the pure SGD and the Adam (refer to also Table 13 in the appendix). Steep fall phenomenon. As we see previously, inordinate magnitude of parameter divergence is one of the notable characteristics for failure cases under federated learning with non-IID data. However, under the non-IID data setting, some of the failure cases have been observed where the test accuracy is still low but the parameter divergence values of the last fully-connected layer decrease (rapidly) over rounds; as the round goes, even the values were sometimes seen to be lower than those of the comparison targets. We refer to this phenomenon as steep fall phenomenon. It is inferred that these (unexpected abnormal) sudden drops of parameter divergence values indicate going into poor local minima (or saddles); this can be supported by the behaviors that test accuracy increases plausibly at very early rounds, but the growth rate quickly stagnates and eventually becomes much lower than the comparison targets. The left plot of Figure 4 shows the effect of the Adam optimizer with respect to its implementations. Through the experiments, we identified that under non-IID data environments, the performance of Adam is very sensitive to the range of model variables to be averaged, unlike the non-adaptive optimizers (e.g., momentum SGD); its moment variables should be also considered in the parameter averaging together with weights and biases (see also Table 3). The poor performance of the Adam-WB under the Non-IID setting would be from twice as many momentum variables as the momentum SGD, which indicates the increased number of them affected by the non-IIDness; thus, originally we had thought that extreme parameter divergence could appear if the momentum variables are not averaged together with weights and biases. However, it was seen that the parameter divergence values under the Adam-WB was seen to be similar or even smaller than under Adam-A (see also Figure 11 in the appendix). Instead, from the left panel we can observe that the parameter divergence of Adam-WB in the last fully-connected layer is bigger than that of Adam-A at the very early rounds (as we expected), but soon it is abnormally sharply reduced over rounds; this is considered the steep fall phenomenon. The middle and the right plots of the figure also show the steep fall phenomenon in the last fullyconnected layer, with respect to network width and whether to use Batch Normalization, respectively. In the case of the NetC models, NetC-Baseline, NetC-Wider, and NetC-Widest use the global average pooling, the max pooling with stride 4, and the max pooling with stride 2, respectively, after the last convolutional layer; the number of neurons in the output layer becomes 2560, 10240, and 40960, respectively (see also Appendix A.1 for their detailed architecture). Under the Non-IID setting, the corresponding test accuracy was found to be 64.06%, 72.61%, and 73.64%, respectively, in order of the degree of wideness. In addition, we can see that under Non-IID, Batch Normalization 9 yields not only big parameter divergence (especially before the first learning rate drop) but also the steep fall phenomenon; the corresponding test accuracy was seen to be very low (see Table 3). The failure of the Batch Normalization stems from that the dependence of batchnormalized hidden activations makes each learner's update too overfitted to the distribution of their local training data. Batch Renormalization, by relaxing the dependence, yields a better outcome; however, it still fails to exceed the performance of the baseline due to the significant parameter divergence. To explain the impact of the steep fall phenomenon in test accuracy, we provide Figure 5, which indicates that the loss landscapes for the failure cases (e.g., Adam-WB and with Batch Normalization) commonly show sharper minima that leads to poorer generalization (Hochreiter & Schmidhuber, 9 For its implementations into the considered federated learning algorithm, we let the server get the proper moving variance by 1997;), and the minimal value in the bowl is relatively greater. 10 Here it is also observed that going into sharp minima starts even in early rounds such as 25th. Excessively high training loss of local updates. The final cause that we consider for the failure cases is excessively high training loss of local updates. For instance, from the left plot of Figure 6, we see that under the Non-IID setting, NetB-Baseline gives much higher training loss than the other models. Here we note that for the NetB-Baseline model, the global average pooling is applied after the last convolutional layer, and the number of neurons in the first fully-connected layer thus becomes 256 · 256; on the other hand, NetB-Wider and NetB-Widest use the max pooling with stride 4 and 2, which make the number of neurons in that layer become 1024 · 256 and 4096 · 256, respectively (see also Appendix A.1 for their details). The experimental were shown that NetB-Baseline has notably lower test accuracy (see Table 4). We additionally remark that for NetBBaseline, very high losses are observed under the IID setting, and their values even are greater than in the non-IID case; however, note that one have to be aware that local updates are extremely easy to be overfitted to each training dataset under non-IID data environments, thus the converged training losses being high is more critical than the IID cases. The middle and the right plot of the figure show the excessive training loss under the non-IID setting when applying the weight decay factor of 0.0005 and the data augmentation, respectively. In the cases of the high level of weight decay, the severe performance degradation appears compared to when the levels are low (i.e., ≤ 0.0001) as already discussed. In addition, we observed that with Nesterov momentum SGD, the data augmentation yields a diminishing return in test accuracy (i.e., with the data augmentation we can achieve +3.36% under IID, but −0.16% is obtained under non-IID, compared to when it is not applied); with Adam the degree of the diminishment becomes higher (refer to Table 12 in the appendix). In the data augmentation cases, judging from that the 10 Based on , the visualization of loss surface was conducted by L(α, β) = (θ * + αδ + βγ), where θ * is a center point of the model parameters, and δ and γ is the orthogonal direction vectors. parameter divergence values are not so different between with and without it, we can identify that the performance degradation stems from the high training loss (see Figures 30 and 31 in the appendix). Here we additionally note that unlike on the CIFAR-10, in the experiments on SVHN it was seen that the generalization effect of the data augmentation is still valid in test accuracy (see Table 12). In this paper, we explored the effects of various hyperparameter optimization strategies for optimizers, network depth/width, and regularization on federated learning of deep networks. Our primary concern in this study was lied on non-IID data, in which we found that under non-IID data settings many of the probed factors show somewhat different behaviors compared to under the IID setting and vanilla training. To explain this, a concept of the parameter divergence was utilized, and its origin was identified both empirically and theoretically. We also provided the internal reasons of our observations with a number of the experimental cases. In the meantime, the federated learning has been vigorously studied for decentralized data environments due to its inherent strength, i.e., high communication-efficiency and privacy-preservability. However, so far most of the existing works mainly dealt with only IID data, and the research to address non-IID data has just entered the beginning stage very recently despite its high real-world possibility. Our study, as one of the openings, handles the essential factors in the federated training under the non-IID data environments, and we expect that it will provide refreshing perspectives for upcoming works. A EXPERIMENTAL DETAILS In the experiments, we consider CNN architectures, as illustrated in Figure 7. In the network configurations, three groups of 3 × 3 convolutional layers are included that have 16 · m, 128, and 256 output channels, respectively; n denotes the number of the layers in each convolutional group. The first two groups are followed by 3 × 3 max pooling with stride 2; the last convolutional layer is followed by either the 3 × 3 max pooling with stride s or the global average pooling. In the case of fully-connected layers, we use two types of the stacks: (i) three layers, of which the output sizes are 256 · u, 256, and 10, respectively; and (ii) a single layer, of which the output size is 10. In addition, we use the ReLU and the softmax activation for the hidden weight layers and the output layer, respectively. Table 6 summarizes the network models used in the experiments. In the experiments, we initialize the network models to mostly follow the truncated normal distribution with a mean of 0 based on , however we fix the standard deviation to 0.05 for the first convolutional group and the last fully-connected layer. For training, minibatch stochastic optimization with cross-entropy loss is considered. Specifically, we use pure SGD, Nesterov momentum SGD , and Adam as optimization methods; initial learning rates are set to 0.05, 0.01, and 0.001, respectively for each optimizer. We drop the learning rate by 0.1 at 50% and 75% of the total training iterations, respectively. Regarding the environmental configurations, we predetermine each of learners' local training dataset in a random seed; the training examples are allocated so that they do not overlap between the learners. To report the experimental , we basically considered to run the trials once, but as for unstable ones in the preliminary tests, we chose the middle of several runs. In every plot, the values are plotted at each round. In relation to the federated learning under non-IID data, so far there have been several works for providing theoretical bounds to explain how does the degree of the non-IIDness of decentralized data affect the performance, with respect to its degree (e.g., ). Inspired by them, here we further study how does the non-IIDness make the model parameters of each learner diverged. In this analysis, we consider training deep networks for multi-class classification. Based on the notations in Algorithm 1, the SGD update of learner k at round t + 1 is given as where f q (x; w) is the posterior probability for class q ∈ Q (Q is the label space), obtained from model parameters w with data examples (x, y), and p k (y = q) is the probability that the label of a data example in P k is q. In this equation, w is the model parameters after the τ -th local iterations in the round t + 1 (R is the number of local iterations of each learner per round). Herein we note that w t+0 k (w t) is the global model parameters received from the server at the round t + 1; we use the term to distinguish it from the term w t k (which indicates the local update that has sent back to the server at round t). Then, by the linearity of the gradient, we obtain where d q denote the neurons, in the (dense) output layer of the model w, that are connected to the output node for class q. with the fixed k. At round t+1, suppose that for learner where (a q). Then, we can get From this, we can identify that the parameter difference, with the fixed q. At round t+1, suppose that for class q, Then, similar with Equation 1, we can have C THE COMPLETE EXPERIMENTAL In this section we provide our complete experimental . Before the main statement, we first note that in the following figures, C ij denotes the j-th convolutional layer in the i-th group, and F j denotes the j-th fully-connected layer (in relation, refer to Appendix A.1). In addition, we remind that in this paper "vanilla" training refers to non-distributed training with a single machine, using the whole data examples; for the vanilla training, we trained the networks for 100 epochs. Here we investigate the effect of optimizers. We importantly note that both momentum SGD and Adam require the additional variables related to momentum as well as weights and biases; the terms (optimizer name)-A and (optimizer name)-WB are used to refer to the parameter averaging being performed for all the variables and only for weights & biases, respectively. The experimental are provided in Table 7 and Figures 10 and 11. From the table, interestingly we can notice that under the non-IID data setting, there exists a huge performance gap between Adam-A and Adam-WB (≈ 7%), unlike the momentum SGD trials. At the initial steps of this study, we had thought that the poor performance of Adam-WB would be from the following: Since Adam requires twice as many momentum variables as momentum SGD, extreme parameter divergence could appear if they are not averaged together with weights and biases. However, unlike our expectations, the parameter divergence values under the Adam-WB was seen to be similar or even smaller than under Adam-A. Nevertheless, we can observe the followings for the non-IID cases: First, the parameter divergence of Adam-WB in F 3 is bigger than that of Adam-A at the very early rounds (as we expected), but soon it is abnormally sharply reduced over rounds; this can be considered the steep fall phenomenon. Second, Adam-WB leads to higher training loss of each learner. We guess that these two caused the severe degradation of Adam-WB in test accuracy. Here we investigate the effect of network depth. Since deepening networks also indicates that there becomes having more parameters to be averaged in the considered federated learning algorithm, we had predicted especially under non-IID data settings that depending on their depth, it would yield bigger parameter divergence in the whole model and the consequent diminishing returns compared to under the vanilla training and the IID data setting; the test accuracy show it as expected (see Table 8). 11 Moreover, it is also seen from Figure 12 that parameter divergence increases also qualitatively (i.e., in a layer level) under the non-IID data setting, as the number of convolutional layers increases. Note that for C 21 and C 31, the divergence pattern is ed as opposed to that of C 11 and F 3; however, the values of C 11 and F 3 would be more impactful as mentioned in Footnote 8. We additionally remark from the figure that the sharp reduction of parameter divergence (in the convolutional layers) at the very early rounds when using NetA-Deepest indicates the parameter averaging algorithm did not work properly. Correspondingly, the test accuracy values in the early period were seen to be not much different from the initial one. Following the previous subsection, from now on we investigate the effect of network width. Contrary to the in the Section C.2, it is seen from Table 9 that widening networks provides positive effects for the considered federated learning algorithm under the non-IID data setting. Especially, one can see that compared to the max pooling trials, while the global average pooling yields higher test accuracy in the vanilla training (with the minibatch size of 50), its performance gets significantly worse under the non-IID data setting (remind that NetB-Baseline and NetC-Baseline use the global average pooling after the last convolutional layer). Focusing on the NetC models, we here make the following observations for the non-IID data setting from Figures 15, 18, and 21: First, the considered federated learning algorithm provides bigger parameter divergence in F 1 as its width decreases (note that each input size of F 1 is 256, 1024, and 4096 for NetC-Baseline, NetC-Wider, and NetC-Widest, respectively), especially during the beginning rounds (e.g., for the NMom-A case, until about 50 rounds). Unlike in Section C.2, here we can identify that even though the parameter size of is the smallest under the global averaging pooling, it rather yields the biggest qualitative parameter divergence. Second, the steep fall phenomenon appears in F 1 for the NetC- Baseline case. Third, the global average pooling gives too high training loss of each learner. All the three observations fit well into the failure of the global average pooling. We additionally note that when using NetC-Baseline, the under the IID data setting shows very high loss values; this leads to diminishing returns for the pure SGD and the NMom-A cases, compared to the vanilla training with the minibatch size of 50. However, the corresponding degradation rate is seen to be much higher under the non-IID data setting. This is because the local updates are extremely easy to be overfitted to the training data under the non-IID data setting; thus the converged training losses being high becomes much more critical. Here we investigate the effect of weight decay. From Table 10, it is seen that under the non-IID data setting we should apply much smaller weight decay for the considered federated learning algorithm than under the vanilla training or the IID data setting. For its internal reason, Figures 22, 23, and 24 show that under the non-IID data setting, the considered federated learning algorithm not only converges to too high training loss (of each learner) but also causes excessive parameter divergence when the weight decay factor is set to 0.0005. Here we note that if a single iteration is considered for each learner's local update per round, the corresponding parameter divergence will be of course the same without regard to degree of weight decay. However, in our experiments, the great number of local iterations per round (i.e., 100) made a big difference of the divergence values under the non-IID data setting; this eventually yielded the accuracy gap. In addition, we further observe under the non-IID data setting that even with the weight decay factor of 0.0005, the test accuracy increases similarly with its smaller values at very early rounds, in which the norm values of the weights are relatively much smaller. Moreover we also conducted additional experiments for the related regularization techniques, FedProx (b). Under the FedProx, in order to make local updates do not deviate excessively from the current global model parameters, at each round t each learner uses the following surrogate loss function that adds a proximal term to the original objective function: (w) + µ 2 w − w t−1 2. Figure 8 shows the experimental ; as seen from the figure, in our implementation FedProx did not provide dramatic improvement in final accuracy, but we can observe that it could yield not only lower parameter divergence but also faster convergence speed (especially before the first learning rate drop). One can find the corresponding complete in Figure 25. Here we investigate the effect of Batch Normalization. For its implementations into the considered federated learning algorithm, we let the server get the proper moving variance by − E φ 2 at each round, by allowing each learner k collect E φ 2 k as well). It is natural to take this strategy especially under the non-IID data setting; otherwise, a huge problem would arise due to bad approximation of the moving statistics. Also, it is additionally remarked that for Batch Renormalization we simply used α = 0.01, r max = 2, and d max = 2 in the experiments (see for the description of the three hyperparameters). Table 11 that under the non-IID data setting, the performance significantly gets worse if Batch Normalization is employed to the baseline; this would be rooted in that the dependence of batch-normalized hidden activations makes each learner's update too overfitted to the distribution of their local training data. The consequent bigger parameter divergence is observed in Figures 26, 27, and 28. On the contrary, Batch Renormalization, by relaxing the dependence, yields a better outcome; although its parameter divergence is seen greater in some layers than under Batch Normalization, it does not lead to the steep fall phenomenon while the Batch Normalization does in F 3. Nevertheless, the Batch Renormalization was still not able to exceed the performance of the baseline due to the significant parameter divergence. In the implementation of data augmentation, we used random horizontal flipping, brightness & contrast adjustment, and 24×24 cropping & resizing in the pipeline. From Table 12, we identify that under the non-IID data setting, the data augmentation yields diminishing returns for the PMom-A, NMom-A, and Adam-A cases on CIFAR-10, compared to under the IID data setting; under Adam-A, especially it gives even a worse outcome. However, it is seen that the corresponding parameter divergence is almost similar between with and without the data augmentation (refer to Figures 30 and 31). Instead, we are able to notice that the diminishing outcomes from the data augmentation had been eventually rooted in local updates' high training losses. Here we note that in the pure SGD case, very high training loss values are found as well under the IID data setting when the data augmentation was applied (see Figure 29); this leads to lower test accuracy compared to the baseline, 83.09 (83.20) 82.01 (82.11) 76.63 (76.68) similar to under the non-IID cases. Also, it is additionally noted that unlike on the CIFAR-10, in the experiments on SVHN it was observed that the generalization effect of the data augmentation is still valid in test accuracy. In the experiments, we employed Dropout with the rates 0.2 and 0.5 for convolutional layers and fully-connected layers, respectively. The show that under the non-IID data setting, the Dropout provides greater parameter divergence compared to the baselines, especially in F 3 (see Figures 32, 33, and 34); this leads to diminishing returns for the PMom-A and NMom-A cases on CIFAR-10, compared to under the IID data setting. However, we can observe from Table 13 that the effect of the Dropout is still maintained positive for the rest of the cases. As remarked in , since the federated learning do not require centralizing local data, data unbalancedness (i.e., each learner has various numbers of local data examples) would be also naturally assumed in the federated learning along with non-IIDness. In relation, we also conducted the experiments under the unbalanced cases. Table 14 summarizes the considered unbalanced data settings; they were constructed similarly to (b) so that the number of data examples per learner follows a power law. The experimental under the unbalanced settings are summarized in Table 15. From the table, it is observed that our findings in Section 4.1 are still valid under the unbalanced data settings. In addition, we can also see that for the unbalanced cases, the performance under Non-IID setting is worse mostly than that of balanced cases while they show similar values under the IID data setting; this indicates that the negative impact of data unbalancedness is not as great as that of the nonIIDness, but it becomes much bigger when the two are combined.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We investigate the internal reasons of our observations, the diminishing effects of the well-known hyperparameter optimization methods on federated learning from decentralized non-IID data.

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Deep learning models are known to be vulnerable to adversarial examples. A practical adversarial attack should require as little as possible knowledge of attacked models T. Current substitute attacks need pre-trained models to generate adversarial examples and their attack success rates heavily rely on the transferability of adversarial examples. Current score-based and decision-based attacks require lots of queries for the T. In this study, we propose a novel adversarial imitation attack. First, it produces a replica of the T by a two-player game like the generative adversarial networks (GANs). The objective of the generative model G is to generate examples which lead D returning different outputs with T. The objective of the discriminative model D is to output the same labels with T under the same inputs. Then, the adversarial examples generated by D are utilized to fool the T. Compared with the current substitute attacks, imitation attack can use less training data to produce a replica of T and improve the transferability of adversarial examples. Experiments demonstrate that our imitation attack requires less training data than the black-box substitute attacks, but achieves an attack success rate close to the white-box attack on unseen data with no query. Deep neural networks are often vulnerable to imperceptible perturbations of their inputs, causing incorrect predictions . Studies on adversarial examples developed attacks and defenses to assess and increase the robustness of models, respectively. Adversarial attacks include white-box attacks, where the attack method has full access to models, and black-box attacks, where the attacks do not need knowledge of models structures and weights. White-box attacks need training data and the gradient information of models, such as FGSM (Fast Gradient Sign Method) , BIM (Basic Iterative Method) (a) and JSMA (Jacobian-based Saliency Map Attack) (b). However, the gradient information of attacked models is hard to access, the white-box attack is not practical in real-world tasks. Literature shows adversarial examples have transferability property and they can affect different models, even the models have different architectures (; a;). Such a phenomenon is closely related to linearity and over-fitting of models (; ; ; Tramèr et al., 2018). Therefore, substitute attacks are proposed to attack models without the gradient information. Substitute black-box attacks utilize pre-trained models to generate adversarial examples and apply these examples to attacked models. Their attack success rates rely on the transferability of adversarial examples and are often lower than that of white-box attacks. Black-box score-based attacks a; b) do not need pre-trained models, they access the output probabilities of the attacked model to generate adversarial examples iteratively. Black-box decisionbased attacks (; ;) require less information than the score-based attacks. They utilize hard labels of the attacked model to generate adversarial examples. Adversarial attacks need knowledge of models. However, a practical attack method should require as little as possible knowledge of attacked models, which include training data and procedure, models weights and architectures, output probabilities and hard labels . The disadvantage of current substitute black-box attacks is that they need pre-trained substitute models trained by the same dataset with attacked model T (; ; a) or a number of images to imitate the outputs of T to produce substitute networks. Actually, the prerequisites of these attacks are hard to obtain in real-world tasks. The substitute models trained by limited images hardly generate adversarial examples with well transferability. The disadvantage of current decision-based and score-based black-box attacks is that every adversarial example is synthesized by numerous queries. Hence, developing a practical attack mechanism is necessary. In this paper, we propose an adversarial imitation training, which is a special two-player game. The game has a generative model G and a imitation model D. The G is designed to produce examples to make the predicted label of the attacked model T and D different, while the imitation model D fights for outputting the same label with T. The proposed imitation training needs much less training data than the T and does not need the labels of these data, and the data do not need to coincide with the training data. Then, the adversarial examples generated by D are utilized to fool the T like substitute attacks. We call this new attack mechanism as adversarial imitation attack. Compared with current substitute attacks, our adversarial imitation attack requires less training data. Score-based and decision-based attacks need a lot of queries to generate each adversarial attack. The similarity between the proposed method and current score-based and decision-based attacks is that adversarial imitation attack also needs to obtain a lot of queries in the training stage. The difference between these two kinds of attack is our method do not need any additional queries in the test stage like other substitute attacks. Experiments show that our proposed method achieves state-of-the-art performance compared with current substitute attacks and decision-based attack. We summarize our main contributions as follows: • The proposed new attack mechanism needs less training data of attacked models than current substitute attacks, but achieves an attack success rate close to the white-box attacks. • The proposed new attack mechanism requires the same information of attacked models with decision attacks on the training stage, but is query-independent on the testing stage. Adversarial Scenes Adversarial attacks happen in two scenes, namely the white-box and the black-box settings. In the white-box settings, the attack method has complete access to attacked models, such as models internal, training strategy and data. While in the black-box settings, the attack method has little knowledge of attacked models. The black-box attack utilizes the transferability property of adversarial examples, only needs the labeled training data, but its attack success rate is often lower than that of the white-box attack method if attacked models have no defense. Actually, attack methods requiring lots of prior knowledge of attacked models are difficult to apply in practical applications . Adversarial Attacks Several methods for generating adversarial examples were proposed. proposed a one-step attack called FGSM. On the basis of the FGSM, Kurakin et al. (2017a) came up with BIM, an iterative optimization-based attack. Another iterative attack called DeepFool aims to find an adversarial example that would cross the decision boundary. Carlini & Wagner (2017b) provided a stronger attack by simultaneously minimizing the perturbation and the L F norm of the perturbation. generated adversarial examples through decoupling the direction and the norm of the perturbation, which is also constrained by the L F norm. showed that targeted adversarial examples hardly have transferability, they proposed ensemble-based methods to generate adversarial examples having stronger transferability. proposed a practical black-box attack which accesses the hard label to train substitute models. For score-based attacks, proposed the zeroth-order based attack (ZOO) which uses gradient estimates to attack a black-box model. Ilyas et al. (2018b) improves the way to estimate gradients. proposed a simple black-box score-based attack on DCT space. For decision-based attacks, first proposed decision-based attacks which do not rely on gradients. and improve the query efficiency of the decision-based attack. Adversarial Defenses To increase the robustness of models, methods for defending against adversarial attacks are being proposed. Adversarial training (; ; b; Tramèr et al., 2018) can be considered as a kind of data augmentation. It applies adversarial examples to the training data, ing in a robust model against adversarial attacks. Defenses based on gradient masking (Tramèr et al., 2018;) provide robustness against optimization-based attacks. Random transformation (a; ;) on inputs of models hide the gradient information, eliminate the perturbation. proposed thermometer encoding based on one-hot encoding, it applied a nonlinear transformation to inputs of models, aiming to reduce the linearity of the model. However, most defenses above are still unsafe against some attacks (a;). showed that defenses based on gradient masking actually are unsafe against attacks. Instead, some researchers focus on detecting adversarial examples. Some use a neural network (; ;) to distinguish between adversarial examples and clean examples. Some achieve statistical properties (; ; ; ; to detect adversarial examples. In this section, we introduce the definition of adversarial examples and then propose a new attack mechanism based on adversarial imitation training. X refers to the samples from the input space of the attacked model T, y true refers to the true labels of the samples X. T (y|X, θ) is the attacked model parameterized by θ. For a non-targeted attack, the objective of the adversarial attack can be formulated as: where the and r are perturbation of the sample and upper bound of the perturbation, respectively. To guarantee that is imperceptible, r is set to a small value in applications. X = X + are the adversarial examples which can fool the attacked model T. θ refers to the parameters of the model T. For white-box attacks, they obtain gradient information of T to generate adversarial examples, and attack the T directly. For substitute attacks, they generate adversarial examples from a substitute model T, and transfer the examples to attack the T. The key point of a successful attack is the transferability of the adversarial examples. To improve the transferability of adversarial examples and avoid output query, we utilize an imitation network to imitate the characteristics of the T by accessing its output labels to improve the transferability of adversarial examples, which are generated by the imitation network. After the adversarial imitation training, adversarial examples generated by imitation network do not need additional query. In the next subsection, we introduce the proposed adversarial imitation training and imitation attack. Inspired by the generative adversarial network (GAN), we use the adversarial framework to copy the attacked model. We propose a two-player game based adversarial imitation training to replicate the information of attacked model T, which is shown in Figure 1. To learn the characteristics of T, we define an imitation network D, and train it using disturbed input X = G(X) + X and the corresponding output label y T (X) of attacked model. X denotes training samples here. The role of G is to create new samples that y T (X) = y D (X). Thus, D, G and T play a special two-player game. To simplify the expression but without loss of generality, we just analyze the case of binary classification. The value function of players can be presented as: The proposed adversarial imitation attack. For the training stage, the objective of G is to generate samples X = G(X) + X and let For the testing stage, the imitation model D is utilized to generate adversarial examples to attack T. Note that the T is equivalent to the referee of this game. The two players G and D optimize their parameters based on the output y T (X). We suppose that adversarial perturbation ≤ r 1, and, our imitation attack will have the same success rate as the white-box attack without the gradient information of T. Therefore, for a well-trained imitation network, adversarial examples generated by D have strong transferability for T. A proper upper bound (r 2 ≥ r 1) of G(X) is the key points for training an efficient imitation network D. Especially in targeted attacks (T outputs the specified wrong label), if the characteristics of D is more similar to that of the attacked model, the transferability of adversarial examples will be stronger. In the training stage, the loss function of D is J D = V G,D. Because G is more hard to train than D, sometimes the ability of D is much stronger than G, so the loss of G fluctuates during the training stage. In order to maintain the stability of training, the loss function of G is designed as J G = e −V G,D. Therefore, the global optimal imitation network D is obtained if and only if ∀ X, D(X) = y T (X). At this point, J D = 0 and J G = e 0 = 1. The loss of G is always in a controllable value in training stage. As we discussed before, if r 1 ≤ r 2, the adversarial examples generated by a well-trained D have strong transferability for T. Because the attack perturbation of adversarial examples is set to be a small value, we constrain the G(X) in training stage to limit the search space of G, which can reduce the number of queries efficiently. For training methodology, we alternately train the G and D in every mini-batch, and use L 2 penalty to constrain the search space of G. The procedure is shown in algorithm 1. Algorithm 1 Mini-batch stochastic gradient descent training of imitation network. 1: for number of training iterations do 2: Sample mini-batch of m examples{X,..., X (m) } from training set. Update the imitation model by descending its loss function: 4: Update the generative model by descending its loss function: 6: e −V G,D + α G(X). In adversarial attacks, when the optimal D is obtained, the adversarial examples generated by D are utilized to attack T. In this subsection, we introduce the settings for our experiments. Datasets: we evaluate our proposed method on MNIST and CIFAR-10 . Because we need to use data different with the training set of T to train the imitation network, we divided the test sets (10k) from MNIST and CIFAR-10 into two parts. One part contains 9500 images for training and another part contains 500 images for evaluating the attack performance. Model architecture and attack method: in order to get the information of the attacked model T as little as possible, we only utilize the output label (not the probability) of the T to train the imitation network. The imitation network has no prior knowledge of the attacked model, which means it does not load any pre-trained model in experiments. For the experiments on MNIST, we design 3 different network architectures with different capacity (small network, medium network and large network) for evaluating the performance of our imitation attack with models having different capacity. We utilize the pre-trained medium network and VGG-16 as the attacked model on MNIST and CIFAR-10, respectively. In order to compare the success rate of the proposed imitation attack with current substitute attack, we utilize 4 attack methods, FGSM, BIM, projected gradient descent (PGD) , C&W to generate adversarial examples. For testing, we use AdverTorch library to generate adversarial examples. On the other hand, for comparing the performance of our method with current decision-based attacks and score-based attacks, we utilize Boundary Attack , HSJA Attack , SimBA-DCT Attack as comparison methods. Note that score-based attacks require output probabilities of T, which contain much more information than labels. Evaluation criteria: the goals of non-targeted attack and targeted attack are to lead the attacked model to output a wrong label and a specific wrong label, respectively. In non-targeted attacks, we only generate adversarial examples on the images classified correctly by the attacked model. In targeted attacks, we only generate adversarial examples on the images which are not classified to the specific wrong labels. The success rates of adversarial attack are calculated by n/m, where n and m are the number of adversarial examples which can fool the attacked model and the total number of adversarial examples, respectively. In this subsection, we utilize the proposed adversarial imitation training to train imitation models and evaluate the performance in terms of attack success rate. To compare our method with substitute attack, We utilize the medium network and VGG-16 as attacked models on MNIST and CIFAR-10, respectively. Then we use the same train dataset to obtain a pre-trained large network (the architecture is also in Table 9) and ResNet-50 to generate substitute adversarial examples. We obtain imitation networks by using the proposed adversarial imitation training. The large network and ResNet-16 are used as the model architectures of imitation networks on MNIST and CIFAR-10, respectively. The imitation models are only trained by 9500 samples of the test dataset, which are much less than the training sets of MNIST (60000 samples) and CIFAR-10 (50000 samples). The of experiments are evaluated on the other 500 samples of the test dataset and are shown in Table 1 and Table 2. The success rates of the proposed imitation attack far exceed the success rates of the substitute attack in all experiments. The experiments of Table 1 and Table 2 show that the proposed new attack mechanism needs less training images than substitute attacks, but achieves an attack success rate close to the white-box attack. The experiments indicate that the adversarial samples generated by a well-trained imitation model have a higher transferability than the substitute attack. To compare our method with decision-based and score-based attacks, we evaluate the performance of these attacks. We utilize 9500 images from the test set of MNIST and CIFAR-10 to train the imitation network, and use other 500 images from the test set as unseen data for our method. The other decision-based and score-based attacks are evaluated on the test set of MNIST and CIFAR-10 dataset. Note that score-based attacks require much more information (output probabilities) than decision-based attacks (output labels). The on MNIST and CIFAR-10 are shown in Table 3 and Table 4, respectively. Because our imitation attack only needs queries on the training stage, we evaluate performances of our method on its train and unseen sets. We set the iteration of adversarial imitation training to 1800, so the average number of query per image is 1800. We utilize BIM attack to generate adversarial examples as our imitation attack in this experiment. This experiment shows that our imitation attack achieves state-of-the-art performance in decisionbased methods. Even compared with the score-based attack, our imitation attack outperforms it in terms of perturbation distance and attack success rate. More importantly, it also obtains good on unseen data, which indicates our imitation attack can be applied to query-independent scenarios. In the above subsection, we utilize a more complex network than the attacked model to replicate the attacked model. In this subsection, we study the impact of model capacity on the ability of imitation. To evaluate the imitation performance of the network with less capacity than the attacked model, we train the small network, medium network, and large network to imitate the pre-trained medium network on MNIST dataset, and train VGG-13, VGG-16 and ResNet-50 to imitate VGG-16 on CIFAR-10. The performance of models with different capacities are shown in Table 5 and 6. The show that an imitation model with a lower capacity than the attacked model can also achieve a good imitation performance. Attack success rates of all imitation models far exceed the substitute attacks in Table 1 and 2. Most experiments show that the larger capacity the imitation network has, the higher attack success rate it can achieve (FGSM, BIM, PGD in MNIST and BIM in CIFAR-10). However, some experiments show models having a larger capacity do not have a higher attack success rate (FGSM and PGD in CIFAR-10). We surmise that the performance of imitating an attacked model is not only influenced by the capacity of the imitation model D, but also influenced by the capability of the G. In this subsection, we only use 200 images (20 samples per class) to train the imitation networks and discuss characteristics of the model replication. We train the imitation network using 200 images from MNIST and CIFAR-10 test set, and compare its performance with Practical Attack on other images from MNIST and CIFAR-10 test set. The are shown in Table 7 and 8. The practical attack uses the output labels of attacked models to train substitute models under the scenario, which they can make an infinite number of queries for attacked models. It is hard to generate adversarial examples to fool the attacked models by limited training samples. Note that what the substitute models imitate is the response for perturbations of the attacked model. A substitute model that can generate adversarial examples with a higher attack success rate is a better replica. Our adversarial imitation attack can produce a substitute model with much higher classification accuracy and attack success rate than Practical Attack for both non-targeted and targeted attacks in this scenario (with an infinite number of queries). We also show the performances of these two methods with limited query numbers in Figure 6. Additionally, the imitation model with low classification accuracy still can produce Practical adversarial attacks should have as little as possible knowledge of attacked model T. Current black-box attacks need numerous training images or queries to generate adversarial images. In this study, to address this problem, we combine the advantages of current black-box attacks and proposed a new attack mechanism, imitation attack, to replicate the information of the T, and generate adversarial examples fooling deep learning models efficiently. Compared with substitute attacks, imitation attack only requires much less data than the training set of T and do not need the labels of the training data, but adversarial examples generated by imitation attack have stronger transferability for the T. Compared with score-based and decision-based attacks, our imitation attack only needs the same information with decision attacks, but achieves state-of-the-art performances and is query-independent on testing stage. Experiments showed the superiority of the proposed imitation attack. Additionally, we observed that deep learning classification model T is easy to be stolen by limited unlabeled images, which are much fewer than the training images of T. In future work, we will evaluate the performance of the proposed adversarial imitation attack on other tasks except for image classification. A NETWORK ARCHITECTURES Figure 2 and Figure 3. The experiments show that adversarial examples generated by the proposed imitation attack can fool the attacked model with a small perturbation.

Represent the TLDR for finding the corresponding scientific article

A novel adversarial imitation attack to fool machine learning models.

Batch normalization reduces adversarial robustness, as well as general robustness in many cases, particularly to noise corruptions.


Question: Represent the article for finding a single-sentence TLDR

Question: Stochastic Gradient Descent (SGD) methods using randomly selected batches are widely-used to train neural network (NN) models. Performing design exploration to find the best NN for a particular task often requires extensive training with different models on a large dataset, which is very computationally expensive. The most straightforward method to accelerate this computation is to distribute the batch of SGD over multiple processors. However, large batch training often times leads to degradation in accuracy, poor generalization, and even poor robustness to adversarial attacks. Existing solutions for large batch training either do not work or require massive hyper-parameter tuning. To address this issue, we propose a novel large batch training method which combines recent in adversarial training (to regularize against ``sharp minima'') and second order optimization (to use curvature information to change batch size adaptively during training). We extensively evaluate our method on Cifar-10/100, SVHN, TinyImageNet, and ImageNet datasets, using multiple NNs, including residual networks as well as compressed networks such as SqueezeNext. Our new approach exceeds the performance of the existing solutions in terms of both accuracy and the number of SGD iterations (up to 1\% and $3\times$, respectively). We emphasize that this is achieved without any additional hyper-parameter tuning to tailor our method to any of these experiments. Finding the right NN architecture for a particular application requires extensive hyper-parameter tuning and architecture search, often times on a very large dataset. The delays associated with training NNs is often the main bottleneck in the design process. One of the ways to address this issue to use large distributed processor clusters; however, to efficiently utilize each processor, the portion of the batch associated with each processor (sometimes called the mini-batch) must grow correspondingly. In the ideal case, the hope is to decrease the computational time proportional to the increase in batch size, without any drop in generalization quality. However, large batch training has a number of well known draw backs. These include degradation of accuracy, poor generalization, and poor robustness to adversarial perturbations BID17 BID36.In order to address these drawbacks, many solutions have been proposed BID14 BID37 BID7 BID29 BID16. However, these methods either work only for particular models on a particular dataset, or they require massive hyperparameter tuning, which is often times not discussed in the presentation of . Note that while extensive hyper-parameter turning may in good tables, it is antithetical to the original motivation of using large batch sizes to reduce training time. One solution to reduce the brittleness of SGD to hyper-parameter tuning is to use second-order methods. Full Newton method with line search is parameter-free, and it does not require a learning rate. This is achieved by using a second-order Taylor series approximation to the loss function, instead of a first-order one as in SGD, to obtain curvature information. BID25; BID34 BID2 show that Newton/quasi-Newton methods outperform SGD for training NNs. However, their re-sults only consider simple fully connected NNs and auto-encoders. A problem with second-order methods is that they can exacerbate the large batch problem, as by construction they have a higher tendency to get attracted to local minima as compared to SGD. For these reasons, early attempts at using second-order methods for training convolutional NNs have so far not been successful. Ideally, if we could find a regularization scheme to avoid local/bad minima during training, this could resolve many of these issues. In the seminal works of El Ghaoui & BID9; BID33, a very interesting connection was made between robust optimization and regularization. It was shown that the solution to a robust optimization problem for least squares is the same as the solution of a Tikhonov regularized problem BID9. This was also extended to the Lasso problem in BID33. Adversarial learning/training methods, which are a special case of robust optimization methods, are usually described as a min-max optimization procedure to make the model more robust. Recent studies with NNs have empirically found that robust optimization usually converges to points in the optimization landscape that are flatter and are more robust to adversarial perturbation BID36.Inspired by these , we explore whether second order information regularized by robust optimization can be used to do large batch size training of NNs. We show that both classes of methods have properties that can be exploited in the context of large batch training to help reduce the brittleness of SGD with large batch size training, thereby leading to significantly improved . In more detail, we propose an adaptive batch size method based on curvature information extracted from the Hessian, combined with a robust optimization method. The latter helps regularize against sharp minima, especially during early stages of training. We show that this combination leads to superior testing performance, as compared to the proposed methods for large batch size training. Furthermore, in addition to achieving better testing performance, we show that the total number of SGD updates of our method is significantly lower than state-of-the-art methods for large batch size training. We achieve these without any additional hyper-parameter tuning of our algorithm (which would, of course, have helped us to tailor our solution to these experiments). Here is a more detailed itemization of the main contributions of this work:• We propose an Adaptive Batch Size method for SGD training that is based on second order information, computed by backpropagating the Hessian operator. Our method automatically changes the batch size and learning rate based on Hessian information. We state and prove a that this method is convergent for a convex problem. More importantly, we empirically test the algorithm for important non-convex problems in deep learning and show that it achieves equal or better test performance, as compared to small batch SGD (We refer to this method as ABS).• We propose a regularization method using robust training by solving a min-max optimization problem. We combine the second order adaptive batch size method with recent of BID36, which show that robust training can be used to regularize against sharp minima. We show that this combination of Hessian-based adaptive batch size and robust optimization achieves significantly better test performance with little computational overhead (we refer to this Adaptive Batch Size Adversarial method as ABSA).• We test the proposed strategies extensively on a wide range of datasets (Cifar-10/100, SVHN, TinyImageNet, and ImageNet), using different NNs, including residual networks. Importantly, we use the same hyper-parameters for all of the experiments, and we do not perform any kind of tuning of our hyper-parameters to tailor our . The empirical show the clear benefit of our proposed method, as compared to the state-of-the-art. The proposed algorithm achieves equal or better test accuracy (up to 1%) and requires significantly fewer SGD updates (up to 5×).• We empirically show that we can use a block approximation of the Hessian operator (i.e. the Hessian of the last fewer layers) to reduce the computational overhead of backpropagating the second order information. This approximation is especially effective for deep NNs. While a number of recent works have discussed adaptive batch size or increasing batch size during training BID7 BID29 BID10 BID1, to the best of our knowledge this is the first paper to introduce Hessian information and adversarial training in adaptive batch size training, with extensive testing on many datasets. We believe that it is important for every work to state its limitations (in general, but in particular in this area). We were particularly careful to perform extensive experiments and repeated all the reported tests multiple times. We test the algorithm on models ranging from a few layers to hundreds of layers, including residual networks as well as smaller networks such as SqueezeNext. An important limitation is that second order methods have additional overhead for backpropagating the Hessian. Currently, most of the existing frameworks do not support (memory) efficient backpropagation of the Hessian (thus providing a structural bias against these powerful methods). However, the complexity of each Hessian matvec is the same as a gradient computation BID21. Our method requires Hessian spectrum, which typically needs ten Hessian matvecs (for power method iterations to reach a tolerance of 1e-2). Thus, the benefits that we show in terms of testing accuracy and reduced number of updates do come at a cost (see Table 3 for details). We measure this additional overhead and report it in terms of wall clock time. Furthermore, we (empirically) show that this power iteration needs to be done only at the end of every epoch, thus significantly reducing the additional overhead. Another limitation is that our theory only holds for convex problems (under certain smoothness assumptions). Proving convergence for non-convex setting requires more involved analysis. Recently, BID32 has provided interesting theoretical guarantees for AdaGrad in the non-convex setting. Exploring a similar direction for our method is of interest for future work. Another point is that adaptive batch size, prevents one from utilizing all of the processes, as compared to using large batch throughout the training. However, a large data center can handle and accommodate a growing number of requests for processor resources, which could alleviate this. Optimization methods based on SGD are currently the most effective techniques for training NNs, and this is commonly attributed to SGD's ability to escape saddle-points and "bad" local minima BID5.The sequential nature of weight updates in synchronous SGD limits possibilities for parallel computing. In recent years, there has been considerable effort on breaking this sequential nature, through asynchronous methods or symbolic execution techniques BID20. A main problem with asynchronous methods is reproducibility, which, in this case, depends on the number of processes used (; BID0 . Due to this issue, recently there have been attempts to increase parallelization opportunities in synchronous SGD by using large batch size training. With large batches, it is possible to distribute more efficiently the computations to parallel compute nodes BID11, thus reducing the total training time. However, large batch training often leads to sub-optimal test performance BID17 BID36 . This has been attributed to the observation that large batch size training tends to get attracted to local minima or sharp curvature directions, which are not robust to (possible) mismatch between training and testing curves BID17. A full understanding of this, however, remains elusive. There have been several solutions proposed for alleviating the problem with large batch size training. The first notable work here is BID14, where it was shown that by scaling the learning rate, it is possible to achieve the same testing accuracy for large batches. In particular, ResNet-50 model was tested on ImageNet dataset, and it was shown that the baseline accuracy could be recovered up to a batch size of 8192. However, this approach does not generalize to other networks such as AlexNet BID37, or other tasks such as NLP. In BID37, an adaptive learning rate method (called LARS) was proposed which allowed scaling training to a much larger batch size of 32K with more hyper-parameter tuning. Another notable work is Smith et al. FORMULA0 (and also BID7), which proposed a hybrid increase of batch size and learning rate to accelerate training. In this approach, one would select a strategy to "anneal" the batch size during the training. This is based on the idea that large batches contain less "noise," and that could be used much the same way as reducing learning rate during training. More recent work BID16; BID24 proposed mix-precision method to further explore the limit of large batch training. A recent study has shown that anisotropic noise injection could also help in escaping sharp minima . The authors showed that the noise from SGD could be viewed as anisotropic, with the Hessian as its covariance matrix. Injecting random noise using the Hessian as covariance was proposed as a method to avoid sharp minima. Another recent work by BID36 has shown that adversarial training (or robust optimization) could be used to "regularize" against these sharp minima, with preliminary showing superior testing performance as compared to other methods. The link between robust optimization and regularization is a very interesting observation that has been theoretically proved in the case of Ridge regression (El Ghaoui & BID9, and Lasso BID2 . BID26 ; BID27 used adversarial training and showed that the model training using robust optimization is often times more robust to perturbations, as compared to normal SGD training. Similar observations have been made by others BID30 BID13 . We consider a supervised learning framework where the goal is to minimize a loss function L(θ): DISPLAYFORM0 where θ are the model weight parameters, Z = X × Y is the training dataset, and l(z, θ) is the loss for a datum z ∈ Z. Here, X is the input, Y is the corresponding label, and N = |Z| is the cardinality of the training set. SGD is typically used to optimize Eqn. FORMULA0 by taking steps of the form: DISPLAYFORM1 where B is a mini-batch of examples drawn randomly from Z, and η t is the step size (learning rate) at iteration t. In the case of large batch size training, the batch size is increased to large values. views the learning rate and batch size as noise injected during optimization. Both a large learning rate as well as a small batch size can be considered to be equivalent to high noise injection. This is explained by modeling the behavior of NNs as a stochastic differential equation (SDE) of the following form: DISPLAYFORM2 where (t) is the noise injected by SGD (see BID28 for details). The authors then argue that the noise magnitude is proportional to g = η t (|Z| |B| − 1). For mini-batch |B| |Z|, the noise magnitude can be estimated as g ≈ η t |Z| |B|. Hence, in order to achieve the benefits from small batch size training, i.e., the noise generated by small batch training, the learning rate η t should increase proportionally to the batch size, and vice versa. That is, the same annealing behavior could be achieved by increasing the batch size, which is the method used by BID29.The need for annealing can be understood by considering a convex problem. When we get closer to a local minimum, a more accurate descent direction with less noise is preferable to a more noisy direction, since less noise helps converge to rather than oscillate around the local minimum. This explains the manual batch size and learning rate changes proposed in BID29 BID7. Ideally, we would like to have an automatic method that could provide us with such information and regularize against local minima with poor generalization. As we show next, this is possible through the use of second order information combined with robust optimization. In this section, we propose a method for utilizing second order information to adaptively change the batch size. We refer to this as the Adaptive Batch Size (ABS) method; see Alg. 1. Intuitively, using a larger batch size in regions where the loss has a "flatter" landscape, and using a smaller batch size in regions with a "sharper" loss landscape, could help to avoid attraction to local minima with poor generalization. This information can be obtained through the lens of the Hessian operator. -Learning rate lr, learning rate decay steps A, learning rate decay ratio ρ -Initial Batch Size B, minimum batch size Bmin, maximum batch size Bmax, input x, label y. -Eigenvalue decreasing ratio α, eigenvalue computation frequency n, i.e., after training n samples compute eigenvalue, batch increasing ratio β, duration factor κ, i.e., if we compute κ times Hessian but eigenvalue does not decrease, we would increase the batch size -If adversarial training is used, perturbation magnitude adv, perturbation ratio γ (γmax) of training data, decay ratio ω, vanishing step τ 2: Initialization: Eig = None, Visiting Sample = 0 DISPLAYFORM0 DISPLAYFORM1 We adaptively increase the batch size as the Hessian eigenvalue decreases or stays stable for several epochs (fixed to be ten in all of the experiments).The second component of our framework is robust optimization. In the seminal works of (El Ghaoui & BID9 BID33), a connection between robust optimization and regularization was proved in the context of ridge and lasso regression. In BID36, the authors empirically showed that adversarial training leads to more robust models with respect to adversarial perturbation. An interesting corollary was that, after adversarial training, the model converges to regions that are considerably flatter, as compared to the baseline. Thus, we can combine our ABS algorithm with adversarial training as a form of regularization against "sharp" minima. We refer to this as the Adaptive Batch Size Adversarial (ABSA) method; see Alg. 1. In practice, ABSA is often more stable than ABS. This corresponds to solving a minmax problem instead of a normal minimization problem BID17 BID36. Solving this min-max problem for NNs is an intractable problem, and thus we approximately solve the maximization problem through the Fast Gradient Sign Method (FGSM) proposed by BID13. This basically corresponds to generating adversarial inputs using one gradient ascent step (i.e., the perturbation is computed by ∆x = ∇ x l(z, θ)). Other possible choices are proposed by BID31 BID4 BID22.1 FIG1 illustrates our ABS schedule as compared to a normal training strategy and the increasing batch size strategy of Smith et al. FORMULA0; BID7. Note that our learning rate adaptively changes based on the Hessian eigenvalue in order to keep the same noise level as in the baseline SGD training. As we show in section 4, our combined approach (second order and robust optimization) not only achieves better accuracy, but it also requires significantly fewer SGD updates, as compared to Smith et al. FORMULA0; BID7. Before discussing the empirical , an important question is whether using ABS is a convergent algorithm for even a convex problem. Here, we show that our ABS algorithm does converge for strongly convex problems. Based on an assumption about the loss (Assumption 2 in Appendix A), it is not hard to prove the following theorem. Theorem 1. Under Assumption 2, let assume at step t, the batch size used for parameter update is b t, the step size is b t η 0, where η 0 is fixed and satisfies, DISPLAYFORM0 where B max is the maximum batch size during training. Then, with θ 0 as the initilization, the expected optimality gap satisfies the following inequality, DISPLAYFORM1 From Theorem 1, if b t ≡ 1, the convergence rate for t steps, based on equation 5, is (1 − η 0 c s). However, the convergence rate of Alg. 1 becomes DISPLAYFORM2 With an adaptive b t, Alg. 1 can converge faster than basic SGD. We show empirical for a logistic regression problem in the Appendix A, which is a simple convex problem. We evaluate the performance of our ABS and ABSA methods on different datasets (ranging from O(1E4) to O(1E7) training examples) and multiple NN models. We compare the baseline performance (i.e., small batch size), along with other state-of-the-art methods proposed for large batch training BID29 BID14. The two main metrics for comparison are the final accuracy and the total number of updates. Preferably we would want a higher testing accuracy along with fewer SGD updates. We emphasize that, for all of the datasets and models we tested, we do not change any of the hyper-parameters in our algorithm. We use the exact same parameters used in the baseline model, and we do not tailor any parameters to suit our algorithm. A detailed explanation of the different NN models, and the datasets is given in Appendix B.Section 4.1 shows the of ABS (ABSA) compared to BaseLine (BL), FB BID14 and GG BID29. Section 4.2 presents the on more challenging datasets of TinyImageNet and ImageNet. The superior performance of our method does come at the cost of backpropagating the Hessian. Thus, in section 4.3, we discuss how approximate Hessian informatino could be used to alleviate teh costs. We first start by discussing the of ABS and ABSA on SVHN and Cifar-10/100 datasets. Notice that GG and our ABS (ABSA) have different batch sizes during training. Hence the batch size reported in our represents the maximum batch size during training. To allow for a direct comparison we also report the number of weight updates in our (lower is better). It should be mentioned that the number of SGD updates is not necessarily the same as the wall-clock time. Therefore, we also report a simulated training time of I3 model in Appendix C. TAB1 report the test accuracy and the number of parameter updates for different datasets and models. First, note the drop in BL accuracy for large batch confirming the accuracy degradation problem. Moreover, note that the FB strategy only works well for moderate batch sizes (it diverges for large batch). However, the GG method has a very consistent performance, but its number of parameter updates are usually greater than our method. Looking at the last two major columns of TAB1 -7, the test performances ABS achieves are similar accuracy as BL. Overall, the number of updates of ABS is 3-10 times smaller than BL with batch size 128. However, for most cases, ABSA achieves superior . This confirms the effectiveness of adversarial training combined with the second order information. SVHN is a very simple dataset, and Cifar-10/100 are relatively small datasets, and one might wonder whether the improvements we reported in section 4.1 hold for more complex problems. Here, we report the ABSA method on more challenging datasets, i.e., TinyImageNet and ImageNet. We use the exact same hyper-parameters in our algorithm, even though tuning them could potentially be preferable for us. TinyImageNet is an image classification problem, with 200 classes and only 500 images per class. Thus it is easy to overfit the training data. The for I1 model is reported in TAB2. Note that with fewer SGD iterations, ABSA can achieve better test accuracy than other methods. The performance of ABSA is actually about 1% higher (the training loss and test performance of I1 on TinyImagenet is shown in FIG4 in appendix). Note that we do not tune the hyper-parameters, e.g., α, β, and perhaps one could close the gap between 70.24% and 70.4% with fine tuning of our hyper-parameters. However, from a practical point of view such tuning is antithetical to the goal of large batch size training as it would increase the total training time, and we specifically did not want to tailor any new parameters for a particular model/dataset. One of the limitations of our ABS (ABSA) method is the additional computational cost for computing the top Hessian eigenvalue. If we use the full Hessian operator, the second backpropagation needs to be done all the way to the first layer of NN. For deep networks this could lead to high cost. Here, we empirically explore whether we could use approximate second order information, and in particular we test a block Hessian approximation Figure 6. The block approximation corresponds to only analyzing the Hessian of the last few layers. In Figure 6 (see Appendix D), we plot the trace of top eigenvalues of full Hessian and block Hessian for C1 model. Although the top eigenvalue of block Hessian has more variance than that of full Hessian, the overall trends are similar for C1. The test performance of C1 on Cifar-10 with block Hessian is 84.82% with 4600 parameter updates (as compared to 84.42% for full Hessian ABSA). The test performance of C4 on Cifar-100 with block Hessian is 68.01% with 12500 parameter updates (as compared to 68.43% for full Hessian ABSA). These suggest that using a block Hessian to estimate the trend of the full Hessian might be a good choice to overcome computation cost, but a more detailed analysis is needed. We introduce an adaptive batch size algorithm based on Hessian information to speed up the training process of NNs, and we combine this approach with adversarial training (which is a form of robust optimization, and which could be viewed as a regularization term for large batch training). We extensively test our method on multiple datasets (SVHN, Cifar-10/100, TinyImageNet and ImageNet) with multiple NN models (AlexNet, ResNet, Wide ResNet and SqueezeNext). As the goal of large batch is to reduce training time, we did not perform any hyper-parameter tuning to tailor our method for any of these tests. Our method allows one to increase batch size and learning rate automatically, based on Hessian information. This helps significantly reduce the number of parameter updates, and it achieves superior generalization performance, without the need to tune any of the additional hyper-parameters. Finally, we show that a block Hessian can be used to approximate the trend of the full Hessian to reduce the overhead of using second-order information. These improvements are useful to reduce NN training time in practice. • L(θ) is continuously differentiable and the gradient function of L is Lipschitz continuous with Lipschitz constant L g, i.e. DISPLAYFORM0 for all θ 1 and θ 2.Also, the global minima of L(θ) is achieved at θ * and L(θ *) = L *.• Each gradient of each individual l i (z i) is an unbiased estimation of the true gradient, i.e. DISPLAYFORM1 where V(·) is the variance operator, i.e. DISPLAYFORM2 From the Assumption 2, it is not hard to get, DISPLAYFORM3 DISPLAYFORM4 With Assumption 2, the following two lemmas could be found in any optimization reference, e.g.. We give the proofs here for completeness. Lemma 3. Under Assumption 2, after one iteration of stochastic gradient update with step size η t at θ t, we have DISPLAYFORM5 where DISPLAYFORM6 Proof. With the L g smooth of L(θ), we have DISPLAYFORM7 From above, the follows. Lemma 4. Under Assumption 2, for any θ, we have DISPLAYFORM8 Proof. Let DISPLAYFORM9 Then h(θ) has a unique global minima atθ DISPLAYFORM10 The following lemma is trivial, we omit the proof here. DISPLAYFORM11 PROOF OF THEOREM 1Given these lemmas, we now proceed with the proof of Theorem 1.Proof. Assume the batch used at step t is b t, according to Lemma 3 and 5, DISPLAYFORM12 where the last inequality is from Lemma 4. This yields DISPLAYFORM13 It is not hard to see, DISPLAYFORM14 which concludes DISPLAYFORM15 Therefore, DISPLAYFORM16 We show a toy example of binary logistic regression on mushroom classification dataset 2. We split the whole dataset to 6905 for training and 1819 for validation. η 0 = 1.2 for SGD with batch size 100 and full gradient descent. We set 100 ≤ b t ≤ 3200 for our algorithm, i.e. ABS. Here we mainly focus on the training losses of different optimization algorithms. The are shown in FIG3. In order to see if η 0 is not an optimal step size of full gradient descent, we vary η 0 for full gradient descent; see in FIG3. In this section, we give the detailed outline of our training datasets, models, strategy as well as hyper-parameter used in Alg 1.Dataset. We consider the following datasets.• SVHN. The original SVHN BID23 dataset is small. However, in this paper, we choose the additional dataset, which contains more than 500k samples, as our training dataset. • Cifar. The two Cifar (i.e., Cifar-10 and Cifar-100) datasets BID18 ) have same number of images but different number of classes.• TinyImageNet. TinyImageNet consists of a subset of ImangeNet images BID6, which contains 200 classes. Each of the class has 500 training and 50 validation images. 3 The size of each image is 64 × 64.• ImageNet. The ILSVRC 2012 classification dataset BID6 ) consists of 1000 images classes, with a total of 1.2 million training images and 50,000 validation images. During training, we crop the image to 224 × 224.Model Architecture. We implement the following convolution NNs. When we use data augmentation, it is exactly same the standard data augmentation scheme as in the corresponding model.• S1. AlexNet like model on SVHN as same as BID36 [C1]. We training it for 20 epochs with initial learning rate 0.01, and decay a factor of 5 at epoch 5, 10 and 15. There is no data augmentation.• C1. ResNet18 on Cifar-10 dataset BID15. We training it for 90 epochs with initial learning rate 0.1, and decay a factor of 5 at epoch 30, 60, 80. There is no data augmentation. • C2. WResNet 16-4 on Cifar-10 dataset . We training it for 90 epochs with initial learning rate 0.1, and decay a factor of 5 at epoch 30, 60, 80. There is no data augmentation.• C3. SqueezeNext on Cifar-10 dataset BID12. We training it for 200 epochs with initial learning rate 0.1, and decay a factor of 5 at epoch 60, 120, 160. Data augmentation is implemented.• C4. ResNet18 on Cifar-100 dataset BID15. We training it for 160 epochs with initial learning rate 0.1, and decay a factor of 10 at epoch 80, 120. Data augmentation is implemented. • I1. ResNet50 on TinyImageNet dataset BID15. We training it for 120 epochs with initial learning rate 0.1, and decay a factor of 10 at epoch 60, 90. Data augmentation is implemented. • I2. AlexNet on ImageNet dataset BID19. We training it for 90 epochs with initial learning rate 0.01, and decay it to 0.0001 quadratically at epoch 60, then keeps it as 0.0001 for the rest 30 epochs. Data augmentation is implemented.• I3 ResNet18 on ImageNet dataset BID15. We training it for 90 epochs with initial learning rate 0.1, and decay a factor of 10 at epoch 30, 60 and 80. Data augmentation is implemented. Training Strategy. We use the following training strategies• BL. Use the standard training procedure.• FB. Use linear scaling rule BID14 with warm-up stage.• GG. Use increasing batch size instead of decay learning rate BID29.• ABS. Use our adaptive batch size strategy without adversarial training.• ABSA. Use our adaptive batch size strategy with adversarial training. For adversarial training, the adversarial data are generated using Fast Gradient Sign Method (FGSM) BID13. The hyper-parameters in Alg. 1 (α and β) are chosen to be 2, κ = 10, adv = 0.005, γ = 20%, and ω = 2 for all the experiments. The only change is that for SVHN, the frequency to compute Hessian information is 65536 training examples as compared to one epoch, due to the small number of total training epochs (only 20).C SIMULATED TRAINING TIMEAs discussed above, the number of SGD updates does not necessarily correlate with wall-clock time, and this is particularly the case because our method require Hessian backpropagation. Here, we use the method suggested in BID11, to approximate the wall-clock time of our algorithm when utilizing p parallel processes. For the ring algorithm BID31, the communication time per SGD iteration for p processes is: DISPLAYFORM0 where α latency is the network latency, β bandwidth is the inverse bandwidth, and |θ| is the size number of model parameters measured in terms of Bits. Moreover, we manually measure the wall-clock time of computing the Hessian information using our in-house code, as well as the cost of forward/backward calculations on a V100 GPU. The total time will consists of this computation time and the communication one along with Hessian computation overhead (if any). Therefore we have: DISPLAYFORM1 where T compute is the time to compute forward and backward propagation, T communication is the time to communicate between different machine, and T Hessian is the time to compute top eigenvalues. We use the latency and bandwidth values of α latency = 2 µs, and β bandwidth = 1 6 Gb/s based on NERSC's Cori2 supercomputing platform. Based on above formulas, we give an example of simulated computation time cost of I3 on ImageNet. Note that for large processes and small latency terms, the communication time formula is simplified as, T comm = 2β bandwidth |θ|.In Table 3 we report the simulation time of I3 on ImageNet on 512 processes. For GG, we assume it increases batch size by a factor of 10 at epoch 30, 60 and 80. The batch size per GPU core is set to 16 for SGD (and 8 for Hessian computation due to memory limit) and the total batch size used for Hessian computation is set to 4096 images. The T comp and T comm is for one SGD update and T Hessian is for one complete Hessian eigenvalue computation (including communication for Hessian computation). Note that the total Hessian computation time for ABS/ABSA is only 1.15 × 90 = 103.5 s even though the Hessian computation is not efficiently implemented in the existing frameworks. Note that even with the additional Hessian overhead ABS/ABSA is still much faster than BL (and these numbers are with an in-house and not highly optimized code for Hessian computations). We furthermore note that we have added the additional computational overhead of adversarial computations to the ABSA method. Table 3: Below we present the breakdown of one SGD update training time in terms of forward/backwards computation (T comp), one step communication time (T comm), one total Hessian spectrum computation (if any T Hess), and the total training time. The correspond to I3 model on ImageNet (for accuracy please see FIG2). In this section, we present additional empirical . TAB2 for details). As one can see, from epoch 60 to 80, the test performance drops due to overfitting. However, ABSA achieves the best performance with apparently less overfitting (it has higher training loss).

Large batch size training using adversarial training and second order information

Represent the TLDR for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.