Question: represent this

Question: The convergence rate and final performance of common deep learning models have significantly benefited from recently proposed heuristics such as learning rate schedules, knowledge distillation, skip connections and normalization layers. In the absence of theoretical underpinnings, controlled experiments aimed at explaining the efficacy of these strategies can aid our understanding of deep learning landscapes and the training dynamics. Existing approaches for empirical analysis rely on tools of linear interpolation and visualizations with dimensionality reduction, each with their limitations. Instead, we revisit the empirical analysis of heuristics through the lens of recently proposed methods for loss surface and representation analysis, viz. mode connectivity and canonical correlation analysis (CCA), and hypothesize reasons why the heuristics succeed. In particular, we explore knowledge distillation and learning rate heuristics of (cosine) restarts and warmup using mode connectivity and CCA. Our empirical analysis suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed in the deeper layers. The introduction of heuristics such as normalization layers BID19 BID0, residual connections BID11, and learning rate strategies BID26 BID9 ) have greatly accelerated progress in Deep Learning. Many of these ingredients are now commonplace in modern architectures, and some of them have also been buttressed with theoretical guarantees BID1 BID28 BID10. However, despite their simplicity and efficacy, why some of these heuristics work is still relatively unknown. Existing attempts at explaining these strategies empirically have been limited to intuitive explanations and the use of tools such as spectrum analysis , linear interpolation between two models and low-dimensional visualizations of the loss surface. In our work, we instead use recent tools built specifically for analyzing deep networks, viz., mode connectivity and singular value canonical correlation analysis (SVCCA) . We investigate three strategies in detail: (a) cosine learning rate decay, (b) learning rate warmup, and (c) knowledge distillation, and list the summary of our contributions at the end of this section. Cosine annealing BID26, also known as stochastic gradient descent with restarts (SGDR), and more generally cyclical learning rate strategies , have been recently proposed to accelerate training of deep networks BID3. The strategy involves reductions and restarts of learning rates over the course of training, and was motivated as means to escape spurious local minima. Experimental have shown that SGDR often improves convergence both from the standpoint of iterations needed for convergence and the final objective. Learning rate warmup BID9 also constitutes an important ingredient in training deep networks, especially in the presence of large or dynamic batch sizes. It involves increasing the learning rate to a large value over a certain number of training iterations followed by decreasing the learning rate, which can be performed using step-decay, exponential decay or other such schemes. The strategy was proposed out of the need to induce stability in the initial phase of training with large learning rates (due to large batch sizes). It has been employed in training of several architectures at scale including ResNets and Transformer networks .Further, we investigate knowledge distillation (KD) BID13. This strategy involves first training a (teacher) model on a typical loss function on the available data. Next, a different (student) model (typically much smaller than the teacher model) is trained, but instead of optimizing the loss function defined using hard data labels, this student model is trained to mimic the teacher model. It has been empirically found that a student network trained in this fashion significantly outperforms an identical network trained with the hard data labels. We defer a detailed discussion of the three heuristics, and existing explanations for their efficacy to sections 3, 4 and 5 respectively. Finally, we briefly describe the tools we employ for analyzing the aforementioned heuristics. Mode connectivity (MC) is a recent observation that shows that, under circumstances, it is possible to connect any two local minima of deep networks via a piecewise-linear curve BID5. This shows that local optima obtained through different means, and exhibiting different local and generalization properties, are connected. The authors propose an algorithm that locates such a curve. While not proposed as such, we employ this framework to better understand loss surfaces but begin our analysis in Section 2 by first establishing its robustness as a framework. Deep network analyses focusing on the weights of a network are inherently limited since there are several invariances in this, such as permutation and scaling. propose using CCA along with some pre-processing steps to analyze the activations of networks, such that the ing comparison is not dependent on permutations and scaling of neurons. They also prove the computational gains of using CCA over alternatives (BID25) for representational analysis and employ it to better understand many phenomenon in deep learning. • We use mode connectivity and CCA to improve understanding of cosine annealing, learning rate warmup and knowledge distillation. For mode connectivity, we also establish the robustness of the approach across changes in training choices for obtaining the modes.• We demonstrate that the reasons often quoted for the success of cosine annealing are not substantiated by our experiments, and that the iterates move over barriers after restarts but the explanation of escaping local minima might be an oversimplification.• We show that learning rate warmup primarily limits weight changes in the deeper layers and that freezing them achieves similar outcomes as warmup.• We show that the latent knowledge shared by the teacher in knowledge distillation is primarily disbursed in the deeper layers.2 EMPIRICAL TOOLS 2.1 MODE CONNECTIVITY introduce a framework, called mode connectivity, to obtain a low loss (or high accuracy, in the case of classification) curve of simple form, such as a piecewise linear curve, that connects optima (modes of the loss function) found independently. This observation suggests that points at the same loss function depth are connected, somewhat contrary to several empirical claiming that minima are isolated or have barriers between them 1.Let w a ∈ R D and w b ∈ R D be two modes in the D-dimensional parameter space obtained by optimizing a given loss function L(w) (like the cross-entropy loss). We represent a curve connecting Validation accuracy corresponding to models on the following 6 different curves -curve GA represents curve connecting mode G (one found with default hyperparameters) and mode A (using large batch size), similarly, curve GB connects mode G and mode B (using Adam), curve GC connects to mode C (using linearly decaying learning rate), curve GD to mode D (with lesser L2 regularization), curve GE to mode E (using a poor initialization), and curve GF to mode F (without using data augmentation). t = 0 corresponds to mode G for all plots.w a and w b by φ θ (t): → R D, such that φ θ = w a and φ θ = w b. To find a low loss path, we find the set of parameters θ ∈ R D that minimizes the following loss: DISPLAYFORM0 where U is the uniform distribution in the interval. To optimize (θ) for θ, we first need to chose a parametric form for φ θ (t). One of the forms proposed by is a polygonal chain with a single bend at θ as follows DISPLAYFORM1 To minimize (θ), we sample t ∼ U at each iteration and use ∇ θ L(φ θ (t)) as an unbiased estimate for the true gradient ∇ θ (θ) to perform updates on θ, where θ is initialized with 1 2 (w a +w b). To demonstrate that the curve-finding approach works in practice, use two optima found using different initializations but a common training scheme which we detail below. We explore the limits of this procedure by connecting optima obtained from different training strategies. Our goal of this investigation is to first establish the robustness of the framework in order to seamlessly use it as a tool for analysis. In particular, we experiment with different initializations, optimizers, data augmentation choices, and hyperparameter settings including regularization, training batch sizes, and learning rate schemes. We note in passing that while the framework was proposed to connect two points in the parameter space that are at equal depth in the loss landscape, it is well-defined to also connect points at different depths; in this case, the path corresponds to one that minimizes the average loss along the curve. Conventional wisdom suggests that the different training schemes mentioned above will converge to regions in the parameter space that are vastly different from each other. Examples of this include size of minibatches used during training BID22, choice of optimizer BID12 ), initialization BID8 and choice of regularizer. Having a high accuracy connection between these pairs would seem counterintuitive. For obtaining the reference model (named mode G), we train the VGG-16 model architecture using CIFAR-10 training data BID23 for 200 epochs with SGD. We then build 6 variants of the reference mode G as follows: we obtain mode A using a training batch size of 4000, mode B by using the Adam optimizer instead of SGD, mode C with a linearly decaying learning rate instead of the step decay used in mode G, mode D using a smaller weight decay of 5 × 10 −6, mode E by increasing the variance of the initialization distribution to 3 × 2/n and mode F using no data augmentation. Note that for the set of modes {A, B, C, D, E, F}, all the other hyper-parameters and settings except the ones mentioned above are kept same as that for mode G. We use the mode connectivity algorithm on each of the 6 pairs of modes including G and another mode, ing in curves GA, GB, GC, GD, GE, and GF. FIG0 shows the validation accuracy for models on each of the 6 connecting curves during the 20th, 40th, 60th and 80th epochs of the mode connectivity training procedure and also for models on the line segment joining the two endpoints (corresponding to the initialization for θ at epoch 0). As described in Section 2.1, for a polychain curve GX (connecting modes G and X using the curve described by θ), model parameters φ θ (t) on the curve are given by p φ θ (t) = 2(tp θ + (0.5 − t)p G ) if 0 ≤ t ≤ 0.5 and p φ θ (t) = 2((t − 0.5)p X + (1 − t)p θ ) if 0.5 < t ≤ 1 where p G, p θ and p X are parameters of the models G, θ, and X respectively. Thus φ θ = G and φ θ = X.In a few epochs of the curve training, for all 6 pairs, we can find a curve such that each point on it generalizes almost as well as models from the pair that is being connected. Note that by virtue of existence of these 6 curves, there exists a high accuracy connecting curve (albeit with multiple bends) for each of the 7 2 pairs of modes. We refer the reader to Appendix 7 for a t-SNE plot of the modes and their connections, and also for additional plots and details. Having established the high likelihood of the existence of these curves, we use this procedure along with interpolation of the loss surface between parameters at different epochs as tools to analyze the dynamics of SGD and SGDR. Canonical correlation analysis (CCA) is a classical tool from multivariate statistics BID16 that investigates the relationships between two sets of random variables. have proposed coupling CCA with pre-processing steps like Singular Value Decomposition (SVD) or Discrete Fourier Transform (DFT) to design a similarity metric for two neural net layers that we want to compare. These layers do not have to be of the same size or belong to the same network. Given a dataset with m examples X = {x 1, . . . x m}, we denote the scalar output of the neuron z l i (i-th neuron of layer l) for the input x i by f z L i (x i). These scalar outputs can be stacked (along n different neurons and m different datapoints) to create a matrix L ∈ R m×n representing the output of a layer corresponding to the entire dataset. This choice of comparing neural network layers using activations instead of weights and biases is crucial to the setup proposed. Indeed, invariances due to re-parameterizations and permutations limit the interpretability of the model weights BID4. However, under CCA of the layers, two activation sets are comparable by design. Given representations corresponding to two layers L a ∈ R ma×n and L b ∈ R m b ×n, SVCCA first performs dimensionality reduction using SVD to obtain L a ∈ R m a ×n and L b ∈ R m b ×n while preserving 99% of the variance. The subsequent CCA step involves transforming L a and L b to a 1 L a and b 1 L b respectively where {a 1, b 1} is found by maximizing the correlation between the transformed subspaces, and the corresponding correlation is denoted by ρ 1. This process continues, using orthogonality constraints, till c = min{m a, m b} leading to the set of correlation values {ρ 1, ρ 2 . . . ρ c} corresponding to c pairs of canonical variables {{a 1, b 1}, {a 2, b 2},... {a c, b c}} respectively. We refer the reader to for details on solving these optimization problems. The average of these c correlations 1 n i ρ i is then considered as a measure of the similarity between the two layers. For convolutional layers, suggest using a DFT pre-processing step before CCA, since they typically have a large number of neurons (m a or m b), where performing raw SVD and CCA would be computationally too expensive. This procedure can then be employed to compare different neural network representations and to determine how representations evolve over training iterations. BID26 introduced SGDR as a modification to the common linear or step-wise decay of learning rates. The strategy decays learning rates along a cosine curve and then, at the end of the decay, restarts them to its initial value. The learning rate at the t-th epoch in SGDR is given by the following expression in where η min and η max are the lower and upper bounds respectively for the learning rate. T cur represents how many epochs have been performed since the last restart and a warm restart is simulated once T i epochs are performed. Also T i = T mult × T i−1, meaning the period T i for the learning rate variation is increased by a factor of T mult after each restart. While the strategy has been claimed to outperform other learning rate schedulers, little is known why this has been the case. One explanation that has been given in support of SGDR is that it can be useful to deal with multi-modal functions, where the iterates could get stuck in a local optimum and a restart will help them get out of it and explore another region; however, BID26 do not claim to observe any effect related to multi-modality. BID18 propose an ensembling strategy using the set of iterates before restarts and claim that, when using the learning rate annealing cycles, the optimization path converges to and escapes from several local minima. We empirically investigate if this is actually the case by interpolating the loss surface between parameters at different epochs and studying the training and validation loss for parameters on the hyperplane passing through 2 the two modes found by SGDR and their connectivity. Further, by employing the CCA framework as described in Section 2.2, we investigate the progression of training, and the effect of restarts on the model activations. We train a VGG-16 network on the CIFAR-10 dataset using SGDR. For our experiments, we choose T 0 = 10 epochs and T mult = 2 (warm restarts simulated every 10 epochs and the period T i doubled at every new warm restart), η max = 0.05 and η min = 10 −6. We also perform VGG training using SGD (with momentum of 0.9) and a step decay learning rate scheme (initial learning rate of η 0 = 0.05, scaled by 5 at epochs 60 and 150). In order to understand the loss landscape on the optimization path of SGDR, the pairs of iterates obtained just before the restarts {w 30, w 70}, {w 70, w 150} and {w 30, w 150} are given as inputs to the mode connectivity algorithm, where w n is the model corresponding to parameters at the n-th epoch of training. FIG1 (b) shows the training loss for models along the line segment joining these pairs and those on the curve found through mode connectivity. For the baseline case of SGD training, we connect the iterates around the epochs when we decrease our learning rate in the step decay learning rate scheme. Thus, we chose {w 55, w 65}, {w 145, w 165} and {w 55, w 165} as input pairs to the mode connectivity algorithm. FIG1 (c) shows the training loss for models along the line segments joining these pairs and the curves found through mode connectivity. From FIG1 (b), it is clear that for the pairs {w 30, w 150} and {w 70, w 150} the training loss for points on segment is much higher than the endpoints suggesting that SGDR indeed finds paths that move over a barrier 3 in the training loss landscape. In contrast, for SGD (without restarts) in FIG1 (c) none of the three pairs show evidence of having a training loss barrier on the line segment joining them. Instead there seems to be an almost linear decrease of training loss along the direction of these line segments, suggesting that SGD's trajectory is quite different from SGDR's. We present additional experiments, including for other metrics, in Appendix 8.To further understand the SGDR trajectory, we evaluate the intermediate points on the hyperplane in the D-dimensional space defined by the three points: w 70, w 150 and w 70−150, where w 70−150 is the bend point that defines the high accuracy connection for the pair {w 70, w 150}. ) suggests that SGDR helps the iterates converge to a different region although neither of w 70 or w 150 are technically a local minimum, nor do they appear to be lying in different basins, hinting that BID18's claims about SGDR converging to and escaping from local minima might be an oversimplification. 4 Another insight we can draw from FIG3 (a) is that the path found by mode connectivity corresponds to lower training loss than the loss at the iterates that SGDR converges to (L(w 150) > L(w 70−150)). However, FIG3 (b) shows that models on this curve seem to overfit and not generalize as well as the iterates w 70 and w 150. Thus, although gathering models from this connecting curve might seem as a novel and computationally cheap way of creating ensembles, this generalization gap alludes to one limitation in doing so; point to other shortcomings of curve ensembling in their original work. In FIG3, the region of the plane between the iterates w 70 and w 150 corresponds to higher training loss but lower validation loss than the two iterates. This hints at a reason why averaging iterates to improve generalization using cyclic or constant learning rates has been found to work well. Finally, in FIG0 in Appendix 9, we present the CCA similarity plots for two pairs of models: epochs 10 and 150 (model at the beginning and end of training), and epochs 150 and 155 (model just before and just after a restart). For standard SGD training, observe that the activations of the shallower layers bear closer resemblance than the deeper layers between a partially and fully trained network from a given training run. For SGDR training, we witness similar (discussed in Appendix 9), meaning that the representational similarities between the network layers at the beginning and end of training are alike for SGDR and SGD, even though restarts lead to a trajectory that tends to cross over barriers. Learning rate warmup is a common heuristic used by many practitioners for training deep neural nets for computer vision BID9 and natural language processing BID2 ) tasks. Theoretically, it can be shown that the learning dynamics of SGD rely on the ratio of the batch size and learning rate (; BID21 BID14 . And hence, an increase in batch size over a baseline requires an accompanying increase in learning rate for comparable training. However, in cases when the batch size is increased significantly, the curvature of the loss function typically does not support a proportional increase in the learning rate. Warmup is hence motivated as a means to use large learning rates without causing training instability. We particularly focus on the importance of the learning rate schedule's warmup phase in the large batch (LB) training of deep convolutional neural networks as discussed in Goyal Using CCA as a tool to study the learning dynamics of neural networks through training iterations, we investigate the differences and similarities for the following 3 training configurations -(a) large batch training with warmup (LB + warmup), (b) large batch training without warmup (LB no warmup) and (c) small batch training without warmup (SB no warmup). We train a VGG-11 architecture on the CIFAR-10 dataset using SGD with momentum of 0.9. Learning rate for the small batch case (batch-size of 100) is set to 0.05, and for the large batch cases (batch-size of 5000) is set to 2.5 as per the scaling rule. For the warmup, we increase the learning rate from 0 to 2.5 over the first 200 iterations. Subsequently, we decrease the learning rate as per the step decay schedule for all runs, scaling it down by a factor of 10 at epochs 60, 120 and 150. We plot the learning rate and validation accuracy for these 3 cases in Figure 4 Figure 4 (c) plots the similarity for layer i of iter a with the same layer of iter b (this corresponds to diagonal elements of the matrices in FIG6) for these three setups. An evident pattern in FIG6, (b) and (c) is the increase in similarity for the last few layers (stack of fully-connected layers) for the LB + warmup and SB cases, which is absent in the LB without warmup case. This suggests that when used with the large batch size and learning rate, warmup tends to avoid unstably large changes in the fully-connected (FC) stack for this network configuration. To validate this proposition, we train using the LB without warmup setup, but freezing the fully-connected stack for the first 20 epochs 5 (LB no warmup + FC freeze). Figure 4(M denotes the i-th layer of network M, T denotes the teacher network (VGG16), S distilled is the student network trained using distillation and S indep. is the student network trained using hard training labels.suggesting the validity our proposition in this case. We refer the reader to Appendix 10 for analogous for ResNet-18 and ResNet-32 BID11; thus also demonstrating the generality of our claim. Finally, note from Figure 4 (d) that no qualitative difference exists in the trajectory beyond the warmup when compared to the standard training approach . We study knowledge distillation as proposed by BID13 using CCA to measure representational similarity between layers of the teacher and student model. Distillation involves training a "student" model using the output probability distribution of a "teacher" model. This has been widely known to help the student model perform better than it would, if it were trained using hard labels due to knowledge transfer from the teacher model. The reason often quoted for the success of distillation is the transfer of dark knowledge from the teacher to the student BID13, and more recently, as an interpretation of importance weighing BID6. We investigate if this knowledge transfer is limited to certain parts of the network, and if representational similarity between layers of the student and teacher model and a student can help answer this question. To construct an example of distillation that can be used for our analysis, we use a VGG-16 model as our teacher network and a shallow convolutional network ([conv, maxpool, relu] x2, fc, relu, fc, fc, softmax) as the student network. We train the shallow network for CIFAR-10 using the teacher's predicted probability distribution (softened using a temperature of 5), (S distilled), and for the baseline, train another instance of the same model in a standard way using hard labels, (S indep.). Over 5 runs for each of the two setups, we find the distillation training attains the best validation accuracy at 85.18% while standard training attains its best at 83.01%. We compare their layer-wise representations with those of the teacher network (T). FIG8 shows the CCA plots and the absolute value of their difference. The scores of these two pairs are quite similar for the shallow layers of the student network relative to the deeper layers, suggesting that the difference that knowledge distillation brings to the training of smaller networks is restricted to the deeper layers (fc stack). Similar are obtained through different configurations for the student and teacher when the student benefits from the teacher's knowledge. We hypothesize that the dark knowledge transferred by the teacher is localized majorly in the deeper (discriminative) layers, and less so in the feature extraction layers. We also note that this is not dissimilar to the hypothesis of BID6, and also relates ot the from the literature on fine-tuning or transfer learning BID8; BID17 which suggest training of only higher layers. Heuristics have played an important role in accelerating progress of deep learning. Founded in empirical experience, intuition and observations, many of these strategies are now commonplace in architectures. In the absence of strong theoretical guarantees, controlled experiments aimed at explaining the the efficacy of these strategies can aid our understanding of deep learning and the training dynamics. The primary goal of our work was the investigation of three such heuristics using sophisticated tools for landscape analysis. Specifically, we investigate cosine annealing, learning rate warmup, and knowledge distillation. For this purpose, we employ recently proposed tools of mode connectivity and CCA. Our empirical analysis sheds light on these heuristics and suggests that: (a) the reasons often quoted for the success of cosine annealing are not evidenced in practice; (b) that the effect of learning rate warmup is to prevent the deeper layers from creating training instability; and (c) that the latent knowledge shared by the teacher is primarily disbursed in the deeper layers. Inadvertently, our investigation also leads to the design of new heuristics for practically improving the training process. Through our on SGDR, we provide additional evidence for the success of averaging schemes in this context. Given the empirical suggesting the localization of the knowledge transfer between teacher and student in the process of distillation, a heuristic can be designed that only trains portions of the (pre-trained) student networks instead of the whole network. For instance, recent on self-distillation BID6 show improved performance via multiple generations of knowledge distillation for the same model. Given our , computational costs of subsequent generations can be reduced if only subsets of the model are trained, instead of training the entire model. Finally, the freezing of weights instead of employing learning rate warmup allows for comparable training performance but with reduced computation during the warmup phase. We note in passing that our also ties in with of Hoffer et al. FORMULA2 The learning rate is initialized to 0.05 and scaled down by a factor of 5 at epochs {60, 120, 160} (step decay). We use a training batch size of 100, momentum of 0.9, and a weight decay of 0.0005. Elements of the weight vector corresponding to a neuron are initialized randomly from the normal distribution N (0, 2/n) where n is the number of inputs to the neuron. We also use data augmentation by random cropping of input images. Figures 7, 8 and 9 show the Validation Loss, Training Accuracy and Training Loss respectively for the curves joining the 6 pairs discussed in Section 2.1.1. These too, confirm the overfitting or poor generalization tendency of models on the curve. We use t-SNE BID27 to visualize these 7 modes and the θ points that define the connectivity for the 6 pairs presented in Section 2.1.1, in a 2-dimensional plot in FIG0. Since t-SNE is known to map only local information correctly and not preserve global distances, we caution the reader about the limited interpretability of this visualization, it is presented simply to establish the notion of connected modes. The W n in FIG3 is equivalent to meaning it is the point on the plane (linear combination of w 70, w 150 and θ) with the least l-2 distance from the original point (iterate in this case). DISPLAYFORM0 8.3 CONNECTING MODES w 30 AND w 70 FROM SGDRIn Section 3, we present some experiments and make observations on the trajectory of SGDR by using the plane defined by the points w 70, w 150 and w 70−150. Here we plot the Training loss and Validation loss surface in FIG0 for another plane defined by SGDR's iterates w 30, w 70 and their connection w 30−70 to ensure the reader that the observations made are general enough. The VGG-16 architecture used in Section 3 does not include Batch Normalization, which has been known to alter properties of the loss surface . Therefore we train VGG-16 with Batch Normalization using SGDR to verify if our observations hold for this case too. As pointed out in Appendix A.2 of, at the test stage, we compute the Batch Normalization statistics for a network on the curve with an additional pass over the data, since these are not collected during training. Except Batch Normalization, other training parameters are kept the same as discussed for Section 3.Figure 13(a) shows the training loss for models along the line segment and MC curve joining the pair of iterates from SGDR. For the two pairs {w 30, w 150} and {w 70, w 150}, we again observe a higher training loss for models on the line segment, suggesting that for this setup too, SGDR finds paths that move over a barrier in the training loss landscape. We further evaluate the intermediate points FIG0, we present the CCA similarity plots comparing two pairs of models: epochs 10 and 150, and epochs 150 and 155. The (i, j) th block of the matrix denotes the correlation between the i th layer of the first model and the j th layer of the other. A high correlation implies that the layers learn similar representations and vice versa. We present the former to compare against the typical stepwise or linear decay of SGD, and the latter to demonstrate the immediate effect of restarting on the model. showed in their work that for typical SGD training, a CCA similarity plot between a partially and completed trained network reveals that the activations of the shallower layers bears closer resemblance in the two models than the deeper layers. We note that, despite the restart, a similar tendency is seen in SGDR training as well. This again suggests that the restart does not greatly impact the model, both in weights and representations, and especially so in the shallower layers. A comparison of epochs 150 and 155, i.e., before and after a restart also stands as evidence for this hypothesis. In Figure 4 (d), we show that the stability induced by warmup when training with large batches and learning rates can also be obtained by holding the FC stack frozen. This experiment was conducted on the VGG-11 network . To demonstrate the generality of our claim, we present additional experiments on two ResNet architectures: 18 and 32. The setup for this experiment is identical to the VGG-11 one with one change: instead of the learning rate being set to 2.5, which is the learning rate for SB (0.05) times the batch size increase (50×), we set it to 5.0 since SB training is better with 0.1. For the warmup case, we linearly increase the learning rate from 0 to 5 again for 20 epochs. Experiments on other configurations yielded similar . Whether these remain true also for training larger datasets, such as ImageNet, remains to be shown and is a topic of future research.

represent the short summary for finding the corresponding scientific article (long)

We use empirical tools of mode connectivity and SVCCA to investigate neural network training heuristics of learning rate restarts, warmup and knowledge distillation.

Represent the TLDR for finding the corresponding scientific article

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the scientific article to find a short TLDR

Question: The increasing demand for neural networks (NNs) being employed on embedded devices has led to plenty of research investigating methods for training low precision NNs. While most methods involve a quantization step, we propose a principled Bayesian approach where we first infer a distribution over a discrete weight space from which we subsequently derive hardware-friendly low precision NNs. To this end, we introduce a probabilistic forward pass to approximate the intractable variational objective that allows us to optimize over discrete-valued weight distributions for NNs with sign activation functions. In our experiments, we show that our model achieves state of the art performance on several real world data sets. In addition, the ing models exhibit a substantial amount of sparsity that can be utilized to further reduce the computational costs for inference. With the advent of deep neural networks (NNs) impressive performances have been achieved in many applications such as computer vision BID13, speech recognition, and machine translation, among others. However, the performance improvements are largely attributed to increasing hardware capabilities that enabled the training of ever-increasing network architectures. On the other side, there is also a growing interest in making NNs available for embedded devices with drastic memory and power limitations -a field with plenty of interesting applications that barely profit from the tendency towards larger and deeper network structures. Thus, there is an emerging trend in developing NN architectures that allow fast and energy-efficient inference and require little storage for the parameters. In this paper, we focus on reduced precision methods that restrict the number of bits per weight while keeping the network structures at a decent size. While this reduces the memory footprint for the parameters accordingly, it can also in drastic improvements in computation speed if appropriate representations for the weight values are used. This direction of research has been pushed towards NNs that require in the extreme case only a single bit per weight. In this case, assuming weights w ∈ {−1, 1} and binary inputs x ∈ {−1, 1}, costly floating point multiplications can be replaced by cheap and hardware-friendly logical XNOR operations. However, training such NNs is inherently different as discrete valued NNs cannot be directly optimized using gradient based methods. Furthermore, NNs with binary weights exhibit their full computational benefits only in case the sign activation function is used whose derivative is zero almost everywhere, and, therefore, is not suitable for backpropagation. Most methods for training reduced precision NNs either quantize the weights of pre-trained full precision NNs BID3 or train reduced precision NNs by maintaining a set of full precision weights that are deterministically or stochastically quantized during forward or backward propagation. Gradient updates computed with the quantized weights are then applied to the full precision weights BID4 BID19 BID8. This approach alone fails if the sign activation function is used. A promising approach is based on the straight through gradient estimator (STE) BID1 which replaces the zero gradient of hard threshold functions by a non-zero surrogate derivative. This allows information in computation graphs to flow backwards such that parameters can be updated using gradient based optimization methods. Encouraging are presented in BID8 where the STE is applied to the weight binarization and to the sign activation function. These methods, although showing The aim is to obtain a single discrete-valued NN (top right) with a good performance. We achieve this by training a distribution over discrete-valued NNs (bottom right) and subsequently deriving a single discrete-valued NN from that distribution. (b) Probabilistic forward pass: The idea is to propagate distributions through the network by approximating a sum over random variables by a Gaussian and subsequently propagating that Gaussian through the sign activation function.convincing empirical performance, have in common that they appear rather heuristic and it is usually not clear whether they optimize any well defined objective. Therefore, it is desired to develop principled methods that support discrete weights in NNs. In this paper, we propose a Bayesian approach where we first infer a distribution q(W) over a discrete weight space from which we subsequently derive discrete-valued NNs. Thus, we can optimize over real-valued distribution parameters using gradient-based optimization instead of optimizing directly over the intractable combinatorial space of discrete weights. The distribution q(W) can be seen as an exponentially large ensemble of NNs where each NN is weighted by its probability q(W).Rather than having a single value for each connection of the NN, we now maintain a whole distribution for each connection (see bottom right of FIG0 (a)). To obtain q(W), we employ variational inference where we approximate the true posterior p(W |D) by minimizing the variational objective KL(q(W)||p(W |D)). Although the variational objective is intractable, this idea has recently received a lot of attention for real-valued NNs due to the reparameterization trick which expresses gradients of intractable expectations as expectations of tractable gradients BID20 BID12 BID2. This allows us to efficiently compute unbiased gradient samples of the intractable variational objective that can subsequently be used for stochastic optimization. Unfortunately, the reparameterization trick is only suitable for real-valued distributions which renders it unusable for our case. The recently proposed Gumbel softmax distribution BID10 BID16 overcomes this issue by relaxing one-hot encoded discrete distributions with probability vectors. Subsequently, the reparameterization trick can again be applied. However, for the sign activation function one still has to rely on the STE or similar heuristics. The log-derivative trick offers an alternative for discrete distributions to express gradients of expectations with expectations of gradients BID18. However, the ing gradient samples are known to suffer from high variance. Therefore, the log-derivative trick is typically impractical unless suitable variance reduction techniques are used. This lack of practical methods has led to a limited amount of literature investigating Bayesian NNs with discrete weights BID23.In this work, we approximate the intractable variational objective with a probabilistic forward pass (PFP) BID26 BID23 BID6 BID21. The idea is to propagate probabilities through the network by successively approximating the distributions of activations with a Gaussian and propagating this Gaussian through the sign activation function FIG0 ). This in a well-defined objective whose gradient with respect to the variational parameters can be computed analytically. This is true for discrete weight distributions as well as for the sign activation function with zero gradient almost everywhere. The method is very flexible in the sense that different weight distributions can be used in different layers. We utilize this flexibility to represent the weights in the first layer with 3 bits and we use ternary weights w ∈ {−1, 0, 1} in the remaining layers. In our experiments, we evaluate the performance of our model by reporting the error of (i) the most probable model of the approximate posterior q(W) and (ii) approximated expected predictions using the PFP. We show that averaging over small ensembles of NNs sampled from W ∼ q(W) can improve the performance while inference using the ensemble is still cheaper than inference using a single full precision NN. Furthermore, our method exhibits a substantial amount of sparsity that further reduces the computational overhead. Compared to BID8, our method requires less precision for the first layer, and we do not introduce a computational overhead by using batch normalization which appears to be a crucial component of their method. The paper is outlined as follows. In Section 2, we introduce the notation and formally define the PFP. Section 3 shows details of our model. Section 4 shows experiments. In Section 5 we discuss important issues concerning our model and Section 6 concludes the paper. The structure of a feed-forward NN with L layers is determined by the number of neurons DISPLAYFORM0 Here, d 0 is the dimensionality of the input, d L is the dimensionality of the output, and d l for 0 < l < L is the number of hidden neurons in layer l. A NN defines a function y = x L = f (x 0) by iteratively applying a linear transformation a l = W l x l−1 to the inputs from the previous layer followed by a non-linear function DISPLAYFORM1 which is applied element-wise to its inputs. For l = L, we use the softmax activation function smax i (a) = exp(a i)/ j exp(a j). Note that the expensive softmax function does not need to be computed at test time. where D l is a finite set.1. For a Bayesian treatment of NNs, we assume a prior distribution p(W) over the discrete weights and interpret the output of the NN after the softmax activation as likelihood p(D|W) for the data set DISPLAYFORM0 is intractable for NNs of any decent size, we employ variational inference to approximate it by a simpler distribution q(W |ν) by minimizing KL(q(W |ν)||p(W |D)) with respect to the variational parameters ν. We adopt the common mean-field assumption where the approximate posterior factorizes into a product of factors q(w|ν w) for each weight w ∈ W.2 The variational objective is usually transformed as DISPLAYFORM1 Minimizing this expression with respect to ν does not involve the intractable posterior p(W |D) and the evidence log p(D) is constant with respect to the variational parameters ν. The KL term can be seen as a regularizer that pulls the approximate posterior q(W) towards the prior distribution p(W) whereas the expected log-likelihood captures the data. While the KL term is tractable if both the prior and the approximate posterior distribution assume independence of the weights, the expected log-likelihood is typically intractable due to a sum over exponentially many terms. We propose a PFP as closed-form approximation to the expected log-likelihood. The approximation of the expected log-likelihood resembles a PFP. In particular, we have DISPLAYFORM0 where we defined DISPLAYFORM1. The overall aim is to successively get rid of the weights in each layer and consequently reduce the exponential number of terms to sum over. In the first approximation in, we approximate the activation distribution with Gaussians using a central limit argument. These Gaussian distributions are propagated through the sign activation function ing in Bernoulli distributions in. These two steps are iterated until a Gaussian approximation of the output activations in FORMULA5 is obtained. This integral is approximated using a second-order Taylor expansion of the log-softmax around µ a L. In the following subsections we provide more details of the individual approximations. The activations of the neurons are computed as weighted sums over the outputs from the previous layers. Since the inputs and the weights are random variables, the activations are random variables as well. Given a sufficiently large number of input neurons, we can apply a central limit argument and approximate the activation distributions with Gaussians N (a DISPLAYFORM0). For computational convenience, we further assume that the activations within a layer are independent. Assuming that the inputs x l−1 j and the weights w l ij are independent, we have DISPLAYFORM1 where DISPLAYFORM2 In case of l = 1, we assume no variance at the inputs and thus the second term of σ a l i in cancels. In the next step, the Gaussian distributions N (a DISPLAYFORM3) over the activations are transformed by the sign activation function. The expectation of the ing Bernoulli distribution with values x ∈ {−1, 1} of the sign activation function is given by µ DISPLAYFORM4 2 ) where erf denotes the error function. The raw second moment as needed for is µ (x l i) 2 = 1. After iterating this approximation up to the last layer, it remains to calculate the expectation of the log-softmax with respect to the Gaussian approximation of the output activations in. Since this integral does not allow for an analytic solution, we approximate the log-softmax by its second-order Taylor approximation around the mean µ a L with a diagonal covariance approximation, ing in DISPLAYFORM0 The maximization of the first term in enforces the softmax output of the true class to be maximized, whereas the second term becomes relevant if there is no output close to one. For softmax outputs substantially different from zero or one, the product inside the sum is substantially larger than zero and the corresponding variance is penalized. In short, the second term penalizes large output variances if their corresponding output activation means are large and close to each other. In this section we provide details of the finite weight sets D l and their corresponding prior and approximate posterior distributions, respectively. As reported in several papers BID8 BID0, it appears to be crucial to represent the weights in the first layer using a higher precision. Therefore, we use three bits for the weights in the first layer and ternary weights in the remaining layers. We use D 1 = {−0.75, −0.5, . . ., 0.75} for the first layer which can be represented as fixed point numbers using three bits. Note that |D 1 | = 7 and we could actually represent one additional value with three bits. However, for the sake of symmetry around zero, we refrain from doing so as we do not want to introduce a bias towards positive or negative values. We empirically observed this range of values to perform well for our problems with inputs x ∈ [−1, 1]. The values in D 1 can be scaled with an arbitrary factor at training time without affecting the output of the NN since only the sign of the activations is relevant. We investigated two different variational distributions q(W 1).(i) General distribution: We store for each weight the seven logits corresponding to the unnormalized log-probabilities for each of the seven values. The normalized probabilities can be recovered using the softmax function. This simple and most general distribution for finite discrete distributions has the advantage that the model can express a tendency towards individual discrete values. Consequently, we expect the maximum of the distribution to be a reasonable value that the model explicitly favors. This is fundamentally different from training full precision networks and quantizing the weights afterwards. The disadvantage of this approach is that the number of variational parameters and the computation time for the means µ w and variances σ w scales with the size of D 1.(ii) Discretized Gaussian: To get rid of the dependency of the number of parameters on |D 1 |, we also evaluated a discretized Gaussian. The distribution is parameterized by a mean m w and a variance v w and the logits of the ing discrete distribution are given by −(w − m w) 2 /(2 v w) for w ∈ D 1 FIG1 ). We denote this distribution as N D1 (m w, v w).3 This parameterization has the advantage that only two parameters are sufficient to represent the ing discrete distribution for an arbitrary size of D 1. Furthermore, the ing distribution is unimodal and neighboring values tend to have similar probabilities which appears natural. Nevertheless, there is no closedform solution for the mean µ w and variance σ w of this distribution and we have to compute a weighted sum involving the |D 1 | normalized probabilities. For the prior distribution p(W 1), we use for both aforementioned variational distributions the discretized Gaussian N D1 (0, γ) with γ being a tunable hyperparameter. Computing the KL-divergence KL(q(W 1)||p(W 1)) also requires computing a weighted sum over |D 1 | values. For the remaining layers we use ternary weights w ∈ D l = {−1, 0, 1}. We use a shifted binomial distribution, i.e. w ∼ Binomial(2, w p) − 1. This distribution requires only a single parameter w p per weight for the variational distribution. The mean µ w is given by 2w p − 1 and the variance σ w is given by 2w p (1 − w p). This makes the Bernoulli distribution an efficient choice for computing the required moments. It is convenient to select a binomial prior distribution p(w) = Binomial(2, 0.5) as it is centered at zero and we get KL(q(w)||p(w)) = |D l |(log(2w p)w p + log(2(1 − w p))(1 − w p)).These favorable properties of the binomial distribution might raise the question why it is not used in the first layer, especially since the required expectations, variances and KL-divergences are available in closed-form independent of the size of D l. We elaborate more on this in Section 5. We normalize the activations of layer l by d l−1. 4 This scales the activation means µ a towards zero and keeps the activation variances σ a independent of the number of incoming neurons. Consequently, the expectation of the Bernoulli distribution after applying the sign activation function µ x = erf(µ a /(2σ a) 1 2 ) is less prone to be in the saturated region of the error function and, thus, gradients can flow backwards in the computation graph. Note that activation normalization influences only the PFP and does not change the classification of individual NNs W ∼ q(W) since only the sign of the activation is relevant. The variational inference objective does not allow to easily trade off between the importance of the expected log-likelihood E q(W) [log p(D|W)] and the KL term KL(q(W)||p(W)). This is problematic since there are usually many more NN weights than there are data samples and the 3 Note that the mean µw and the variance σw of N D 1 (mw, vw) are in general different from mw and vw. 4 The activation variance σa is normalized by d l−1. KL term tends to be orders of magnitudes larger than the expected log-likelihood. As a , the optimization procedure mainly focuses on keeping the approximate posterior q(W) close to the prior p(W) whereas the influence of the data is too small. We propose to counteract this issue by trading off between the expected log-likelihood and the KL term using a convex combination, i.e., DISPLAYFORM0 Here, λ ∈ is a tunable hyperparameter that can be interpreted as creating λ/(1 − λ) copies of the data set D. A similar technique is used in BID17 to avoid getting stuck in poor local optima. Another approach to counteract this issue is the KL-reweighting scheme proposed in BID2. Due to the exponential weight decay, only a few minibatches are influenced by the KL term whereas the vast majority is mostly influenced by the expected log-likelihood. We evaluated the performance of our model (NN VI) on MNIST , variants of the MNIST data set BID14, and a TIMIT data set BID27. Details about the individual data sets can be found in the supplementary material. We selected a three layer structure with d 1 = d 2 = 1200 hidden units for all experiments. We evaluated both the general parameterization and the Gaussian parameterization as variational distribution q(W >1) for the first layer (Section 3.1), and a binomial variational distribution q(W >1) for the following layers (Section 3.2). We optimized the variational distribution using ADAM BID11 without KL reweighting BID2, and using rmsprop with KL reweighting, and report the experiment ing in the better classification performance. For both optimization algorithms, we employ an exponential decay of the learning rate η where we multiply η after every epoch by a factor α ≤ 1. We use dropout BID24 with dropout rate p in for the input layer and a common dropout rate p hid for the remaining hidden layers. We normalize the activation by d l−1 p where p is either p in or p hid to consider dropout in the activation normalization. We tuned the hyperparameters λ DISPLAYFORM0 hid ∈ [0, 0.8] with 50 iterations of Bayesian optimization BID22. We report the for the single most probable model from the approximate posterior W = arg max W q(W). This model is indeed a low precision network that can efficiently be implemented in hardware. We also report by computing predictions using the PFP which can be seen as approximating the expected prediction arg max t E q(W) [p(t|x, W)] as it is desired in Bayesian inference. We compare our model with real-valued NNs (NN real) trained with batch normalization BID9, dropout, and ReLU activation function. We also evaluated the model from BID8 (NN STE) which uses batch normalization, dropout, real-valued weights in the first layer and binary weights w ∈ {−1, 1} in the subsequent layers. The binary weights and the sign activation function are handled using the STE. For NN (real) and NN STE, we tuned the hyperparameters η ∈ [10 −4, 10 DISPLAYFORM1, and p hid ∈ [0, 0.8] on a separate held-out validation set using 50 iterations of Bayesian optimization. The are shown in TAB0. Our model (single) performs on par with NN (real) and it outperforms NN STE on the TIMIT data set and the more challenging variants of MNIST with different kinds of artifacts. Interestingly, our model outperforms the other models on MNIST Background and MNIST Background Random by quite a large margin which could be due to the Bayesian nature of our model. The PFP outperforms the single most probable model. This is no surprise since the PFP uses all the available information from the approximate posterior q(W). Overall, the performance of the general variational distribution seems to be slightly better than the discretized Gaussian at the cost of more computational overhead at training time. On a Nvidia GTX 1080 graphics card, a training epoch on the MNIST data set took approximately 8.8 seconds for NN VI general and 7.5 seconds for NN VI Gauss compared to 1.3 seconds for NN (real). The computational bottleneck of our method is the first layer since here the moments require computing weighted sums over all discrete values w ∈ D 1. Next, we approximate the expected predictions arg max t E q(W) [p(t|x, W)] by sampling several models W ∼ q(W). We demonstrate this on the MNIST Rotated Background data set. NNs with batch normalization, dropout, sign activation function, real-valued weights in the first layer and binary weights in the remaining layers. NN VI (our method): 3 bits for the first layer and ternary weights for the remaining layers. We evaluated the single most probable model, the probabilistic forward pass (pfp), the general 3 bit distribution for the first layer, and the discretized Gaussian for the first layer. shows the classification error of Bayesian averaging over 1000 NNs sampled from the model with the best PFP performance using the discretized Gaussian. We see that the performance approaches the PFP which indicates that the PFP is a good approximation to the true expected predictions. However, the size of the ensemble needed to approach the PFP is quite large and the computation time of evaluating a large ensemble is much larger than a single PFP. Therefore, we investigated a greedy forward selection strategy, where we sample 100 NNs out of which we include only the NN in the ensemble which leads to the lowest error. This is shown in FIG1 (c). Using this strategy in a slightly better performance than Bayesian averaging. Most importantly, averaging only a few models in a decent performance increase while still allowing for faster inference than full precision NNs. Our NNs obtained by taking the most probable model from q(W) can be efficiently implemented in hardware. They require only multiplications with 3 bit fixed point values as opposed to multiplications with floating point values in NN (real) and NN STE. In the special case of image data, the inputs are also given as 8 bit fixed point numbers. By scaling the inputs and the weights from the first layer appropriately, this in an ordinary integer multiplication while leaving the output of the sign activation function unchanged. In the following layers we only have to compute multiplications as logical XNOR operations and accumulate -1 and +1 values for the activations. TAB2 shows the fraction of non-zero weights of the best performing single models from TAB0. Especially in the input layer where we have our most costly 3 bit weights, there are a lot of zero weights on most data sets. This can be utilized to further reduce the computational costs. For example, on the MNIST Background Random data set, evaluating a single NN requires only approximately 23000 integer multiplications and 1434000 XNOR operations instead of approximately 2393000 floating point multiplications. The presented model has many tunable parameters, especially the type of variational distributions for the individual layers, that heavily influence the behavior in terms of convergence at training time and performance at test time. The binomial distribution appears to be a natural choice for evenly spaced values with many desirable properties. It is fully specified by only a single parameter, and its mean, variance, and KL divergence with another binomial has nice analytic expressions. Furthermore, neighboring values have similar probabilities which rules out odd cases in which, for instance, there is a value with low probability in between of two values with high probability. Unfortunately, the binomial distribution is not suited for the first layer as here it is crucial to be able to set weights with high confidence to zero. However, when favoring zero weights by setting w p = 0.5, the variance of the binomial distribution takes on its largest possible value. This might not be a problem in case predictions are computed as the true expectations with respect to q(W) as in the PFP, but it in bad classification errors when deriving a single model from q(W). We also observed that using the binomial distribution in deeper layers favor the weights −1 and 1 over 0 (cf. TAB2). This might indicate that binary weights w ∈ {−1, 1} using a Bernoulli distribution could be sufficient, but in our experiments we observed this to perform worse. We believe this to stem partly from the importance of the zero weight and partly from the larger variance of 4w p (1 − w p) of the Bernoulli distribution compared to the variance of 2w p (1 − w p) of the binomial distribution. Furthermore, there is a general issue with the sign activation functions if the activations are close to zero. In this case, a small change to the inputs can cause the corresponding neuron to take on a completely different value which might have a large impact on the following layers of the NN. We found dropout to be a very helpful tool to counteract this issue. FIG1 shows histograms of the activations of the second hidden layer for both a model trained with dropout and the same model trained without dropout. We can see that without dropout the activations are much closer to zero whereas dropout introduces a much larger spread of the activations and even causes the histogram to decrease slightly in the vicinity of zero. Thus, the activations are much more often in regions that are stable with respect to changes of their inputs which makes them more robust. We believe that such regularization techniques are crucial if the sign activation function is used. We introduced a method to infer NNs with low precision weights. As opposed to existing methods, our model neither quantizes the weights of existing full precision NNs nor does it rely on heuristics to compute "approximated" gradients of functions whose gradient is zero almost everywhere. We perform variational inference to obtain a distribution over a discrete weight space from which we subsequently derive a single discrete-valued NN or a small ensemble of discrete-valued NNs. Our method propagates probabilities through the network which in a well defined function that allows us to optimize the discrete distribution even for the sign activation function. The weights in the first layer are modeled using fixed point values with 3 bits precision and the weights in the remaining layers have values w ∈ {−1, 0, 1}. This reduces costly floating point multiplications to cheaper multiplications with fixed point values of 3 bits precision in the first layer, and logical XNOR operations in the following layers. In general, our approach allows flexible bit-widths for each individual layer. We have shown that the performance of our model is on par with state of the art methods that use a higher precision for the weights. Furthermore, our model exhibits a large amount of sparsity that can be utilized to further reduce the computational overhead. A DATA SETS The MNIST data set BID15 contains grayscale images of size 28 × 28 showing handwritten digits. It is split into 50000 training samples, 10000 validation samples, and 10000 test samples. The task is to classify the images to digits. The pixel intensities are normalized to the range [−1, 1] by dividing through 128 and subtracting 1. We use the MNIST data set in the permutationinvariant setting where the model is not allowed to use prior knowledge about the image structure, i.e., convolutional NNs are not allowed. The variants of the MNIST data set BID14 contain images of size 28 × 28 showing images of the original MNIST data set that have been transformed by various operations in order to obtain more challenging data sets. The variants of the MNIST data set are split into 10000 training samples, 2000 validation samples and 50000 test samples. In particular, there are the following variants:• MNIST Basic: This data set has not been transformed. The data set is merely split differently into training, validation, and test set, respectively.• MNIST Background: The pixels of the images have been replaced by random image patches.• MNIST Background Random: The pixels of the images have been set to a uniformly random pixel value.• MNIST Rotated: The images are randomly rotated.• MNIST Rotated Background: The transformations from MNIST Rotated and MNIST Background are combined Some samples of the individual data sets are shown in FIG3. We also normalized the pixel intensities of these data sets to lie in the range [−1, 1]. The TIMIT data set BID27 contains samples of 92 features representing a phonetic segment. The task is to classify the phonetic segment to one of 39 phonemes. The data is split into 140173 training samples, 50735 validation samples (test) and 7211 test samples (core test). Details on data preprocessing can be found in BID5. We normalized the features to have zero mean and unit variance.

represent the tldr for finding the corresponding scientific article
The provided query could be "We propose a novel combination of adversarial training and provable defenses which produces a model with state-of-the-art accuracy and certified robustness on CIFAR-10. " and the positive "We propose a new method to train neural networks based on a novel combination of adversarial training and provable defenses. The key idea is to model training as a procedure which includes both, the verifier and the adversary. In every iteration, the verifier aims to certify the network using convex relaxation while the adversary tries to find inputs inside that convex relaxation which cause verification to fail. We experimentally show that this training method is promising and achieves the best of both worlds – it produces a model with state-of-the-art accuracy (74.8%) and certified robustness (55.9%) on the challenging CIFAR-10 dataset with a 2/255 L-infinity perturbation. This is a significant improvement over the currently known best of 68.3% accuracy and 53.9% certified robustness, achieved using a 5 times larger network than our work. The discovery of adversarial examples in deep learning has increased the importance of creating new training methods which produce accurate and robust neural networks with provable guarantees. Existing work: adversarial and provable defenses Adversarial training provides a basis framework which augments the training procedure with adversarial inputs produced by an adversarial attack. instantiated adversarial training using a strong iterative adversary and showed that their approach can train models which are highly robust against the strongest known adversarial attacks . This method has also been used to train robust ImageNet models . While promising, the main drawback of the method is that when instantiated in practice, via an approximation of an otherwise intractable optimization problem, it provides no guarantees -it does not produce a certificate that there are no possible adversarial attacks which could potentially break the model. To address this lack of guarantees, recent line of work on provable defenses; has proposed to train neural networks which are certifiably robust under a specific attacker threat model. However, these guarantees come at the cost of a significantly lower standard accuracy than models trained using adversarial training. This setting raises a natural question -can we leverage ideas from both, adversarial training techniques and provable defense methods, so to obtain models with high accuracy and certified robustness? This work: combining adversarial and provable defenses In this work, we take a step towards addressing this challenge. We show that it is possible to train more accurate and provably robust neural networks using the same convex relaxations as those used in existing, state-of-the-art provable defense methods, but with a new, different optimization procedure inspired by adversarial training. Our optimization works as follows: (i) to certify a property (e.g., robustness) of the network, the verifier produces a convex relaxation of all possible intermediate vector outputs in the neural network, then (ii) an adversary now searches over this (intermediate) convex regions in order to find, what we refer to as a latent adversarial example -a concrete intermediate input contained in the relaxation that when propagated through the network causes a misclassification that prevents verification, and finally (iii) the ing latent adversarial examples are now incorporated into our training scheme using adversarial training. Overall, we can see this method as bridging the gap between adversarial training and provable defenses (it can conceptually be instantiated with any convex relaxation). We experimentally show that the method is promising and in a neural network with state-of-theart 78.8% accuracy and 58.1% certified robustness on the challenging CIFAR-10 dataset with 2/255 L ∞ perturbation (the best known existing are 68.3% accuracy and 53.9% certified robustness using 5 times larger network). • A new method we refer to as layerwise adversarial training which can train provably robust neural networks and conceptually bridges the gap between adversarial training and existing provable defense methods. • Instantiation of layerwise adversarial training using linear convex relaxations used in prior work (accomplished by introducing a projection operator). • Experimental showing layerwise adversarial training can train neural network models which achieve both, state-of-the-art accuracy and certified robustness on CIFAR-10 with 2/255 L ∞ perturbation. Overall, we believe the method presented in this work is a promising step towards training models that enjoy both, higher accuracy and higher certification guarantees. An interesting item for future work would be to explore instantiations of the method with other convex relaxations than the one considered here. We now discuss some of the closely related work on robustness of neural networks. Heuristic adversarial defenses After the first introduction of adversarial examples , defense mechanisms to train robust neural networks were built based on the inclusion of adversarial examples to the training set . Models trained using adversarial training with projected gradient descent (PGD) were shown to be robust against the strongest known attacks . This is in contrast to other defense mechanisms which have been broken by new attack techniques . While models trained using adversarial training achieve robustness against strong adversaries, there are no guarantees that model is robust against any kind of adversarial attack under the threat model considered. Provable adversarial defenses Another line of work proposes to learn classifiers which come with robustness guarantees. These approaches are based on linear or semidefinite (; a) relaxations, hybrid zonotope or interval bound propagation . While these approaches currently obtain robustness guarantees, accuracy of these networks is relatively small and limits practical use of these methods. There has also been recent work on certification of general neural networks, not necessarily trained in a special way. These methods are based on SMT solvers , mixed-integer linear programs, abstract interpretation, restricted polyhedra (b) or combinations of those (a). Another line of work proposes to replace neural networks with a randomized classifier (; ; a) which comes with probabilistic guarantees on its robustness. While these approaches scale to larger datasets such as ImageNet (although with probabilistc instead of exact guarantees), their bounds come from the relationship between L 2 robustness and Gaussian distribution. In this paper, we consider general verification problem where input is not necessarily limited to an L p ball, but arbitrary convex set. In this work we consider a threat model where an adversary is allowed to transform an input x ∈ R d0 into any point from a convex set S 0 (x) ⊆ R d0. For example, for a threat model based on L ∞ perturbations, the convex set will be defined as Figure 1: An iteration of layerwise adversarial training. Latent adversarial example x 1 is found in the convex region C 1 (x) and propagated through the rest of the layers in a forward pass which is shown with the blue line. During backward pass, gradients are propagated through the same layers, shown with the red line. Note that the first convolutional layer does not receive any gradients. We now describe our layerwise adversarial training approach which yields a provable defense that bridges the gap between standard adversarial training and existing provable defenses. Motivation: latent adversarial examples Consider an already trained neural network model h θ which we would like to certify using convex relaxations. A fundamental issue here is that certification methods based on convex relaxations can struggle to prove the target property (e.g., robustness) due to the iterative accumulation of errors introduced by the relaxation. More precisely, assume the neural network actually satisfies the property from Equation 1 for an input x, meaning that. Naturally, this also implies that the neural network behaves correctly in the latent space of its first hidden layer in the region S 1 (x). Formally, this means that c T h 2:k θ (x 1)+d < 0, ∀x 1 ∈ S 1 (x). However, if one would use a certification method which replaces the region S 1 (x) by its convex relaxation C 1 (x), then it is possible that we would fail to certify our desired property. This is due to the fact that there may exist an input Of course, we could repeat the above thought experiment and possibly find more violating latent inputs in the set C i (x) \ S i (x) of any hidden layer i. The existence of points found in the difference between a convex relaxation and the true region is a fundamental reason for the failure of certification methods based on convex approximations. For convenience, we refer to such points as latent adversarial examples. Next, we describe a method which trains the neural network in a way that aims to minimize the number of latent adversarial examples. Layerwise provable optimization via convex relaxations Our key observation is that the two families of defense methods described earlier are in fact different ends of the same spectrum: methods based on adversarial training maximize the cross-entropy loss in the first convex region C 0 (x) while provable defenses maximize the same loss, but in the last convex region C k (x). Both methods then backpropagate the loss through the network and update the parameters using SGD. However, as explained previously, certification methods may fail even before the last layer due to the presence of latent adversarial examples in the difference of the regions C i (x) and S i (x). A natural question then is -can we leverage adversarial training so to eliminate latent adversarial examples from hidden layers and obtain a provable network? To this end, we propose adversarial training in layerwise fashion. The initial phase of training is equivalent to adversarial training as used by. In this phase in the inner loop we repeatedly find an input in C 0 (x) which maximizes the cross-entropy loss and update the parameters of the neural network so to minimize this loss using SGD. Note that the outcome of this phase is a model which is highly robust against strong multi-step adversaries. However, certification of this fact often fails due to the previously mentioned accumulation of errors in the particular convex relaxation being used. The next step of our training method is visually illustrated in Figure 1. Here, we propagate the initial convex region through the first layer of the network and obtain the convex relaxation C 1 (x). We then solve the optimization problem to find a concrete point x 1 inside of C 1 (x) which produces Algorithm 1: Layerwise adversarial training via convex relaxations Data: k-layer neural network h θ, training set (X, Y), learning rate η, step size α, inner steps n Result: Update in parallel n times: Freeze parameters θ l+1 of layer l + 1; 12 end the maximum loss when this point is propagated further through the network (this forward pass is shown with the blue line). Finally, we backpropagate the final loss (red line) and update the parameters of the network so to minimize the loss. Critically, we do not backpropagate through the convex relaxation in the first layer as standard provable defenses do ). We instead freeze the first layer and stop backpropagation after the update of the second layer. Because of this, our optimization problem is significantly easier -the neural network only has to learn to behave well on the concrete points that were found in the convex region C l (x). This can be viewed as an extension of the robust optimization method that found to work well in practice. We then proceed with the above process for later layers. Formally, this training process amounts to (approximately) solving the following min-max optimization problem at the l-th step: Note that for l = 0 this formulation is equivalent to the standard min-max formulation in Equation 2 because C 0 (x) = S 0 (x). Our approach to solve this min-max optimization problem for every layer l is shown in Algorithm 1. We initialize every batch by random sampling from the corresponding convex region. Then, in every iteration we use projected gradient descent (PGD) to maximize the inner loss in 3. We first update x j in the direction of the gradient of the loss and then project it back to C l (x j) using the projection operator Π. Note that this approach assumes the existence of an efficient projection method to the particular convex relaxation the method is instantiated with. In the next section, we show how to instantiate the training algorithm described above to a particular convex relaxation which is generally tighter than a hyperrectangle and where we derive an efficient projection operation. So far we described the general approach of layerwise adversarial training. Now we show how to instantiate it for a particular convex relaxation based on linear approximations. If instead one would use interval approximation ) as the convex relaxation, then all regions C l (x) will be hyperrectangles and projection to these sets is fast and simple. However, the interval relaxation provides a coarse approximation which motivates the need to train with relaxations that provide tighter bounds. Thus, we consider linear relaxations which are generally tighter than those based on intervals. In particular we leverage the same relaxation which was previously proposed in;; as an effective way to certify neural networks. Here, each convex region is represented as a set Vector a l represents the center of the set and the matrix A l represents the affine transformation of the hypercube [−1, 1] m l. The initial convex region C 0 (x) is represented using a 0 = x and A 0 = I d0 is a diagonal matrix. Propagation of these convex regions through the network is out of the scope of this paper -a full description can be found in or. At a high level, the convolutional and fully connected layers are handled by multiplying A l and a l by appropriate matrices. To handle the ReLU activation, for ReLU units which cross 0, we apply a convex relaxation which amounts to multiplying A l and a l by appropriately chosen scalar values, depending whether the ReLU is activated or not. Using this relaxation of ReLU, we can recursively obtain all convex regions C l (x). In practice, A l e can be computed without explicitly constructing matrix A l because A l e = W l Λ l−1 W l−2 · · · M 0 e. Then we can perform matrix-vector multiplication right to left to obtain vector A l e. We provide more detailed description of this propagation in Appendix A. Projection to linear convex regions To use our training method we now need to instantiate Algorithm 1 with a suitable projection operator Π C l (x). The key insight here is that the vector x ∈ C l (x) is uniquely determined by auxiliary vector e ∈ [−1, 1] m l where x = a l + A l e. Then instead of directly solving for x which requires projecting to C l (x), we can solve for e instead which would uniquely determine x. Crucially, the domain of e is a hyperrectangle [−1, 1] m l which is easy to project to. To visualize this further we provide an example in Figure 2. The goal is to project the red point x in the right picture to the convex region C l (x). To project, we first perform change of variables to substitute x with e and then project e to the square [−1, 1] × [−1, 1] to obtain the blue point Π(e) on the left. Then, we again perform change of variables to obtain the blue point Π(x) on the right, the projection of x we were looking for. Based on these observations, we modify Line 7 of Algorithm 1 to first update the coefficients e j using the following update rule: e j ← clip(e j + αA Here clip is function which thresholds its argument between -1 and 1, formally clip(x, −1, 1) = min(max(x, −1), 1). This is followed by an update to x j via x j ← a l + A l e j, completing the update step. Sparse representation While our representation of convex regions with matrix A l and vector a l has clean mathematical properties, in practice, a possible issue is that the matrix A l can grow to be quite large. Because of this, propagating it through the network can be memory intensive and prohibit the use of larger batches. To overcome this difficulty, we first observe that A l is quite sparse. We start with a very sparse, diagonal matrix A 0 at the input. After each convolution, an element of matrix A l+1 is non-zero only if there is a non-zero element inside of its convolutional kernel in matrix A l. We can leverage this observation to precompute positions of all non-zero elements in matrix A l+1 and compute their values using matrix multiplication. This optimization is critical to enabling training to take place altogether. An interesting item for future work is further optimizing the current relaxation (via a specialized GPU implementation) or developing more memory friendly relaxations, so to scale the training to larger networks. After training a neural network via layerwise adversarial training, our goal is to certify the target property (e.g., robustness). Here we leverage certification techniques which are not fast enough to be incorporated into the training procedure, but which can significantly boost the certification performance. The linear relaxation of ReLU that we are using is parameterized by slopes λ of the linear relaxation. Prior work which employed this relaxation; ) chose these slopes in a greedy manner by minimizing the area of the relaxation. During training we also choose λ in the same way. However, during certification, we can also optimize for the values of λ that give rise to the convex region inside of which the maximum loss is minimized. This optimization problem can be written as: Solving this is computationally too expensive inside the training loop, but during certification it is feasible to approximate the solution. We solve for λ using the Adam optimizer and clipping the elements between 0 and 1 after each update. We remark that the idea of learning the slope is similar to Dvijotham et al. (2018b) who propose to optimize dual variables in a dual formulation, however here we stay in the primal formulation. Combining convex relaxations with exact bound propagation During layerwise adversarial training we essentially train the network to be certified on all regions C 0 (x),..., C k (x). While computing exact regions S l (x) ⊆ C l (x) is not feasible during training, we can afford it to some extent during certification. The idea is to first propagate the bounds using convex relaxations until one of the hidden layers l and obtain a region C l (x). If training was successful, there should not exist a concrete point x l ∈ C l (x) which, if propagated through the network, violates the correctness property in Equation 1. We can encode both, the property and the propagation of the exact bounds S l (x) using a Mixed-Integer Linear Programming (MILP) solver. Note that we can achieve this because we represent the region C l (x) using a set of linear constraints, however, for general convex shapes this may not be possible. We perform the MILP encoding using the formulation from. It is usually possible to encode only the last two layers using MILP due to the poor scalability of these solvers for realistic network sizes. One further improvement we also include is to tighten the convex regions C l (x) using refinement via linear programming as described in Singh et al. (2019a). We remark that this combination of convex relaxation and exact bound propagation does not fall under the recently introduced convex barrier to certification Salman et al. (2019b). We now present an evaluation of our training method on the challenging CIFAR-10 dataset. Experimental setup We evaluate on a desktop PC with 2 GeForce RTX 2080 Ti GPU-s and 16-core Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz. We use Gurobi as a MILP solver. Our method is implemented in PyTorch and we plan to release both, the code and the trained models. Neural network architecture All presented are on a 4-layer convolutional network with 49 402 neurons: first 3 layers are convolutional layers with filter sizes 32, 32 and 128, kernel sizes 3, 4, 4 and strides 1, 2, 2 respectively. These are followed by a fully connected layer with 250 hidden units. After each of these layers, there is a ReLU activation. Training We use batch size 50 and L1 regularization 0.00001 for training. We perform optimization using Adam with initial learning rate 0.001 which is decreased by 10× every 100 epochs. During layerwise training we start with perturbation which is 10% higher than the one we certify and we decrease it by 5% when the training progresses to the next layer. Certification After training completes, we perform certification as follows: for every image, we first try to certify it using only linear relaxations (with the improvement of learned slopes, Section 6). If this fails, we encode the last layer as MILP and try again. Finally, if this fails we encode the ReLU activation after the last convolution using additional 300 binary variables and the rest using the triangle formulation. We consider an image to be not certifiable if we fail to verify it using these methods. We always evaluate on the first 1 000 images from the test set. Comparison to prior work We first train and certify using our method for the L ∞ perturbation 2/255. Results are shown in Table 1. We always compare to the best reported and reproducible et al. 68.3 53.9 70.1 50.0 59.9 46.1 62.3 45.5 61.1 45.9, as this improvement is also orthogonal to the method here. Thus, we only consider their best single network architecture (inline with prior work which compares to a single architecture). We believe all methods listed in Table 1, including ours, would benefit from additional techniques such as cascades, pre-training and leveraging unlabeled data. Experimentally, we find that the neural network trained using our method substantially outperforms all existing approaches, both in terms of standard accuracy and certified robustness for 2/255. Note that here we are using the same linear relaxation as, but our optimization procedure is different and shows significant improvements over the one used in their work. We also run the same experiment for L ∞ perturbation 8/255. Here we do not include comparison with as their were found to be not reproducible (; ;). These are presented in Table 2. Here we substantially outperform all existing approaches in terms of standard accuracy. However, in terms of certified robustness we are not able to achieve similar to whose method is based on a combination of interval approximation and linear relaxation. The main issue is that our 4-layer network lacks capacity to solve this task -even if training only using standard adversarial training our empirical robustness does not go above ∼ 34%. We remark that capacity was found to be one of the key components necessary to obtain a robust classifier . Due to promising for 2/255, we believe achieving state-of-the-art for 8/255 is very likely an issue of instantiating our method with a convex relaxation that is more memory efficient, which we believe is an interesting item for future work. We presented a new method to train certified neural networks. The key concept was to combine techniques from provable defenses using convex relaxations with those of adversarial training. Our method achieves state-of-the-art 78.8% accuracy and 58.1% certified robustness on CIFAR-10 with a 2/255 L ∞ perturbation, significantly outperforming prior work when considering a single network (it also achieves competitive on 8/255 L ∞). The method is general and can be instantiated with any convex relaxation. A promising future work item is scaling to larger networks: this will require tight convex relaxations with a low memory footprint that allow for efficient projection. Here we provide additional details that were omitted in the main body of the paper. In this section, we describe how to propagate convex relaxations of the form C l (x) = {a l + A l e | e ∈ [−1, 1] m l } through the network, for a single input x. As explained before, these relaxations were previously proposed in;. For the sake of completeness we describe them here using our notation. Depending on the form of function h i θ representing operation applied at layer i we distinguish different cases. Here we assume we have obtained region C i−1 (x) and our goal is to compute the region C i (x) using convex relaxation g i θ of the function h i θ. Initial convex region Let be L ∞ radius that we are certifying. We then compute minimum and maximum pixel values for each pixel as x l = max(0, x −) and x u = min(1, x +). We define initial convex region as: Convolutional and fully connected layers For both convolutional and fully connected layers, the update is given by mi. We can then compute: Using this formula, we define convex region C i+1 (x) = {a i+1 + A i+1 e | e ∈ [−1, 1] mi+1 } where: We will explain the transformation of a single element x i,j = a i,j + A T i,j e. We first compute lower bound l i,j and upper bound u i,j of element x i in the set C i (x): In the other case where 0 is between l i,j and u i,j we define ReLU (x i,j) = λ i,j x i,j + µe mi+j where e mi+j ∈ [−1, 1] is a coefficient for a new error term. Formulas for λ i,j and µ i,j are the following: This computation can also be written in the matrix form as ReLU (x i) = Λ i+1 x i + M i+1 e new where Λ i+1 and M i+1 are diagonal matrices with elements computed as above. Finally, new convex region C i+1 (x) = {a i+1 + A i+1 e | e ∈ [−1, 1] mi+1 } is defined as: where [] denotes concatenation of matrices. Here we describe how we apply random projection approach from to estimate the bounds during training. While operate in dual framework, their method to statistically estimate the bounds during training can also be applied in primal framework which we use in this paper. Recall that lower and upper bound for each neuron are computed as Thus, we need to compute ||A i || 1, which is L 1 norm of each row in the matrix A i. Using the method from, based on the from , we estimate ||A i || 1 with the method of random projections. Here A i is a matrix with d i rows and m i columns, where d i is dimensionality of the output vector in the i-th layer and m i is number of unit terms in the definition of region C i (x). The method of random projections samples standard Cauchy random matrix R of dimensions m i × k (k is number of projections) and then estimates ||A i || 1 ≈ median(|A i R|). To avoid computing the entire matrix A i we substitute: In the formula above, we set M 0 = A 0. Now, Dvijotham et al. (2018a) 83.4 62.4 To calculate A i R we split R = [R 0, R 2, ..., R i−1] and compute: Crucially, each summand can now be efficiently computed due to the associativity of matrix multiplication by performing the multiplication backwards. In this section we present additional on SVHN and MNIST datasets. SVHN We evaluated on SVHN dataset. For this experiment, we used convolutional network with 2 convolutional layers of kernel size 4 and stride 1 with 32 and 128 filters respectively. These convolutional layers are followed by a fully connected layer with 200 neurons. Each of the layers is followed by a ReLU activation function. For our training, we started with perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer. We trained each layer for 50 epochs and used L 1 regularization factor 0.0001. Results of this experiment are shown in Table 3. We certified first 1 000 images in SVHN test dataset. Our network has both, higher accuracy and higher certified robustness, than networks trained using other techniques for provable defense. MNIST In the next experiment, we evaluted on MNIST dataset. For this experiment, we used convolutional network with 2 convolutional layers with kernel sizes 5 and 4, and strides 2 followed by 1 fully connected layer. Each of the layers is followed by a ReLU activation function. For our training, we started with perturbation 10% higher than the one we are certifying and decreased it by 5% when progressing to the next layer. We trained each layer for 50 epochs and used L 1 regularization factor 0.00001. We certified first 1 000 images in MNIST test dataset. For perturbation 0.1, convolutional layers have filter sizes 32 and 64, and fully connected layer has 150 neurons. Results are presented in Table 4. Here, our numbers are comparable to those of state-of-the-art approaches. For perturbation 0.3, convolutional layers have filter sizes 32 and 128, and fully connected layer has 400 neurons. Results are presented in Table 4. Here, our certified robustness is somewhat lower than state-of-the-art. We believe this is due to the imprecise estimates of lower and upper bund via random projections. This is also reflected in relatively poor performance of who also rely on the same statistical estimates. Thus, for MNIST dataset and perturbation 0.3 it is likely necessary to use exact propagation instead of the estimates. However, this also induces large cost to the runtime. During training, we use two regularization mechanisms to make our convex relaxation tighter, both previosuly proposed in. First, we use L 1 -norm regularization which is known to induce sparsity in the weights of the network. has shown that weight sparsity helps induce more stable ReLU units which in turn makes our convex relaxation tighter (as for stable ReLU units it is already precise). 98.9 97.7 99.0 94. 4 98.9 96.3 Dvijotham et al. (2018a) 98.8 95. 6 98.7 95.8 99.0 95.6 Table 5: Evaluation on MNIST dataset with L ∞ perturbation 0.3 Method Accuracy (%) Certified robustness (%) Our work 97.6 84. 6 98.3 91.9 98.5 91.5 85.1 56.9 96.6 89.3 97.3 80.7 Second, in the i-th phase of training, we explicitly introduce a loss based on the volume of convex region C i+1. To make the relaxation tighter and minimize the volume, for each neuron j in layer i + 1 we add a loss of the form max(0, −l i+1,j) max(0, u i+1,j). This loss corresponds to the area under ReLU relaxation, see e.g. for a derivation."

Variational Inference for infering a discrete distribution from which a low-precision neural network is derived

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.