Question: Represent the scientific article to find a short TLDR

Question: Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data. Stability analysis lies at the heart of many scientific and engineering disciplines. In an unstable system, infinitesimal perturbations amplify and have substantial impacts on the performance of the system. It is especially critical to perform a thorough stability analysis on complex engineered systems deployed in practice, or else what may seem like innocuous perturbations can lead to catastrophic consequences such as the Tacoma Narrows Bridge collapse and the Space Shuttle Challenger disaster . As a rule of thumb, well-engineered systems should be robust against any input shifts -expected or unexpected. Most models in machine learning are complex nonlinear systems and thus no exception to this rule. For instance, a reliable model must withstand shifts from training data to unseen test data, bridging the so-called generalization gap. This problem is severe especially when training data are strongly biased with respect to test data, as in domain-adaptation tasks, or when only sparse sampling of a true underlying distribution is available, as in few-shot learning. Any instability in the system can further be exploited by adversaries to render trained models utterly useless (; ; ; a; ; ; ;). It is thus of utmost importance to ensure that models be stable against perturbations in the input space. Various regularization schemes have been proposed to improve the stability of models. For linear classifiers and support vector machines , this goal is attained via an L 2 regularization which maximizes classification margins and reduces overfitting to the training data. This regularization technique has been widely used for neural networks as well and shown to promote generalization (; ;). However, it remains unclear whether or not L 2 regularization increases classification margins and stability of a network, especially for deep architectures with intertwining nonlinearity. In this paper, we suggest ensuring robustness of nonlinear models via a Jacobian regularization scheme. We illustrate the intuition behind our regularization approach by visualizing the classification margins of a simple MNIST digit classifier in Figure 1 (see Appendix A for more). Decision cells of a neural network, trained without regularization, are very rugged and can be unpredictably unstable (Figure 1a). On average, L 2 regularization smooths out these rugged boundaries but does not necessarily increase the size of decision cells, i.e., does not increase classification margins (Figure 1b). In contrast, Jacobian regularization pushes decision boundaries farther away from each training data point, enlarging decision cells and reducing instability (Figure 1c). The goal of the paper is to promote Jacobian regularization as a generic scheme for increasing robustness while also being agnostic to the architecture, domain, or task to which it is applied. In support of this, after presenting the Jacobian regularizer, we evaluate its effect both in isolation as well as in combination with multiple existing approaches that are intended to promote robustness and generalization. Our intention is to showcase the ease of use and complimentary nature of our proposed regularization. Domain experts in each field should be able to quickly incorporate our regularizer into their learning pipeline as a simple way of improving the performance of their state-of-the-art system. The rest of the paper is structured as follows. In Section 2 we motivate the usage of Jacobian regularization and develop a computationally efficient algorithm for its implementation. Next, the effectiveness of this regularizer is empirically studied in Section 3. As regularlizers constrain the learning problem, we first verify that the introduction of our regularizer does not adversely affect learning in the case when input data remain unperturbed. Robustness against both random and adversarial perturbations is then evaluated and shown to receive significant improvements from the Jacobian regularizer. We contrast our work with the literature in Section 4 and conclude in Section 5. Here we introduce a scheme for minimizing the norm of an input-output Jacobian matrix as a technique for regularizing learning with stochastic gradient descent (SGD). We begin by formally defining the input-output Jacobian and then explain an efficient algorithm for computing the Jacobian regularizer using standard machine learning frameworks. Let us consider the set of classification functions, f, which take a vectorized sensory signal, x ∈ R I, as input and outputs a score vector, z = f (x) ∈ R C, where each element, z c, is associated with likelihood that the input is from category, c. 1 In this work, we focus on learning this classification function as a neural network with model parameters θ, though our findings should generalize to any parameterized function. Our goal is to learn the model parameters that minimize the classification objective on the available training data while also being stable against perturbations in the input space so as to increase classification margins. 1 Throughout the paper, the vector z denotes the logit before applying a softmax layer. The probabilistic output of the softmax pc relates to zc via pc ≡ The input-output Jacobian matrix naturally emerges in the stability analysis of the model predictions against input perturbations. Let us consider a small perturbation vector, ∈ R I, of the same dimension as the input. For a perturbed input x = x +, the corresponding output values shift to where in the second equality the function was Taylor-expanded with respect to the input perturbation and in the third equality the input-output Jacobian matrix, was introduced. As the function f is typically almost everywhere analytic, for sufficiently small perturbations the higher-order terms can be neglected and the stability of the prediction is governed by the input-output Jacobian. From Equation, it is straightforward to see that the larger the components of the Jacobian are, the more unstable the model prediction is with respect to input perturbations. A natural way to reduce this instability then is to decrease the magnitude for each component of the Jacobian matrix, which can be realized by minimizing the square of the Frobenius norm of the input-output Jacobian, For linear models, this reduces exactly to L 2 regularization that increases classification margins of these models. For nonlinear models, however, Jacobian regularization does not equate to L 2 regularization, and we expect these schemes to affect models differently. In particular, predictions made by models trained with the Jacobian regularization do not vary much as inputs get perturbed and hence decision cells enlarge on average. This increase in stability granted by the Jacobian regularization is visualized in Figure 1, which depicts a cross section of the decision cells for the MNIST digit classification problem using a nonlinear neural network . The Jacobian regularizer in Equation can be combined with any loss objective used for training parameterized models. Concretely, consider a supervised learning problem modeled by a neural network and optimized with SGD. At each iteration, a mini-batch B consists of a set of labeled examples, {x α, y α} α∈B, and a supervised loss function, L super, is optimized possibly together with some other regularizer R(θ) -such as L 2 regularizer λWD 2 θ 2 -over the function parameter space, by minimizing the following bare loss function To integrate our Jacobian regularizer into training, one instead optimizes the following joint loss where λ JR is a hyperparameter that determines the relative importance of the Jacobian regularizer. By minimizing this joint loss with sufficient training data and a properly chosen λ JR, we expect models to learn both correctly and robustly. 2 Minimizing the Frobenius norm will also reduce the L 1 -norm, since these norms satisfy the inequalities ||J(x)||F ≤ i,c Jc;i (x) ≤ √ IC||J(x)||F. We prefer to minimize the Frobenius norm over the L 1 -norm because the ability to express the former as a trace leads to an efficient algorithm [see Equations through]. In the previous section we have argued for minimizing the Frobenius norm of the input-output Jacobian to improve robustness during learning. The main question that follows is how to efficiently compute and implement this regularizer in such a way that its optimization can seamlessly be incorporated into any existing learning paradigm. Recently, Sokolić et al. also explored the idea of regularizing the Jacobian matrix during learning, but only provided an inefficient algorithm requiring an increase in computational cost that scales linearly with the number of output classes, C, compared to the bare optimization problem (see explanation below). In practice, such an overhead will be prohibitively expensive for many large-scale learning problems, e.g. ImageNet classification has C = 1000 target classes . (Our scheme, in contrast, can be used for ImageNet: see Appendix H.) Here, we offer a different solution that makes use of random projections to efficiently approximate the Frobenius norm of the Jacobian. 3 This only introduces a constant time overhead and can be made very small in practice. When considering such an approximate algorithm, one naively must trade off efficiency against accuracy for computing the Jacobian, which ultimately trades computation time for robustness. Prior work by briefly considers an approach based on random projection, but without providing any analysis on the quality of the Jacobian approximation. Here, we describe our algorithm, analyze theoretical convergence guarantees, and verify empirically that there is only a negligible difference in model solution quality between training with the exact computation of the Jacobian as compared to training with the approximate algorithm, even when using a single random projection (see Figure 2). Given that optimization is commonly gradient based, it is essential to efficiently compute gradients of the joint loss in Equation and in particular of the squared Frobenius norm of the Jacobian. First, we note that automatic differentiation systems implement a function that computes the derivative of a vector such as z with respect to any variables on which it depends, if the vector is first contracted with another fixed vector. To take advantage of this functionality, we rewrite the squared Frobienus norm as where a constant orthonormal basis, {e}, of the C-dimensional output space was inserted in the second equality and the last equality follows from definition and moving the constant vector inside the derivative. For each basis vector e, the quantity in the last parenthesis can then be efficiently computed by differentiating the product, e · z, with respect to input parameters, x. Recycling that computational graph, the derivative of the squared Frobenius norm with respect to the model parameters, θ, can be computed through backpropagation with any use of automatic differentiation. Sokolić et al. essentially considers this exact computation, which requires backpropagating gradients through the model C times to iterate over the C orthonormal basis vectors {e}. Ultimately, this incurs computational overhead that scales linearly with the output dimension C. Instead, we further rewrite Equation in terms of the expectation of an unbiased estimator where the random vectorv is drawn from the (C − 1)-dimensional unit sphere S C−1. Using this relationship, we can use samples of n proj random vectorsv µ to estimate the square of the norm as which converges to the true value as O(n −1/2 proj). The derivation of Equation and the calculation of its convergence make use of random-matrix techniques and are provided in Appendix B. Finally, we expect that the fluctuations of our estimator can be suppressed by cancellations within a mini-batch. With nearly independent and identically distributed samples in a mini-batch of size The difference between the exact method (cyan) and the random projection method with n proj = 1 (blue) and n proj = 3 (red orange) is negligible both in terms of accuracy (a) and the norm of the input-output Jacobian (b) on the test set for LeNet' models trained on MNIST with λ JR = 0.01. Shading indicates the standard deviation estimated over 5 distinct runs and dashed vertical lines signify the learning rate quenches. Algorithm 1 Efficient computation of the approximate gradient of the Jacobian regularizer. Inputs: mini-batch of |B| examples x α, model outputs z α, and number of projections n proj. Outputs: Square of the Frobenius norm of the Jacobian J F and its gradient ∇ θ J F. Uniform sampling from the unit sphere for each α. 1, we expect the error in our estimate to be of order (n proj |B|) −1/2. In fact, as shown in Figure 2, with a mini-batch size of |B| = 100, single projection yields model performance that is nearly identical to the exact method, with computational cost being reduced by orders of magnitude. The complete algorithm is presented in Algorithm 1. With a straightforward implementation in PyTorch and n proj = 1, we observed the computational cost of the training with the Jacobian regularization to be only ≈ 1.3 times that of the standard SGD computation cost, while retaining all the practical benefits of the expensive exact method. In this section, we evaluate the effectiveness of Jacobian regularization on robustness. As all regularizers constrain the learning problem, we begin by confirming that our regularizer effectively reduces the value of the Frobenius norm of the Jacobian while simultaneously maintaining or improving generalization to an unseen test set. We then present our core , that Jacobian regularization provides significant robustness against corruption of input data from both random and adversarial perturbations (Section 3.2). In the main text we present mostly with the MNIST dataset; the corresponding experiments for the CIFAR-10 and ImageNet datasets are relegated to Appendices E and H. The following specifications apply throughout our experiments: Datasets: The MNIST data consist of black-white images of hand-written digits with 28-by-28 pixels, partitioned into 60,000 training and 10,000 test samples . We preprocess the data by subtracting the mean (0.1307) and dividing by the variance (0.3081) of the training data. No regularization 49.2 ± 1.9 67.0 ± 1.7 83.3 ± 0.7 90.4 ± 0.5 98.9 ± 0.1 32.9 ± 3.3 L 2 49.9 ± 2.1 68.1 ± 1.9 84.3 ± 0.8 91.2 ± 0.5 99.2 ± 0.1 4.6 ± 0.2 Dropout 49.7 ± 1.7 67.4 ± 1.7 83.9 ± 1.8 91.6 ± 0.5 98.6 ± 0.1 21.5 ± 2.3 Jacobian 49.3 ± 2.1 68.2 ± 1.9 84.5 ± 0.9 91.3 ± 0.4 99.0 ± 0.0 1.1 ± 0.1 All Combined 51.7 ± 2.1 69.7 ± 1.9 86.3 ± 0.9 92.7 ± 0.4 99.1 ± 0.1 1.2 ± 0.0 Implementation Details: For the MNIST dataset, we use the modernized version of LeNet-5 , henceforth denoted LeNet' (see Appendix D for full details). We optimize using SGD with momentum, ρ = 0.9, and our supervised loss equals the standard cross-entropy with one-hot targets. The model parameters θ are initialized at iteration t = 0 by the Xavier method and the initial descent value is set to 0. The hyperparameters for all models are chosen to match reference implementations: the L 2 regularization coefficient (weight decay) is set to λ WD = 5 · 10 −4 and the dropout rate is set to p drop = 0.5. The Jacobian regularization coefficient λ JR = 0.01, is chosen by optimizing for clean performance and robustness on the white noise perturbation. (See Appendix G for performance dependence on the coefficient λ JR .) The main goal of supervised learning involves generalizing from a training set to unseen test set. In dealing with such a distributional shift, overfitting to the training set and concomitant degradation in test performance is the central concern. For neural networks one of the most standard antidotes to this overfitting instability is L 2 reguralization (; ;). More recently, dropout regularization has been proposed as another way to circumvent overfitting . Here we show how Jacobian regualarization can serve as yet another solution. This is also in line with the observed correlation between the input-output Jacobian and generalization performance . We first verify that in the clean case, where the test set is composed of unseen samples drawn from the same distribution as the training data, the Jacobian regularizer does not adversely affect classification accuracy. Table 1 reports performance on the MNIST test set for the LeNet' model trained on either a subsample or all of the MNIST train set, as indicated. When learning using all 60,000 training examples, the learning rate is initially set to η 0 = 0.1 with mini-batch size |B| = 100 and then decayed ten-fold after each 50,000 SGD iterations; each simulation is run for 150,000 SGD iterations in total. When learning using a small subsample of the full training set, training is carried out using SGD with full batch and a constant learning rate η = 0.01, and the model performance is evaluated after 10,000 iterations. The main observation is that optimizing with the proposed Jacobian regularizer or the commonly used L 2 and dropout regularizers does not change performance on clean data within domain test samples in any statistically significant way. Notably, when few samples are available during learning, performance improved with increased regularization in the form of jointly optimizing over all criteria. Finally, in the right most column of Table 1, we confirm that the model trained with all data and regularized with the Jacobian minimization objective has an order of magnitude smaller Jacobian norm than models trained without Jacobian regularization. This indicates that while the model continues to make the same predictions on clean data, the margins around each prediction has increased as desired. We test the limits of the generalization provided by Jacobian regularization by evaluating an MNIST learned model on data drawn from a new target domain distribution -the USPS test set. Here, models are trained on the MNIST data as above, and the USPS test dataset consists of 2007 black-white images of hand-written digits with Table 2: Generalization on clean test data from an unseen domain. LeNet' models learned with all MNIST training data are evaluated for accuracy on data from the novel input domain of USPS test set. Here, each regularizer, including Jacobian, increases accuracy over an unregularized model. In addition, the regularizers may be combined for the strongest generalization effects. Averages and 95% confidence intervals are estimated over 5 distinct runs. No regularization L 16-by-16 pixels; images are upsampled to 28-by-28 pixels using bilinear interpolation and then preprocessed following the MNIST protocol stipulated above. Table 2 offers preliminary evidence that regularization, of each of the three forms studied, can be used to learn a source model which better generalizes to an unseen target domain. We again find that the regularizers may be combined to increase the generalization property of the model. Such a regularization technique can be immediately combined with state-of-the-art domain adaptation techniques to achieve further gains. This section showcases the main robustness of the Jacobian regularizer, highlighted in the case of both random and adversarial input perturbations. The real world can differ from idealized experimental setups and input data can become corrupted by various natural causes such as random noise and occlusion. Robust models should minimize the impact of such corruption. As one evaluation of stability to natural corruption, we perturb each test input image x to x = x + crop where each component of the perturbation vector is drawn from the normal distribution with variance σ noise as and the perturbed image is then clipped to fit into the range before preprocessing. As in the domain-adaptation experiment above, models are trained on the clean MNIST training data and then tested on corrupted test data. Results in Figure 3a show that models trained with the Jacobian regularization is more robust against white noise than others. This is in line with -and indeed quantitatively validates -the embiggening of decision cells as shown in Figure 1. Adversarial Perturbations: The world is not only imperfect but also possibly filled with evil agents that can deliberately attack models. Such adversaries seek a small perturbation to each input example that changes the model predictions while also being imperceptible to humans. Obtaining the actual smallest perturbation is likely computationally intractable, but there exist many tractable approxima-tions. The simplest attack is the white-box untargeted fast gradient sign method (FGSM) , which distorts the image as x = x + crop with This attack aggregates nonzero components of the input-output Jacobian to a substantial effect by adding them up with a consistent sign. In Figure 3b we consider a stronger attack, projected gradient descent (PGD) method , which iterates the FGSM attack in Equation k times with fixed amplitude ε FGSM = 1/255 while also requiring each pixel value to be within 32/255 away from the original value. Even stronger is the Carlini-Wagner (CW) attack presented in Figure 3c, which yields more reliable estimates of distance to the closest decision boundary (see Appendix F). Results unequivocally show that models trained with the Jacobian regularization is again more resilient than others. As a baseline defense benchmark, we implemented adversarial training, where the training image is corrupted through the FGSM attack with uniformly drawn amplitude ε FGSM ∈ [0, 0.01]; the Jacobian regularization can be combined with this defense mechanism to further improve the robustness. 5 Appendix A additionally depicts decision cells in adversarial directions, further illustrating the stabilizing effect of the Jacobian regularizer. To our knowledge, double backpropagation (; is the earliest attempt to penalize large derivatives with respect to input data, in which (∂L super /∂x) 2 is added to the loss in order to reduce the generalization gap. 6 Different incarnations of a similar idea have appeared in the following decades (; ; ; ; ; ; ;). Among them, Jacobian regularization as formulated herein was proposed by to combat against adversarial attacks. However, the authors did not implement it due to a computational concern -resolved by us in Section 2 -and instead layer-wise Jacobians were penalized. Unfortunately, minimizing layer-wise Jacobians puts a stronger constraint on model capacity than minimizing the input-output Jacobian. In fact, several authors subsequently claimed that the layer-wise regularization degrades test performance on clean data (; b) and in marginal improvement of robustness . Very recently, full Jacobian regularization was implemented in Sokolić et al., but in an inefficient manner whose computational overhead for computing gradients scales linearly with the number of output classes C compared to unregularized optimization, and thus they had to resort back to the layer-wise approximation above for the task with a large number of output classes. This computational problem was resolved by in exactly the same way as our approach (referred to as spherical SpectReg in). As emphasized in Section 2, we performed more thorough theoretical and empirical convergence analysis and showed that there is practically no difference in model solution quality between the exact and random projection method in terms of test accuracy and stability. Further, both of these two references deal only with the generalization property and did not fully explore strong distributional shifts and noise/adversarial defense. In particular, we have visualized (Figure 1) and quantitatively borne out (Section 3) the stabilizing effect of Jacobian regularization on classification margins of a nonlinear neural network. In this paper, we motivated Jacobian regularization as a task-agnostic method to improve stability of models against perturbations to input data. Our method is simply implementable in any open source automatic differentiation system, and additionally we have carefully shown that the approximate nature of the random projection is virtually negligible. Furthermore, we have shown that Jacobian regularization enlarges the size of decision cells and is practically effective in improving the generalization property and robustness of the models, which is especially useful for defense against input-data corruption. We hope practitioners will combine our Jacobian regularization scheme with the arsenal of other tricks in machine learning and prove it useful in pushing the (decision) boundary of the field and ensuring stable deployment of models in everyday life. We show in Figure S1 plots similar to the ones shown in Figure 1 in the main text, but with different seeds for training models and around different test data points. Additionally, shown in Figure S2 are similar plots but with different scheme for hyperplane slicing, based on adversarial directions. Interestingly, the adversarial examples constructed with unprotected model do not fool the model trained with Jacobian regularization. Figure S2: Cross sections of decision cells in the input space for LeNet' models trained on the MNIST dataset along adversarial hyperplanes. Namely, given a test sample (black dot), the hyperplane through it is spanned by two adversarial examples identified through FGSM, one for the model trained with L 2 regularization λ WD = 0.0005 and dropout rate 0.5 but no defense (dark-grey dot; left figure) and the other for the model with the same standard regularization methods plus Jacobian regularization λ JR = 0.01 and adversarial training (white-grey dot; right figure). Let us denote by Ev ∼S C−1 [F (v) ] the average of the arbitrary function F over C-dimensional vectorsv sampled uniformly from the unit sphere S C−1. As in Algorithm 1, such a unit vector can be sampled by first sampling each component v c from the standard normal distribution N and then normalizing it asv ≡ v/||v||. In our derivation, the following formula proves useful: where e is an arbitrary C-dimensional unit vector and dµ (O) First, let us derive Equation. Using Equation, the square of the Frobenius norm can then be written as where in the second line we insert the identity matrix in the form I = O T O and make use of the cyclicity of the trace; in the third line we rewrite the trace as a sum over an orthonormal basis {e} of the C-dimensional output space; in the forth line Equation was used; and in the last line we note that the expectation no longer depends on the basis vectors e and perform the trivial sum. This completes the derivation of Equation. Next, let us compute the variance of our estimator. Using tricks as before, but in reverse order, yields In this form, we use the following formula (Collins andŚniady, 2006;) to evaluate the first term After the dust settles with various cancellations, the expression for the variance simplifies to We can strengthen our claim by using the relation ||AB|| The right-hand side is independent of J and thus independent of the details of model architecture and particular data set considered. In the end, the relative error of the random-projection estimate for ||J(x)|| 2 F with n proj random vectors will diminish as some order-one number divided by n −1/2 proj. In addition, upon averaging ||J(x)|| 2 F over a mini-batch of samples of size |B|, we expect the relative error of the Jacobian regularization term to be additionally suppressed by ∼ 1/ |B|. Finally, we speculate that in the large-C limit -possibly relevant for large-class datasets such as the ImageNet -there might be additional structure in the Jacobian traces (e.g. the central-limit concentration) that leads to further suppression of the variance. It is also possible to derive a closed-form expression for the derivative of the Jacobian regularizer, thus bypassing any need for random projections while maintaining computational efficiency. The expression is here derived for multilayer perceptron, though we expect similar computations may be done for other models of interest. We provide full details in case one may find it practically useful to implement explicitly in any open-source packages or generalize it to other models. Let us denote the input x i and the output z c = z Defining the layer-wise Jacobian as the total input-output Jacobian is given by The Jacobian regularizer of interest is defined as (up to the magnitude coefficient λ JR) Its derivatives with respect to biases and weights are denoted as Some straightforward algebra then yields and where we have set B Algorithmically, we can iterate the following steps for = L, L − 1,..., 1: 2. Compute Note that the layer-wise Jacobians, J 's, are calculated within the standard backpropagation algorithm. The core of the algorithm is in the computation of Ω j −1,j in Equation. It is obtained by first backpropagating from − 1 to 1, then forwardpropagating from 1 to L, and finally backpropagating from L to + 1. It thus makes the cycle around, hence the name cyclopropagation. In order to describe architectures of our convolutional neural networks in detail, let us associate a tuple [F, C in → C out, S, P ; M] to a convolutional layer with filter width F, number of in-channels C in and out-channels C out, stride S, and padding P, followed by nonlinear activations and then a max-pooling layer of width M (note that M = 1 corresponds to no pooling). Let us also associate a pair [N in → N out] to a fully-connected layer passing N in inputs into N out units with activations and possibly dropout. With these notations, our LeNet' model used for the MNIST experiments consists of a input followed by a convolutional layer with [5, 1 → 6, 1, 2; 2], another one with [5, 6 → 16, 1, 0; 2], a fully-connected layer with [2100 → 120] and dropout rate p drop, another fully-connected layer with [120 → 84] and dropout rate p drop, and finally a fully-connected layer with [84 → 10], yielding 10-dimensional output logits. For our nonlinear activations, we use the hyperbolic tangent. For the CIFAR-10 dataset, we use the model architecture specified in the paper on defensive distillation (b), abbreviated as DDNet. Specifically, the model consists of a input followed by convolutional layers with In addition, we experiment with a version of ResNet-18 modified for the 32-by-32 input size of CIFAR-10 and shown to achieve strong performance on clean image recognition. 9 For this architecture, we use the standard PyTorch initialization of the parameters. Data preproceessing and optimization hyperparameters for both architectures are specified in the next section. For our ImageNet experiments, we use the standard ResNet-18 model available within PyTorch (torchvision.models.resnet) together with standard weight initialization. Note that there is typically no dropout regularization in the ResNet models but we still examine the effect of L 2 regularization in addition to Jacobian regularization. is vacuous. 9 Model available at: https://github.com/kuangliu/pytorch-cifar. No regularization 12.9 ± 0.7 15.5 ± 0.7 20.5 ± 1.3 26.6 ± 1.0 76.8 ± 0.4 115.1 ± 1.8 L • Datasets: the CIFAR-10 dataset consists of color images of objects -divided into ten categories -with 32-by-32 pixels in each of 3 color channels, each pixel ranging in, partitioned into 50,000 training and 10,000 test samples . The images are preprocessed by uniformly subtracting 0.5 and multiplying by 2 so that each pixel ranges in [−1, 1]. • Optimization: essentially same as for the LeNet' on MNIST, except the initial learning rate for full training. Namely, model parameters θ are initialized at iteration t = 0 by the Xavier method for DDNet and standard PyTorch initialization for ResNet-18, along with the zero initial velocity v(t = 0) = 0. They evolve under the SGD dynamics with momentum ρ = 0.9, and for the supervised loss we use cross-entropy with one-hot targets. For training with the full training set, mini-batch size is set as |B| = 100, and the learning rate η is initially set to η 0 = 0.01 for the DDNet and η 0 = 0.1 for the ResNet-18 and in both cases quenched ten-fold after each 50,000 SGD iterations; each simulation is run for 150,000 SGD iterations in total. For few-shot learning, training is carried out using full-batch SGD with a constant learning rate η = 0.01, and model performance is evaluated after 10,000 iterations. • Hyperparameters: the same values are inherited from the experiments for LeNet' on the MNIST and no tuning was performed. Namely, the weight decay coefficient λ WD = 5·10 −4; the dropout rate p drop = 0.5; the Jacobian regularization coefficient λ JR = 0.01; and adversarial training with uniformly drawn FGSM amplitude ε FGSM ∈ [0, 0.01]. The relevant for generalization properties are shown in Table S3. One difference from the MNIST counterparts in the main text is that dropout improves test accuracy more than L 2 regularization. Meanwhile, for both setups the order of stability measured by ||J|| F on the test set more or less stays the same. Most importantly, turning on the Jacobian regularizer improves the stability by orders of magnitude, and combining it with other regularizers do not compromise this effect. The relevant for robustness against input-data corruption are plotted in Figures S3 and S4. The success of the Jacobian regularizer is retained for the white-noise and CW adversarial attack. For the PGD attack are mixed at high degradation level when Jacobian regularization is combined with adversarial training. This might be an artifact stemming from the simplicity of the PGD search algorithm, which overestimates the shortest distance to adversarial examples in comparison to the CW attack (see Appendix F), combined with Jacobian regularization's effect on simplifying the loss landscape with respect to the input space that the attack methods explore. In Figure S5, we compare the effects of various input perturbations on changing model's decision. For each attack method, fooling L 2 distance in the original input space -before preprocessing -is measured between the original image and the fooling image as follows (for all attacks, cropping is performed to put pixels in the range in the orignal space): (i) for the white noise attack, a random direction in the input space is chosen and the magnitude of the noise is cranked up until the model yields wrong prediction; (ii) for the FGSM attack, the gradient is computed at a clean sample and then the magnitude ε FGSM is cranked up until the model is fooled; (iii) for the PGD attack, the attack step with ε FGSM = 1/255 is iterated until the model is fooled [as is customary for PGD and described in the main text, there is saturation constraint that demands each pixel value to be within 32/255 (MNIST) and 16/255 (CIFAR-10) away from the original clean value]; and (iv) the CW attack halts when fooling is deemed successful. Here, for the CW attack (see for details of the algorithm) the Adam optimizer on the logits loss (their f 6) is used with the learning rate 0.005, and the initial value of the conjugate variable, c, is set to be 0.01 and binary-searched for 10 iterations. For each model and attack method, the shortest distance is evaluated for 1,000 test samples, and the test error (= 100% − test accuracy) at a given distance indicates the amount of test examples misclassified with the fooling distance below that given distance. Below, we highlight various notable features. • The most important highlight is that, in terms of effectiveness of attacks, CW > PGD > FGSM > white noise, duly respecting the complexity of the search methods for finding adversarial examples. Compared to CW attack, the simple methods such as FGSM and PGD attacks could sometime yield erroneous picture for the geometry of the decision cells, especially regarding the closest decision boundary., we used 10,000 test examples (rather than 1,000 used for other figures) to compensate for the lack of multiple runs.

We analyze and develop a computationally efficient implementation of Jacobian regularization that increases the classification margins of neural networks.

Represent the TLDR for finding the corresponding scientific article

We propose a hypothesis for why gradient descent generalizes based on how per-example gradients interact with each other.


Question: Represent the article for finding a single-sentence TLDR

Question: With the increasing demand to deploy convolutional neural networks (CNNs) on mobile platforms, the sparse kernel approach was proposed, which could save more parameters than the standard convolution while maintaining accuracy. However, despite the great potential, no prior research has pointed out how to craft an sparse kernel design with such potential (i.e., effective design), and all prior works just adopt simple combinations of existing sparse kernels such as group convolution. Meanwhile due to the large design space it is also impossible to try all combinations of existing sparse kernels. In this paper, we are the first in the field to consider how to craft an effective sparse kernel design by eliminating the large design space. Specifically, we present a sparse kernel scheme to illustrate how to reduce the space from three aspects. First, in terms of composition we remove designs composed of repeated layers. Second, to remove designs with large accuracy degradation, we find an unified property named~\emph{information field} behind various sparse kernel designs, which could directly indicate the final accuracy. Last, we remove designs in two cases where a better parameter efficiency could be achieved. Additionally, we provide detailed efficiency analysis on the final 4 designs in our scheme. Experimental validate the idea of our scheme by showing that our scheme is able to find designs which are more efficient in using parameters and computation with similar or higher accuracy. CNNs have achieved unprecedented success in visual recognition tasks. The development of mobile devices drives the increasing demand to deploy these deep networks on mobile platforms such as cell phones and self-driving cars. However, CNNs are usually resource-intensive, making them difficult to deploy on these memory-constrained and energy-limited platforms. To enable the deployment, one intuitive idea is to reduce the model size. Model compression is the major research trend for it. Previously several techniques have been proposed, including pruning BID18, quantization BID28 and low rank approximation BID6. Though these approaches can can offer a reasonable parameter reduction with minor accuracy degradation, they suffer from the three drawbacks: 1) the irregular network structure after compression, which limits performance and throughput on GPU; 2) the increased training complexity due to the additional compression or re-training process; and 3) the heuristic compression ratios depending on networks, which cannot be precisely controlled. Recently the sparse kernel approach was proposed to mitigate these problems by directly training networks using structural (large granularity) sparse convolutional kernels with fixed compression ratios. The idea of sparse kernel was originally proposed as different types of convolutional approach. Later researchers explore their usages in the context of CNNs by combining some of these sparse kernels to save parameters/computation against the standard convolution. For example, MobileNets BID12 realize 7x parameter savings with only 1% accuracy loss by adopting the combination of two sparse kernels, depthwise convolution BID26 and pointwise convoluiton BID20, to replace the standard convolution in their networks. However, despite the great potential with sparse kernel approach to save parameters/computation while maintaining accuracy, it is still mysterious in the field regarding how to craft an sparse kernel design with such potential (i.e., effective sparse kernel design). Prior works like MobileNet BID12 and Xception BID1 just adopt simple combinations of existing sparse kernels, and no one really points out the reasons why they choose such kind of design. Meanwhile, it has been a long-existing question in the field whether there is any other sparse kernel design that is more efficient than all state-of-the-art ones while also maintaining a similar accuracy with the standard convolution. To answer this question, a native idea is to try all possible combinations and get the final accuracy for each of them. Unfortunately, the number of combination will grow exponentially with the number of kernels in a design, and thus it is infeasible to train each of them. Specifically, even if we limit the design space to four common types of sparse kernels -group convolution BID16, depthwise convolution BID26, pointwise convolution BID20 and pointwise group convolution ) -the total number of possible combinations would be 4 k, given that k is the number of sparse kernels we allow to use in a design (note that each sparse kernel can appear more than once in a design).In this paper, we craft the effective sparse kernel design by efficiently eliminating poor candidates from the large design space. Specifically, we reduce the design space from three aspects: composition, performance and efficiency. First, observing that in normal CNNs it is quite common to have multiple blocks which contain repeated patterns such as layers or structures, we eliminate the design space by ignoring the combinations including repeated patterns. Second, realizing that removing designs with large accuracy degradation would significantly reduce the design space, we identify a easily measurable quantity named information field behind various sparse kernel designs, which is closely related to the model accuracy. We get rid of designs that lead to a smaller information field compared to the standard convolution model. Last, in order to achieve a better parameter efficiency, we remove redundant sparse kernels in a design if the same size of information field is already retained by other sparse kernels in the design. With all aforementioned knowledge, we present a sparse kernel scheme that incorporates the final four different designs manually reduced from the original design space. Additionally, in practice, researchers would also like to select the most parameter/computation efficient sparse kernel designs based on their needs, which drives the demand to study the efficiency for different sparse kernel designs. Previously no research has investigated on the efficiency for any sparse kernel design. In this paper, three aspects of efficiency are addressed for each of the sparse kernel designs in our scheme: 1) what are the factors which could affect the efficiency for each design? 2) how does each factor affect the efficiency alone? 3) when is the best efficiency achieved combining all these factors in different real situations?Besides, we show that the accuracy of models composed of new designs in our scheme are better than that of all state-of-the-art methods under the same constraint of parameters, which implies that more efficient designs are constructed by our scheme and again validates the effectiveness of our idea. The contributions of our paper can be summarized as follows:• We are the first in the field to point out that the information field is the key for the sparse kernel designs. Meanwhile we observe the model accuracy is positively correlated to the size of the information field.• We present a sparse kernel scheme to illustrate how to eliminate the original design space from three aspects and incorporate the final 4 types of designs along with rigorous mathematical foundation on the efficiency.• We provide some potential network designs which are in the scope of our scheme and have not been explored yet and show that they could have superior performances. We first give a brief introduction to the standard convolution and the four common styles of sparse kernels. Standard convolution is the basic component in most CNN models, kernels of which can be described as a 4-dimensional tensor: W ∈ R C×X×Y ×F, where C and F are the numbers of the input and the output channels and X and Y are the spatial dimensions of the kernels. Let I ∈ R C×U ×V be the input tensor, where U and V denote the spatial dimensions of the feature maps. Therefore, the output activation at the output feature map f and the spatial location (x, y) can be expressed as, DISPLAYFORM0 Group convolution is first used in AlexNet BID16 ) for distributing the model over two GPUs. The idea of it is to split both input and output channels into disjoint groups and each output group is connected to a single input group and vice versa. By doing so, each output channel will only depend on a fraction of input channels instead of the entire ones, thus a large amount of parameters and computation could be saved. Considering the number of group as M, the output activation (f, x, y) can be calculated as, DISPLAYFORM0 The idea of depthwise convolution is similar to the group convolution, both of which sparsifies kernels in the channel extent. In fact, depthwise convolution can be regarded as an extreme case of group convolution when the number of groups is exactly the same with the number of input channels. Also notice that in practice usually the number of channels does not change after the depthwise convolution is applied. Thus, the equation above can be further rewritten as, DISPLAYFORM1 Pointwise convolution is actually a 1 × 1 standard convolution. Different from the group convolution, pointwise convolution achieves the sparsity over the spatial extent by using kernels with 1 × 1 spatial size. Similarly, the equation below shows how to calculate one output activation from the pointwise convolution in detail, DISPLAYFORM0 To sparsify kernels in both the channel and the spatial extents, the group convolution can be combined together with the pointwise convolution, i.e., pointwise group convolution. Besides the use of 1 × 1 spatial kernel size, in pointwise group convolution each output channel will also depend on a portion of input channels. The specific calculations for one output activation can be found from the equation below, DISPLAYFORM0 3 SPARSE KERNEL SCHEMERecall that the total number of combinations will grow exponentially with the number of kernels in a design, which could in a large design space. In this paper, we craft the effective sparse kernel design (i.e., design that consumes less parameters but maintains accuracy with the standard convolution) by efficiently examining the design space. Specifically, first we determine the initial design space by setting the maximum number of sparse kernels (length). To decide this number, two aspects are considered: 1) in order to give the potential to find more efficient designs which have not been explored yet, the maximum length of sparse kernel design should be greater than the numbers of all state-of-the-art ones; 2) it is also obvious that the greater length is more likely to consume more parameters, which contradicts our goal to find more efficient designs. Therefore combining the two aspects together, we set the maximum length to 6, which is not only greater than the largest number (i.e., 3) in all current designs, but also makes designs with the maximum length could still be able to be more efficient than the standard convolution. We then start to reduce the design space from three aspects: composition, performance and efficiency. In the following paragraphs, we will introduce the three aspects in detail. Composition. The overall layout in CNNs provides a good insight for us to quickly reduce the design space. Specifically, in normal CNNs it is quite common to have multiple stages/blocks which contain repeated patterns such as layers or structures. For example, in both VGG BID27 and ResNet BID9 there are 4 stages and inside each stage are several same repeated layers. Inspired by the fact, when we replace the standard convolution using various sparse kernel designs intuitively there is no need to add these repeated patterns to the original place of each standard convolutional layer. For example, suppose there are three types of sparse kernels, A, B and C, then the following combinations should be removed as containing repeated patterns: AAAAAA, ABABAB and ABCABC. AAAAAA contains the repeated pattern of A, while ABABAB and ABCABC have the patterns of AB and ABC respectively. Repeated patterns are also easy to detect, which makes the entire process extremely fast. To find such patterns, we can use the regular expression matching. The corresponding expression for the matched combinations should be (.+?)1+, where (.+?) denotes the first capturing group which contains at least one character, but as few as possible, and 1+ means try to match the same character(s) as most recently matched by the first group as many times as possible. As a , we can efficiently eliminate the design space with the help of the regular expression. Performance. There are lots of sparse kernel designs that could in large accuracy degradation, which gives us another opportunity to greatly reduce the design space. To get rid of them, we need an easily measurable (i.e., no training) property behind various designs that could directly indicate the final accuracy. Fortunately, after analyzing many prior works and conducting many experimental studies, we do find such property. We name it information field. Definition 1. (Information Field) Information field is the area in input tensor which one or more convolutional layers use to generate one output activation. For one output tensor, sizes of information fields for all activations are usually the same. FIG0 shows the spatial and channel dependency for the standard convolution, from which we can also find out the size of information field. Assuming the spatial kernel size is 3 × 3, starting from any output node in the figure we can see that in terms of the channel dimension each output channel will connect to all input channels and for the spatial dimensions one output activation will depend on activations inside a 3 × 3 spatial area. Therefore the information field for the standard convolution will be (3, 3, C) where C is the number of input channels. We find that information field is the key behind all sparse kernel designs, and also observe the model accuracy is positively correlated to the size of information field, the idea of which is also validated by later experiments in Section 4.2.With the help of information field, sparse kernel designs that would in large accuracy degradation could be easily removed from the original design space without actually training the models. Specifically, first for each design we calculate the size of information field by adding up it sequentially from the leftmost kernel to the rightmost one. For example, we use a three-dimensional vector,, to represent the initial values of information field on three different dimensions (i.e., two spatial dimensions and one channel dimension), then corresponding values of the vector will be updated based on the known properties of the sparse kernel encountered. After the rightmost kernel, the final vector we get will be the size of information field for the design. Finally we compare it with that of the standard convolution. If the two sizes are the same, we will keep the design, otherwise we will simply discard it. For instance, the design composed of one depthwise convolution will be removed since the information field of it only contains one channel area instead of the full channel space from the standard convolution. Efficiency. Considering a better parameter efficiency more designs could be removed. In paragraphs above, we only eliminate designs in terms of the accuracy via checking the size of information field. Recall that our goal is also to find efficient designs. Thus, while ensuring the accuracy we also need to take the efficiency into consideration. In fact, there are two cases that could worsen the efficiency and should be regarded as redundant designs: 1) it can be easily verified that the size of information field will never decrease when passing through sparse kernels in a design, thus there could be one situation that after one kernel, the size of information field still remains the same, which means the kernel does not help with regards to the information field even if the final size is the same as the standard convolution; 2) it is also possible that the same size of information field with the standard convolution is already retained by a fraction of sparse kernels in a design, in which case, other kernels can also be considered as not contributing to the information field. In terms of parameter efficiency designs in both of the two cases contain non-contributed kernels, therefore we can remove them from the original design space. To detect designs within the two cases, we introduce a early-stop mechanism during the process to check the size of information field above. Specifically, as per the two cases we check two things when adding up information field from the leftmost kernel in a design: 1) we record the size of information field before entering each kernel and compare it with the new size calculated after that kernel. If the two sizes are the same, we stop adding up information field for the design and directly go to the next one; 2) we add another conditional check every time we get a new size of information field. If the size is still less than or equal to that of the standard convolution, we will continue to add up information field from the next kernel, otherwise we will stop and go to the next design. With all aforementioned knowledge, we manually reduce the original design space (4 1 +4 2 +· · ·+4 6) to 4 different types of sparse kernel designs 1. In the next section we will present the 4 final designs respectively. Also notice that other techniques to save parameters such as bottleneck structure BID9 appear to be complimentary to our approach, which can be combined together to further improve parameter efficiency while maintaining accuracy. To validate this idea, we also consider the bottleneck structure when reducing the design space. Depthwise Convolution + Pointwise Convolution. Unlike the standard convolution which combines spatial and channel information together to calculate the output, the combination of depthwise convolution (DW) and pointwise convolution (PW) split the two kinds of information and deal with them separately. The output activation at location (f, x, y) can be written as DISPLAYFORM0 where W 1 and W 2 correspond to the kernels of depthwise convolution and pointwise convolution respectively. The dependency of such design is depicted in FIG0, from which we can easily verify that the size of information field is the same with the standard convolution. Group Convolution + Pointwise Group Convolution. The combination of group convolution (GC) and pointwise group convolution (PWG) can be regarded as an extension for the design above, where group convolution is applied on the pointwise convolution. However, simply using pointwise group convolution would reduce the size of information field on the channel dimension since depthwise convolution will not deal with any channel information. To recover the information field depthwise convolution is replaced with the group convolution. Meanwhile channel permutation should be added between the two layers. Assuming the number of channels does not change after the first group convolution, the output activation can be calculated as DISPLAYFORM1 DISPLAYFORM2 and N denote numbers of groups for group convolution and pointwise group convolution and W 1 and W 2 correspond to the kernels of group convolution and pointwise group convolution respectively. FIG0 shows the information field of this design clearly. Pointwise Convolution + Depthwise Convolution + Pointwise Convolution. Although two pointwise convolutions do not ensure a better efficiency in our scheme, the combination with bottleneck structure can help ease the problem, which makes it survive as one of the last designs. Following the normal practice we set bottleneck ratio to 1: 4, which implies the ratio of bottleneck channels to output channels. Also notice that more parameters could be saved if we place the depthwise convolution between the two pointwise convolutions since now depthwise convolution would only apply on a reduced number of channels. As a , the output activation T (f, x, y) is calculated as DISPLAYFORM3 where K denote the number of bottleneck channels and W 1, W 2 and W 3 correspond to the kernels of first pointwise convolution, depthwise convolution and second pointwise convolution respectively. Along with the equation FIG0 shows that the information field of such design is same with the standard convolution. Pointwise Group Convolution + Depthwise Convolution + Pointwise Group Convolution. The combination of two pointwise group convolutions and one depthwise convolution can also ensure the same size of information field. Similarly, channel permutation is needed. The bottleneck structure is also adopted to achieve a better efficiency. The output activation is calculated as DISPLAYFORM4 DISPLAYFORM5 and N represent the number of bottleneck channels and numbers of groups for first pointwise group convolution and second pointwise group convolution and W 1, W 2 and W 3 correspond to the kernels of first pointwise group convolution, depthwise convolution and second pointwise group convolution respectively. Both the equation and FIG0 could verify the same size of information field with the standard convolution. In addition, we find that the efficiency for different designs in our scheme do not always overlap. Thus to save the pain for researchers to find the most parameter/computation efficient designs based on their needs, we study the efficiency for each of the designs. Specifically, we consider two real situations which are frequently encountered by researchers when applying sparse kernel designs (i.e., given the input and the output for a layer and given the total number of parameters for a layer) and give accurate conditions when the best efficiency could be achieved. Efficiency given the input and the output. Given the numbers of input and output channels C and F. The total number of parameters after applying this design is 9C + CF, and the number of parameters for standard convolution is 9CF. Therefore the parameter efficiency of such method is 1/F + 1/9 represented by the ratio of parameters after and before applying such design. Clearly, given C and F, the parameter efficiency is always the same. Efficiency given the total amount of parameters. It can be easily verified that given the total number of parameters the greatest width is reached when the best efficiency is achieved. Thus the condition for the best efficiency given the total amount of parameters should be the same with the one when the greatest width is reached. The total number of parameters P for the design can be expressed as DISPLAYFORM0 when studying the greatest width, we need to assume the ratio between C and F does not change, thus the number of output channels F could be written like F = α · C where usually α ∈ N +. As a , from the equation above when P is fixed, the greatest width G (i.e., DISPLAYFORM1) will also be fixed, which indicates that the parameter efficiency is always the same. Efficiency given the input and the output. Similarly, we use the ratio of parameters to show parameter efficiency of this design. Given C and F, the number of parameters after using such design can be written as 3 · 3 · DISPLAYFORM0 Since the number of parameters for standard convolution is 9CF, the ratio will become DISPLAYFORM1. Notice that to ensure the same size of information field with standard convolution, in any input group of the second layer there should be at least one output channel from each one of the output groups of the first layer, therefore M · N should be less than or equal to the number of output channels from the first layer, i.e., M · N ≤ C. To further illustrate the relationship between the best parameter efficiency and the choices of M and N, we have the following theorem (the proof is given in the Appendix): Theorem 1. With the same size of information field, the best parameter efficiency is achieved if and only if the product of the two group numbers equals the channel number of the intermediate layer. As per the theorem, the best parameter efficiency can be achieved only when M · N = C. Thus the ratio will become Efficiency given the total amount of parameters. Given the total number of parameters P for one design, both M and N could affect the width of the network. As per Theorem 1 the greatest C can be reached only when C = M · N. When F = α · C, P could be written like DISPLAYFORM2 Given the number of parameters P, width C has a upper bound when 9N = αM, which is also the condition for the best efficiency. The greatest width G is (DISPLAYFORM3 Efficiency given the input and the output. Same as before, given the number of input channels C, bottleneck channels K and output channels F . After applying the design, the total amount of parameters is reduced to DISPLAYFORM0 The number of parameters for standard convolution is still 9CF . Notice that K = F/4, therefore the ratio can be further expressed as DISPLAYFORM1 36C . Clearly, given C, K and F, such design will also in a fixed efficiency. Efficiency given the total amount of parameters. When F = α · C and K = F/4, the total number of parameters P will be DISPLAYFORM2 when P is fixed, the greatest width G is also fixed, i.e., −9α+ √ 81α 2 +16α 2 P +16αP 2(α 2 +α). Efficiency given the input and the output. We use the same way to evaluate parameter efficiency for this design. First, the number of parameters after applying such method is 1 DISPLAYFORM0 The number for standard convolution is 9CF. Since K = F/4 and as per Theorem 1 the best parameter efficiency can be achieved only when K = M · N, the ratio of parameters can then be represented as DISPLAYFORM1. Thus given C, K and F, the best parameter efficiency can be reached by setting DISPLAYFORM2 Efficiency given the total amount of parameters. Similarly, according to the Theorem 1 the greatest C can be reached only when the number of bottleneck channels K = M · N. Since F = α · C and K = F/4, the total number of parameters of one design P can be expressed as DISPLAYFORM3 Given the number of parameters P, the greatest width G exists when αM = N. 4.1 IMPLEMENTATION DETAILS DISPLAYFORM0 The overall layout of the network is shown in TAB0. Identity mapping BID10 ) is used over each block. When building the models, we can simply replace every block in the layout with the standard convolution or the sparse kernel designs mentioned in Section 3. Batch normalization (BN) BID14 ) is adopted right after each layer in the block and as suggested by BID1 ) nonlinear activation ReLU is only performed after the summation of the identity shortcut and the output of each block. We evaluate our models on ImageNet 2012 dataset BID4 BID25, which contains 1.2 million training images and 50000 validation images from 1000 categories. We follow the same data augmentation scheme in BID10 a) which includes randomized cropping, color jittering and horizontal flipping. All models are trained for 100 epochs with batch size 256. SGD optimizer is used with the Nesterov momentum. The weight decay is 0.0001 and the momentum is 0.9. We adopt the similar weight initialization method from BID8 BID9 BID13. The learning rate starts with 0.1 and is divided by 10 every 30 epochs. All reported are single center crop top-1 performances. Relationship between the information field and the model accuracy. In Section 3, we have shown that all the sparse kernel designs generated by our scheme share the same size of the infor- mation field when the size of input is fixed. Meanwhile different sparse kernel designs could save different amount of parameters/computation compared to the standard convolution and the saved computation/parameters can then be used to increase the number of channels, enlarge the information field, and increase the final accuracy. The fundamental idea behind this is that we believe the information field is an essential property of all sparse kernel designs and could directly affect the final accuracy. To verify this idea we choose a bottleneck-like design and conduct some comparisons by tuning different number of groups. We adopt the same overall network layout in TAB0. It can be easily verified that given the same size of the input tensor the change of the number of groups in the bottleneck-like design will not affect the size of the information field in the output. Results are shown in TAB1. Specifically, compare on row 2 and row 5, we can see that by increasing the number of group from 2 to 32, more than a half amount of parameters will be saved to generate the same width, however the model accuracy will only decrease slightly. Meanwhile a further comparison on row 5 and row 6 indicate that if we use the saved parameters to increase the network width, the accuracy could still be improved. Since both of the two networks contain the same amount of parameters, overall network layout and type of sparse kernel design, the performance gains should only come from the increase of network width (information field). Same phenomenon could also be found by comparing on row 1 and row 2.Besides we investigate on different usages of parameters, on row 3 and row 4 show that the increase of network width has better potential for the improvement of accuracy than that of the depth, which also indicates that the size of the information field could play a more important role on the model accuracy. Additionally in TAB1 can further explain the sparse kernel design (PW+DW+PW) in Section 3.2 where we directly apply the most parameter-efficient depthwise convolution in the middle since it has the same size of the information field with other group numbers. Comparisons of different sparse kernel designs. We also compare different sparse kernel designs mentioned in Section 3. Results are shown in TAB2. As mentioned in Section 3 all designs have the same-sized information field given the same input. Results from TAB2 show that given the close amount of parameters by choosing different sparse kernel designs or group numbers models with different widths can be constructed, and the final accuracy is positively correlated to the model width (the size of the information field), which also coincides with our analysis above. Also notice that here do not necessarily indicate one type of sparse kernel design is always better than the other one in terms of the parameter efficiency since as per the analysis in Section 3 the efficiency also depends on other factors like the number of groups. For example, considering the same number of parameters and overall network layout, there could be a combination of group numbers M and N such that the network with the design GConv(M)+PWGConv(N) is wider than that of DW+PW. Based on the sparse kernel scheme, we are also able to construct more efficient designs than the state-of-the-art ones. TAB3 shows comparisons between the sparse kernel designs generated by our scheme and the state-of-the-art ones. For fair comparisons, we use the same network layout as shown in TAB0 and replace blocks in it with corresponding designs, and the model size around 11.0M is selected as it is the size that different models (e.g., Xception, ResNeXt and ShuffleNet) can be (a) 11.3 192 29.9 ResNeXt BID31 11. 1 192 29.8 Xception BID1 11.2 280 28.5 ShuffleNet 11.3 560 25.6GConv+PWGConv 8.6 200 27.0 PWGConv+DW+PWGConv 10.4 700 24.9 easily configured to. Results in TAB3 indicate that sparse kernel designs in our scheme could even yield better accuracy with a smaller model size, which also validates the idea of our sparse kernel scheme. Also notice that the choices of group numbers used in our designs are chosen to help easily accommodate both the similar model size and the overall network layout, which may not be the most efficient ones that are supposed to in a wider network with better accuracy under the same limitation of parameters. Model Compression. Traditional model compression techniques include pruning, quantization and low-rank approximation. Pruning BID0 BID22 BID21 reduces redundant weights, network connections or channels in a pre-trained model. However, it could face difficulty for deploying on hardware like GPU since some pruning methods may be only effective when the weight matrix is sufficiently sparse. Quantization BID35 BID2 BID1 BID5 BID23 reduces the number of bits required to represent weights. Unfortunately, this technique will require specialized hardware support. Low rank approximation BID17 BID15 BID32 BID24 BID7 uses two or more matrices to approximate the original matrix values in a pre-trained model. Nevertheless, since the process is an approximation of original matrix values maintaining a similar accuracy will always need additional re-training. The focus of this paper, the sparse kernel approach, mitigates all these problems by directly training networks using structural sparse convolutional kernels. In this paper, we present a scheme to craft the effective sparse kernel design by eliminating the large design space from three aspects: composition, performance and efficiency. During the process to reduce the design space, we find an unified property named information field behind various designs, which could directly indicate the final accuracy. Meanwhile we show the final 4 designs in our scheme along with detailed efficiency analysis. Experimental also validate the idea of our scheme. Proof of Theorem 1Proof. Without loss of generality we use the example in Section 3.3.2 to prove the theorem. Recall that the total number of parameters for such design can be expressed as DISPLAYFORM0 then the problem could be interpreted as proving that the minimum value of P can be achieved if and only if M · N = C.We prove the theorem by contradiction. Assume the minimum value of P could be achieved when M · N < C. Then we can always find a N = C/M > N such that the combination of M and N could in a smaller value of P, which contradicts our assumption. The theorem is hence proved.

represent the short summary for finding the corresponding scientific article (long)

We are the first in the field to show how to craft an effective sparse kernel design from three aspects: composition, performance and efficiency.

Represent the short summary for finding the corresponding scientific article (long)

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.