Question: represent the scientific article to find a short tldr!

Question: Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemeshave been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTi-vation (PACT), uses an activation clipping parameter α that is optimized duringtraining to find the right quantization scale. PACT allows quantizing activations toarbitrary bit precisions, while achieving much better accuracy relative to publishedstate-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance dueto a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories. Deep Convolutional Neural Networks (CNNs) have achieved remarkable accuracy for tasks in a wide range of application domains including image processing (He et al. (2016b) ), machine translation , and speech recognition . These state-of-the-art CNNs use very deep models, consuming 100s of ExaOps of computation during training and GBs of storage for model and data. This poses a tremendous challenge to widespread deployment, especially in resource constrained edge environments -leading to a plethora of explorations in compressed models that minimize memory footprint and computation while preserving model accuracy as much as possible. Recently, a whole host of different techniques have been proposed to alleviate these computational costs. Among them, reducing the bit-precision of key CNN data structures, namely weights and activations, has gained attention due to its potential to significantly reduce both storage requirements and computational complexity. In particular, several weight quantization techniques (and) showed significant reduction in the bit-precision of CNN weights with limited accuracy degradation. However, prior work (Hubara et al. (2016b); ) has shown that a straightforward extension of weight quantization schemes to activations incurs significant accuracy degradation in large-scale image classification tasks such as ImageNet . Recently, activation quantization schemes based on greedy layer-wise optimization were proposed (; ;), but achieve limited accuracy improvement. In this paper, we propose a novel activation quantization technique, PArameterized Clipping acTivation function (PACT), that automatically optimizes the quantization scales during model training. PACT allows significant reductions in the bit-widths needed to represent both weights and activations and opens up new opportunities for trading off hardware complexity with model accuracy. The primary contributions of this work include: 1) PACT: A new activation quantization scheme for finding the optimal quantization scale during training. We introduce a new parameter α that is used to represent the clipping level in the activation function and is learnt via back-propagation. α sets the quantization scale smaller than ReLU to reduce the quantization error, but larger than a conventional clipping activation function (used in previous schemes) to allow gradients to flow more effectively. In addition, regularization is applied to α in the loss function to enable faster convergence. We provide reasoning and analysis on the expected effectiveness of PACT in preserving model accuracy.3) Quantitative demonstrating the effectiveness of PACT on a spectrum of models and datasets. Empirically, we show that: (a) for extremely low bit-precision (≤ 2-bits for weights and activations), PACT achieves the highest model accuracy compared to all published schemes and (b) 4-bit quantized CNNs based on PACT achieve accuracies similar to single-precision floating point representations.4) System performance analysis to demonstrate the trade-offs in hardware complexity for different bit representations vs. model accuracy. We show that a dramatic reduction in the area of the computing engines is possible and use it to estimate the achievable system-level performance gains. The rest of the paper is organized as follows: Section 2 provides a summary of related prior work on quantized CNNs. Challenges in activation quantization are presented in Section 3. We present PACT, our proposed solution for activation quantization in Section 4. In Section 5 we demonstrate the effectiveness of PACT relative to prior schemes using experimental on popular CNNs. Overall system performance analysis for a representative hardware system is presented in Section 6 demonstrating the observed trade-offs in hardware complexity for different bit representations. Recently, a whole host of different techniques have been proposed to minimize CNN computation and storage costs. One of the earliest studies in weight quantization schemes (and) show that it is indeed possible to quantize weights to 1-bit (binary) or 2-bits (ternary), enabling an entire DNN model to fit effectively in resource-constrained platforms (e.g., mobile devices). Effectiveness of weight quantization techniques has been further improved (and), by ternarizing weights using statistical distribution of weight values or by tuning quantization scales during training. However, gain in system performance is limited when only weights are quantized while activations are left in high precision. This is particularly severe in convolutional neural networks (CNNs) since weights are relatively smaller in convolution layers in comparison to fully-connected (FC) layers. To reduce the overhead of activations, prior work (, Hubara et al. (2016a), and ) proposed the use of fully binarized neural networks where activations are quantized using 1-bit as well. More recently, activation quantization schemes using more general selections in bit-precision (Hubara et al. (2016b); Zhou et al. (2016; 2017);; ) have been studied. However, these techniques show significant degradation in accuracy (> 1%) for ImageNet tasks when bit precision is reduced significantly (≤ 2 − bits). Improvements to previous logarithmic quantization schemes using modified base and offset based on "weighted entropy" of activations have also been studied . recommends that normalized activation, in the process of batch normalization (, BatchNorm), is a good candidate for quantization. further exploits the statistics of activations and proposes variants of the ReLU activation function for better quantization. However, such schemes typically rely on local (and greedy) optimizations, and are therefore not adaptable or optimized effectively during training. This is further elaborated in Section 3 where we present a detailed discussion on the challenges in quantizing activations. Quantization of weights is equivalent to discretizing the hypothesis space of the loss function with respect to the weight variables. Therefore, it is indeed possible to compensate weight quantization errors during model training . Traditional activation Figure 1: (a) Training error, (b) Validation error across epochs for different activation functions (relu and clipping) with and without quantization for the ResNet20 model using the CIFAR10 dataset functions, on the other hand, do not have any trainable parameters, and therefore the errors arising from quantizing activations cannot be directly compensated using back-propagation. Activation quantization becomes even more challenging when ReLU (the activation function most commonly used in CNNs) is used as the layer activation function (ActFn). ReLU allows gradient of activations to propagate through deep layers and therefore achieves superior accuracy relative to other activation functions . However, as the output of the ReLU function is unbounded, the quantization after ReLU requires a high dynamic range (i.e., more bit-precision). In Fig. 1 we present the training and validation errors of ResNet20 with the CIFAR10 dataset using ReLU and show that accuracy is significantly degraded with ReLU quantizations It has been shown that this dynamic range problem can be alleviated by using a clipping activation function, which places an upper-bound on the output (Hubara et al. (2016b); ). However, because of layer to layer and model to model differences -it is difficult to determine a globally optimal clipping value. In addition, as shown in Fig 1, even though the training error obtained using clipping with quantization is less than that obtained with quantized ReLU, the validation error is still noticeably higher than the baseline. Recently, this challenge has been partially addressed by applying a half-wave Gaussian quantization scheme to activations . Based on the observation that activation after BatchNorm normalization is close to a Gaussian distribution with zero mean and unit variance, they used Lloyd's algorithm to find the optimal quantization scale for this Gaussian distribution and use that scale for every layer. However, this technique also does not fully utilize the strength of backpropagation to optimally learn the clipping level because all the quantization parameters are determined offline and remain fixed throughout the training process. Building on these insights, we introduce PACT, a new activation quantization scheme in which the ActFn has a parameterized clipping level, α. α is dynamically adjusted via gradient descent-based training with the objective of minimizing the accuracy degradation arising from quantization. In PACT, the conventional ReLU activation function in CNNs is replaced with the following: DISPLAYFORM0 where α limits the range of activation to [0, α]. The truncated activation output is then linearly quantized to k bits for the dot-product computations, where DISPLAYFORM1 With this new activation function, α is a variable in the loss function, whose value can be optimized during training. For back-propagation, gradient ∂yq ∂α can be computed using the Straight-Through Estimator (STE) to estimate ∂yq ∂y as 1. Thus, DISPLAYFORM2 The larger the α, the more the parameterized clipping function resembles a ReLU Actfn. To avoid large quantization errors due to a wide dynamic range, we include a L2-regularizer for α in the loss function. FIG6 illustrates how the value of α changes during full-precision training of CIFAR10-ResNet20 starting with an initial value of 10 and using the L2-regularizer. It can be observed that α converges to values much smaller than the initial value as the training epochs proceed, thereby limiting the dynamic range of activations and minimizing quantization loss. To provide further reasoning on why PACT works, we provide in-depth analysis in Appendix A and B. In particular, we show in Appendix A that PACT is as expressive as ReLU when it is used as an activation function. Further we explain in Appendix B that PACT finds a balancing point between clipping and quantization errors to minimize their impact to classification accuracy. When activation is quantized, the overall behavior of network parameters is affected by the quantization error during training. To observe the impact of activation quantization during network training, we sweep the clipping parameter α and record the training loss with and without quantization. Figs. 3 a,b and 3c show cross-entropy and training loss (cross entropy + regularization), respectively, over a range of α for the pre-trained SVHN network. The loaded network is trained with the proposed quantization scheme in which ReLU is replaced with the proposed parameterized clipping ActFn for each of its seven convolution layers. We sweep the value of α one layer at a time, keeping all other parameters (weight (W), bias (b), BatchNorm parameters (β, γ), and the α of other layers) fixed when computing the cross-entropy and training loss. The cross-entropy computed via full-precision forward-pass of training is shown in FIG6. In this case, the cross-entropy converges to a small value in many layers as α increases, indicating that ReLU is a good activation function when no quantization is applied. But even for the full-precision case, training clipping parameter α may help reduce the cross-entropy for certain layers; for example, ReLU (i.e., α = ∞) is not optimal for act0 and act6 layers. Next, the cross-entropy computed with quantization in the forward-pass is shown in FIG1. With quantization, the cross-entropy increases in most cases as α increases, implying that ReLU is no longer effective. We also observe that the optimal α has different ranges for different layers, motivating the need to "learn" the quantization scale via training. In addition, we observe plateaus of cross-entropy for the certain ranges of α (e.g., act6), leading to difficulties for gradient descent-based training. Finally, in FIG1, we show the total training loss including both the cross-entropy discussed above and the cost from α regularization. The regularization effectively gets rid of the plateaus in the training loss, thereby favoring convergence for gradient-descent based training. At the same time, α regularization does not perturb the global minimum point. For example, the solid circles in FIG1, which are the optimal α extracted from the pre-trained model, are at the minimum of the training loss curves. The regularization coefficient, λ α, discussed in the next section, is an additional hyper-parameter which controls the impact of regularization on α. For this new quantization approach, we studied the scope of α, the choice of initial values of α,and the impact of regularizing α. We briefly summarize our findings below, and present more detailed analysis in Appendix C.From our experiments, the best scope for α was to share α per layer. This choice also reduces hardware complexity because α needs to be multiplied only once after all multiply-accumulate (MAC) operations in reduced-precision in a layer are completed. Among initialization choices for α, we found it to be advantageous to initialize α to a larger value relative to typical values of activation, and then apply regularization to reduce it during training. Finally, we observed that applying L2-regularization for α with the same regularization parameter λ used for weight works reasonably well. We also observed that, as expected, the optimal value for λ α slightly decreases when higher bit-precision is used because more quantization levels in higher resolution for activation quantization. Additionally, we follow the practice of many other quantized CNN studies (e.g., Hubara et al. (2016b); Zhou et al. FORMULA0 ), and do not quantize the first and last layers, as these have been reported to significantly impact accuracy. We implemented PACT in Tensorflow BID0 ) using Tensorpack (Zhou et al. FORMULA0). To demonstrate the effectiveness of PACT, we studied several well-known CNNs. The following is a summary of the Dataset-Network for the tested CNNs. More implementation details can be found in Appendix. D. Note that the baseline networks use the same hyper-parameters and ReLU activation functions as described in the references. For PACT experiments, we only replace ReLU into PACT but the same hyper-parameters are used. All the time the networks are trained from scratch.• CIFAR10-ResNet20 (CIFAR10,): a convolution (CONV) layer followed by 3 ResNet blocks (16 CONV layers with 3x3 filter) and a final fully-connected (FC) layer.• SVHN-SVHN (SVHN, Netzer et al. FORMULA0): 7 CONV layers followed by 1 FC layer.• IMAGENET-AlexNet (AlexNet, Krizhevsky et al. FORMULA0): 5 parallel-CONV layers followed by 3 FC layers. BatchNorm is used before ReLU.• IMAGENET-ResNet18 (ResNet18, He et al. FORMULA0): a CONV layer followed by 8 ResNet blocks (16 CONV layers with 3x3 filter) and a final FC layer. "full pre-activation" ResNet structure (He et al. (2016a) ) is employed.• IMAGENET-ResNet50 (ResNet50, He et al. FORMULA0): a CONV layer followed by 16 ResNet "bottleneck" blocks (total 48 CONV layers) and a final FC layer. "full pre-activation" ResNet structure (He et al. FORMULA0) is employed. For comparisons, we include accuracy reported in the following prior work: DoReFa (Zhou et al. FORMULA0 FORMULA0). Detailed experimental setting for each of these papers, as well as full comparison of accuracy (top-1 and top5) for AlexNet, ResNet18, ResNet50, can be found in Appendix E. In the following section, we present key demonstrating the effectiveness of PACT relative to prior work. We first evaluate our activation quantization scheme using various CNNs. FIG3 training and validation error of PACT for the tested CNNs. Overall, the higher the bit-precision, the closer the training/validation errors are to the full-precision reference. Specifically it can be seen that training using bit-precision higher than 3-bits converges almost identically to the full-precision baseline. The final validation error has less than 1% difference relative to the full-precision validation error for all cases when the activation bit-precision is at least 4-bits. We further compare activation quantization performance with 3 previous schemes, DoReFa, LPBN, and HWGQ. We use accuracy degradation as the quantization performance metric, which is calculated as the difference between full-precision accuracy and the accuracy for each quantization bit-precision. FIG3 shows accuracy degradation (top-1) for ResNet18 (left) and ResNet50 (right) for increasing activation bit-precision, when the same weight bit-precision is used for each quantization scheme (indicated within the parenthesis). Overall, we observe that accuracy degradation is reduced as we increase the bit-precision of activations. For both ResNet18 and ResNet50, PACT achieves consistently lower accuracy degradation compared to the other quantization schemes, demonstrating the robustness of PACT relative to prior quantization approaches. In this section, we demonstrate that although PACT targets activation quantization, it does not preclude us from using weight quantization as well. We used PACT to quantize activation of CNNs, and DoReFa scheme to quantize weights. TAB0 summarizes top-1 accuracy of PACT for the tested CNNs (CIFAR10, SVHN, AlexNet, ResNet18, and ResNet50). We also show the accuracy of CNNs when both the weight and activation are quantized by DoReFa's scheme. As can be seen, with 4 bit precision for both weights and activation, PACT achieves full-precision accuracy consistently across the networks tested. To the best of our knowledge, this is the lowest bit precision for both weights and activation ever reported, that can achieve near (≤ 1%) full-precision accuracy. We further compare the performance of PACT-based quantized CNNs with 7 previous quantization schemes (DoReFa, BalancedQ, WRPN, FGQ, WEP, LPBN, and HWGQ). Fig. 5 shows comparison of accuracy degradation (top-1) for AlexNet, ResNet18, and ResNet50. Overall, the accuracy degradation decreases as bit-precision for activation or weight increases. For example, in Fig. 5a, the accuracy degradation decreases when activation bit-precision increases given the same weight precision or when weight bit-precision increases given the same activation bit-precision. PACT outperforms other schemes for all the cases. In fact, AlexNet even achieves marginally better accuracy (i.e., negative accuracy degradation) using PACT instead of full-precision. In this section, we demonstrate the gain in system performance as a of the reduction in bit-precision achieved using PACT-CNN. To this end, as shown in Fig. 6(a), we consider a DNN accelerator system comprising of a DNN accelerator chip, comprising of multiple cores, interfaced with an external memory. Each core consists of a 2D-systolic array of fixed-point multiply-andaccumulate (MAC) processing elements on which DNN layers are executed. Each core also contains an on-chip memory, which stores the operands that are fed into the MAC processing array. To estimate system performance at different bit precisions, we studied different versions of the DNN accelerator each comprising the same amount of on-chip memory, external memory bandwidth, and occupying iso-silicon area. First, using real hardware implementations in a state of the art technology (14 nm CMOS), we accurately estimate the reduction in the MAC area achieved by aggressively scaling bit precision. As shown in Fig. 6(b), we achieve ∼14× improvement in density when the bit-precisions of both activations and weights are uniformly reduced from 16 bits to 2 bits. Next, to translate the reduction in area to improvement in overall performance, we built a precisionconfigurable MAC unit, whose bit precision can be modulated dynamically. The peak compute capability (FLOPs) of the MAC unit varied such that we achieve iso-area at each precision. Note that the total on-chip memory and external bandwidth remains constant at all precisions. We estimate the overall system performance using DeepMatrix, a detailed performance modelling framework for DNN accelerators (Venkataramani et al.). Fig. 6(c) shows the gain in inference performance for the ResNet50 DNN benchmark. We study the performance improvement using different external memory bandwidths, namely, a bandwidth unconstrained system (infinite memory bandwidth) and two bandwidth constrained systems at 32 and 64 GBps. In the bandwidth unconstrained scenario, the gain in performance is limited by how amenable it is to parallelize the work. In this case, we see a near-linear increase in performance for upto 4 bits and a small drop at extreme quantization levels (2 bits).Practical systems, whose bandwidths are constrained, (surprisingly) exhibit a super-linear growth in performance with quantization. For example, when external bandwidth is limited to 64 GBps, quantizing from 16 to 4 bits leads to a 4× increase in peak FLOPs but a 4.5× improvement in performance. This is because, the total amount of on-chip memory remains constant, and at very low precision some of the data-structures begin to fit within the memory present in the cores, thereby avoiding data transfers from the external memory. Consequently, in bandwidth limited systems, reducing the amount of data transferred from off-chip can provide an additional boost in system performance beyond the increase in peak FLOPs. Note that for the 4 and 2 bit precision configurations, we still used 8 bit precision to execute the first and last layers of the DNN. If we are able to quantize the first and last layers as well to 4 or 2 bits, we estimate an additional 1.24× improvement in performance, motivating the need to explore ways to quantize the first and last layers. In this paper, we propose a novel activation quantization scheme based on the PArameterized Clipping acTivation function (PACT). The proposed scheme replaces ReLU with an activation function with a clipping parameter, α, that is optimized via gradient descent based training. We provide analysis on why PACT outperforms ReLU when quantization is applied during training. Extensive empirical evaluation using several popular convolutional neural networks, such as CIFAR10, SVHN, AlexNet, ResNet18 and ResNet50, shows that PACT quantizes activations very effectively while simultaneously allowing weights to be heavily quantized. In comparison to all previous quantization schemes, we show that both weights and activations can be quantized much more aggressively (down to 4-bits) -while achieving near (≤ 1%) full-precision accuracy. In addition, we have shown that the area savings from using reduced-precision MAC units enable a dramatic increase in the number of accelerator cores in the same area, thereby, significantly improving overall system-performance. When used as an activation function of the neural network, PACT is as expressive as ReLU. This is because clipping parameter, α, introduced in PACT, allows flexibility in adjusting the dynamic range of activation for each layer. We demonstrate in the simple example below that PACT can reach the same solution as ReLU via SGD.Lemma A.1. Consider a single-neuron network with PACT; x = w · a, y = P ACT (x), where a is input and w is weight. This network can be trained with SGD to find the output the network with ReLU would produce. Proof. Consider a sample of training data (a, y *). For illustration purposes consider mean-squareerror (MSE) as the cost function: DISPLAYFORM0 Therefore, when α is updated by SGD, DISPLAYFORM1 where η is a learning rate. Note that during this update, the weight is not updated as DISPLAYFORM2 From MSE, ∂L ∂y = (y − y *). Therefore, if y * > x, α is increased for each update of until α ≥ x, then the PACT network behaves the same as the ReLU network. Interestingly, if y * ≤ y or y < y * < x, α is decreased or increased to converge to y *. Note that in this case, ReLU would pass erroneous output x to increase cost function, which needs to be fixed by updating w with ∂L ∂w. PACT, on the other hand, ignores this erroneous output by directly adapting the dynamic range to match the target output y *. In this way, the PACT network can be trained to produce output which converges to the same target that the ReLU network would achieve via SGD.In general cases, ∂L ∂α = i ∂L ∂yi, and PACT considers output of neurons together to change the dynamic range. There are two options: if output x i is not clipped, then the network is trained via back-propagation of gradient to update weight, if output x i is clipped, then α is increased or decreased based on how close the overall output is to the target. Hence, there are configurations under which SGD would lead to a solution close to the one which the network with ReLU would achieve. FIG6 demonstrates that ResNet20 with PACT converges almost identical to the network with ReLU. In Section 3, when we briefly discussed the challenges in activation quantization, we mentioned that there is a trade-off between errors due to clipping and quantization. As the clipping level increases, larger range of activation can be passed to the next layer of the neural network causing less clipping error (ErrClip i = max(x i − α, 0)). However, the increased dynamic range incurs larger quantization error, since its magnitude is proportional to the clipping level (DISPLAYFORM0, with k-bit quantization). This imposes the challenge of finding a proper clipping level to balance between clipping and quantization errors. This trade-off can be better observed in FIG7, which shows normalized mean-square-error caused by clipping and quantization during training of the CIFAR10-ResNet20 with different clipping levels. It can be seen that activation functions with large dynamic range, such as ReLU, would suffer quantization errors whose magnitude increases exponentially as the bit-precision k decreases. This explains why the network with ReLU fails to converge when the activation is quantized (Fig. 1).PACT can find a balancing point between clipping and quantization errors. As explained in Section A, PACT adjusts dynamic range based on how close the output is to the target. As both clipping and quantization errors distort output far from the target, PACT would increase or decrease the dynamic range during training to minimize both clipping and quantization errors. FIG7 shows how PACT balances the clipping and quantization errors during training. CIFAR10-ResNet20 is trained with clipping activation function with varying clipping level α from 1 to 16. When activation is quantized, the network trained with clipping activation shows significant accuracy degradation as α increases. This is consistent with the trend in quantization error we observed in FIG7. In this case, PACT achieves the best accuracy one of the clipping activation could achieve, but without exhaustively sweeping over different clipping levels. In other words, PACT auto-tunes the clipping level to achieve best accuracy without incurring significant computation overhead. PACT's auto-tuning of dynamic range is critical in efficient yet robust training of large scale quantized neural networks, especially because it does not increase the burden for hyper-parameter tuning. In fact, we used the same hyper-parameters as well as the original network structure for all the models we tested, except replacing ReLU to PACT, when we applied activation quantization. Without quantization, there is a trend that validation error decreases as α increases. Surprisingly, some of the cases even outperforms the ReLU network. In this case, PACT also achieves comparable accuracy as ReLU, confirming its expressivity discussed in Section A. In this section, we present details on the hyper-parameters and design choices studied for PACT. One of key questions is the optimal scope for α. In other words, determining which neuron activations should share the same α. We considered 3 possible choices: (a) Individual α for each neuron activation, (b) Shared α among neurons within the same output channel, and (c) Shared α within a layer. We empirically studied each of these choices of α (without quantization) using CIFAR10-ResNet20 and determined training and validation error for PACT. As shown in FIG8, sharing α per layer is the best choice in terms of accuracy. This is in fact a preferred option from the perspective of hardware complexity as well, since α needs to be multiplied only once after all multiply-accumulate(MAC) operations in a layer are completed. The optimization behavior of α can be explained from the formulation of the parameterized clipping function. From Eq. 3 it is clear that, if α is initialized to a very small value, more activations fall into the range for the nonzero gradient, leading to unstable α in the early epochs, potentially causing accuracy degradation. On the other hand, if α is initialized to a very large value, the gradient becomes too small and α may be stuck at a large value, potentially suffering more on quantization error. Therefore, it is intuitive to start with a reasonably large value to cover a wide dynamic range and avoid unstable adaptation of α, but apply regularizer to reduce the value of α so as to alleviate quantization error. In practice, we found that applying L2-regularization for α while setting its coefficient λ α the same as the L2-regularization coefficient for weight, λ, works well. FIG9 shows that validation error for PACT-quantized CIFAR10-ResNet20 does not significantly vary for a wide range of λ α. We also observed that, as expected, the optimal value for λ α slightly decreases when higher bit-precision is used because more quantization levels in higher resolution for activation quantization. FORMULA0 ) follow the convention to keep the first and last layer in full precision during training, since quantizing those layers lead to substantial accuracy degradation. We empirically studied this for the proposed quantization approach for CIFAR10-ResNet20. In FIG10, the only difference among the curves is whether input activation and weight of the first convolution layer or the last fully-connected layer are quantized. As can be seen from the plots, there can be noticeable accuracy degradation if the first or last layers are aggressively quantized. But computation in floating point is very expensive in hardware. Therefore, we further studied the option of quantizing the first and last layers with higher quantization bit-precision than the bit-precision of the other layers. TAB2 shows that independent of the quantization level for the other layers, there is little accuracy degradation if the first and last layer are quantized with 8-bits. This motivates us to employ reduced precision computation even for the first/last layers. In this section, we summarize details of our CNN implementation as well as our training settings, which is based on the default networks provided by Tensorpack . Unless mentioned otherwise, ReLU following BatchNorm is used for ActFn of the convolution (CONV).0 12.9 11.1 10.9 17.4 10.0 9.4 8.9 15.9 9.7 9.2 8.9 18.2 9.0 8.4 8.5 FL/M/NQ 21.3 11.5 11.5 10.7 17.6 9.7 9.2 9.0 16.5 9.7 8.7 8.7 16.3 9.3 8.6 8.5 NQ/M/FL 12.1 11.2 11.0 11.5 9.8 8.9 9.2 9.2 8.4 8.4 8.7 8.8 8.5 9.0 8.5 8.5 layers, and Softmax is used for the fully-connected (FC) layer. Note that the baseline networks use the same hyper-parameters and ReLU activation functions as described in the references. For PACT experiments, we only replace ReLU into PACT but the same hyper-parameters are used. All the time the networks are trained from scratch. The CIFAR10 dataset is an image classification benchmark containing 32 × 32 pixel RGB images. It consists of 50K training and 10K test image sets. We used the "standard" ResNet structure (He et al. (2016a) ) which consists of a CONV layer followed by 3 ResNet blocks (16 CONV layers with 3x3 filter) and a final FC layer. We used stochastic gradient descent (SGD) with momentum of 0.9 and learning rate starting from 0.1 and scaled by 0.1 at epoch 60, 120. L2-regularizer with decay of 0.0002 is applied to weight. The mini-batch size of 128 is used, and the maximum number of epochs is 200.The SVHN dataset is a real-world digit recognition dataset containing photos of house numbers in Google Street View images, where the "cropped" 32 × 32 colored images (resized to 40 × 40 as input to the network) centered around a single character are used. It consists of 73257 digits for training and 26032 digits for testing. We used a CNN model which contains 7 CONV layers followed by 1 FC layer. We used ADAM(Kingma & Ba FORMULA0) with epsilon 10 −5 and learning rate starting from 10 −3 and scaled by 0.5 every 50 epoch. L2-regularizer with decay of 10 −7 is applied to weight. The mini-batch size of 128 is used, and the maximum number of epochs is 200.The IMAGENET dataset consists of 1000-categories of objects with over 1.2M training and 50K validation images. Images are first resized to 256 256 and randomly cropped to 224224 prior to being used as input to the network. We used a modified AlexNet, ResNet18 and ResNet50.We used AlexNet network in which local contrast renormalization (RNorm) layer is replaced with BatchNorm layer. We used ADAM with epsilon 10 −5 and learning rate starting from 10 −4 and scaled by 0.2 at epoch 56 and 64. L2-regularizer with decay factor of 5 × 10 −6 is applied to weight. The mini-batch size of 128 is used, and the maximum number of epochs is 100.ResNet18 consists of a CONV layer followed by 8 ResNet blocks (16 CONV layers with 3x3 filter) and a final FC layer. "full pre-activation" ResNet structure (He et al. (2016a) ) is employed. ResNet50 consists of a CONV layer followed by 16 ResNet "bottleneck" blocks (total 48 CONV layers) and a final FC layer. "full pre-activation" ResNet structure (He et al. (2016a) ) is employed. For both ResNet18 and ResNet50, we used stochastic gradient descent (SGD) with momentum of 0.9 and learning rate starting from 0.1 and scaled by 0.1 at epoch 30, 60, 85, 95. L2-regularizer with decay of 10 −4 is applied to weight. The mini-batch size of 256 is used, and the maximum number of epochs is 110. • DoReFa-Net : A general bit-precision uniform quantization schemes for weight, activation, and gradient of DNN training. We compared the experimental of DoReFa for CIFAR10, SVHN, AlexNet and ResNet18 under the same experimental setting as PACT. Note that a clipped absolute activation function is used for SVHN in DoReFa.• Balanced Quantization : A quantization scheme based on recursive partitioning of data into balanced bins. We compared the reported top-1/top-5 validation accuracy of their quantization scheme for AlexNet and ResNet18.• Quantization using Wide Reduced-Precision Networks : A scheme to increase the number of filter maps to increase robustness for activation quantization. We compared the reported top-1 accuracy of their quantization with various weight/activation bit-precision for AlexNet.• Fine-grained Quantization : A direct quantization scheme (i.e., little re-training needed) based on fine-grained grouping (i.e., within a small subset of filter maps). We compared the reported top-1 validation accuracy of their quantization with 2-bit weight and 4-bit activation for AlexNet and ResNet50.• Weighted-entropy-based quantization : A quantization scheme that considers statistics of weight/activation. We compared the top-1/top-5 reported accuracy of their quantization with various bit-precision for AlexNet, where the first and last layers are not quantized.• Low-precision batch normalization : A scheme for activation quantization in the process of batch normalization. We compared the top-1/top-5 reported accuracy of their quantization with 3-5 bit precision for activation. The first layer activation is not quantized.• Half-wave Gaussian quantization : A quantization scheme that finds the scale via Lloyd search on Normal distribution. We compared the top-1/top-5 reported accuracy for their quantization with 1-bit weight and varying activation bit-precision for AlexNet, and 2-bit weight for ResNet18 and ResNet50. The first and last layers are not quantized. In this section, we present full comparison of accuracy (top-1 and top-5) of the tested CNNs (AlexNet, ResNet18, ResNet50) for image classification on IMAGENET dataset. All the data points for PACT and DoReFa are obtained by running experiments on Tensorpack. All the other data points are accuracy reported in the corresponding papers. As can be seen, PACT achieves the best accuracy across the board for various flavors of quantization. We also observe that using PACT for activation quantization enables more aggressive weight quantization without loss in accuracy.

represent this text to find a much much longer article about it with say hundreds-thousands of words:

A new way of quantizing activation of Deep Neural Network via parameterized clipping which optimizes the quantization scale via stochastic gradient descent.

represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes massive reductions in parameter counts. To better understand this phenomenon, we analyze the behavior of pruning over the course of training, finding that pruning's effect on generalization relies more on the instability it generates (defined as the drops in test accuracy immediately following pruning) than on the final size of the pruned model. We demonstrate that even the pruning of unimportant parameters can lead to such instability, and show similarities between pruning and regularizing by injecting noise, suggesting a mechanism for pruning-based generalization improvements that is compatible with the strong generalization recently observed in over-parameterized networks. Pruning weights and/or convolutional filters from deep neural networks (DNNs) can substantially shrink parameter counts with minimal loss in accuracy (; ; a; ; ; ; ;), enabling broader application of DNNs via reductions in memory-footprint and inference-FLOPs requirements. Moreover, many pruning methods have been found to actually improve generalization (measured by model accuracy on previously unobserved inputs) (; ;). Consistent with this, pruning was originally motivated as a means to prevent over-parameterized networks from overfitting to comparatively small datasets . Concern about over-parameterizing models has weakened, however, as many recent studies have found that adding parameters can actually reduce a DNN's generalization-gap (the drop in performance when moving from previously seen to previously unseen inputs), even though it has been shown that the same networks have enough parameters to fit large datasets of randomized data . Potential explanations for this unintuitive phenomenon have come via experiments (; ; ; ;), and the derivation of bounds on DNN generalization-gaps that suggest less overfitting might occur as parameter counts increase . This research has implications for neural network pruning, where a puzzling question has arisen: if larger parameter counts don't increase overfitting, how does pruning parameters throughout training improve generalization? To address this question we first introduce the notion of pruning instability, which we define to be the size of the drop in network accuracy caused by a pruning iteration (Section 3). We then empirically analyze the instability and generalization associated with various magnitude-pruning (b) algorithms in different settings, making the following contributions: 1. We find a tradeoff between the stability and potential generalization benefits of pruning, and show iterative pruning's similarity to regularizing with noise-suggesting a mechanism unrelated to parameter counts through which pruning appears to affect generalization. 2. We characterize the properties of pruning algorithms which lead to instability and correspondingly higher generalization. There are various approaches to pruning neural networks. Pruning may be performed post-hoc (; ; b;), or iteratively throughout training, such that there are multiple pruning events as the model trains (; ;). Most methods prune parameters that appear unimportant to the function computed by the neural network, though means of identifying importance vary. Magnitude pruning (b) uses small-magnitude to indicate unimportance and has been shown to perform competitively with more sophisticated approaches . Many pruning studies have shown that the pruned model has heightened generalization (; ;), consistent with the fact that pruning may be framed as a regularization (rather than compression) approach. For example, variational Bayesian approaches to pruning via sparsity-inducing priors can describe weight removal as a process that reduces model description length, which in theory may help improve generalization . Similarly, the idea that models may be described more succinctly at flat minima has motivated pruning in service of flat minimum search . notes, however, that flatness can be arbitrarily modified by reparameterizing the function, and sharp minima can generalize well. VC dimension (a measure of model capacity) has motivated the use of iterative pruning to improve generalization . Overfitting can be bounded above by an increasing function of VC dimension, which itself often increases with parameter counts, so fewer parameters can lead to a guarantee of less overfitting . Unfortunately, such bounds can be so loose in practice that tightening them by reducing parameter counts need not translate to better generalization . Rather than support parameter-count-based arguments for generalization in DNNs, our suggest iterative DNN pruning may improve generalization by creating various noisy versions of the internal representation of the data, which unpruned parameters try to fit to, as in noise-injection regularization . Dropout creates particularly similar noise, as it temporarily sets random subsets of layer outputs to zero (likely changing an input's internal representation every epoch). Indeed, applying dropout-like zeroing noise to a subset of features during training can encourage robustness to a post-hoc pruning of that subset . Iterative DNN pruning noise ultimately differs, however, as it is: applied less frequently, not temporary (except in algorithms with weight re-entry), usually not random, and less well studied. Given a neural network and set of test data, let t be the top-1 test accuracy, the fraction of test data examples correctly classified multiplied by 100. We define a pruning algorithm's instability on pruning iteration i in terms of t measured immediately before (t pre,i) and immediately after (t post,i) pruning: instability i = t pre,i − t post,i. In other words, the instability is the size of the accuracy drop caused by a particular pruning event. (tpost,i) This measure is related to a weight's importance (sometimes referred to as "saliency"; ;) to the test accuracy, in that less stable pruning algorithms target more important sets of weights (all else equal). The stability of a pruning algorithm may be affected by many factors. Our experiments (Section 4) explore the effects of the following: pruning target, pruning schedule, iterative pruning rate, and model. The remainder of this section provides an overview of these factors and demonstrates a need for a novel pruning target, which we derive. In all of our experiments, we use iterative magnitude pruning (b), which removes weights according to some magnitude-based rule, retrains the ing smaller network to recover from the pruning, and repeats until the desired size reduction is met. We denote pruning algorithms that target the smallest-magnitude parameters with an "S" subscript (e.g. prune S), random parameters with an "R" subscript, and the largest-magnitude parameters with an "L" subscript. The usual approach to pruning involves removing parameters that have the smallest magnitudes , or, similarly, those parameters least important to the loss function as determined by some other metric (; ; ; ; ; ;). The correlation between parameter magnitude and importance weakens in the presence of batch normalization (BN) . Without batch normalization, a convolutional filter with weights W will produce feature map activations with half the magnitude of a filter with weights 2W: filter magnitude clearly scales the output. With batch normalization, however, the feature maps are normalized to have zero mean and unit variance, and their ultimate magnitudes depend on the BN affine-transformation parameters γ and β. As a , in batch normalized networks, filter magnitude does not scale the output, and equating small magnitude and unimportance may therefore be particularly flawed. This has motivated approaches to use the scale parameter γ's magnitude to find the convolutional filters that are important to the network's output . Here, we derive a novel approach to determining filter importance/magnitude that incorporates both γ and β. To approximate the expected value/magnitude of a batch-normalized, post-ReLU feature map activation, we start by defining the 2D feature map produced by convolution with BN: We approximate the activations within this feature map as M ij ∼ N (β, γ). This approximation is justified if central limit theorem assumptions are met by the dot products in W * x, and we empirically show in Figure A.1 that this approximation is highly accurate early in training, though it becomes less accurate as training progresses. Given this approximation, the post-ReLU feature map R = max{0, M} has elements R ij that are either 0 or samples from a truncated normal distribution with left truncation point l = 0, right truncation point r = ∞, and mean µ where and φ(x) and Φ(x) are the standard normal distribution's PDF and CDF (respectively) evaluated at x. Thus, an approximation to the expected value of R ij is given by We use the phrase "E[BN] pruning" to denote magnitude pruning that computes filter magnitude using this derived estimate of E[R ij]. E[BN] pruning has two advantages. First, this approach avoids the problematic assumption that filter importance is tied to filter 2 norm in a batch-normalized network. Accordingly, we hypothesize that E[BN] pruning can grant better control of the stability of the neural network's output than pruning based on filters' 2 norms. Second, the complexity of the calculation is negligible as it requires (per filter) just a handful of arithmetic operations on scalars, and two PDF and CDF evaluations, which makes it cheaper than a data-driven approach (e.g. approximating the expected value via the sample mean of feature map activations for a batch of feature maps). We consider three basic model classes: a simple network with convolutions (2x32, pool, 2x64, pool) and fully connected layers that we denote Conv4, VGG11 with its fully-connected layers replaced by a single fully-connected layer, and ResNet18 . All convolutions are 3x3. We trained these models using Adam with initial learning rate lr = 0.001, as we found Adam more helpful than SGD for recovering from unstable pruning (seemingly consistent with the observation in that recovery from pruning is more difficult when learning rates are low). , it allowed the network to recover more easily. The pruning algorithms we consider are iterative: we define a pruning schedule that describes the epochs on which pruning events occur, and set a corresponding (constant) iterative pruning rate that will ensure the total pruning percentage is met by the end of training (please see Appendix A.5 for rate and schedule details). Thus, throughout training, pruning steadily removes DNN parameters, with the iterative pruning rate determining the number pruned per event. While our plots label each pruning configuration with its iterative pruning rate, the total pruning percentages were: 42% of VGG11, 46% of ResNet18, and 10% of Conv4 (except in Appendix A.4, wherein we prune 85% of Conv4). Pruning studies often aim to compress pre-trained models that generalize well, and consequently, much work has focused on metrics to identify parameter importance: if you can find the parameters that matter the least to the function computed by the DNN, then you can prune more parameters without significantly harming accuracy. As a bonus, such pruning methods can sometimes even increase generalization (; ;). However, the mechanism by which pruning induces higher generalization remains unclear. Here, rather than investigate how to best maintain accuracy when pruning the network, we instead focus on understanding the mechanisms underlying these generalization improvements. Can improved generalization in pruned DNNs be explained by parameter-count reduction alone, or rather, do the properties of the pruning algorithm play an important role in generalization? As removing parameters from a DNN via pruning may make the DNN less capable of fitting to the noise in the training data, as originally suggested in; , we might expect that the generalization improvements observed in pruned DNNs are entirely explained by the number of parameters removed. In which case, methods that prune equal amounts of parameters would generalize similarly. Alternatively, perhaps some aspect of the pruning algorithm itself is responsible for increased generalization. This seems plausible as the reported generalization benefits of pruning vary widely across studies. One possible explanation for this variability is differences in the pruning algorithms themselves. A key differentiator of these algorithms is their stability: more stable approaches may compute a very close approximation to the way the loss changes with respect to each parameter and prune a single parameter at a time , while less stable approaches may assume that parameter magnitude and importance are roughly similar and prune many weights all at once (b). Therefore, to the extent that differences in pruning algorithms explain differences in pruning-based generalization improvements, we might expect to observe a relationship between generalization and pruning stability. To determine whether pruning algorithm stability affects generalization, we compared the instability and final top-1 test accuracy of several pruning algorithms with varying pruning targets and iterative pruning rates (Figure 1). Consistent with the nature of the pruning algorithm playing a role in generalization, we observed that more unstable pruning algorithms created higher final test accuracies than those which were stable (Figure 1, right; VGG11: Pearson's correlation r = .84, p-value = 1.6e−11; ResNet18: r = .65, p-value = 5e−5). While many pruning approaches have aimed to induce as little instability as possible, these suggest that pruning techniques may actually facilitate better generalization when they induce more instability. Furthermore, these suggest that parameter-count based arguments may not be sufficient to explain generalization in pruned DNNs, and suggest that the precise pruning method plays a critical role in this process. Figure 1 also demonstrates that pruning events for prune L with a high iterative pruning rate (red curve, pruning as much as 13% of a given convolutional layer per pruning iteration) are substantially more destabilizing than other pruning events, but despite the dramatic pruning-induced drops in performance, the network recovers to higher performance within a few epochs. Several of these pruning events are highlighted with red arrows. Please see Appendix A.2 for visualization of the epoch-wise instabilities of each method in VGG11, and Appendix A.3 for an 2 -norm pruning version of Figure 1, which has qualitatively similar . Interestingly, we initially observed that ResNet18 adapted to pruning events more quickly than VGG11 (accuracy rebounded after pruning then flattened soon after instead of climbing steadily). Thinking that shortcut connections were allowing the network to adapt to pruning events too easily, we tried pruning a larger amount of the penultimate block's output layer: this reduced the number of shortcut connections to the final block's output layer, lengthened the adaptation period, and improved generalization. This simple improvement of pruning hyperparameters suggests a potential for further optimization of the shown. Please see Appendix A.5.1 for all hyperparameters/details of these experiments. We have demonstrated that, perhaps surprisingly, pruning larger magnitude weights via the E[BN] algorithm can in larger test accuracy improvements (Figure 1). This suggests a positive correlation between pruning target magnitude and pruning's regularization effect. However, it's not clear whether this relationship holds more generally; i.e., perhaps it was caused by a feature of our Figure 2: When pruning 10% of Conv4's largest dense layer, the final generalization gap depends on the magnitude of the weights that were pruned during training. This is particularly true when using unstructured pruning (left) rather than structured pruning (right). E[BN] algorithm or the networks examined. Alternatively, this effect may be dependent on whether nodes/filters (structured pruning) or individual parameters (unstructured pruning) are pruned. As such, we tested whether target weight magnitude correlates with pruning's regularizing effect when using both unstructured and structured magnitude pruning on the penultimate linear layer of a small network without batch normalization (Conv4). Specifically, we constructed a pruning target for each weight-magnitude decile (see Appendix A.5.2 for details), used each target to prune ten separate networks as they trained, and compared the generalization gaps (test-train accuracy) of the pruned networks to the target pruned (Figure 2). For both unstructured and structured pruning (Figure 2 left and right, respectively), we found that pruning larger weights led to better generalization gaps, though, interestingly, this effect was much more dramatic in the context of unstructured pruning than structured pruning. One possible explanation for this is that, in structured pruning, the 2 norm of pruned neurons did not vary dramatically past the fifth decile, whereas the unstructured deciles were approximately distributed exponentially. As a , the top 50% of filters for the structured case were not clearly distinguished, making magnitude pruning much more susceptible to small sources of noise. These suggest that, when weight magnitudes vary considerably, pruning large magnitude weights may lead to improved generalization. Interestingly, for ResNet18, we actually found that structured prune L (red line) performed better than unstructured prune L (green line) (Figure 3). The worse performance of unstructured prune L may stem from its harming the helpful inductive bias provided by convolutional filters (i.e., perhaps removing the most important connections in all convolutional filters degrades performance more than pruning the same number of connections via removal of several entire filters) or its lower instability. While pruning large magnitude weights appears to play a role in pruning's ability to improve generalization, more commonly used pruning algorithms often see generalization improvements when targeting the smallest magnitude or least important parameters, suggesting that target magnitude/importance is not the only characteristic of pruning algorithms relevant to generalization. One possibility is that, given a pruning target, pruning more parameters per pruning iteration (while holding constant the total pruning percentage) may lead to greater instability. If this is the case, the generalization-stability tradeoff suggests that the increase in instability from raising the iterative pruning rate would coincide with improved generalization performance. Alternatively, if the pruning target or total pruning percentage is all that matters, we may expect that changing the iterative: In VGG11, increasing the iterative pruning rate (and decreasing the number of pruning events in order to hold total pruning percentage constant) leads to more instability, and can allow methods that target less important parameters to generalize better. Additionally, E[BN] magnitude better approximates parameter importance than 2 -norm magnitude (see Figure A2 for another example and discussion of this phenomenon). An unpruned baseline model has 85.21% accuracy. pruning rate (while keeping the pruning target and total pruning percentage fixed) would not affect generalization. To test this, we plotted mean instability and test accuracy as a function of different iterative pruning rates for both 2 -norm and E[BN] pruning (Figure 4). Consistent with iterative pruning rate playing a role in instability, we find that (given a pruning target) more instability is induced by using larger iterative pruning rates (Figure 4 left). Moreover, pruning random or small magnitude parameters performs best at the largest iterative rate (30%), supporting the idea that these methods require a source of instability to boost generalization. Note this suggests that, when targeting less important weights, higher iterative pruning rates during training can be an effective way to induce additional instability and generalization. (Algorithm and experiment details are available in Appendix A.5.4.) Perhaps strangely, higher iterative pruning rates did not translate to improved generalization when targeting the largest magnitude weights (prune L) with 2 -norm pruning. The fact that prune L does not generalize the best at the highest iterative pruning rate may be due to the reduction in pruning iterations required by the large iterative pruning rate (i.e., when the iterative rate is at 30%, the number of pruning events is capped at three, which removes 90% of a layer). Thus, while this rate grants more instability (Figure 4 left) per iteration, pruning affects the network less often. The idea that the regularizing effect of pruning is enhanced by pruning more often may also help explain the observation that methods that prune iteratively can generalize better (b). Another possibility is that, since raising the iterative pruning rate (and consequently the duration between pruning events) tends to make the 2 -norm worse for differentiating parameters by their importance to accuracy 1, raising the iterative pruning rate causes prune L with 2 -norm pruning to target less important weights. Consequently, prune L with 2 -norm pruning may generalize worse at higher iterative rates by leaving unpruned more important weights, the presence of which can harm model generalization . Relatedly, this also means that prune S with 2 -norm pruning may increase (in networks with batch normalization at least) instability and generalization by failing to avoid the pruning of important parameters. Our thus far suggest that pruning improves generalization when it creates instability throughout training. These prior , though, involved damaging model capacity simply by the nature of pruning, which decreases the number of model parameters. It therefore remains possible that the generalization benefits we've seen rely upon the reduction in capacity conferred by pruning. Here, we examine this critical question. We first note that iterative pruning can be viewed as noise injection Figure 5: Generalization improvements from pruning bear resemblance to those obtained by using temporary (Left) multiplicative zeroing noise, and (Right) additive Gaussian noise, as long as the noise is applied for enough batches/steps. the permanence of this zeroing can mitigate some of the capacity effect of pruning 2 and, therefore, help us isolate and study how iterative pruning regularizes through noise injection. As a baseline, we consider prune L applied to VGG11, judging filter magnitude via the 2 -norm (additional experimental details are in Appendix A.5.5). We then modify this algorithm such that, rather than permanently prune filters, it simply multiplies the filter weights by zero, then allows the zeroed weights to immediately resume training in the network ("Zeroing 0" in Figure 5 Left). However, by allowing pruned weights to immediately recover, this experiment also removes a key, potentially regularizing aspect of pruning noise: the requirement that the rest of the network adapts to fit the new representations generated by pruning. To encourage this potentially important facet of pruning noise, we also added variants that held weights to zero for 50 and 1500 consecutive batches 3. As a related experiment, we also measured the impact of adding Gaussian noise to the weights in Figure 5, right. Noise was applied either once (Gaussian 0) or repeatedly over a series of training batches (Gaussian 50/1500). If the capacity effects of weight removal are not necessary to explain pruning's effect on generalization, then we would expect that the generalization behavior of these non-permanent noise injection algorithms could mimic the generalization behavior of prune L. Alternatively, if weight removal is a necessary component of pruning-based generalization improvements, then we would not expect close similarities between the generalization phenomena of prune L and non-permanent pruning noise injection. Consistent with the capacity effects of weight removal not being necessary to explain generalization in pruned DNNs, applying zeroing noise for 50 batches to filters (rather than pruning them completely) generates strikingly similar accuracy to prune L (Figure 5 Left). Specifically, the patterns in instability are qualitatively and quantitatively similar, as are the generalization levels throughout training. Importantly, we found that applying zeroing noise once (Zeroing 0; brown line) was not sufficient to generate better performance, suggesting that the regularization induced by forcing weights to adapt to noised representations is critical to pruning's ability to improve generalization. Moreover, we found that, while applying Gaussian noise could increase generalization if applied for long enough (Gaussian 1500; purple line), it still did not match the performance of prune L, suggesting that multiplicative zeroing noise is substantially more effective than additive Gaussian noise 4. Together, these demonstrate that pruning induced generalization benefits are not merely explained by weight removal, but rather are dependent on the regularization conferred by forcing networks to adapt to noised representations over a sufficiently long period throughout training. In this study, we defined the notion of pruning algorithm instability, and applied several pruning approaches 5 to multiple neural networks, assessing the approaches' effects on instability and generalization. Throughout these experiments, we observed that pruning algorithms that generated more instability led to better generalization (as measured by test accuracy). For a given pruning target and total pruning percentage, instability and generalization could be fueled by raising iterative pruning rates (Figure 4, Section 4.3). Additionally, targeting more important weights, again holding total parameters pruned constant, led to more instability and generalization than targeting less important weights (Figure 1, Section 4.1). These support the idea that the generalization benefits of pruning cannot be explained solely by pruning's effect on parameter counts-the properties of the pruning algorithm must be taken into account. Our analysis also suggests that the capacity effects of weight-removal may not even be necessary to explain how pruning improves generalization. Indeed, we provide an interpretation of iterative pruning as noise injection, a popular approach to regularizing DNNs, and find that making pruning noise impermanent provides pruning-like generalization benefits while not removing as much capacity as permanent pruning (Figure 5, Section 4.4). While not emphasized in our discussion, pruning algorithm stability can be a desirable property, as recovery from pruning damage is not guaranteed. Indeed, pruning too many large/important weights can lead to worse final generalization . Recovery appears to be a function of several factors, including: learning rate ); presence of an ongoing regularization effect (Figure 3, Section 4.2); preservation of helpful inductive biases (Figure 3, Section 4.2); and damage to network capacity (e.g., removing too much of an important layer could cause underfitting). A better understanding of the factors which aid recovery from pruning instability could aid the design of novel pruning algorithms. For example, pruning methods that allow weights to re-enter the network could perhaps prune important weights occasionally to enhance generalization improvements, without risking permanent damage to the pruned networks (see Appendix A.4). In describing how pruning regularizes a model, we touched on similarities between pruning and noise injection. Our , however, may also be consistent with other parameter-count-independent approaches to understanding generalization in neural networks, as pruning may reduce the information stored in the network's weights , and make the network more distributed . This raises the possibility that pruning noise engenders helpful properties in DNNs, though it remains unclear whether such properties might be identical to those achieved with more common noise injection schemes . Further exploration will be necessary to better understand the relationship between these approaches. One important caveat of our is that they were generated with CIFAR-10, a relatively small dataset, so future work will be required to evaluate whether the presented phenomena hold in larger datasets. Relatedly, we only studied pruning's regularizing effect in isolation and did not include commonly used regularizers (e.g., weight decay) in our setups. In future work, it would be interesting to examine whether pruning complements the generalization improvements of other commonly used regularization techniques. Figure A1: We examined the normalized activations (shown in blue histograms) of feature maps in the final eight convolutional layers of VGG19 before (left) and after (right) training to convergence. We found that the approximation to standard normality (shown in orange) of these activations is reasonable early on but degrades with training (particularly in layers near the output). The main drawback to the E[BN] approach (Section 3.1.1) is the sometimes poor approximation M ij ∼ N (β, γ). In Figure A.1, the approximation's quality depends on layer and training epoch. A less serious drawback is that this approach does not account for the strength of connections to the post-BN feature map, which could have activations with a large expected value but low importance if relatively small-magnitude weights connected the map to the following layer. Figure A2: In VGG11, prune S E[BN] is more stable than prune S, which uses filter-2 -norm to compare parameter magnitudes. Methods with higher iterative pruning rates create more instability on a given iteration. Means reduce along the run dimension (10 runs per configuration). Note that this graph uses a method (prune S) that was included in Figure 1 Here we use the same training and pruning configurations that were used for Figure 1, but we replace E[BN] pruning with 2 -norm pruning. Qualitatively, the two figures' are similar. Interestingly, though, the correlation between instability and generalization is somewhat weaker with 2 -norm pruning. This may be explained by the fact that 2 -norm pruning generates a narrower spectrum of instabilities, which is perhaps due to 2 -norm scoring's inability to accurately assess parameter importance (illustrated in Figure 4). Top-1 Accuracy % No Pruning Prune_S Prune_L Train Test Figure A4: When training Conv4 on CIFAR10, unstable pruning can significantly improve the baseline's generalization. The training accuracies and test accuracies (the latter were calculated immediately after pruning) illustrate how much each pruning algorithm disturbs the neural network's output during training. While it's unclear how much unstable pruning, which is particularly damaging to capacity, can be sustained at high sparsity levels, prune L can lead to generalization several percentage points above the baseline/prune S when pruning 85% of Conv4. Please see Section A.5.6 for experimental setup details. Unstructured magnitude pruning entails removing individual weights (subsets of filters/neurons), which are selected for pruning based on their magnitude. Our unstructured pruning approach does not allow previously pruned weights to reenter the network (; ;). Structured magnitude pruning removes entire filters/neurons, which are selected based on their 2 -norms or via the E[BN] calculation. Except where noted, we use structured pruning for VGG11 and ResNet18. We denote the pruning of n layers of a network by specifying a series of epochs at which pruning starts s = (s 1, ..., s n), a series of epochs at which pruning ends e = (e 1, ..., e n), a series of fractions of parameters to remove p = (p 1, ..., p n), and an inter-pruning-iteration retrain period r ∈ N. For a given layer l, the retrain period r and fraction p l jointly determine the iterative pruning percentage i l. Our experiments prune the same number of parameters i l × size(layer l) per pruning iteration, ultimately removing p l × 100% of the parameters by the end of epoch e l. Our approach is designed to study the effects of changing factors such as the iterative pruning rate and lacks some practically helpful features, e.g. hyperparameters indicating how many parameters can be safely pruned . When layerwise iterative pruning percentages differ (i.e., when there exists an a and b such that i a and i b are unequal), our figures state the largest iterative pruning rate that was used in any of the layers. For ResNet, our pruning algorithms did not account for the magnitude of incoming shortcut connections when judging filter magnitude/importance. Though we did prune the incoming and outgoing shortcut connections associated with any pruned feature maps. We used only the CIFAR-10 dataset in our experiments, a limitation of our study. We used batch size 128, and only used data augmentation in the decile experiment (Figure 2). For some experiments, we give multi-step learning rate schedules lr s = (x, y), which means we shrink the learning rate by a factor of 10 at epochs x and y. A.5.1 FIGURE 1 We used E [BN] pruning in all models that were pruned, except for one model that used 2 -norm magnitude pruning, which was included in Figure 1 right but not displayed in Figure 1 left due to its qualitative similarity to prune S E[BN]. We leave out "E[BN]" in the legend of Figure 1 left, but all models nonetheless used E[BN] pruning. The models were trained on CIFAR-10 with Adam for 325 epochs with lr s =. The error bars are 95% confidence intervals for the mean, bootstrapped from 10 distinct runs of each experiment. Since the layerwise pruning percentages varied, pruning required multiple iterative pruning percentages, the largest of which is denoted in the legend (rounded to the nearest integer). VGG11 Pruning targeted the final four convolutional layers during training with (layerwise) starting epochs s =, ending epochs e =, and pruning fractions p = (0.3, 0.3, 0.3, 0.9). To allow for the same amount of pruning among models with differing iterative pruning percentages, we adjusted the number of inter-pruning retraining epochs. The models with the maximum iterative pruning percentage of 1% had r = 4, while the models with the maximum iterative pruning percentage of 13% had r = 40. The model pruned with 2 -norm magnitude pruning, which only appeared in Figure 1 right, had r = 4 as well. ResNet18 Pruning targeted the final four convolutional layers of ResNet18 during training with (layerwise) starting epochs s =, ending epochs e =, and pruning fractions p = (0.25, 0.4, 0.25, 0.95). As noted in Section 4.1, we increased the pruning rate of the output layer of the penultimate block to remove shortcut connections to the last layer, thinking that it should increase the duration of adaptation to pruning. The models with the maximum iterative pruning percentage of 1% had r = 4, while the models with the maximum iterative pruning percentage of 13% had r = 40. Each experiment in Figure 2 targeted one of ten weight-magnitude deciles in the post-convolutional linear layer of the Conv4 network during training on CIFAR-10 with data augmentation. While there are just ten deciles, the iterative nature of our pruning algorithms required the creation of eleven different pruning targets: ten methods pruned from the bottom of the decile upward (one experiment for each decile's starting point: 0th percentile, 10th percentile, etc.), and one (D10) pruned from the last decile's ending point downward (pruning the very largest collection of weights each iteration). In other words, D9 and D10 targeted the same decile (90th percentile to maximum value), but only D10 actually removed the largest weights on a given iteration (weights in the 100th-99th percentiles, for example). The D9 experiment would target weights starting from the 90th percentile (e.g. it may prune the 90th-91st percentiles on a particular iteration). The training/pruning setup used the Adam optimizer, s =, e =, p = (0.1), r = 3, and lr s =. We calculated the generalization gap on epoch 54 and sampled average pruned magnitudes on epoch 35. We obtained qualitatively similar regardless of whether we used fewer training epochs or data augmentation. The error bars are 95% confidence intervals for the means, bootstrapped from 10 distinct runs of each configuration. A.5.3 FIGURE 3 In Figure 3, prune L was applied to the final four convolutional layers of ResNet18 during training with (layerwise) starting epochs s =, ending epochs e =, and pruning fractions p = (0.25, 0.4, 0.25, 0.95). Since the layerwise pruning percentages varied, pruning required multiple iterative pruning percentages, the largest of which is denoted in the legend (rounded to the nearest integer). The models with the maximum iterative pruning percentage of 1% had r = 4, the models with the maximum iterative pruning percentage of 13% had r = 40, and the "One Shot" model pruned all its targeted parameters at once on epoch 246. When performing unstructured pruning, we pruned individual weights from filters based on their magnitude. The structured pruning experiments used E[BN] pruning. The models were trained on CIFAR-10 with Adam for 325 epochs with lr s =. The error bars are 95% confidence intervals for the means, bootstrapped from 10 distinct runs of each experiment. A.5.4 FIGURE 4 In Figure 4, pruning targeted the final four convolutional layers of VGG11 during training with (layerwise) starting epochs s =, ending epochs e =, and pruning fractions p = (0.3, 0.3, 0.3, 0.9). To create the different iterative pruning rates, we used models with inter-pruning retrain periods r = 4, r = 20, r = 40, r = 60, and r = 100. Since the layerwise pruning percentages varied, pruning required multiple iterative pruning percentages, the largest of which is denoted on the horizontal axis. An unpruned baseline model average (10 runs) is plotted on the dotted line. The models were trained on CIFAR-10 with Adam for 325 epochs with lr s =. The error bars are 95% confidence intervals for the means, bootstrapped from 10 distinct runs of each experiment. A.5.5 FIGURE 5 In Figure 5, pruning targeted the final four convolutional layers of VGG11 during training with (layerwise) starting epochs s =, ending epochs e =, pruning fractions p = (0.3, 0.3, 0.3, 0.9), and inter-pruning-iteration retrain period r = 40. When injecting pruning noise, we used the same pruning schedule and percentages, but applied noise to the parameters instead of removing them. The Gaussian noise had mean 0 and standard deviation equal to the empirical standard deviation of a noiseless filter from the same layer. Prune L used 2 -norm pruning. The models were trained on CIFAR-10 with Adam for 325 epochs with lr s =. The error bars are 95% confidence intervals for the means, bootstrapped from 10 distinct runs of each experiment. A.5.6 APPENDIX A.4 Each experiment in Appendix A.4 targeted the post-convolutional linear layer of the Conv4 network during training on CIFAR-10 with the Adam optimizer. The pruning algorithms start on epoch s =, end on epoch e =, prune the percentage p = (0.9), and prune every epoch via retrain period r = 1. These relatively simple experiments were conducted to show that, at higher sparsity (pruning 85% of the model's parameters), unstable pruning can improve the generalization of the baseline. The error bars are 95% confidence intervals for the means, bootstrapped from 20 distinct runs of each configuration.

We demonstrate that pruning methods which introduce greater instability into the loss also confer improved generalization, and explore the mechanisms underlying this effect.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Neural networks trained with backpropagation, the standard algorithm of deep learning which uses weight transport, are easily fooled by existing gradient-based adversarial attacks. This class of attacks are based on certain small perturbations of the inputs to make networks misclassify them. We show that less biologically implausible deep neural networks trained with feedback alignment, which do not use weight transport, can be harder to fool, providing actual robustness. Tested on MNIST, deep neural networks trained without weight transport have an adversarial accuracy of 98% compared to 0.03% for neural networks trained with backpropagation and generate non-transferable adversarial examples. However, this gap decreases on CIFAR-10 but is still significant particularly for small perturbation magnitude less than 1 ⁄ 2. Deep neural networks trained with backpropagation (BP) are not robust against certain hardly perceptible perturbation, known as adversarial examples, which are found by slightly altering the network input and nudging it along the gradient of the network's loss function. The feedback-path synaptic weights of these networks use the transpose of the forward-path synaptic weights to run error propagation. This problem is commonly named the weight transport problem. Here we consider more biologically plausible neural networks introduced by Lillicrap et al. to run error propagation using feedbackpath weights that are not the transpose of the forward-path ones i.e. without weight transport. This mechanism was called feedback alignment (FA). The introduction of a separate feedback path in in the form of random fixed synaptic weights makes the feedback gradients a rough approximation of those computed by backpropagation. Since gradient-based adversarial attacks are very sensitive to the quality of gradients to perturb the input and fool the neural network, we suspect that the gradients computed without weight transport cannot be accurate enough to design successful gradient-based attacks. Here we compare the robustness of neural networks trained with either BP or FA on three well-known gradient-based attacks, namely the fast gradient sign method (FGSM), the basic iterative method (BIM) and the momentum iterative fast gradient sign method (MI-FGSM). To the best of our knowledge, no prior adversarial attacks have been applied for deep neural networks without weight transport. A typical neural network classifier, trained with the backpropagation algorithm, computes in the feedback path the error signals δ and the weight update ∆W according to the error-backpropagation equations: where y l is the output signal of layer l, φ is the derivative of the activation function φ and η W is a learning-rate factor. For neuroscientists, the weight update in equation 1 is a biologically implausible computation: the backward error δ requires W T, the transposed synapses of the forward path, as the feedback-path synapses. However, the synapses in the forward and feedback paths are physically distinct in the brain and we do not know any biological mechanism to keep the feedback-path synapses equal to the transpose of the forward-path ones. To solve this modeling difficulty, Lillicrap et al. made the forward and feedback paths physically distinct by fixing the feedback-path synapses to different matrices B that are randomly fixed (not learned) during the training phase. This solution, called feedback alignment, enables deep neural networks to compute the error signals δ without weight transport problem by the rule In the rest of this paper, we add the superscript "bp" and "fa" in the notation of any term computed respectively with backpropagation and feedback alignment to avoid any confusion. We call a "BP network" a neural network trained with backpropagation and "FA network" a neural network trained with feedback alignment. Authors in showed recently that the angles between the gradients ∆W f a and ∆W bp stay > 80°for most layers of ResNet-18 and ResNet-50 architectures. This means that feedback alignment provides inaccurate gradients ∆W f a that are mostly not aligned with the true gradients ∆W bp. Since gradient-based adversarial attacks rely on the true gradient to maximize the network loss function, can less accurate gradients computed by feedback alignment provide less-effective adversarial attacks? 3 Gradient-based adversarial attacks The objective of gradient-based attacks is to find gradient updates to the input with the smallest perturbation possible. We compare the robustness of neural networks trained with either feedback alignment or backpropagation using three techniques mentioned in the recent literature. Goodfellow et al. proposed an attack called Fast Gradient Sign Method to generate adversarial examples x by perturbing the input x with one step gradient update along the direction of the sign of gradient, which can be summarized by where is the magnitude of the perturbation, J is the loss function and y * is the label of x. This perturbation can be computed through transposed forward-path synaptic weights like in backpropagation or through random synaptic weights like in feedback alignment. While the Fast Gradient Sign method computes a one step gradient update for each input x, Kurakin et al. extended it to the Basic Iterative Method. It runs the gradient update for multiple iterations using small step size and clips pixel values to avoid large changes on each pixel in the beginning of each iteration as follows where α is the step size and Clip X (·) denotes the clipping function ensuring that each pixel x i,j is in the interval [x ij -, x ij +]. This method is also called the Projected Gradient Descent Method because it "projects" the perturbation onto its feasible set using the clip function. This method is a natural extension to the Fast Gradient Sign Method by introducing momentum to generate adversarial examples iteratively. At each iteration t, the gradient g t is computed by the rule All the experiments were performed on neural networks with the LeNet architecture with the cross-entropy loss function. We vary the perturbation magnitude from 0 to 1 with a step size of 0.1. All adversarial examples were generated using the number of iterations n = 10 for BIM and MI-FGSM attacks and µ = 0.8 for the MI-FGSM attack. The of the accuracy as function of the perturbation magnitude on MNIST are given in Figure 1a. We find that when performing the three gradient-based adversarial attacks (FGSM, BIM and MI-FGSM) on a FA neural network, the accuracy does not decrease and stays around 97%. This suggests that MNIST adversarial examples generated by FA gradients cannot fool FA neural networks for ∈ unlike BP neural networks whose accuracy drastically decreases to 0% as the perturbation increases. In the legend, we denote by "BP → F A" the generation of adversarial examples using BP to fool the FA network, and "F A → BP " the generation of adversarial examples using FA to fool the BP network Additionally, we investigate the transferability of the adversarial examples generated with either BP or FA networks using each one of the three studied attacks. As shown in Figure 1b, we find that the generated adversarial examples by the FA network don't fool the BP network. This means that these adversarial examples are not transferable. The mutual is not true since adversarial examples generated by the BP network can fool the FA network. Results on CIFAR-10 about the robustness of FA and BP networks to the three gradient-based adversarial attacks can be found in Figure 2a. Unlike the on MNIST, the accuracy of FA networks as function of the perturbation magnitude does decrease but still with a lower rate than the accuracy of BP networks. For the transferability of adversarial examples, we still find that the generated adversarial examples by the BP network do fool the FA network. However, unlike the non-transferability of adversarial examples from the FA network to the BP network on MNIST, the BP network is significantly fooled as long as the perturbation magnitude increases. We perform an empirical evaluation investigating both the robustness of deep neural networks without weight transport and the transferability of adversarial examples generated with gradient-based attacks. The on MNIST clearly show that FA networks are robust to adversarial examples generated with FA and the adversarial examples generated by FA are not transferable to BP networks. On the other hand, we find that these two are not true on CIFAR-10 even if FA networks showed a significant robustness to Figure 1b, we denote by "BP → F A" the generation of adversarial examples using BP to fool the FA network, and "F A → BP " the generation of adversarial examples using FA to fool the BP network gradient-based attacks. Therefore, one should consider performing more exhaustive analysis on more complex datasets to understand the impact of the approximated gradients provided by feedback alignment on the adversarial accuracy of biologically plausible neural networks attacked with gradient-based methods.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Less biologically implausible deep neural networks trained without weight transport can be harder to fool.

Represent the short summary for finding the corresponding scientific article (long)

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.