Question: Represent this

Question: Training neural networks with verifiable robustness guarantees is challenging. Several existing approaches utilize linear relaxation based neural network output bounds under perturbation, but they can slow down training by a factor of hundreds depending on the underlying network architectures. Meanwhile, interval bound propagation (IBP) based training is efficient and significantly outperforms linear relaxation based methods on many tasks, yet it may suffer from stability issues since the bounds are much looser especially at the beginning of training. In this paper, we propose a new certified adversarial training method, CROWN-IBP, by combining the fast IBP bounds in a forward bounding pass and a tight linear relaxation based bound, CROWN, in a backward bounding pass. CROWN-IBP is computationally efficient and consistently outperforms IBP baselines on training verifiably robust neural networks. We conduct large scale experiments on MNIST and CIFAR datasets, and outperform all previous linear relaxation and bound propagation based certified defenses in L_inf robustness. Notably, we achieve 7.02% verified test error on MNIST at epsilon=0.3, and 66.94% on CIFAR-10 with epsilon=8/255. The success of deep neural networks (DNNs) has motivated their deployment in some safety-critical environments, such as autonomous driving and facial recognition systems. Applications in these areas make understanding the robustness and security of deep neural networks urgently needed, especially their resilience under malicious, finely crafted inputs. Unfortunately, the performance of DNNs are often so brittle that even imperceptibly modified inputs, also known as adversarial examples, are able to completely break the model . The robustness of DNNs under adversarial examples is well-studied from both attack (crafting powerful adversarial examples) and defence (making the model more robust) perspectives (; a; b; ; ; ; b; 2018b; c; ; ; ; b). Recently, it has been shown that defending against adversarial examples is a very difficult task, especially under strong and adaptive attacks. Early defenses such as distillation have been broken by stronger attacks like C&W (b). Many defense methods have been proposed recently (; ; ; ; ; a; 2019a), but their robustness improvement cannot be certified -no provable guarantees can be given to verify their robustness. In fact, most of these uncertified defenses become vulnerable under stronger attacks . Several recent works in the literature seeking to give provable guarantees on the robustness performance, such as linear relaxations (; ; a; b; ;), interval bound propagation , ReLU stability regularization (c), and distributionally robust optimization and semidefinite relaxations (a; Dvijotham et al.). Linear relaxations of neural networks, first proposed by , is one of the most popular categories among these certified defences. They use the dual of linear programming or several similar approaches to provide a linear relaxation of the network (referred to as a "convex adversarial polytope") and the ing bounds are tractable for robust optimization. However, these methods are both computationally and memory intensive, and can increase model training time by a factor of hundreds. On the other hand, interval bound propagation (IBP) is a simple and efficient method for training verifiable neural networks , which achieved state-of-the-art verified error on many datasets. However, since the IBP bounds are very loose during the initial phase of training, the training procedure can be unstable and sensitive to hyperparameters. In this paper, we first discuss the strengths and weakness of existing linear relaxation based and interval bound propagation based certified robust training methods. Then we propose a new certified robust training method, CROWN-IBP, which marries the efficiency of IBP and the tightness of a linear relaxation based verification bound, CROWN . CROWN-IBP bound propagation involves a IBP based fast forward bounding pass, and a tight convex relaxation based backward bounding pass (CROWN) which scales linearly with the size of neural network output and is very efficient for problems with low output dimensions. Additional, CROWN-IBP provides flexibility for exploiting the strengths of both IBP and convex relaxation based verifiable training methods. The efficiency, tightness and flexibility of CROWN-IBP allow it to outperform state-of-the-art methods for training verifiable neural networks with ∞ robustness under all settings on MNIST and CIFAR-10 datasets. In our experiment, on MNIST dataset we reach 7.02% and 12.06% IBP verified error under ∞ distortions = 0.3 and = 0.4, respectively, outperforming the state-of-the-art baseline by IBP (8.55% and 15.01%). On CIFAR-10, at = 2 255, CROWN-IBP decreases the verified error from 55.88% (IBP) to 46.03% and matches convex relaxation based methods; at a larger, CROWN-IBP outperforms all other methods with a noticeable margin. Neural network robustness verification algorithms seek for upper and lower bounds of an output neuron for all possible inputs within a set S, typically a norm bounded perturbation. Most importantly, the margins between the ground-truth class and any other classes determine model robustness. However, it has already been shown that finding the exact output range is a non-convex problem and NP-complete . Therefore, recent works resorted to giving relatively tight but computationally tractable bounds of the output range with necessary relaxations of the original problem. Many of these robustness verification approaches are based on linear relaxations of non-linear units in neural networks, including CROWN , DeepPoly , Fast-Lin , DeepZ and Neurify (b). We refer the readers to (b) for a comprehensive survey on this topic. After linear relaxation, they bound the output of a neural network f i (·) by linear upper/lower hyper-planes: where a row vector is the product of the network weight matrices W (l) and diagonal matrices D (l) reflecting the ReLU relaxations for output neuron i; b L and b U are two bias terms unrelated to ∆x. Additionally, Dvijotham et al. (2018c; a); solve the Lagrangian dual of verification problem; Raghunathan et al. (2018a; b); Dvijotham et al. propose semidefinite relaxations which are tighter compared to linear relaxation based methods, but computationally expensive. Bounds on neural network local Lipschitz constant can also be used for verification (c;). Besides these deterministic verification approaches, randomized smoothing can be used to certify the robustness of any model in a probabilistic manner (; a; ;). To improve the robustness of neural networks against adversarial perturbations, a natural idea is to generate adversarial examples by attacking the network and then use them to augment the training set . More recently, showed that adversarial training can be formulated as solving a minimax robust optimization problem as in. Given a model with parameter θ, loss function L, and training data distribution X, the training algorithm aims to minimize the robust loss, which is defined as the maximum loss within a neighborhood {x + δ|δ ∈ S} of each data point x, leading to the following robust optimization problem: proposed to use projected gradient descent (PGD) to approximately solve the inner max and then use the loss on the perturbed example x + δ to update the model. Networks trained by this procedure achieve state-of-the-art test accuracy under strong attacks (; a;). Despite being robust under strong attacks, models obtained by this PGD-based adversarial training do not have verified error guarantees. Due to the nonconvexity of neural networks, PGD attack can only compute the lower bound of robust loss (the inner maximization problem). Minimizing a lower bound of the inner max cannot guarantee is minimized. In other words, even if PGD-attack cannot find a perturbation with large loss, that does not mean there exists no such perturbation. This becomes problematic in safety-critical applications since those models need certified safety. Verifiable adversarial training methods, on the other hand, aim to obtain a network with good robustness that can be verified efficiently. This can be done by combining adversarial training and robustness verification-instead of using PGD to find a lower bound of inner max, certified adversarial training uses a verification method to find an upper bound of the inner max, and then update the parameters based on this upper bound of robust loss. Minimizing an upper bound of the inner max guarantees to minimize the robust loss. There are two certified robust training methods that are related to our work and we describe them in detail below. Dvijotham et al. (2018b). Since the bound propagation process of a convex adversarial polytope is too expensive, several methods were proposed to improve its efficiency, like Cauchy projection and dynamic mixed training (a). However, even with these speed-ups, the training process is still slow. Also, this method may significantly reduce a model's standard accuracy (accuracy on natural, unmodified test set). As we will demonstrate shortly, we find that this method tends to over-regularize the network during training, which is harmful for obtaining good accuracy. Interval Bound Propagation (IBP). Interval Bound Propagation (IBP) uses a very simple rule to compute the pre-activation outer bounds for each layer of the neural network. Unlike linear relaxation based methods, IBP does not relax ReLU neurons and does not consider the correlations between neurons of different layers, yielding much looser bounds. proposed a variety of abstract domains to give sound over-approximations for neural networks, including the "Box/Interval Domain" (referred to as IBP in) and showed that it could scale to much larger networks than other works (a) could at the time. demonstrated that IBP could outperform many state-of-the-art by a large margin with more precise approximations for the last linear layer and better training schemes. However, IBP can be unstable to use and hard to tune in practice, since the bounds can be very loose especially during the initial phase of training, posing a challenge to the optimizer. To mitigate instability, use a mixture of regular and minimax robust cross-entropy loss as the model's training loss. Notation. We define an L-layer feed-forward neural network recursively as: where h (x) = x, n 0 represents input dimension and n L is the number of classes, σ is an elementwise activation function. We use z to represent pre-activation neuron values and h to represent Table 1: IBP trained models have low IBP verified errors but when verified with a typically much tighter bound, including convex adversarial polytope (CAP) and CROWN , the verified errors increase significantly. CROWN is generally tighter than convex adversarial polytope however the gap between CROWN and IBP is still large, especially at large. We used a 4-layer CNN network for all datasets to compute these bounds. 1 post-activation neuron values. Consider an input example x k with ground-truth label y k, we define a set of S(x k,) = {x| x − x k ∞ ≤} and we desire a robust network to have the property We define element-wise upper and lower bounds for z (l) and Verification Specifications. Neural network verification literature typically defines a specification vector c ∈ R n L, that gives a linear combination for neural network output: c f (x). In robustness verification, typically we set c i = 1 where i is the ground truth class label, c j = −1 where j is the attack target label and other elements in c are 0. This represents the margin between class i and class j. For an n L class classifier and a given label y, we define a specification matrix C ∈ R n L ×n L as: otherwise (note that the y-th row contains all 0) Importantly, each element in vector m:= Cf (x) ∈ R n L gives us margins between class y and all other classes. We define the lower bound of Cf (x) for all x ∈ S(x k,) as m(x k,), which is a very important quantity: when all elements of m(x k,) > 0, x k is verifiably robust for any perturbation with ∞ norm less than. m(x k,) can be obtained by a neural network verification algorithm, such as convex adversarial polytope, IBP, or CROWN. showed that for cross-entropy (CE) loss: gives us the opportunity to solve the robust optimization problem via minimizing this tractable upper bound of inner-max. This guarantees that max x∈S(x k,) L(f (x), y) is also minimized. Interval Bound Propagation (IBP) Interval Bound Propagation (IBP) uses a simple bound propagation rule. For the input layer we set x L ≤ x ≤ x U element-wise. For affine layers we have: where |W (l) | takes element-wise absolute value. Note that h = x U and h = x L 2. And for element-wise monotonic increasing activation functions σ, 1 We implemented CROWN with efficient CNN support on GPUs: https://github.com/huanzhang12/CROWN-IBP 2 For inputs bounded with general norms, IBP can be applied as long as this norm can be converted to per-neuron intervals after the first affine layer. For example, for p norms (1 ≤ p ≤ ∞) Hölder's inequality can be applied at the first affine layer to obtain h and h, and IBP rule for later layers do not change. We found that IBP can be viewed as training a simple augmented ReLU network which is friendly to optimizers (see Appendix A for more discussions). We also found that a network trained using IBP can obtain good verified errors when verified using IBP, but it can get much worse verified errors using linear relaxation based verification methods, including convex adversarial polytope (CAP) by (equivalently, Fast-Lin by) and CROWN . Table 1 demonstrates that this gap can be very large on large. However, IBP is a very loose bound during the initial phase of training, which makes training unstable and hard to tune; purely using IBP frequently leads to divergence. proposed to use a schedule where is gradually increased during training, and a mixture of robust cross-entropy loss with natural cross-entropy loss as the objective to stabilize training: In Figure 1 we train a small 4-layer MNIST model and we linearly increase from 0 to 0.3 in 60 epochs. We plot the ∞ induced norm of the 2nd CNN layer during the training process of CROWN-IBP and . The norm of weight matrix using does not increase. When becomes larger (roughly at = 0.2, epoch 40), the norm even starts to decrease slightly, indicating that the model is forced to learn smaller norm weights. Meanwhile, the verified error also starts to ramp up possibly due to the lack of capacity. We conjecture that linear relaxation based training over-regularizes the model, especially at a larger. However, in CROWN-IBP, the norm of weight matrices keep increasing during the training process, and verifiable error does not significantly increase when reaches 0.3. Another issue with current linear relaxation based training or verification methods is their high computational and memory cost, and poor scalability. For the small network in Figure 1, convex adversarial polytope (with 50 random Cauchy projections) is 8 times slower and takes 4 times more memory than CROWN-IBP (without using random projections). Convex adversarial polytope scales even worse for larger networks; see Appendix J for a comparison. Overview. We have reviewed IBP and linear relaxation based methods above. As shown in , IBP performs well at large with much smaller verified error, and also efficiently scales to large networks; however, it can be sensitive to hyperparameters due to its very imprecise bound at the beginning phase of training. On the other hand, linear relaxation based methods can give tighter lower bounds at the cost of high computational expenses, but it over-regularizes the network at large and forbids us to achieve good standard and verified accuracy. We propose CROWN-IBP, a new certified defense where we optimize the following problem (θ represents the network parameters): where our lower bound of margin m(x,) is a combination of two bounds with different natures: IBP, and a CROWN-style bound (which will be detailed below); L is the cross-entropy loss. Note that the combination is inside the loss function and is thus still a valid lower bound; thus still holds and we are within the minimax robust optimization theoretical framework. Similar to IBP and TRADES (a), we use a mixture of natural and robust training loss with parameter κ, allowing us to explicitly trade-off between clean accuracy and verified accuracy. In a high level, the computation of the lower bounds of CROWN-IBP (m CROWN-IBP (x,)) consists of IBP bound propagation in a forward bounding pass and CROWN-style bound propagation in a backward bounding pass. We discuss the details of CROWN-IBP algorithm below. Forward Bound Propagation in CROWN-IBP. In CROWN-IBP, we first obtain z (l) and z (l) for all layers by applying, and. Then we will obtain m IBP (x,) = z (L) (assuming C is merged into W (L) ). The time complexity is comparable to two forward propagation passes of the network. Linear Relaxation of ReLU neurons Given z (l) and z (l) computed in the previous step, we first check if some neurons are always active (z where propose to adaptively select α k = 1 when z k | and 0 otherwise, which minimizes the relaxation error. Following, for an input vector z (l), we effectively replace the ReLU layer with a linear layer, giving upper or lower bounds of the output: where D (l) and D (l) are two diagonal matrices representing the "weights" of the relaxed ReLU layer. Other general activation functions can be supported similarly. In the following we focus on conceptually presenting the algorithm, while more details of each term can be found in the Appendix. Backward Bound Propagation in CROWN-IBP. Unlike IBP, CROWN-style bounds start bounding from the last layer, so we refer to it as backward bound propagation (not to be confused with the back-propagation algorithm to obtain gradients). Suppose we want to obtain the lower bound (we assume the specification matrix C has been merged into), which can be bounded linearly by Eq.. CROWN-style bounds choose the lower bound of σ(z i,k is positive, and choose the upper bound otherwise. We then merge W (L) and the linearized ReLU layer together and define: where b with the next linear layer, which is straight forward by plugging in Then we continue to unfold the next ReLU layer σ(z (L−2) ) using its linear relaxations, and compute in a similar manner as in. Along with the bound propagation process, we need to compute a series of matrices,. At this point, we merged all layers of the network into a linear layer: z For ReLU networks, convex adversarial polytope uses a very similar bound propagation procedure. CROWN-style bounds allow an adaptive selection of α i in, thus often gives better bounds (e.g., see Table 1). We give details on each term in Appendix L. Computational Cost. Ordinary CROWN and convex adversarial polytope use as the final layer of the network. For each layer m, we need a different set of m A matrices, defined as A m,(l), l ∈ {m − 1, · · ·, 0}. This causes three computational issues: • Computation of all A m,(l) matrices is expensive. Suppose the network has n neurons for all L − 1 intermediate and input layers and n L n neurons for the output layer (assuming L ≥ 2), the time complexity of ordinary CROWN or convex adversarial polytope is O(A ordinary forward propagation only takes O(Ln 2) time per example, thus ordinary CROWN does not scale up to large networks for training, due to its quadratic dependency in L and extra Ln times overhead. • When both W (l) and W (l−1) represent convolutional layers with small kernel tensors K (l) and, there are no efficient GPU operations to form the matrix and K (l−1). Existing implementations either unfold at least one of the convolutional kernels to fully connected weights, or use sparse matrices to represent W (l) and W (l−1). They suffer from poor hardware efficiency on GPUs. In CROWN-IBP, we use IBP to obtain bounds of intermediate layers, which takes only twice the regular forward propagate time (O(Ln 2)), thus we do not have the first and second issues. The time complexity of the backward bound, only n L times slower than forward propagation and significantly more scalable than ordinary CROWN (which is Ln times slower than forward propagation, where typically n n L). The third convolution issue is also not a concern, since we start from the last specification layer W (L) which is a small fully connected layer. Suppose we need to compute on GPUs using the transposed convolution operator with kernel K (L−1), without unfolding any convoluational layers. Conceptually, the backward pass of CROWN-IBP propagates a small specification matrix W (L) backwards, replacing affine layers with their transposed operators, and activation function layers with a diagonal matrix product. This allows efficient implementation and better scalability. Benefits of CROWN-IBP. Tightness, efficiency and flexibility are unique benefits of CROWN-IBP: • CROWN-IBP is based on CROWN, a tight linear relaxation based lower bound which can greatly improve the quality of bounds obtained by IBP to guide verifiable training and improve stabability; • CROWN-IBP avoids the high computational cost of convex relaxation based methods: the time complexity is reduced from O(, well suited to problems where the output size n L is much smaller than input and intermediate layers' sizes; also, there is no quadratic dependency on L. Thus, CROWN-IBP is efficient on relatively large networks; • The objective is strictly more general than IBP and allows the flexibility to exploit the strength from both IBP (good for large) and convex relaxation based methods (good for small). We can slowly decrease β to 0 during training to avoid the over-regularization problem, yet keeping the initial training of IBP more stable by providing a much tighter bound; we can also keep β = 1 which helps to outperform convex relaxation based methods in small regime (e.g., = 2/255 on CIFAR-10). Models and training schedules. We evaluate CROWN-IBP on three models that are similar to the models used in on MNIST and CIFAR-10 datasets with different ∞ perturbation norms. Here we denote the small, medium and large models in as DM-small, DM-medium and DM-large. During training, we first warm up (regular training without robust loss) for a fixed number of epochs and then increase from 0 to train using a ramp-up schedule of R epochs. Similar techniques are also used in many other works (; a;). For both IBP and CROWN-IBP, a natural cross-entropy (CE) loss with weight κ (as in Eq) may be added, and κ is scheduled to linearly decrease from κ start to κ end within R ramp-up epochs. used κ start = 1 and κ end = 0.5. To understand the trade-off between verified accuracy and standard (clean) accuracy, we explore two more settings: κ start = κ end = 0 (without natural CE loss) and κ start = 1, κ end = 0. For β, a linear schedule during the ramp-up period is used, but we always set β start = 1 and β end = 0, except that we set β start = β end = 1 for CIFAR-10 at = 2 255. Detailed model structures and hyperparameters are in Appendix C. Our training code for IBP and CROWN-IBP, and pre-trained models are publicly available 3. Metrics. Verified error is the percentage of test examples where at least one element in the lower bounds m(x k,) is < 0. It is an guaranteed upper bound of test error under any ∞ perturbations. We obtain m(x k,) using IBP or CROWN-IBP (Eq. 13). We also report standard (clean) errors and errors under 200-step PGD attack. PGD errors are lower bounds of test errors under ∞ perturbations. Comparison to IBP. Table 2 represents the standard, verified and PGD errors under different for each dataset with different κ settings. We test CROWN-IBP on the same model structures in Table 1 of. These three models' architectures are presented in Table A in the Appendix. Here we only report the DM-large model structure in as it performs best under all setttings; small and medium models are deferred to Table C in the Appendix. When both κ start = κ end = 0, no natural CE loss is added and the model focuses on minimizing verified error, but the lack of natural CE loss may lead to unstable training, especially for IBP; the κ start = 1, κ end = 0.5 setting emphasizes on minimizing standard error, usually at the cost of slightly higher verified error rates. κ start = 1, κ end = 0 typically achieves the best balance. We can observe that under the same κ settings, CROWN-IBP outperforms IBP in both standard error and verified error. The benefits of CROWN-IBP is significant especially when model is large and is large. We highlight that CROWN-IBP reduces the verified error rate obtained by IBP from 8.21% to 7.02% on MNIST at = 0.3 and from 55.88% to 46.03% on CIFAR-10 at = 2/255 (it is the first time that an IBP based method outperforms from , and our model also has better standard error). We also note that we are the first to obtain verifiable bound on CIFAR-10 at = 16/255. Trade-off Between Standard Accuracy and Verified Accuracy. To show the trade-off between standard and verified accuracy, we evaluate DM-large CIFAR-10 model with test = 8/255 under different κ settings, while keeping all other hyperparameters unchanged. For each κ end = {0.5, 0.25, 0}, we uniformly choose 11 κ start ∈ [1, κ end] while keeping all other hyper-parameters unchanged. A larger κ start or κ end tends to produce better standard errors, and we can explicitly control the trade-off between standard accuracy and verified accuracy. In Figure 2 we plot the standard and verified errors of IBP and CROWN-IBP trained models with different κ settings. Each cluster on the figure has 11 points, representing 11 different κ start values. Models with lower verified errors tend to have higher standard errors. However, CROWN-IBP clearly outperforms IBP with improvement on both standard and verified accuracy, and pushes the Pareto front towards the lower left corner, indicating overall better performance. To reach the same verified error of 70%, CROWN-IBP can reduce standard error from roughly 55% to 45%. Training Stability. To discourage hand-tuning on a small set of models and demonstrate the stability of CROWN-IBP over a broader range of models, we evaluate IBP and CROWN-IBP on a variety of small and medium sized model architectures (18 for MNIST and 17 for CIFAR-10), detailed in Appendix D. To evaluate training stability, we compare verified errors under different ramp-up schedule length (R = {30, 60, 90, 120} on CIFAR-10 and R = {10, 15, 30, 60} on MNIST) Table 4 of are evaluated using mixed integer programming (MIP) and linear programming (LP), which are strictly smaller than IBP verified errors but computationally expensive. For a fair comparison, we use the IBP verified errors reported in their Table 3. † According to direct communications with , achieving the 68.44% IBP verified error requires to adding an extra PGD adversarial training loss. Without adding PGD, the verified error is 72.91% (LP/MIP verified) or 73.52% (IBP verified). Our should be compared to 73.52%. ‡ Although not explicitly mentioned, the CIFAR-10 models in are trained using train = 1.1 test. We thus follow their settings. § We use βstart = βend = 1 for this setting, and thus CROWN-IBP bound (β = 1) is used to evaluate the verified error. and different κ settings. Instead of reporting just the best model, we compare the best, worst and median verified errors over all models. Our are presented in Figure 3: (a) is for MNIST with = 0.3; (c),(d) are for CIFAR with = 8/255. We can observe that CROWN-IBP achieves better performance consistently under different schedule length. In addition, IBP with κ = 0 cannot stably converge on all models when schedule is short; under other κ settings, CROWN-IBP always performs better. We conduct additional training stability experiments on MNIST and CIFAR-10 dataset under other model and settings and the observations are similar (see Appendix H). We propose a new certified defense method, CROWN-IBP, by combining the fast interval bound propagation (IBP) bound and a tight linear relaxation based bound, CROWN. Our method enjoys high computational efficiency provided by IBP while facilitating the tight CROWN bound to stabilize training under the robust optimization framework, and provides the flexibility to trade-off between the two. Our experiments show that CROWN-IBP consistently outperforms other IBP baselines in both standard errors and verified errors and achieves state-of-the-art verified test errors for ∞ robustness. Given a fixed neural network (NN) f (x), IBP gives a very loose estimation of the output range of f (x). However, during training, since the weights of this NN can be updated, we can equivalently view IBP as an augmented neural network, which we denote as an IBP-NN (Figure A). Unlike a usual network which takes an input x k with label y k, IBP-NN takes two points x L = x k − and x U = x k + as inputs (where x L ≤ x ≤ x U, element-wisely). The bound propagation process can be equivalently seen as forward propagation in a specially structured neural network, as shown in Figure A. After the last specification layer C (typically merged into W (L) ), we can obtain m(x k,). Then, −m(x k,) is sent to softmax layer for prediction. Importantly, since [m(x k,)] y k = 0 (as the y k -th row in C is always 0), the top-1 prediction of the augmented IBP network is y k if and only if all other elements of m(x k,) are positive, i.e., the original network will predict correctly for all x L ≤ x ≤ x U. When we train the augmented IBP network with ordinary cross-entropy loss and desire it to predict correctly on an input x k, we are implicitly doing robust optimization (Eq.). The simplicity of IBP-NN may help a gradient based optimizer to find better solutions. On the other hand, while the computation of convex relaxation based bounds can also be cast as an equivalent network (e.g., the "dual network" in), its construction is significantly more complex, and sometimes requires non-differentiable indicator functions (the sets I +, I − and I in). As a consequence, it can be challenging for the optimizer to find a good solution, and the optimizer tends to making the bounds tighter naively by reducing the norm of weight matrices and over-regularizing the network, as demonstrated in Figure 1. Both IBP and CROWN-IBP produce lower bounds m(x,), and a larger lower bound has better quality. To measure the relative tightness of the two bounds, we take the average of all bounds of training examples: A positive value indicates that CROWN-IBP is tighter than IBP. In Figure B we plot this averaged bound differences during schedule for one MNIST model and one CIFAR-10 model. We can observe that during the early phase of training when the schedule just starts, CROWN-IBP produces significantly better bounds than IBP. A tighter lower bound m(x,) gives a tighter upper bound for max δ∈S L(x + δ; y; θ), making the minimax optimization problem more effective to solve. When the training schedule proceeds, the model gradually learns how to make IBP bounds tighter and eventually the difference between the two bounds become close to 0. Why CROWN-IBP stabilizes IBP training? When taking a randomly initialized network or a naturally trained network, IBP bounds are very loose. But in Table 1, we show that a network trained using IBP can eventually obtain quite tight IBP bounds and high verified accuracy; the network can adapt to IBP bounds and learn a specific set of weights to make IBP tight and also correctly classify examples. However, since the training has to start from weights that produce loose bounds for IBP, the beginning phase of IBP training can be challenging and is vitally important. We observe that IBP training can have a large performance variance across models and initializations. Also IBP is more sensitive to hyper-parameter like κ or schedule length; in Figure 3, many IBP models converge sub-optimally (large worst/median verified error). The reason for instability is that during the beginning phase of training, the loose bounds produced by IBP make the robust loss ineffective, and it is challenging for the optimizer to reduce this loss and find a set of good weights that produce tight IBP verified bounds in the end. Conversely, if our bounds are much tighter at the beginning, the robust loss always remains in a reasonable range during training, and the network can gradually learn to find a good set of weights that make IBP bounds increasingly tighter (this is obvious in Figure B). Initially, tighter bounds can be provided by a convex relaxation based method like CROWN, and they are gradually replaced by IBP bounds (using β start = 1, β end = 0), eventually leading to a model with learned tight IBP bounds in the end. The goal of these experiments is to reproduce the performance reported in and demonstrate the advantage of CROWN-IBP under the same experimental settings. Specifically, to reproduce the IBP , for CIFAR-10 we train using a large batch size and long training schedule on TPUs (we can also replicate these on multi-GPUs using a reasonable amount of training time; see Section F). Also, for this set of experiments we use the same code base as in. For model performance on a comprehensive set of small and medium sized models trained on a single GPU, please see Table D in Section F, as well as the training stability experiments in Section 4 and Section H. The models structures (DM-small, DM-medium and DM-large) used in Table C and Table 2 are listed in Table A. These three model structures are the same as in. Training hyperparameters are detailed below: • For MNIST IBP baseline , we follow exact the same set of hyperparameters as in . We train 100 epochs (60K steps) with a batch size of 100, and use a warm-up and ramp-up duration of 2K and 10K steps. Learning rate for Adam optimizer is set to 1 × 10 −3 and decayed by 10X at steps 15K and 25K. Our IBP match their reported numbers. Note that we always use IBP verified errors rather than MIP verified errors. We use the same schedule for CROWN-IBP with train = 0.2 (test = 0.1) in Table C and Table 2. For train = 0.4, this schedule can obtain verified error rates 4.22%, 7.01% and 12.84% at test = {0.2, 0.3, 0.4} using the DM-Large model, respectively. • For MNIST CROWN-IBP with train = 0.4 in Table C and Table 2, we train 200 epochs with a batch size of 256. We use Adam optimizer and set learning rate to 5 × 10 −4. We warm up with 10 epochs' regular training, and gradually ramp up from 0 to train in 50 epochs. We reduce the learning rate by 10X at epoch 130 and 190. Using this schedule, IBP's performance becomes worse (by about 1-2% in all settings), but this schedule improves verified error for CROWN-IBP at test = 0.4 from 12.84% to to 12.06% and does do affect verified errors at other test levels. • For CIFAR-10, we follow the setting in and train 3200 epochs on 32 TPU cores. We use a batch size of 1024, and a learning rate of 5 × 10 −4. We warm up for 320 epochs, and ramp-up for 1600 epochs. Learning rate is reduced by 10X at epoch 2600 and 3040. We use random horizontal flips and random crops as data augmentation, and normalize images according to per-channel statistics. Note that this schedule is slightly different from the schedule used in ; we use a smaller batch size due to TPU memory constraints (we used TPUv2 which has half memory capacity as TPUv3 used in ), and also we decay learning rates later. We found that this schedule improves both IBP baseline performance and CROWN-IBP performance by around 1%; for example, at = 8/255, this improved schedule can reduce verified error from 73.52% to 72.68% for IBP baseline (κ start = 1.0, κ end = 0.5) using the DM-Large model. Hyperparameter κ and β. We use a linear schedule for both hyperparameters, decreasing κ from κ start to κ end while increasing β from β start to β end. The schedule length is set to the same length as the schedule. In both IBP and CROWN-IBP, a hyperparameter κ is used to trade-off between clean accuracy and verified accuracy. Figure 2 shows that κ end can significantly affect the trade-off, while κ start has minor impacts compared to κ end. In general, we recommend κ start = 1 and κ end = 0 as a safe starting point, and we can adjust κ end to a larger value if a better standard accuracy is desired. The setting κ start = κ end = 0 (pure minimax optimization) can be challenging for IBP as there is no natural loss as a stabilizer; under this setting CROWN-IBP usually produces a model with good (sometimes best) verified accuracy but noticeably worse standard accuracy (on CIFAR-10 = 8 255 the difference can be as large as 10%), so this setting is only recommended when a model with best verified accuracy is desired at a cost of noticeably reduced standard accuracy. Compared to IBP, CROWN-IBP adds one additional hyperparameter, β. β has a clear meaning: balancing between the convex relaxation based bounds and the IBP bounds. β start is always set to 1 as we want to use CROWN-IBP to obtain tighter bounds to stabilize the early phase of training when IBP bounds are very loose; β end determines if we want to use a convex relaxation based bound (β end = 1) or IBP based bound (β end = 0) after the schedule. Thus, we set β end = 1 for the case where convex relaxation based method can outperform IBP (e.g., CIFAR-10 = 2/255, and β end = 0 for the case where IBP outperforms convex relaxation based bounds. We do not tune or grid-search this hyperparameter. . "CONV k w×h+s" represents a 2D convolutional layer with k filters of size w×h using a stride of s in both dimensions. "FC n" = fully connected layer with n outputs. Last fully connected layer is omitted. All networks use ReLU activation functions. In all our training stability experiments, we use a large number of relatively small models and train them on a single GPU. These small models cannot achieve state-of-the-art performance but they can be trained quickly and cheaply, allowing us to explore training stability over a variety of settings, and report min, median and max statistics. We use the following hyperparameters: • For MNIST, we train 100 epochs with batch size 256. We use Adam optimizer and the learning rate is 5 × 10 −4 . The first epoch is standard training for warming up. We gradually increase linearly per batch in our training process with a schedule length of 60. We reduce the learning rate by 50% every 10 epochs after schedule ends. No data augmentation technique is used and the whole 28 × 28 images are used (normalized to 0 -1 range). • For CIFAR, we train 200 epoch with batch size 128. We use Adam optimizer and the learning rate is 0.1%. The first 10 epochs are standard training for warming up. We gradually increase linearly per batch in our training process with a schedule length of 120. We reduce the learning rate by 50% every 10 epochs after schedule ends. We use random horizontal flips and random crops as data augmentation. The three channels are normalized with mean (0.4914, 0.4822, 0.4465) and standard deviation (0.2023, 0.1914, 0.2010). These numbers are per-channel statistics from the training set used in . All verified error numbers are evaluated on the test set using IBP, since the networks are trained using IBP (β = 0 after reaches the target train), except for CIFAR = 2 255 where we set β = 1 to compute the CROWN-IBP verified error. Table B gives the 18 model structures used in our training stability experiments. These model structures are designed by us and are not used in. Most CIFAR-10 models share the same structures as MNIST models (unless noted on the table) except that their input dimensions are different. Model A is too small for CIFAR-10 thus we remove it for CIFAR-10 experiments. Models A -J are the "small models" reported in Figure 3. Models K -T are the "medium models" reported in Figure 3. For in Table 1, we use a small model (model structure B) for all three datasets. These MNIST, CIFAR-10 models can be trained on a single NVIDIA RTX 2080 Ti GPU within a few hours each. In Table 2 we report from the best DM-Large model. Table C presents the verified, standard (clean) and PGD attack errors for all three model structures used in (DM-Small, DM-Medium and DM-Large) trained on MNIST and CIFAR-10 datasets. We evaluate IBP and CROWN-IBP under the same three κ settings as in Table 2. We use hyperparameters detailed in Section C to train these models. We can see that given any model structure and any κ setting, CROWN-IBP consistently outperforms IBP. Table B: Model structures used in our training stability experiments. We use ReLU activations for all models. We omit the last fully connected layer as its output dimension is always 10. In the table, "Conv k w × w + s" represents to a 2D convolutional layer with k filters of size w × w and a stride of s. Model A -J are referred to as "small models" and model K to T are referred to as "medium models". In this section we present additional experiments on a variety of smaller MNIST and CIFAR-10 models which can be trained on a single GPU. The purpose of this experiment is to compare model performance statistics (min, median and max) on a wide range of models, rather than a few hand selected models. The model structures used in these experiments are detailed in Table B. In Table D, we present the best, median and worst verified and standard (clean) test errors for models trained on MNIST and CIFAR-10 using IBP and CROWN-IBP. Although these small models cannot achieve state-of-the-art performance, CROWN-IBP's best, median and worst verified errors among all model structures consistently outperform those of IBP. Especially, in many situations the worst case verified error improves significantly using CROWN-IBP, because IBP training is not stable on some of the models. It is worth noting that in this set of experiments we explore a different setting: train = test. We found that both IBP and CROWN-IBP tend to overfit to training dataset on MNIST with small, thus verified errors are not as good as presented in Table C. This overfitting issue can be alleviated by using train > test (as used in Table 2 and Table C), or using an explicit 1 regularization, which will be discussed in detail in Section I. To further test the training stability of CROWN-IBP, we run each MNIST experiment (using selected models in Table B) 5 times to get the mean and standard deviation of the verified and standard errors on test set. Results are presented in Table E Table E: Means and standard deviations of verified and standard errors of 10 MNIST models trained using CROWN-IBP. The architectures of these models are presented in Table B. We run each model 5 times to compute its mean and standard deviation. regularization term in CROWN-IBP training helps when train = 0.1 or 0.2. The verified and standard errors on the training and test sets with and without regularization can be found in Table F. We can see that with a small 1 regularization added (λ = 5 × 10 −5) we can reduce verified errors on test set significantly. This makes CROWN-IBP comparable to the numbers reported in convex adversarial polytope ; at = 0.1, the best model using convex adversarial polytope training can achieve 3.67% verified error, while CROWN-IBP achieves 3.60% best certified error on the models presented in Table F. The overfitting is likely caused by IBP's strong learning power without over-regularization, which also explains why IBP based methods significantly outperform linear relaxation based methods at larger values. Using early stopping can also improve verified error on test set; see Figure D. J TRAINING TIME In Table G we present the training time of CROWN-IBP, IBP and convex adversarial polytope on several representative models. All experiments are measured on a single RTX 2080 Ti GPU with 11 GB RAM except for 2 DM-Large models where we use 4 RTX 2080 Ti GPUs to speed up training. We can observe that CROWN-IBP is practically 1.5 to 3.5 times slower than IBP. Theoretically, CROWN-IBP is up to n L = 10 times slower 4 than IBP; however usually the total training time is less than 10 times since the CROWN-IBP bound is only computed during the ramp-up phase, and CROWN-IBP has higher GPU computation intensity and thus better GPU utilization than IBP. convex adversarial polytope . Using random projections alone is not sufficient to scale purely linear relaxation based methods to larger datasets, thus we advocate a combination of IBP bounds with linear relaxation based methods as in CROWN-IBP, which offers good scalability and stability. We also note that the random projection based acceleration can also be applied to the backward bound propagation (CROWN-style bound) in CROWN-IBP to further speed up CROWN-IBP. The use of 32 TPUs for our CIFAR-10 experiments is not necessary. We use TPUs mainly for obtaining a completely fair comparison to IBP , as their implementation was TPU-based. Since TPUs are not widely available, we additionally implemented CROWN-IBP using multi-GPUs. We train the best models in Table 2 on 4 RTX 2080Ti GPUs. As shown in Table H, we can achieve comparable verified errors using GPUs, and the differences between GPU and TPU training are around ±0.5%. Training time is reported in Table G. L EXACT FORMS OF THE CROWN-IBP BACKWARD BOUND CROWN is a general framework that replaces non-linear functions in a neural network with linear upper and lower hyperplanes with respect to pre-activation variables, such that the entire neural network function can be bounded by a linear upper hyperplane and linear lower hyperplane for all x ∈ S (S is typically a norm bounded ball, or a box region): CROWN achieves such linear bounds by replacing non-linear functions with linear bounds, and utilizing the fact that the linear combinations of linear bounds are still linear, thus these linear bounds 1 We use β start = β end = 1 for this setting, the same as in Table 2, and thus CROWN-IBP bound is used to evaluate the verified error. can propagate through layers. Suppose we have a non-linear vector function σ, applying to an input (pre-activation) vector z, CROWN requires the following bounds in a general form: In general the specific bounds A σ, b σ, A σ, b σ for different σ needs to be given in a case-by-case basis, depending on the characteristics of σ and the preactivation range z ≤ z ≤ z. In neural network common σ can be ReLU, tanh, sigmoid, maxpool, etc. Convex adversarial polytope is also a linear relaxation based techniques that is closely related to CROWN, but only for ReLU layers. For ReLU such bounds are simple, where A σ, A σ are diagonal matrices, b σ = 0: where D and D are two diagonal matrices:, if z k > 0, i.e., this neuron is always active 0, if z k < 0, i.e., this neuron is always inactive α, otherwise, any 0 ≤ α ≤ 1 1, if z k > 0, i.e., this neuron is always active 0, if z k < 0, i.e., this neuron is always inactive if z k > 0, i.e., this neuron is always active 0, if z k < 0, i.e., this neuron is always inactive Note that CROWN-style bounds require to know all pre-activation bounds z (l) and z (l). We assume these bounds are valid for x ∈ S. In CROWN-IBP, these bounds are obtained by interval bound propagation (IBP). With pre-activation bounds z (l) and z (l) given (for x ∈ S), we rewrite the CROWN lower bound for the special case of ReLU neurons: Theorem L.1 (CROWN Lower Bound). For a L-layer neural network function f (x): R n0 → R n L, ∀j ∈ [n L], ∀x ∈ S, we have f j (x) ≤ f j (x), where if l ∈ {0, · · ·, L − 1}.

We propose a new certified adversarial training method, CROWN-IBP, that achieves state-of-the-art robustness for L_inf norm adversarial perturbations.

This paper identifies classes of problems for which adversarial examples are inescapable, and derives fundamental bounds on the susceptibility of any classifier to adversarial examples.


Question: Represent this text.

Question: Network pruning is widely used for reducing the heavy inference cost of deep models in low-resource settings. A typical pruning algorithm is a three-stage pipeline, i.e., training (a large model), pruning and fine-tuning. During pruning, according to a certain criterion, redundant weights are pruned and important weights are kept to best preserve the accuracy. In this work, we make several surprising observations which contradict common beliefs. For all state-of-the-art structured pruning algorithms we examined, fine-tuning a pruned model only gives comparable or worse performance than training that model with randomly initialized weights. For pruning algorithms which assume a predefined target network architecture, one can get rid of the full pipeline and directly train the target network from scratch. Our observations are consistent for multiple network architectures, datasets, and tasks, which imply that: 1) training a large, over-parameterized model is often not necessary to obtain an efficient final model, 2) learned ``important'' weights of the large model are typically not useful for the small pruned model, 3) the pruned architecture itself, rather than a set of inherited ``important'' weights, is more crucial to the efficiency in the final model, which suggests that in some cases pruning can be useful as an architecture search paradigm. Our suggest the need for more careful baseline evaluations in future research on structured pruning methods. We also compare with the "Lottery Ticket Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate, the "winning ticket" initialization as used in does not bring improvement over random initialization. Over-parameterization is a widely-recognized property of deep neural networks , which leads to high computational cost and high memory footprint for inference. As a remedy, network pruning BID12;;; BID14 has been identified as an effective technique to improve the efficiency of deep networks for applications with limited computational budget. A typical procedure of network pruning consists of three stages: 1) train a large, over-parameterized model (sometimes there are pretrained models available), 2) prune the trained large model according to a certain criterion, and 3) fine-tune the pruned model to regain the lost performance. 2015). Thus most existing pruning techniques choose to fine-tune a pruned model instead of training it from scratch. The preserved weights after pruning are usually considered to be critical, as how to accurately select the set of important weights is a very active research topic in the literature (; BID14 ; b; BID4).In this work, we show that both of the beliefs mentioned above are not necessarily true for structured pruning methods, which prune at the levels of convolution channels or larger. Based on an extensive empirical evaluation of state-of-the-art pruning algorithms on multiple datasets with multiple network architectures, we make two surprising observations. First, for structured pruning methods with predefined target network architectures (Figure 2), directly training the small target model from random initialization can achieve the same, if not better, performance, as the model obtained from the three-stage pipeline. In this case, starting with a large model is not necessary and one could instead directly train the target model from scratch. Second, for structured pruning methods with autodiscovered target networks, training the pruned model from scratch can also achieve comparable or even better performance than fine-tuning. This observation shows that for these pruning methods, what matters more may be the obtained architecture, instead of the preserved weights, despite training the large model is needed to find that target architecture. Interestingly, for a unstructured pruning method that prunes individual parameters, we found that training from scratch can mostly achieve comparable accuracy with pruning and fine-tuning on smaller-scale datasets, but fails to do so on the large-scale ImageNet benchmark. Note that in some cases, if a pretrained large model is already available, pruning and fine-tuning from it can save the training time required to obtain the efficient model. The contradiction between some of our and those reported in the literature might be explained by less carefully chosen hyper-parameters, data augmentation schemes and unfair computation budget for evaluating baseline approaches. Predefined: prune x% channels in each layer Automatic: prune a%, b%, c%, d% channels in each layer A 4-layer model Figure 2: Difference between predefined and automatically discovered target architectures, in channel pruning as an example. The pruning ratio x is userspecified, while a, b, c, d are determined by the pruning algorithm. Unstructured sparse pruning can also be viewed as automatic. Our advocate a rethinking of existing structured network pruning algorithms. It seems that the over-parameterization during the first-stage training is not as beneficial as previously thought. Also, inheriting weights from a large model is not necessarily optimal, and might trap the pruned model into a bad local minimum, even if the weights are considered "important" by the pruning criterion. Instead, our suggest that the value of automatic structured pruning algorithms sometimes lie in identifying efficient structures and performing implicit architecture search, rather than selecting "important" weights. For most structured pruning methods which prune channels/filters, this corresponds to searching the number of channels in each layer. In section 5, we discuss this viewpoint through carefully designed experiments, and show the patterns in the pruned model could provide design guidelines for efficient architectures. The rest of the paper is organized as follows: in Section 2, we introduce and some related works on network pruning; in Section 3, we describe our methodology for training the pruned model from scratch; in Section 4 we experiment on various pruning methods and show our main for both pruning methods with predefined or automatically discovered target architectures; in Section 5, we discuss the value of automatic pruning methods in searching efficient network architectures; in Section 6 we discuss some implications and conclude the paper. Recent success of deep convolutional networks BID13; BID8;; 2017a) has been coupled with increased requirement of computation resources. In particular, the model size, memory footprint, the number of computation operations (FLOPs) and power usage are major aspects inhibiting the use of deep neural networks in some resource-constrained settings. Those large models can be infeasible to store, and run in real time on embedded systems. To address this issue, many methods have been proposed such as low-rank approximation of weights (; BID11, weight quantization , knowledge distillation BID2 ) and network pruning (; BID14, among which network pruning has gained notable attention due to their competitive performance and compatibility. One major branch of network pruning methods is individual weight pruning, and it dates back to Optimal Brain Damage BID12 and Optimal Brain Surgeon , which prune weights based on Hessian of the loss function. More recently, proposes to prune network weights with small magnitude, and this technique is further incorporated into the "Deep Compression" pipeline (b) to obtain highly compressed models. proposes a data-free algorithm to remove redundant neurons iteratively. uses Variatonal Dropout (P.) to prune redundant weights. learns sparse networks through L 0 -norm regularization based on stochastic gate. However, one drawback of these unstructured pruning methods is that the ing weight matrices are sparse, which cannot lead to compression and speedup without dedicated hardware/libraries (a).In contrast, structured pruning methods prune at the level of channels or even layers. Since the original convolution structure is still preserved, no dedicated hardware/libraries are required to realize the benefits. Among structured pruning methods, channel pruning is the most popular, since it operates at the most fine-grained level while still fitting in conventional deep learning frameworks. Some heuristic methods include pruning channels based on their corresponding filter weight norm BID14 and average percentage of zeros in the output BID3. Group sparsity is also widely used to smooth the pruning process after training (; BID0 BID10). BID4 and impose sparsity constraints on channel-wise scaling factors during training, whose magnitudes are then used for channel pruning. BID6 uses a similar technique to prune coarser structures such as residual blocks. He et al. (2017b) and minimizes next layer's feature reconstruction error to determine which channels to keep. optimizes the reconstruction error of the final response layer and propagates a "importance score" for each channel. uses Taylor expansion to approximate each channel's influence over the final loss and prune accordingly. analyzes the intrinsic correlation within each layer and prune redundant channels. proposes a layer-wise compensate filter pruning algorithm to improve commonly-adopted heuristic pruning metrics. He et al. (2018a) proposes to allow pruned filters to recover during the training process. BID15 prune certain structures in the network based on the current input. Our work is also related to some recent studies on the characteristics of pruning algorithms. shows that random channel pruning can perform on par with a variety of more sophisticated pruning criteria, demonstrating the plasticity of network models. In the context of unstructured pruning, The Lottery Ticket Hypothesis conjectures that certain connections together with their randomly initialized weights, can enable a comparable accuracy with the original network when trained in isolation. We provide comparisons between and this work in Appendix A. shows that training a small-dense model cannot achieve the same accuracy as a pruned large-sparse model with identical memory footprint. In this work, we reveal a different and rather surprising characteristic of structured network pruning methods: fine-tuning the pruned model with inherited weights is not better than training it from scratch; the ing pruned architectures are more likely to be what brings the benefit. In this section, we describe in detail our methodology for training a small target model from scratch. Target Pruned Architectures. We first divide network pruning methods into two categories. In a pruning pipeline, the target pruned model's architecture can be determined by either a human (i.e., predefined) or the pruning algorithm (i.e., automatic) (see Figure 2). When a human predefines the target architecture, a common criterion is the ratio of channels to prune in each layer. For example, we may want to prune 50% channels in each layer of VGG. In this case, no matter which specific channels are pruned, the pruned target architecture remains the same, because the pruning algorithm only locally prunes the least important 50% channels in each layer. In practice, the ratio in each layer is usually selected through empirical studies or heuristics. Examples of predefined structured pruning include BID14, , He et al. (2017b) and He et al. (2018a) When the target architecture is automatically determined by a pruning algorithm, it is usually based on a pruning criterion that globally compares the importance of structures (e.g., channels) across layers. Examples of automatic structured pruning include BID4, BID6, and.Unstructured pruning (; ;) also falls in the category of automatic methods, where the positions of pruned weights are determined by the training process and the pruning algorithm, and it is usually not possible to predefine the positions of zeros before training starts. Datasets, Network Architectures and Pruning Methods. In the network pruning literature, CIFAR-10, CIFAR-100 BID9 ImageNet datasets are the de-facto benchmarks, while VGG , ResNet and DenseNet BID4 are the common network architectures. We evaluate four predefined pruning methods, BID14, , He et al. (2017b), He et al. (2018a), two automatic structured pruning methods, BID4, BID6, and one unstructured pruning method . For the first six methods, we evaluate using the same (target model, dataset) pairs as presented in the original paper to keep our comparable. For the last one , we use the aforementioned architectures instead, since the ones in the original paper are no longer state-of-the-art. On CIFAR datasets, we run each experiment with 5 random seeds, and report the mean and standard deviation of the accuracy. Training Budget. One crucial question is how long we should train the small pruned model from scratch. Naively training for the same number of epochs as we train the large model might be unfair, since the small pruned model requires significantly less computation for one epoch. Alternatively, we could compute the floating point operations (FLOPs) for both the pruned and large models, and choose the number of training epoch for the pruned model that would lead to the same amount of computation as training the large model. Note that it is not clear how to train the models to "full convergence" given the stepwise decaying learning rate schedule commonly used in the CIFAR/ImageNet classification tasks. In our experiments, we use Scratch-E to denote training the small pruned models for the same epochs, and Scratch-B to denote training for the same amount of computation budget (on ImageNet, if the pruned model saves more than 2× FLOPs, we just double the number of epochs for training Scratch-B, which amounts to less computation budget than large model training). When extending the number of epochs in Scratch-B, we also extend the learning rate decay schedules proportionally. One may argue that we should instead train the small target model for fewer epochs since it may converge faster. However, in practice we found that increasing the training epochs within a reasonable range is rarely harmful. In our experiments we found in most times Scratch-E is enough while in other cases Scratch-B is needed for a comparable accuracy as fine-tuning. Note that our evaluations use the same computation as large model training without considering the computation in fine-tuning, since in our evaluated methods fine-tuning does not take too long; if anything this still favors the pruning and fine-tuning pipeline. Implementation. In order to keep our setup as close to the original papers as possible, we use the following protocols: 1) ff a previous pruning method's training setup is publicly available, e.g. BID4, BID6 and He et al. (2018a), we adopt the original implementation; 2) otherwise, for simpler pruning methods, e.g., BID14 and , we re-implement the three-stage pruning procedure and generally achieve similar as in the original papers; 3) for the remaining two methods (; b), the pruned models are publicly available but without the training setup, thus we choose to re-train both large and small target models from scratch. Interestingly, the accuracy of our re-trained large model is higher than what is reported in the original papers. This could be due to the difference in the deep learning frameworks: we used Pytorch while the original papers used Caffe BID8. In this case, to accommodate the effects of different frameworks and training setups, we report the relative accuracy drop from the unpruned large model. We use standard training hyper-parameters and data-augmentation schemes, which are used both in standard image classification models (; BID4 and network pruning methods BID14 BID4 BID6 a). The optimization method is SGD with Nesterov momentum, using an stepwise decay learning rate schedule. For random weight initialization, we adopt the scheme proposed in . For of models fine-tuned from inherited weights, we either use the released models from original papers (case 3 above) or follow the common practice of fine-tuning the model using the lowest learning rate when training the large model BID14 b). For CIFAR, training/fine-tuning takes 160/40 epochs. For ImageNet, training/fine-tuning takes 90/20 epochs. For reproducing the and a more detailed knowledge about the settings, see our code at: https://github. com/Eric-mingjie/rethinking-network-pruning. In this section we present our experimental comparing training pruned models from scratch and fine-tuning from inherited weights, for both predefined and automatic (Figure 2) structured pruning, as well as a magnitude-based unstructured pruning method . We also include a comparison with the Lottery Ticket Hypothesis , and an experiment on transfer learning from image classification to object detection in Appendix, due to space limit. We also put the and discussions on a pruning method (Soft Filter pruning (a) ) in Appendix. L 1 -norm based Filter Pruning BID14 ) is one of the earliest works on filter/channel pruning for convolutional networks. In each layer, a certain percentage of filters with smaller L 1 -norm will be pruned. Table 1 shows our . The Pruned Model column shows the list of predefined target models (see BID14 for configuration details on each model). We observe that in each row, scratch-trained models achieve at least the same level of accuracy as fine-tuned models, with Scratch-B slightly higher than Scratch-E in most cases. On ImageNet, both Scratch-B models are better than the fine-tuned ones by a noticeable margin. Table 1: Results (accuracy) for L1-norm based filter pruning BID14. "Pruned Model" is the model pruned from the large model. Configurations of Model and Pruned Model are both from the original paper. ThiNet greedily prunes the channel that has the smallest effect on the next layer's activation values. As shown in TAB2, for VGG-16 and ResNet-50, both Scratch-E and Scratch-B can almost always achieve better performance than the fine-tuned model, often by a significant margin. The only exception is Scratch-E for VGG-Tiny, where the model is pruned very aggressively from VGG-16 (FLOPs reduced by 15×), and as a , drastically reducing the training budget for Scratch-E. The training budget of Scratch-B for this model is also 7 times smaller than the original large model, yet it can achieve the same level of accuracy as the fine-tuned model. Regression based Feature Reconstruction (b) prunes channels by minimizing the feature map reconstruction error of the next layer. In contrast to ThiNet , this optimization problem is solved by LASSO regression. Results are shown in TAB3. Again, in terms of relative accuracy drop from the large models, scratch-trained models are better than the fine-tuned models. (b). Pruned models such as "VGG-16-5x" are defined in He et al. (2017b). Similar to TAB2, we compare relative accuracy drop from unpruned large models. Network Slimming BID4 imposes L 1 -sparsity on channel-wise scaling factors from Batch Normalization layers BID7 during training, and prunes channels with lower scaling factors afterward. Since the channel scaling factors are compared across layers, this method produces automatically discovered target architectures. As shown in Table 4, for all networks, the small models trained from scratch can reach the same accuracy as the fine-tuned models. More specifically, we found that Scratch-B consistently outperforms (8 out of 10 experiments) the finetuned models, while Scratch-E is slightly worse but still mostly within the standard deviation. Table 4: Results (accuracy) for Network Slimming BID4. " Prune ratio" stands for total percentage of channels that are pruned in the whole network. The same ratios for each model are used as the original paper. Sparse Structure Selection BID6 ) also uses sparsified scaling factors to prune structures, and can be seen as a generalization of Network Slimming. Other than channels, pruning can be on residual blocks in ResNet or groups in ResNeXt . We examine residual blocks pruning, where ResNet-50 are pruned to be ResNet-41, ResNet-32 and ResNet-26. Table 5 shows our . On average Scratch-E outperforms pruned models, and for all models Scratch-B is better than both. Table 5: Results (accuracy) for residual block pruning using Sparse Structure Selection BID6. In the original paper no fine-tuning is required so there is a "Pruned" column instead of "Fine-tuned" as before. Unstructured magnitude-based weight pruning can also be treated as automatically discovering architectures, since the positions of exact zeros cannot be determined before training, but we highlight its differences with structured pruning using another subsection. Because all the network architectures we evaluated are fully-convolutional (except for the last fully-connected layer), for simplicity, we only prune weights in convolution layers here. Before training the pruned sparse model from scratch, we re-scale the standard deviation of the Gaussian distribution for weight initialization, based on how many non-zero weights remain in this layer. This is to keep a constant scale of backward gradient signal as in , which however in our observations does not bring gains compared with unscaled counterparts. Table 6: Results (accuracy) for unstructured pruning . " Prune Ratio" denotes the percentage of parameters pruned in the set of all convolutional weights. As shown in Table 6, on the smaller-scale CIFAR datasets, when the pruned ratio is small (≤ 80%), Scratch-E sometimes falls short of the fine-tuned , but Scratch-B is able to perform at least on par with the latter. However, we observe that in some cases, when the prune ratio is large (95%), fine-tuning can outperform training from scratch. On the large-scale ImageNet dataset, we note that the Scratch-B is mostly worse than fine-tuned by a noticable margin, despite at a decent accuracy level. This could be due to the increased difficulty of directly training on the highly sparse networks (CIFAR), or the scale/complexity of the dataset itself (ImageNet). Another possible reason is that compared with structured pruning, unstructured pruning significantly changes the weight distribution (more details in Appendix G). The difference in scratch-training behaviors also suggests an important difference between structured and unstructured pruning. While we have shown that, for structured pruning, the inherited weights in the pruned architecture are not better than random, the pruned architecture itself turns out to be what brings the efficiency benefits. In this section, we assess the value of architecture search for automatic network pruning algorithms (Figure 2) by comparing pruning-obtained models and uniformly pruned models. Note that the connection between network pruning and architecture learning has also been made in prior works (; BID4 ;, but to our knowledge we are the first to isolate the effect of inheriting weights and solely compare pruningobtained architectures with uniformly pruned ones, by training both of them from scratch. Parameter Efficiency of Pruned Architectures. In FIG0 (left), we compare the parameter efficiency of architectures obtained by an automatic channel pruning method (Network Slimming BID4), with a naive predefined pruning strategy that uniformly prunes the same percentage of channels in each layer. All architectures are trained from random initialization for the same number of epochs. We see that the architectures obtained by Network Slimming are more parameter efficient, as they could achieve the same level of accuracy using 5× fewer parameters than uniformly pruning architectures. For unstructured magnitude-based pruning , we conducted a similar experiment shown in FIG0 (right). Here we uniformly sparsify all individual weights at a fixed probability, and the architectures obtained this way are much less efficient than the pruned architectures. Figure 4: The average sparsity pattern of all 3×3 convolutional kernels in certain layer stages in a unstructured pruned VGG-16. Darker color means higher probability of weight being kept. We also found the channel/weight pruned architectures exhibit very consistent patterns (see TAB8 and Figure 4). This suggests the original large models may be redundantly designed for the task and the pruning algorithm can help us improve the efficiency. This also confirms the value of automatic pruning methods for searching efficient models on the architectures evaluated. More Analysis. However, there also exist cases where the architectures obtained by pruning are not better than uniformly pruned ones. We present such in FIG1, where the architectures obtained by pruning (blue) are not significantly more efficient than uniform pruned architectures (red). This phenomenon happens more likely on modern architectures like ResNets and DenseNets. When we investigate the sparsity patterns of those pruned architectures (shown in TAB13, 19 and 20 in Appendix H), we find that they exhibit near-uniform sparsity patterns across stages, which might be the reason why it can only perform on par with uniform pruning. In contrast, for VGG, the pruned sparsity patterns can always beat the uniform ones as shown in FIG0 and Figure 6. We also show the sparsity patterns of VGG pruned by Network Slimming BID4 in TAB2 of Appendix H, and they are rather far from uniform. Compared to ResNet and DenseNet, we can see that VGG's redundancy is rather imbalanced across layer stages. Network pruning techniques may help us identify the redundancy better in the such cases. Generalizable Design Principles from Pruned Architectures. Given that the automatically discovered architectures tend to be parameter efficient on the VGG networks, one may wonder: can we derive generalizable principles from them on how to design a better architecture? We conduct several experiments to answer this question. For Network Slimming, we use the average number of channels in each layer stage (layers with the same feature map size) from pruned architectures to construct a new set of architectures, and we call this approach "Guided Pruning"; for magnitude-based pruning, we analyze the sparsity patterns (Figure 4) in the pruned architectures, and apply them to construct a new set of sparse models, which we call "Guided Sparsification". The are shown in Figure 6. It can be seen that for both Network Slimming (Interestingly, these guided design patterns can sometimes be transferred to a different VGG-variant and/or dataset. In Figure 6, we distill the patterns of pruned architectures from VGG-16 on CIFAR-10 and apply them to design efficient VGG-19 on CIFAR-100. These sets of architectures are denoted as "Transferred Guided Pruning/Sparsification". We can observe that they (brown) may sometimes be slightly worse than architectures directly pruned (blue), but are significantly better than uniform pruning/sparsification (red). In these cases, one does not need to train a large model to obtain an efficient model as well, as transferred design patterns can help us achieve the efficiency directly. Discussions with Conventional Architecture Search Methods. Popular techniques for network architecture search include reinforcement learning and evolutionary algorithms (; BID16 . In each iteration, a randomly initialized network is trained and evaluated to guide the search, and the search process usually requires thousands of iterations to find the goal architecture. In contrast, using network pruning as architecture search only requires a one-pass training, however the search space is restricted to the set of all "subnetworks" inside a large network, whereas traditional methods can search for more variations, e.g., activation functions or different layer orders. Figure 6: Pruned architectures obtained by different approaches, all trained from scratch, averaged over 5 runs. "Guided Pruning/Sparsification" means using the average sparsity patterns in each layer stage to design the network; "Transferred Guided Pruning/Sparsification" means using the sparsity patterns obtained by a pruned VGG-16 on CIFAR-10, to design the network for VGG-19 on CIFAR-100. Following the design guidelines provided by the pruned architectures, we achieve better parameter efficiency, even when the guidelines are transferred from another dataset and model. uses a similar pruning technique to Network Slimming BID4 to automate the design of network architectures; BID1 prune channels using reinforcement learning and automatically compresses the architecture. On the other hand, in the network architecture search literature, sharing/inheriting trained parameters (; b) during searching has become a popular approach for reducing the training budgets, but once the target architecture is found, it is still trained from scratch to maximize the accuracy. Our encourage more careful and fair baseline evaluations of structured pruning methods. In addition to high accuracy, training predefined target models from scratch has the following benefits over conventional network pruning procedures: a) since the model is smaller, we can train the model using less GPU memory and possibly faster than training the original large model; b) there is no need to implement the pruning criterion and procedure, which sometimes requires fine-tuning layer by layer and/or needs to be customized for different network architectures BID14 BID4; c) we avoid tuning additional hyper-parameters involved in the pruning procedure. Our do support the viewpoint that automatic structured pruning finds efficient architectures in some cases. However, if the accuracy of pruning and fine-tuning is achievable by training the pruned model from scratch, it is also important to evaluate the pruned architectures against uniformly pruned baselines (both training from scratch), to demonstrate the method's value in identifying efficient architectures. If the uniformly pruned models are not worse, one could also skip the pipeline and train them from scratch. Even if pruning and fine-tuning fails to outperform the mentioned baselines in terms of accuracy, there are still some cases where using this conventional wisdom can be much faster than training from scratch: a) when a pre-trained large model is already given and little or no training budget is available; we also note that pre-trained models can only be used when the method does not require modifications to the large model training process; b) there is a need to obtain multiple models of different sizes, or one does not know what the desirable size is, in which situations one can train a large model and then prune it by different ratios. The Lottery Ticket Hypothesis conjectures that inside the large network, a sub-network together with their initialization makes the training particularly effective, and together they are termed the "winning ticket". In this hypothesis, the original initialization of the sub-network (before large model training) is needed for it to achieve competitive performance when trained in isolation. Their experiments show that training the sub-network with randomly re-initialized weights performs worse than training it with the original initialization inside the large network. In contrast, our work does not require reuse of the original initialization of the pruned model, and shows that random initialization is enough for the pruned model to achieve competitive performance. The seem to be contradictory, but there are several important differences in the evaluation settings: a) Our main is drawn on structured pruning methods, despite for smallscale problems (CIFAR) it also holds on unstructured pruning; only evaluates on unstructured pruning. b) Our evaluated network architectures are all relatively large modern models used in the original pruning methods, while most of the experiments in use small shallow networks (< 6 layers). c) We use momentum SGD with a large initial learning rate (0.1), which is widely used in prior image classification (; BID4 and pruning works BID14 BID4 b; ; a; BID6) to achieve high accuracy, and is the de facto default optimization setting on CIFAR and ImageNet; while with two initial learning rates 0.1 and 0.01, on CIFAR-10 dataset. Each point is averaged over 5 runs. Using the winning ticket as initialization only brings improvement when the learning rate is small (0.01), however such small learning rate leads to a lower accuracy than the widely used large learning rate (0.1).In this section, we show that the difference in learning rate is what causes the seemingly contradicting behaviors between our work and , in the case of unstructured pruning on CIFAR. For structured pruning, when using both large and small learning rates, the winning ticket does not outperform random initialization. We test the Lottery Ticket Hypothesis by comparing the models trained with original initialization ("winning ticket") and that trained from randomly re-initialized weights. We experiment with two choices of initial learning rate (0.1 and 0.01) with a stepwise decay schedule, using momentum SGD. 0.1 is used in our previous experiments and most prior works on CIFAR and ImageNet. , we investigate both iterative pruning (prune 20% in each iteration) and one-shot pruning for unstructured pruning. We show our for unstructured pruning in FIG3 and TAB13, and L 1 -norm based filter pruning BID14 in Table 9. with two initial learning rates: 0.1 and 0.01. The same are visualized in FIG3. Using the winning ticket as initialization only brings improvement when the learning rate is small (0.01), however such small learning rate leads to a lower accuracy than the widely used large learning rate (0.1).From FIG3 and TAB13, we see that for unstructured pruning, using the original initialization as in only provides advantage over random initialization with small initial learning rate 0.01. For structured pruning as BID14, it can be seen from Table 9 that using the original initialization is only on par with random initialization for both large and small initial learning rates. In both cases, we can see that the small learning rate gives lower accuracy than the widely-used large learning rate. To summarize, in our evaluated settings, the winning ticket only brings improvement in the case of unstructured pruning, with small initial learning rate, but this small learning rate yields inferior accuracy compared with the widely-used large learning rate. Note that also report in their Section 5, that the winning ticket cannot be found on ResNet-18/VGG using the large learning rate. The reason why the original initialization is helpful when the learning rate is small, might be the weights of the final trained model are not too far from the original initialization due to the small parameter updating stepsize. Table 9: Experiments on the Lottery Ticket Hypothesis on a structured pruning method (L1-norm based filter pruning BID14) with two initial learning rates: 0.1 and 0.01. In both cases, using winning tickets does not bring improvement on accuracy. Soft Filter Pruning (SFP) (a) prunes filters every epoch during training but also keeps updating the pruned filters, i.e., the pruned weights have the chance to be recovered. In the original paper, SFP can either run upon a random initialized model or a pretrained model. It falls into the category of predefined methods (Figure 2). TAB16 shows our without using pretrained models and TAB17 shows the with a pretrained model. We use authors' code (b) for obtaining the . It can be seen that Scratch-E outperforms pruned models for most of the time and Scratch-B outperforms pruned models in nearly all cases. Therefore, our also holds on this method. (a) using pretrained models. We have shown that for structured pruning the small pruned model can be trained from scratch to match the accuracy of the fine-tuned model in classification tasks. To see whether this phenomenon would also hold for transfer learning to other vision tasks, we evaluate the L 1 -norm based filter pruning method BID14 on the PASCAL VOC object detection task, using Faster-RCNN . Object detection frameworks usually require transferring model weights pre-trained on ImageNet classification, and one can perform pruning either before or after the weight transfer. More specifically, the former could be described as "train on classification, prune on classification, fine-tune on classification, transfer to detection", while the latter is "train on classification, transfer to detection, prune on detection, fine-tune on detection". We call these two approaches Prune-C (classification) and Prune-D (detection) respectively, and report the in TAB2. With a slight abuse of notation, here Scratch-E/B denotes "train the small model on classification, transfer to detection", and is different from the setup of detection without ImageNet pre-training as in. TAB2: Results (mAP) for pruning on detection task. The pruned models are chosen from BID14. Prune-C refers to pruning on classifcation pre-trained weights, Prune-D refers to pruning after the weights are transferred to detection task. Scratch-E/B means pre-training the pruned model from scratch on classification and transfer to detection. For this experiment, we adopt the code and default hyper-parameters from , and use PASCAL VOC 07 trainval/test set as our training/test set. For backbone networks, we evaluate ResNet-34-A and ResNet-34-B from the L 1 -norm based filter pruning BID14, which are pruned from ResNet-34. TAB2 shows our , and we can see that the model trained from scratch can surpass the performance of fine-tuned models under the transfer setting. Another interesting observation from TAB2 is that Prune-C is able to outperform Prune-D, which is surprising since if our goal task is detection, directly pruning away weights that are considered unimportant for detection should presumably be better than pruning on the pre-trained classification models. We hypothesize that this might be because pruning early in the classification stage makes the final model less prone to being trapped in a bad local minimum caused by inheriting weights from the large model. This is in line with our observation that Scratch-E/B, which trains the small models from scratch starting even earlier at the classification stage, can achieve further performance improvement. It would be interesting to see whether our observation still holds if the model is very aggressively pruned, since they might not have enough capacity to be trained from scratch and achieve decent accuracy. Here we provide using Network Slimming BID4 and L 1 -norm based filter pruning BID14. From TAB3, it can be seen that when the prune ratio is large, training from scratch is better than fine-tuned models by an even larger margin, and in this case fine-tuned models are significantly worse than the unpruned models. Note that in TAB2, the VGG-Tiny model we evaluated for ThiNet is also a very aggressively pruned model (FLOPs reduced by 15× and parameters reduced by 100×). BID4 when the models are aggressively pruned. " Prune ratio" stands for total percentage of channels that are pruned in the whole network. Larger ratios are used than the original paper of BID4 Generally, pruning algorithms use fewer epochs for fine-tuning than training the large model BID14 b; ). For example, L 1 -norm based filter pruning BID14 uses 164 epochs for training on CIFAR-10 datasets, and only fine-tunes the pruned networks for 40 epochs. This is due to that mostly small learning rates are used for fine-tuning to better preserve the weights from the large model. Here we experiment with fine-tuning for more epochs (e.g., for the same number of epochs as Scratch-E) and show it does not bring noticeable performance improvement. Table 16: "Fine-tune-40" stands for fine-tuning 40 epochs and so on. Scratch-E models are trained for 160 epochs. We observe that fine-tuning for more epochs does not help improve the accuracy much, and models trained from scratch can still perform on par with fine-tuned models. We use L 1 -norm filter pruning BID14 for this experiment. Table 16 shows our with different number of epochs for fine-tuning. It can be seen that fine-tuning for more epochs gives negligible accuracy increase and sometimes small decrease, and Scratch-E models are still on par with models fine-tuned for enough epochs. In our experiments, we use the standard training schedule for both CIFAR (160 epochs) and ImageNet (90 epochs). Here we show that our observation still holds after we extend the standard training schedule. We use L 1 -norm based filter pruning BID14 for this experiment. TAB8 shows our when we extend the standard training schedule of CIFAR from 160 to 300 epochs. We observe that scratch trained models still perform at least on par with fine-tuned models. BID14 when the training schedule of the large model is extended from 160 to 300 epochs. Figure 8: Weight distribution of convolutional layers for different pruning methods. We use VGG-16 and CIFAR-10 for this visualization. We compare the weight distribution of unpruned models, fine-tuned models and scratch-trained models. Top: Results for Network Slimming BID4. Bottom: Results for unstructured pruning . Accompanying the discussion in subsection 4.3, we show the weight distribution of unpruned large models, fine-tuned pruned models and scratch-trained pruned models, for two pruning methods: (structured) Network Slimming BID4 and unstructured pruning . We choose VGG-16 and CIFAR-10 for visualization and compare the weight distribution of unpruned models, fine-tuned models and scratch-trained models. For Network Slimming, the prune ratio is 50%. For unstructured pruning, the prune ratio is 80%. Figure 8 shows our . We can see that the weight distribution of fine-tuned models and scratch-trained pruned models are different from the unpruned large models -the weights that are close to zero are much fewer. This seems to imply that there are less redundant structures in the found pruned architecture, and support the view of architecture learning for automatic pruning methods. For unstructured pruning, the fine-tuned model also has significantly different weight distribution from the scratch-trained model -it has nearly no close-to-zero values. This might be a potential reason why sometimes models trained from scratch cannot achieve the accuracy of the fine-tuned models, as shown in Table 6. In this section we provide the additional on sparsity patterns for the pruned models, accompanying the discussions of "More Analysis" in Section 5. TAB13: Sparsity patterns of PreResNet-164 pruned on CIFAR-10 by Network Slimming shown in FIG1 (left) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of channels to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages). Table 19: Average sparsity patterns of 3×3 kernels of PreResNet-110 pruned on CIFAR-100 by unstructured pruning shown in FIG1 (middle) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of weights to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages). TAB2: Average sparsity patterns of 3×3 kernels of DenseNet-40 pruned on CIFAR-100 by unstructured pruning shown in FIG1 (right) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of weights to be kept. We can observe that for a certain prune ratio, the sparsity patterns are close to uniform (across stages). TAB2: Sparsity patterns of VGG-16 pruned on CIFAR-10 by Network Slimming shown in FIG0 (left) under different prune ratio. The top row denotes the total prune ratio. The values denote the ratio of channels to be kept. For each prune ratio, the latter stages tend to have more redundancy than earlier stages.

Represent the short summary for finding the corresponding scientific article (long)

In structured network pruning, fine-tuning a pruned model only gives comparable performance with training it from scratch.

Represent the TLDR for finding the corresponding scientific article

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.