Question: represent the text to find a very short summary (1-2 sentences max)

Question: {\em Saliency methods} attempt to explain a deep net's decision by assigning a {\em score} to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input. Recently \citet{adebayosan} questioned the validity of many of these methods since they do not pass simple {\em sanity checks}, which test whether the scores shift/vanish when layers of the trained net are randomized, or when the net is retrained using random labels for inputs. % for the inputs. %Surprisingly, the tested methods did not pass these checks: the explanations were relatively unchanged. We propose a simple fix to existing saliency methods that helps them pass sanity checks, which we call {\em competition for pixels}. This involves computing saliency maps for all possible labels in the classification task, and using a simple competition among them to identify and remove less relevant pixels from the map. Some theoretical justification is provided for it and its performance is empirically demonstrated on several popular methods. Saliency methods attempt to explain a deep net's decision to humans by assigning a score to each feature/pixel in the input, often doing this credit-assignment via the gradient of the output with respect to input (from now on refered to as just "gradient"). Here we are interested in tasks involving multiclass classification, and for simplicity the exposition will assume the input is an image. Then a saliency method assigns scores to input pixels, which are presented as a heat map. (Extensions of these ideas to higher-level features of the net will not be discussed here.) While gradient alone is often too noisy, it as well as related notions are the basis of other more successful methods. In Gradient Input the pixel score is the product of the corresponding coordinate of gradient vector with the pixel value. Layer-wise Relevance Propagation (LRP) uses a back-propagation technique where every node in the deep net receives a share of the output which it distributes to nodes below it. This happens all the way to the input layer, whereby every pixel gets assigned a share of the output, which is its score. Another rule Deep-Lift does this in a different way and is related to Shapley values of cooperative game theory. is a state of the art method that performs an efficient approximation of the Shapley values. The perceived limitations of these methods in turn motivated a long list of new ones. provides a survey of existing methods, and brief discussion is presented in Section 2. The focus of the current paper is an evaluation of saliency methods called sanity checks in. This involves randomizing the model parameters or the data labels (see Section 2 for details). The authors show that maps produced using corrupted parameters and data are often difficult to visually distinguish from those produced using the original parameters and data. The authors concluded that "...widely deployed saliency methods are independent of both the data the model was trained on, and the model parameters." The current paper shows how to pass sanity checks via a simple modification to existing methods: Competition for pixels. Section 3 motivates this idea by pointing out a significant issue with previous methods: they produce saliency maps for a chosen output (label) node using gradient information only for that node while ignoring the gradient information from the other (non-chosen) outputs. To incorporate information from non-chosen labels/outputs in the multiclass setting we rely on an axiom called completeness satisfied by many saliency methods, according to which the sum of pixel scores in a map is equal to the value of the chosen node (see Section 3). Existing methods design saliency maps for all outputs and the map for each label satisfies completeness. One can then view the various scores assigned to a single pixel as its "votes" for different labels. The competition idea is roughly to zero out any pixel whose vote for the chosen label was lower than for another (nonchosen) label. Section 4 develops theory to explain why this modification helps pass sanity checks in the multi-class setting, and yet produces maps not too different from existing saliency maps. It also introduces a notion called approximate completeness and suggests that it is both a reasonable alternative to completeness in practice, and also allows our analysis of the competition idea to go through. We the present an new empirical finding that saliency methods that were not designed to satisfy completeness in practice seem to satisfy approximate completeness anyway. This may be relevant for future research in this area. Section 5 reports experiments applying the competition idea to three well-regarded methods, Gradient Input, LRP, and DASP, and shows that they produce sensible saliency maps while also passing the sanity checks. List of testbeds and methods is largely borrowed from , except for inclusion of DASP, which draws inspiration from cooperative game theory. and provide surveys of saliency methods. Brief descriptions of some methods used in our experiments appear in Appendix Section 7.1. Here we briefly discuss the issue most relevant to the current paper, which is the interplay between tests/evaluations of saliency methods and principled design of new methods. Controversies here often boil down to interpretations of the word "saliency," which method designers have sought to codify via axioms. (Completeness is a simple example.) Evaluations then suggest that the axioms fail to ensure other desirable properties. To give an example, does the map change significantly if we blank out or modify a portion of the image that humans find insignificant, as depicted in? But in such cases it is sometimes unclear if the failure is due to the method alone, or traces to other unsolved issues in deep learning related to distribution shift, domain adaptation, adversarial examples, etc. ROAR evaluation greys out pixels/features found informative by a saliency method, and retrains the classifier on these modified inputs. The method is considered low quality if the accuracy drop is less than greying out the same fraction of randomly-chosen pixels. Many popular methods (including some tested here) do poorly on this, and a subset of ensemble methods outperform a random assignment of saliency scores. But clearly this evaluation is using a different definition of saliency than the axiomatic approaches. The axiomatic approaches seek to find a set of pixels that are sufficient to justify the output label. Since real-life images have high redundancy, multiple sets of pixels could be sufficient to justify the label. ROAR defines the goal as identifying every pixel that is potentially relevant, and then ensembling would clearly help. (To see this, imagine taking any dataset and duplicating each feature k times.) The current paper sidesteps some of these controversies by focusing solely on the sanity checks evaluation of , and their exact testbed and framework. At first sight the problems uncovered by the sanity checks appear quite serious. While the distribution-shift based evaluations were asking "Does the map shift too much upon changes to the input that humans find insignificant?," the sanity checks asks the simpler question "Does the map fail to shift when the model/labels are dramatically changed to become nonsensical?" So it is surprising that most methods already failed the following sanity checks (the authors suggest there could be others). The model parameter randomization test. According to the authors, this "compares the output of a saliency method on a trained model with the output of the saliency method on a randomly initialized untrained network of the same architecture." The saliency method fails if the maps are similar for trained models and randomized models. The randomization can be done in a cascading or layerwise fashion. The data randomization test "compares a given saliency method applied to a model trained on a labeled data set with the method applied to the same model architecture but trained on a copy of the data set in which we randomly permuted all labels." Clearly the model in the second case has learnt no useful relationship between the data and the labels and does not generalize. The saliency method fails if the maps are similar in the two cases on test data. To the best of our knowledge, no subsequent paper has designed reasonable saliency methods that pass the sanity checks. The idea of competition suggests itself naturally when one examines saliency maps produced using all possible labels/logits in a multiclass problem, rather than just the chosen label. Figure 1 shows some Gradient Input maps produced using AlexNet trained on MNIST, where the first layer was modified to accept one color channel instead of 3. Notice: Many pixels found irrelevant by humans receive heat (i.e., positive value) in all the maps, and many relevant pixels receive heat in more than one map. Our experiments showed similar phenomenon on more complicated datasets such as ImageNet. Figure 1 highlights an important point of , which is that many saliency maps pick up a lot of information about the input itself -e.g., presence of sharp edgesthat may be incidental to the final classification. Furthermore, these incidental features can survive during the various randomization checks, leading to failure in the sanity check. Thus it is a natural idea to create a saliency map by combining information from all labels, in the process filtering out or downgrading the importance of incidental features. Suppose the input is x and the net is solving a k-way classification. We assume a standard softmax output layer whose inputs are k logits, one per label. Let x be an input, y be its label and y denote the corresponding logit. The input image is of the number 3, which is clearly visible in all maps. Note how maps computed using logits/labels "7" and " 5" assign red color (resp., blue color) to pixels that would have been expected to be present (resp., absent) in those digits. The last figure shows the map produced using our CGI method. Usually prior methods do not combine information from maps from different labels, but we wish to do so to design a simple competition among labels for pixels. A priori it can be unclear how to compare scores across labels, since this could end up being an "apples vs oranges" comparison due to potentially different scaling. However, prior work has identified a property called completeness: this requires that the sum of the pixel scores is exactly the logit value. For example, Gradient Input satisfies completeness exactly for ReLU nets with zero bias 1. (Recall that the ReLU function with bias a is ReLU (z, a) max{z − a, 0}. ) Ensuring completeness for nonzero bias requires more computationally expensive operations. Enter competition. Completeness (and also approximate completeness, a property explained below in Section 4.1) allows an apples-to-apples comparison of saliency scores from different labels, and to view them as "votes" for a label. Now consider the case where y is the label predicted by the net for input x. Suppose pixel i has a positive score for label y and an even more positive score for label y 1. This pixel contributes positively to both logit values. But remember that since label y 1 was not predicted by the net as the label, the logit y1 is less than than logit y, so the contribution of pixel x i's "vote" to y1 is proportionately even higher than its contribution to y. This perhaps should make us realize that this pixel may be less relevant or even irrelevant to label y since it is effectively siding with label y 1 (recall Figure 1). We conclude that looking at saliency maps for non-chosen labels should allow us to fine-tune our estimate of the relevance of a pixel to the chosen label. Now we formalize the competition idea. Note that positive and negative pixel scores should in general be interpreted differently; the former should be viewed as supporting the chosen label, and the latter as opposing that label. Competitive version of saliency method Input: An image, I, an underlying saliency method, S u and some chosen label y Output: A saliency map S of same dimensions as the input, I For each pixel p i in the input 1. Calculate the saliency scores assigned by each label to pixel p i by S u 2. Zero out pixel p i in the saliency map S if either (a) p i has a positive saliency score for label y that is not maximal among its saliency scores for all labels (b) p i has a negative saliency score for label y that is not minimal among its saliency scores for all labels 4 WHY COMPETITION WORKS: SOME THEORY Figure 1 suggests that it is a good idea to zero out some pixels in existing saliency maps, as the existing saliency maps are reflecting information incidental to classification, such as edges. Here we develop a more principled understanding of why adding competition (a) is aggressive enough to zero out enough pixels to help pass sanity checks on randomized nets and (b) not too aggressive so as to retain a reasonable saliency map for properly trained nets. used linear models to explain why methods like Gradient Input fail their randomization tests. These tests turn the gradient into a random vector, and if ξ 1, ξ 2 are random vectors, then x ξ 1 and x ξ 2 are visually quite similar when x is an image. (See Figure 10 in their appendix.) Thus the saliency map retains a strong sense of x after the randomization test, even though the gradient is essentially random. Now it is immediately clear that with k-way competition among the labels, the saliency map would be expected to become almost blank in the randomization tests since each pixel is equally likely to receive its highest score from each label, and thus in each saliency map the pixel becomes zero with probability 1 − 1/k. Thus we would expect that adding competition enables the map to pass the sanity checks in multiclass settings, even for relatively small number of classes. Our experiments later show that the final map is indeed very sparse. But one cannot use the simple model to understand why competition yields a reasonable saliency map for properly trained nets. The reason being that the saliency map is not random and depends on the input -since completeness property requires that the sum of pixel saliencies is the label logit. We give a simple model for thinking about this dependence, which we hope will motivate further theory on the best way to aggregate information from different labels. We rely upon the completeness property, although Section 4.1 shows that a weaker property should suffice. Mental model for saliency maps. One should keep in mind that a saliency map is the outcome of a stochastic process: training a deep net, with stochasticity coming from dataset choice, random initialization, and SGD. Let y and µ x,y be random variables denoting respectively the logit value for label y on input x and the corresponding saliency map. For simplicity we assume the saliency method correlates with a certain ground truth saliency as follows. There exists a subset S x,y of pixels (e.g., the pixels of x that a human may find salient for label y) such that for each pixel in S x,y the corresponding coordinate of µ x,y is distributed as N (c, σ 2 1) and each pixel in S x,y is distributed as N (0, σ 2 2) where c = c(x, y) and σ 1, σ 2 are constants. This modeling assumption -"noisy signal" mixed with "white noise"-makes intuitive sense and has some experimental support; see Figure 2 where distribution of saliency scores does seem similar to mixture of two Gaussians (maybe three?), one centered at zero and other at some positive value. One could consider variants of our model, e.g., allowing means and variances to differ across pixels so the maps can include certain image portions (e.g., sharp edges) with very high probability. These don't qualitatively affect the phenomena discussed below. Now µ x,y · 1 is the sum of pixel saliencies (where 1 is the all-1's vector), which by completeness, is y. By linearity of expectation E[y] = c · |S x,y |. By measure concentration, with probability at least 1 − exp(−t 2 /4) we have Since the sets are fairly large (say, reasonable fraction of all pixels), this concentration should be good 2. After applying competition, the saliency map changes to µ x,y I x,y where I x,y is the vector with 1 in coordinates where label y wins the competition for pixels, and 0 elsewhere. Now we note various plausible conditions under which µ x,y ⊗ I x,y can be seen as a reasonable approximation to µ x,y. For simplicity we consider how well competition preserves completeness, though obviously other properties can be analysed. Theorem 1 (Informal meta-theorem) For many plausible conditions on the S x,y's there is a scale factor γ > 0 (depending on the conditions) such that the vector γµ x,y I x,y satisfies completeness up to a small additive error. Proof (sketch) One uses measure concentration. As an illustrative example suppose S x,y's are disjoint for different y's. For each coordinate in S x,y there is a certain probability p = p(c, σ 1, σ 2) that the coordinate is nonzero in I x,y. Thus the expected sum of coordinates in the map µ x,y I x,y is p|S x,y |, and so rescaling coordinates by c/p makes them satisfy completeness, up to an additive error given by measure concentration. Qualitatively similar are obtained when S x,y's are not disjoint but for S x,y ∩ ∪ y =y S x,y is small compared to S x,y. Remarks: Obviously, the additive error will have smaller impact for the larger logit values than for smaller ones. The above proof suggests that adding competition is akin to sampling a subset of the salient pixels. At least for image data, where pixel-level information has a lot of redundancy, the essence of the original map survives, as seen in the experiments later. While some methods satisfy completeness by design , others don't. Can our theory apply to the latter class of methods? While an exhaustive study was not performed, randomly sampling a few methods suggests that the saliency maps in practice satisfy approximate completeness anyway (theory is of course lacking), in sense of the following definition. Figure 3 depicts two saliency methods satisfying approximate completeness. Theorem 3 (Modified meta theorem) If a saliency method satisfies (α, β)-approximate completeness, then under various plausible conditions on the S x,y's similar to Theorem 1 there are constants (α, β) such that after applying competition the maps satisfy (α, β)-approximate completeness. We consider saliency methods Gradient Input, LRP, and DASP, and adding competition to them to get methods CGI, CLRP, and CDASP, respectively. CGI is tested using VGG-19 architecture trained on; LRP using VGG-16 architecture with Imagenet; DASP using a CNN model on MNIST. While these architectures are not the most powerful ones around, this is not a limitation for testing saliency methods, which are supposed to work for all architectures. Figure 4 shows underlying saliency maps versus maps with our modifications for VGG-19 on Imagenet. Applying competition does not visibly degrade the map. Some more examples (labeled "original") also appear in Figure 5. The goal of these experiments is to determine whether our modification may be applied to an underlying saliency method to pass the first sanity check in. We conduct cascading and layerwise randomization as described in from top to bottom. • The top figure in Figure 5 shows the of layerwise randomization on Gradient Input. (Figure 8 in the Appendix shows the full figure).The text underneath each image represents which layer of the model was randomized, with the leftmost label of'original' representing the original saliency map of the fully trained model. The top panel shows the saliency maps produced by CGI, and the bottom panel the maps produces by Gradient Input. We find that the Gradient Input method displays the bird no matter which layer is randomized, and that our method immediately stops revealing the structure of the bird in the saliency maps as soon as any layer is randomized. Figure 10 in the Appendix shows a similar but utilizing absolute value visualization. Notice that CGI's sensitivity to model parameters still holds. • The second figure in Figure 5 shows the for cascading randomization on Gradient Input. The rightmost column represents the original saliency map when all layer weights and biases are set to their fully trained values. The leftmost saliency map represents the map produced when only the softmax layer has been randomized. The image to the right of that when everything up to and including conv5 4 has been randomized, and so on. Again we find that CGI is much more sensitive to parameter randomization than Gradient Input. • The bottom figure in Figure 5 shows our for cascading randomization on LRP. We find that our competitive selection process (CLRP) benefits the LRP maps as well. The LRP maps show the structure of the bird even after multiple blocks of randomization, while our maps greatly reduce the prevalence of the bird structure in the images. • Figure 6 shows our for cascading randomization on , a state of the art method that satisfies approximate completeness. DASP still shows the structure of the digit even after randomization, while CDASP eliminates much of this structure. Figure 5: The first two figures depict layerwise and cascading randomization, respectively, for a VGG19 model trained on Imagenet for Gradient Input versus CGI. We find that in CGI, the saliency map is almost blank when any weights are reinitialized. By contrast, we find that the original Gradient Input method displays the structure of the bird, no matter which layer is randomized. The third figure depicts saliency map for cascading randomization on VGG -16 on Imagenet LRP versus CLRP. We notice that LRP shows the structure of the bird even after multiple blocks of randomization. CLRP eliminates much of the structure of the bird. We run experiments to determine whether our saliency method is sensitive to model training. We use a version of adjusted to accept one color channel instead of three On the middle image we find that using the original gradient times input method in an image where the original structure of the number 3 is still visible. On the right hand side image we find that our modification removes the structure of the original input image, as we would expect for a model that had been fitted on randomized data. and train on MNIST. We randomly permute the labels in the training data set and train the model to greater than 98 % accuracy and examine the saliency maps. Figure 7 shows our . On the left hand side is the original image. In the middle is the map produced by Gradient Input. We find that the input structure, the number 3, still shows through with the Gradient Input method. On the other hand, CGI removes the underlying structure of the number. Competition among labels is a simple modification to existing saliency methods that produces saliency maps by combining information from maps from all labels, instead of just the chosen label. Our modification keeps existing methods relevant for visual evaluation (as shown on three wellknown methods Gradient Input, LRP, and DASP) while allowing them to pass sanity checks of , which had called into question the validity of saliency methods. Possibly our modification even improves the quality of the map, by zero-ing out irrelevant features. We gave some theory in Section 4 to justify the competition idea for methods which satisfy approximate completeness. Many methods satisfy completeness by design, and experimentally we find other methods satisfy approximate completeness. We hope the simple analysis of Section 4-modeling the saliency map as "noisy signal" mixed with "white noise"-will inspire design of other new saliency maps. We leave open the question of what is the optimum way to design saliency maps by combining information from all labels 3. When pixel values are spatially correlated it is natural to involve that in designing the competition. This is left for future work. The sanity checks of randomize the net in a significant way, either by randomizing a layer or training on corrupted data. It is an interesting research problem to devise sanity checks that are less disruptive. also computes the gradient of the chosen class's logit. However, instead of evaluating this gradient at one fixed data point, integrated gradients consider the path integral of this value as the input varies from a baseline,x, to the actual input, x along a straight line. proposed an approach for propagating importance scores called Layerwise Relevance Propagation (LRP). LRP decomposes the output of the neural network into a sum of the relevances of coordinates of the input. Specifically, if a neural network computes a function f (x) they attempt to find relevance scores R p 4. Taylor decomposition As stated for special classes of piecewise linear functions that satisfy f (tx) = tf (x), including ReLU networks with no biases, one can always find a root point near the origin such that DeepLIFT explanation The DeepLIFT explanation calculates the importance of the input by comparing each neuron's activation to some'reference' activation. Each neuron is assigned an attribution that represents the amount of difference from the baseline that that neuron is responsible for. Reference activations are determined by propagating some reference input,x, through the neural network. Relationships between different methods. and showed that if modifications for numerical stability are not taken into account, the LRP rules are equivalent within a scaling factor to Gradient Input. showed that for ReLU networks (with zero baseline and no biases) the -LRP and DeepLIFT (Rescale) explanation methods are equivalent to the Gradient Input. , like DeepLIFT relies on the Shapley values, but designs an efficient way to approximately compute these values. Layerwise Randomization original Gradient Input conv1_1 conv1_2 conv2_1 conv2_2 conv3_1 conv3_2 conv3_3 conv3_4 conv4_1 conv4_2 conv4_3 conv4_4 conv5_1 conv5_2 conv5_3 conv5_4 fc1 fc2 softmax Figure 8: Saliency map for layer-wise randomization of the learned weights. Diverging visualization where we plot the positive importances in red and the negative importances in blue. We find that with CGI, the saliency map is almost blank when any layer is reinitialized. By contrast, we find that Gradient Input displays the structure of the bird, no matter which layer is randomized. Cascading randomization for CLRP versus LRP original LRP dense_1 fc2 fc1 block5_conv3 block5_conv2 block5_conv1 block4_conv3 block4_conv2 block4_conv1 block3_conv3 block3_conv2 block3_conv1 block2_conv2 block2_conv1 block1_conv2 block1_conv1 conv1_1 conv1_2 conv2_1 conv2_2 conv3_1 conv3_2 conv3_3 conv3_4 conv4_1 conv4_2 conv4_3 conv4_4 conv5_1 conv5_2 conv5_3 conv5_4 fc1 fc2 softmax Figure 10: Saliency map for layer-wise randomization of the learned weights. Absolute value visualization where we plot the absolute value of the saliency map. We find that using CGI, the saliency map is almost blank when any layer is reinitialized. By contrast, we find that Gradient Input displays the structure of the bird, no matter which layer is randomized.

Represent the short summary for finding the corresponding scientific article (long).

We devise a mechanism called competition among pixels that allows (approximately) complete saliency methods to pass the sanity checks.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR.

Question: Classification systems typically act in isolation, meaning they are required to implicitly memorize the characteristics of all candidate classes in order to classify. The cost of this is increased memory usage and poor sample efficiency. We propose a model which instead verifies using reference images during the classification process, reducing the burden of memorization. The model uses iterative non-differentiable queries in order to classify an image. We demonstrate that such a model is feasible to train and can match baseline accuracy while being more parameter efficient. However, we show that finding the correct balance between image recognition and verification is essential to pushing the model towards desired behavior, suggesting that a pipeline of recognition followed by verification is a more promising approach towards designing more powerful networks with simpler architectures. FIG3: Overview of our hybrid model in contrast with two opposing approaches. (a) Recognition network directly predicts the class given the input. (b) Verification network predicts binary output indicating the amount of similarity or likelihood that they belong in the same class. The verification network can be used to compare to all reference images from each class to produce the final class prediction. (c) Our approach, RVNN, queries for reference image from a particular class at each time step, and makes a class prediction at the last time step. BID9. Then to classify, the image is pair-wise compared with a support image from every class and 37 image with the maximum similarity score is chosen. Matching networks extends the verification-38 based approach by outputting a prediction based on a weighted-sum of similarity across classes. Additionally the work introduces an episodic-training regime which encourages the model to better 40 learn for the one-shot learning scenario. Prototypical Networks uses Euclidean Distance in embedding 41 space as a verification metric rather than a learned metric, while maintaining the same training regime 42 as Matching Networks to encourage different classes to have distant means in embedding space. One recent work outside of few-shot learning domain is the Retrieval-Augmented Convolutional Neural Networks (RaCNN), which combines CNN recognition network with a retrieval engine 45 for support images to help increase adversarial robustness. For all the above few shot learning approaches, verification with support images from all classes are required before a classification decision is made. Hence the classification decision is solely derived 48 from verifications. RaCNN is closer to our approach, which uses a hybrid between verification 49 and recognition. However, RaCNN simply retrieves the K closest support image neighbours in the 50 embedding space, whereas our model is required to form a hypothesis as to which class to compare 51 with. In cases in which there are a large number of classes, we expect our approach to excel. As well, this introduces a non-differentiable component in our model not present in previous work. Prior work has also looked at the concept of incorporating external knowledge into the decision 55 making process of neural networks, often with non-differentiable components. Buck order to produce a prediction C for the class. FIG2 illustrates the full process. The subsequent sections detail the implementations of the three components as well as training 84 considerations. The recurrent querying model f rnn was implemented using a Gated Recurrent Unit (GRU). We 105 also considered passing in additional information to f rnn, such as the query that was used in the 106 previous time steps. We implemented f q as sampling a class based on the categorical probability given by the softmax of 109 the logits from the f rnn. Therefore, f q can be written as: DISPLAYFORM0 DISPLAYFORM1 To sample S n+1, we can use the Gumbel-Max trick: DISPLAYFORM2 where g i...g k are i.i.d samples drawn from Gumbel distribution. However, the arg max operator 112 is not differentiable, so instead we explored two approaches during training. The first is to use the Gumbel-Softmax trick, also known as the Concrete estimator. We 114 relaxed the arg max operator to a differentiable softmax function with temperature parameter τ: DISPLAYFORM0 The τ parameter is annealed exponentially from τ = 1 to τ = 0.5 as the training iterations progresses. The second approach is to use simple Straight-Through estimator BID0. In the forward pass, we apply 117 the Gumbel-Max trick to take discrete query choices. Then on the backward pass, we set the derivative 118 of the query with respect to the softmax probabilities to be identity so that the out-going gradient 119 from the arg max operator is equal to the incoming gradient during backpropagation: DISPLAYFORM0 We highlight that our model differ from several existing networks in various aspects. The performance of the model is assessed by both reduced parameter usage and sample efficiency. Reduced parameter usage is measured relative to a baseline model, in this case the CNN architecture The performance of the model alone does not indicate whether our approach is functioning as intended. We also experiment with several small modifications to our architecture as well as a few hyper- parameters that are unique to our model. We list them here below. • Architectural Considerations -Query Memory (QM): The query from the past time step is passed to the RNN. the RNN learns to adapt its policy to varying levels of accuracy, and see whether a better can 184 be achieved when they are used in tandem. We also experiment with fixing query policy to assess 185 whether the RNN actually learns intelligent query behavior. From our overall performance metrics we observe that at both smaller and larger sizes of model, our usage. This is in agreement with our hypotheses that our model would be more parameter efficient. RNN's learned policy is compared against a random query policy and the optimal query policy (never 218 repeating a query). From the figure we see that the model is able to conduct a better than random 219 query policy, but is not able to achieve optimal performance. We also observed that performance 220 increases with a higher RNN size up to 200. This suggests that the model is in some part able to track 221 previously unsuccessful queries and remember if there was a match. However, its memory is not 222 perfect and it cannot achieve optimal performance. and a real (neural network) comparator. Of the query-based models, informed queries performed the best over random and no query models. appropriate pipeline for our model is to perform a recognition operation which is then followed by 241 verification, rather than perform them simultaneously. This new pipeline would also imply that our model may be less well suited for the one-shot learning 243 task than initially believed, as a reasonably-well trained recognition module is required as the first tasks of this form are pre-requisite if neural-network models are considered to be as such BID5.

represent the short summary for finding the corresponding scientific article (long)

Image classification via iteratively querying for reference image from a candidate class with a RNN and use CNN to compare to the input image

theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: To reduce memory footprint and run-time latency, techniques such as neural net-work pruning and binarization have been explored separately. However, it is un-clear how to combine the best of the two worlds to get extremely small and efficient models. In this paper, we, for the first time, define the filter-level pruning problem for binary neural networks, which cannot be solved by simply migrating existing structural pruning methods for full-precision models. A novel learning-based approach is proposed to prune filters in our main/subsidiary network frame-work, where the main network is responsible for learning representative features to optimize the prediction performance, and the subsidiary component works as a filter selector on the main network. To avoid gradient mismatch when training the subsidiary component, we propose a layer-wise and bottom-up scheme. We also provide the theoretical and experimental comparison between our learning-based and greedy rule-based methods. Finally, we empirically demonstrate the effectiveness of our approach applied on several binary models, including binarizedNIN, VGG-11, and ResNet-18, on various image classification datasets. For bi-nary ResNet-18 on ImageNet, we use 78.6% filters but can achieve slightly better test error 49.87% (50.02%-0.15%) than the original model Deep neural networks (DNN), especially deep convolution neural networks (DCNN), have made remarkable strides during the last decade. From the first ImageNet Challenge winner network, AlexNet, to the more recent state-of-the-art, ResNet, we observe that DNNs are growing substantially deeper and more complex. These modern deep neural networks have millions of weights, rendering them both memory intensive and computationally expensive. To reduce computational cost, the research into network acceleration and compression emerges as an active field. A family of popular compression methods are the DNN pruning algorithms, which are not only efficient in both memory and speed, but also enjoy relatively simple procedure and intuition. This line of research is motivated by the theoretical analysis and empirical discovery that redundancy does exist in both human brains and several deep models BID7 BID8. According to the objects to prune, we can categorize existing research according to the level of the object, such as connection (weights)-level pruning, unit/channel/filter-level pruning, and layer-level pruning BID28. Connection-level pruning is the most widely studied approach, which produces sparse networks whose weights are stored as sparse tensors. Although both the footprint memory and the I/O consumption are reduced BID12, Such methods are often not helpful towards the goal of computation acceleration unless specifically-designed hardware is leveraged. This is because the dimensions of the weight tensor remain unchanged, though many entries are zeroed-out. As a wellknown fact, the MAC operations on random structured sparse matrices are generally not too much faster than the dense ones of the same dimension. In contrast, structural pruning techniques BID28, such as unit/channel/filter-level pruning, are more hardware friendly, since they aim to produce tensors of reduced dimensions or having specific structures. Using these techniques, it is possible to achieve both computation acceleration and memory compression on general hardware and is common for deep learning frameworks. We consider the structural network pruning problem for a specific family of neural networks -binary neural networks. A binary neural network is a compressed network of a general deep neural network through the quantization strategy. Convolution operations in DCNN 1 inherently involve matrix multiplication and accumulation (MAC). MAC operations become much more energy efficient if we use low-precision (1 bit or more) fixed-point number to approximate weights and activation functions (i.e., to quantify neurons) BID3. To the extreme extent, the MAC operation can even be degenerated to Boolean operations, if both weights and activation are binarized. Such binary networks have been reported to achieve ∼58x computation saving and ∼32x memory saving in practice. However, the binarization operation often introduces noises into DNNs, thus the representation capacity of DNNs will be impacted significantly, especially if we also binarize the activation function. Consequently, binary neural networks inevitably require larger model size (more parameters) to compensate for the loss of representation capacity. Although Boolean operation in binary neural networks is already quite cheap, even smaller models are still highly desired for low-power embedded systems, like smart-phones and wearable devices in virtual reality applications. Even though quantization (e.g., binarization) has significantly reduced the redundancy of each weight/neuron representation, our experiment shows that there is still heavy redundancy in binary neural networks, in terms of network topology. In fact, quantization and pruning are orthogonal strategies to compress neural networks: Quantization reduces the precision of parameters such as weights and activations, while pruning trims the connections in neural networks so as to attain the tightest network topology. However, previous studies on network pruning are all designed for full-precision models and cannot be directly applied for binary neural networks whose both weights and activations are 1-bit numbers. For example, it no longer makes any sense to prune filters by comparing the magnitude or L 1 norm of binary weights, and it is nonsensical to minimize the distance between two binary output tensors. We, for the first time, define the problem of simplifying binary neural networks and try to learn extremely efficient deep learning models by combining pruning and quantization strategies. Our experimental demonstrate that filters in binary neural networks are redundant and learning-based pruning filter selection is constantly better than those existing rule-based greedy pruning criteria (like by weight magnitude or L 1 norm).We propose a learning-based method to simplify binary neural network with a main-subsidiary framework, where the main network is responsible for learning representative features to optimize the prediction performance, whereas the subsidiary component works as a filter selector on the main network to optimize the efficiency. The contributions of this paper are summarized as follows:• We propose a learning-based structural pruning method for binary neural networks to significantly reduce the number of filters/channels but still preserve the prediction performance on large-scale problems like the ImageNet Challenge.• We show that our non-greedy learning-based method is superior to the classical rule-based methods in selecting which objects to prune. We design a main-subsidiary framework to iteratively learn and prune feature maps. Limitations of the rule-based methods and advantages of the learning-based methods are demonstrated by theoretical and experimental . In addition, we also provide a mathematical analysis for L 1 -norm based methods.• To avoid gradient mismatch of the subsidiary component, we train this network in a layerwise and bottom-up scheme. Experimentally, the iterative training scheme helps the main network to adopt the pruning of previous layers and find a better local optimal point.2 RELATED WORK 2.1 PRUNING Deep Neural Network pruning has been explored in many different ways for a long time. BID13 proposed Optimal Brain Surgeon (OBS) to measure the weight importance using the second-order derivative information of loss function by Taylor expansion. BID9 further adapts OBS for deep neural networks and has reduced the retraining time. Deep Compression BID12 prunes connections based on weight magnitude and achieved great compression ratio. The idea of dynamic masks BID10 is also used for pruning. Other approaches used Bayesian methods and exploited the diversity of neurons to remove weights BID23 BID22. However, these methods focus on pruning independent connection without considering group information. Even though they harvest sparse connections, it is still hard to attain the desired speedup on hardware. To address the issues in connection-level pruning, researchers proposed to increase the groupsparsity by applying sparse constraints to the channels, filters, and even layers BID28 BID0 BID25 BID1. used LASSO constraints and reconstruction loss to guide network channel selection. introduced L 1 -Norm rank to prune filters, which reduces redundancy and preserves the relatively important filters using a greedy policy. BID21 leverages a scaling factor from batch normalization to prune channels. To encourage the scaling factor to be sparse, a regularization term is added to the loss function. On one hand, methods mentioned above are all designed for full-precision models and cannot be trivially transferred to binary networks. For example, to avoid introducing any non-Boolean operations, batch normalization in binary neural networks (like XNOR-Net) typically doesn't have scaling (γ) and shifting (β) parameters BID3. Since all weights and activation only have two possible values {1, −1}, it is also invalid to apply classical tricks such as ranking filters by their L 1 -Norms, adding a LASSO constraint, or minimizing the reconstruction error between two binary vectors. On the other hand, greedy policies that ignore the correlations between filters cannot preserve all important filters. Recent work shows that full precision computation is not necessary for the training and inference of DNNs BID11. Weights quantization is thus widely investigated, e.g., to explore 16-bit BID11 and 8-bit fixed-point numbers. To achieve higher compression and acceleration ratio, extremely low-bit models like binary weights BID5 BID18 and ternary weights BID30 BID29 BID27 have been studied, which can remove all the multiplication operations during computation. Weight quantization has relatively milder gradient mismatch issue as analyzed in Section 3.1.2, and lots of methods can achieve comparable accuracy with full-precision counterparts on even large-scale tasks. However, the ultimate goal for quantization networks is to replace all MAC operations by Boolean operations, which naturally desires that both activation and weights are quantized, even binarized. The activation function of quantized network has the form of a step function, which is discontinuous and non-differentiable. Gradient cannot flow through a quantized activation function during backpropagation. The straight-through estimator (STE) is widely adopted to circumvents this problem, approximating the gradient of step function as 1 in a certain range BID16 BID2. BID4 proposed the Half-wave Gaussian Quantization (HWGQ) to further reduce the mismatch between the forward quantized activation function and the backward ReLU BID24. Binary Neural Networks (BNN) proposed in BID6 and BID3 use only 1 bit for both activation functions and weights, which ends up with an extremely small and faster network. BNNs inherit the drawback of acceleration via quantization strategy and their accuracy also need to be further improved.. Because both weights and activations are binary, we remove the subscripts of F b and W b for clarity. The goal of pruning is to remove certain filters W i n,:,:,:, n ∈ Ω, where Ω is the indices of pruned filters. If a filter is removed, the corresponding output feature map of this layer (which is also the input feature map of next layer) will be removed, too. Furthermore, the input channels of all filters in the next layer would become unnecessary. If all filters in one layer can be removed, the filter-level pruning will upgrade to layerlevel pruning naturally. The goal of our method is to remove as many filters as possible for binary neural networks which are already compact and have inferior numerical properties, thus this task is more challenging compared with pruning a full-precision model. We borrow the ideas from binary network optimization to simplify binary networks. While it sounds tautological, note that the optimization techniques were originally invented to solve the quantization problem, but we will show that it can be crafted to solve the pruning problem for binary networks. A new binary network, called subsidiary component, acts as learnable masks to screen out redundant features in the main network, which is the network to complete classification tasks. Each update of the subsidiary component can be viewed as the exploration in the mask search space. We try to find a (local) optimal mask in that space with the help of the subsidiary component. The process of training subsidiary and main networks is as follows: For layer i, the weights of subsidiary component M i ∈ R Ni+1×Ni×Ki+1×Ki+1 are initialized by the uniform distribution: DISPLAYFORM0 In practice, σ is chosen to be less than 10 −5. To achieve the goal of pruning filters, all elements whose first index is the same share the same value. DISPLAYFORM1 is an output tensor from the subsidiary component. In the first stage, we use the Iden(·) function (identity transforma- DISPLAYFORM2 We apply the filter mask O i to screen main network's weights W i, DISPLAYFORM3, where ⊗ is element-wise product. Ŵ i denotes the weights of the main network after transformation, which is used to be convolved with the input feature maps, F i, to produce the output feature maps F i+1 . Then, weights of the main network, W j, j ∈ [1, I], are set to be trainable while weights of the subsidiary component, M j, j ∈ [1, I], are fixed. Because subsidiary weights are fixed and initialized to be near-zero, it will not function in the Feature Learning stage, thuŝ DISPLAYFORM4 The whole main binary neural network will be trained from scratch. Training Subsidiary Component within a Single Layer i: After training the whole main network from scratch, we use a binary operator to select features in a layer-wise manner. In opposite to the previous Feature Learning stage, the weights of all layers W j, j ∈ [1, I] of the main network and the weights except layer i of the subsidiary component M j, j ∈ [1, I]/[i] are set to be fixed, while the subsidiary component's weights at the current layer M i are trainable when selecting features for Layer i. The transformation function for the filter mask O i is changed from Iden(·) to Bin(·) (sign transformation + linear affine), DISPLAYFORM0 2 By doing this, we project the float-point M i to binarized numbers ranging from 0 to 1. Elements in O i which are equal to 0 indicate that the corresponding filters are removed and the elements of value 1 imply to keep this filter. Since Bin(·) is not differentiable, we use the following function instead of the sign function in back propagation when training the subsidiary component M i BID16 BID2, DISPLAYFORM1 Apart from the transformation, we also need to add regularization terms to prevent all O i from degenerating to zero, which is a trivial solution. So the loss function of training Layer i in the subsidiary component is, arg min DISPLAYFORM2 where L cross entropy is the loss on data and L distill is the distillation loss defined in.Finally, we fix the layers M j, j ∈ [1, I] in the subsidiary component and layers before i in the main network (i.e., W j, j ∈ [1, i − 1]), and retrain the main layers after Layer i (i.e., W j, j ∈ [i, I]).Bottom-up Layer-wise Training for Multiple Layers: We showed how to train a layer in the subsidiary component above. To alleviate the gradient mismatch and keep away from the trivial solution during Features Selection, next, we propose a layer-wise and bottom-up training scheme for the subsidiary component: Layers closer to the input in the subsidiary component will be trained with priority. As Layer i is under training, all previous layers (which should have already been trained) will be fixed and subsequent layers will constantly be the initial near-zero value during training. There are three advantages of this training scheme. First, as in, we use STE as in BID16 BID2 to approximate the gradient of the sign function. By chain rule, for each activation node j in Layer i, we would like to compute an "error term" δ i j = ∂L ∂a i j which measures how much that node is responsible for any errors in the output. For binary neural networks, activation is also binarized by a sign function which need STE for back-propagation. The "Error term" for binary neural networks is given by, DISPLAYFORM3 where and can be obtained by the chain rule, and and are estimated from STE, which will introduce gradient mismatch into back-propagation. We refer as weight gradient mismatch issue and as activation gradient mismatch issue. They are two open problems in the optimization of binary neural networks, both caused by the quantization transform functions like Sign(·). Starting from bottom layers, we can train and fix layers who are harder to train as early as possible for the subsidiary component. In addition, because of the retraining part in Features Selection, bottom-up training scheme allows bottom layers to be fixed earlier, as well. In practice, this scheme in more stable training curves and can find a better local optimal point. Second, the bottom-up layer-wise training scheme helps the main network to better accommodate the feature distribution shift caused by the pruning of previous layers. As mentioned before, the main difference in the motivation between our pruning method and rule-based methods is that we have more learnable parameters to fit the data by focusing on the final network output. With the bottom-up and layer-wise scheme, even if the output of Layer i changes, subsequent layers in the main network can accommodate this change by modifying their features. Lastly and most importantly, we achieve higher pruning ratio by this scheme. According to our experiments, a straight-forward global training scheme leads to limited pruning ratio. Some layers are pruned excessively and hence damaged the accuracy, while some layers are barely pruned, which hurts the pruning ratio. The layer-wise scheme would enforce all layer to be out of the comfort zone and allow balancing between accuracy and pruning ratio. The pipeline of our method is as follows:1. Initialize weights of subsidiary component M j, j ∈ [1, I] with near-zero σ's.2. Set M j, j ∈ [1, I] to be fixed, and train the whole main network from scratch.3. Train starting from the first binary kernel. Each layer is the same as in the algorithm shown below:• Change the activation function for M i from Iden(·) to Bin(·). And all other parameters apart from M i are fixed. Train subsidiary component according to.• Fix the subsidiary layers M j, j ∈ [1, I] and main layers before i-th layer W j, j ∈ [1, i − 1], and retrain main layers after i-th layer W j, j ∈ [i, I]. Though pruning network filters is not an explicit transfer learning task, the aim is to guide the thin network to learn more similar output distributions with the original network. The model is supposed to learn a soft distribution but not a hard one as proposed in previous traditional classifier networks. Hence, we add a distillation loss to guide the training subsidiary component to be more stable, as shown in FIG2. DISPLAYFORM0 We set p to be the original binary neural network distribution. Because the distribution is fixed, the H(p) is a constant and can be removed from L distill. It means that the distillation loss can be written as DISPLAYFORM1 ) where z i and t i represent the final output of the pruned and original networks before the softmax layer. T is a temperature parameter for the distillation loss defined in BID17. We set T as 1 in practice. M is the number of classes. Previous methods use rules to rank the importance of each filter and then remove the top k least important filters. The rules can be weight magnitude, e.g., measured by the L 1 norm, or some other well-designed criteria. Studies in this line share the same motivation that individual filters have their own importance indication, and filters with less importance can be removed relatively safely. This assumption ignores interactions among filters. As mentioned before, rule-based pruning algorithms use a greedy way to prune filters, i.e., they assume that individual filters behave independently and their own importance (or function) for representation learning. We give a theoretical analysis in Section 3.3 about this point. In fact, pruning filters independently may cause problems when filter are strongly correlated. For example, if two filters have learned the same features (or concepts), these two filters may be pruned out together by rule-based methods, because their rankings are very close. Clearly, pruning one of them is a better choice. However, almost all these criteria are based on value statistics and are completely unsuitable for the binary scenario with only two discrete values. One possible pruning method is to exhaustively search the optimal pruning set, but this is NP-Hard and prohibitive for modern DNNs that have thousands of filters. Our method uses the subsidiary component to "search" the optimal solution. Our soft "search" strategy is gradient-based and batch-based compared to exhaustive search, and it is much more efficient. If our main network is full-precision, the L 1 -Norm based pruning technique would be strongly relevant to our method, except that we target at optimizing the final output of the network, whereas the L 1 -Norm based method greedily controls the perturbation of the feature map in the next layer. Suppose that W = [w 1 ; . . . ; w n] is the original filter blocked by rows, W = [w 1 ; . . . ; w n] is the pruned filter, and x is the input feature map. Let ∆w i ≡ w i − w i. Then, the L 1 -Norm approach minimizes the upper bound of the following problem: max x ∞ <τ W x − W x. To see this, note DISPLAYFORM0 To minimize i ∆w 1 by zeroing-out a single row w i, obviously, the solution is to select the one with the smallest L 1 -Norm. However, note that this strategy cannot be trivially applied for binary networks, because the L 1 -Norm for any filter that is a {−1, +1} tensor of the same shape is always identical. Previous work (He et al.) uses the LASSO regression to minimize the reconstruction error of each layer: DISPLAYFORM1 β 0 ≤ C. Solving this L 0 minimization problem is NPhard, so the L 0 regularization is usually relaxed to L 1. In the binary/quantization scenario, activations only have two/several values and the least reconstruction error is not applicable. Instead of minimizing the reconstruction error of a layer, our method pays attention on the final network output with the help of the learnable subsidiary component. We directly optimize the discrete variables of masks (a.k.a subsidiary component) without the relaxation. To evaluate our method, we conduct several pruning experiments for VGG-11, Net-InNet (NIN), and ResNet-18 on CIFAR-10 and ImageNet. Since our goal is to simplify binary neural networks, whose activation and weights are both 1-bit, all main models and training settings in our experiments inherit from XNOR-Net BID26 ). Since we are, to the best of our knowledge, the first work to define filter-level pruning for binary neural networks, we proposed a rule-based method by ourselves as the baseline. Instead of ranking filters according to the L 1 -Norm, we use the magnitude of each filter's scaling factor (MSF) as our pruning criterion. Inspired by, we test both the "prune once and retrain" scheme 2 and the "prune and retrain iteratively" scheme 3. Figure 3: Learning curve for subsidiary component. We train the subsidiary component with different learning rate. These curves are smoothed for the directly seeing the trend of the learning Subsidiary Component. All the dotted lines represent the learning curve of the large learning rate 10 −3, the normal lines represent the learning curves of the small learning rate 10 −4.As pointed out in BID26 we set weights of the first layer and last layer as full-precision, which also means that we only do pruning for the intermediate binary layers. We measure effectiveness of pruning methods in terms of PFR, the ratio of the number of pruned filters to original filter number, and error rate before and after retraining. For error ratio, smaller is better. For PFR, larger is better. For CIFAR-10, when training the main network, learning rate starts from 10 −4, and learningrate-decay is equal to 0.1 for every 20 epochs. Learning rate is fixed with 10 −3 when training the subsidiary component. For ImageNet, we set a constant learning rate of 10 −3 for the subsidiary component and main work. For fair comparison, we control PFR for each layer of these methods to be the same to observe the final Retrain-Error. In FIG3, MSF-Layerwise refers to the "prune once and retrain" scheme, and the MSF-Cascade refers the "prune and retrain iteratively" scheme. The first three figures of experiments were done on the CIFAR-10 dataset. The last figure refers to on Imagenet. 4.1 NIN AND VGG-11 ON CIFAR-10 NIN is a fully convolutional network, using two 1 × 1 convolution layers instead of fully connected layer, and has quite compact architecture. VGG-11 is a high-capacity network for classification. VGG-11 on CIFAR-10 consists of 8 convolutional layers(including 7 binary layers) and 1 fully connected layers. Batch normalization is used between every binary convolution and activation layer, which makes the training process more stable and converge with high performance. For both MSF-Layerwise and MSF-Cascade, with the same PCR, the performance is worse than us. With 30% ∼ 40% of pruning filter ratio, the pruned network error rate only increased 1% ∼ 2%. An interesting phenomenon is observed when training subsidiary components for different models. We try different learning rates in our experiments and observe it impacts final convergent point a lot as shown in Figure 3. The relatively smaller learning rate (10 −4) will converge with lower accuracy and higher pruning number; however, the larger learning rate (10 −3) leads to the opposite . One possible explanation is that the solution space of the high-dimensional manifold for binary neural networks is more discrete compared to full-precision networks, so it is difficult for a subsidiary component to jump out of a locally optimal point to a better one. Moreover, in the binary scenario, larger learning rate will increase the frequency of value changing for weights. Our motivation is to use a learnable subsidiary components to approximate exhaustive search, so using a larger learning rate will enable the subsidiary component to "search" more aggressively. A large learning rate may be unsuitable for normal binary neural networks like the main network in this paper, but it is preferred by the subsidiary component. As mentioned in section 3.1.1, we use the uniform distribution to initialize the mask. According to the expectation of the uniform distribution, E(SP) = 0.5, where SP is the ratio of the number of positive elements in subsidiary weights to size of weights. However, since we use Sign(·), different SP may impact the a lot. We conduct six experiments on different models across different layers and show that initialization with 0.4, 0.6, 1.0 SP will all converge to the same state. However, when SP is 0.2, final performance will be very poor. A possible reason is that the number of filters thrown out by the initialization is too large, and due to the existence of the regularization term, the network's self-adjustment ability is limited and cannot converge to a good state. Hence we recommend the SP to be intialized to greater than 0.4. Compared with NIN and VGG-11, ResNet has identity connections within residual block and much more layers. As the depth of network increases, the capacity of network also increases, which then leads to more redundancy. From experimental , we find that when the identification mapping network has a downsampling layer, the overall sensitivity of the residual block will increase. Overall for ResNet on CIFAR-10 is shown in table, and statistics for each layer can be found in Appendix. We further verify our method with ResNet-18 on ImageNet. α can be set from 10 −7 to 10 −9 depending on the expected PFR, the accuracy and pruning ratio are balanced before retraining. After 20 epoches retraining for each layer, the final PFR is 21.4%, with the retrained error has decreased from 50.02% to 49.87%. Using STE, weights gradient mismatch will be introduced here. Using STE, activation gradient mismatch will be introduced here. Figure 5: Gradient flow of binary neural networks during back-propagation. Rectangles represent the weight tensor and ellipses represent functional operation. In this paper, we use binary operation as a special quantization function. MAC is short for multiplication and accumulate operations, or the equivalent substitution like XNOR BID3 in BNN. For fair comparison, we control PFR for each layer of these methods to be the same to observe the final Retrain-Error. In TAB1, MSF-Layerwise refers to the "prune once and retrain" scheme, and the MSF-Cascade refers the "prune and retrain iteratively" scheme. The first three groups of experiments were done on the CIFAR-10 dataset. The last group refers to on Imagenet.

we define the filter-level pruning problem for binary neural networks for the first time and propose method to solve it.

Represent the short summary for finding the corresponding scientific article (long).

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.