Question: Represent the scientific article to find a short TLDR

Question: Pruning large neural networks while maintaining their performance is often desirable due to the reduced space and time complexity. In existing methods, pruning is done within an iterative optimization procedure with either heuristically designed pruning schedules or additional hyperparameters, undermining their utility. In this work, we present a new approach that prunes a given network once at initialization prior to training. To achieve this, we introduce a saliency criterion based on connection sensitivity that identifies structurally important connections in the network for the given task. This eliminates the need for both pretraining and the complex pruning schedule while making it robust to architecture variations. After pruning, the sparse network is trained in the standard way. Our method obtains extremely sparse networks with virtually the same accuracy as the reference network on the MNIST, CIFAR-10, and Tiny-ImageNet classification tasks and is broadly applicable to various architectures including convolutional, residual and recurrent networks. Unlike existing methods, our approach enables us to demonstrate that the retained connections are indeed relevant to the given task. Despite the success of deep neural networks in machine learning, they are often found to be highly overparametrized making them computationally expensive with excessive memory requirements. Pruning such large networks with minimal loss in performance is appealing for real-time applications, especially on resource-limited devices. In addition, compressed neural networks utilize the model capacity efficiently, and this interpretation can be used to derive better generalization bounds for neural networks BID0 ).In network pruning, given a large reference neural network, the goal is to learn a much smaller subnetwork that mimics the performance of the reference network. The majority of existing methods in the literature attempt to find a subset of weights from the pretrained reference network either based on a saliency criterion BID29; BID22; BID11 ) or utilizing sparsity enforcing penalties BID3; BID2 ). Unfortunately, since pruning is included as a part of an iterative optimization procedure, all these methods require many expensive prune -retrain cycles and heuristic design choices with additional hyperparameters, making them non-trivial to extend to new architectures and tasks. In this work, we introduce a saliency criterion that identifies connections in the network that are important to the given task in a data-dependent way before training. Specifically, we discover important connections based on their influence on the loss function at a variance scaling initialization, which we call connection sensitivity. Given the desired sparsity level, redundant connections are pruned once prior to training (i.e., single-shot), and then the sparse pruned network is trained in the standard way. Our approach has several attractive properties:• Simplicity. Since the network is pruned once prior to training, there is no need for pretraining and complex pruning schedules. Our method has no additional hyperparameters and once pruned, training of the sparse network is performed in the standard way.• Versatility. Since our saliency criterion chooses structurally important connections, it is robust to architecture variations. Therefore our method can be applied to various architectures including convolutional, residual and recurrent networks with no modifications.• Interpretability. Our method determines important connections with a mini-batch of data at single-shot. By varying this mini-batch used for pruning, our method enables us to verify that the retained connections are indeed essential for the given task. We evaluate our method on MNIST, CIFAR-10, and Tiny-ImageNet classification datasets with widely varying architectures. Despite being the simplest, our method obtains extremely sparse networks with virtually the same accuracy as the existing baselines across all tested architectures. Furthermore, we investigate the relevance of the retained connections as well as the effect of the network initialization and the dataset on the saliency score. Classical methods. Essentially, early works in network pruning can be categorized into two groups BID34 ): 1) those that utilize sparsity enforcing penalties; and 2) methods that prune the network based on some saliency criterion. The methods from the former category BID3; BID38; BID15 ) augment the loss function with some sparsity enforcing penalty terms (e.g., L 0 or L 1 norm), so that back-propagation effectively penalizes the magnitude of the weights during training. Then weights below a certain threshold may be removed. On the other hand, classical saliency criteria include the sensitivity of the loss with respect to the neurons BID29 ) or the weights BID17 ) and Hessian of the loss with respect to the weights BID22; BID12 ). Since these criteria are heavily dependent on the scale of the weights and are designed to be incorporated within the learning process, these methods are prohibitively slow requiring many iterations of pruning and learning steps. Our approach identifies redundant weights from an architectural point of view and prunes them once at the beginning before training. Modern advances. In recent years, the increased space and time complexities as well as the risk of overfitting in deep neural networks prompted a surge of further investigation in network pruning. While Hessian based approaches employ the diagonal approximation due to its computational simplicity, impressive (i.e., extreme sparsity without loss in accuracy) are achieved using magnitude of the weights as the criterion BID11 ). This made them the de facto standard method for network pruning and led to various implementations BID9 BID2 ). The magnitude criterion is also extended to recurrent neural networks BID30 ), yet with heavily tuned hyperparameter setting. Unlike our approach, the main drawbacks of magnitude based approaches are the reliance on pretraining and the expensive pruneretrain cycles. Furthermore, since pruning and learning steps are intertwined, they often require highly heuristic design choices which make them non-trivial to be extended to new architectures and different tasks. Meanwhile, Bayesian methods are also applied to network pruning BID37; BID27 ) where the former extends the soft weight sharing in BID32 to obtain a sparse and compressed network, and the latter uses variational inference to learn the dropout rate which can then be used to prune the network. Unlike the above methods, our approach is simple and easily adaptable to any given architecture or task without modifying the pruning procedure. Network compression in general. Apart from weight pruning, there are approaches focused on structured simplification such as pruning filters BID24; BID28 ), structured sparsity with regularizers BID39 ), low-rank approximation BID16 ), matrix and tensor factorization BID31 ), and sparsification using expander graphs BID33 ) or Erdős-Rényi random graph BID26 ). In addition, there is a large body of work on compressing the representation of weights. A non-exhaustive list includes quantization BID7 ), reduced precision BID10 ) and binary weights BID14 ). In this work, we focus on weight pruning that is free from structural constraints and amenable to further compression schemes. The main hypothesis behind the neural network pruning literature is that neural networks are usually overparametrized, and comparable performance can be obtained by a much smaller network BID34 ) while improving generalization BID0 ). To this end, the objective is to learn ii a sparse network while maintaining the accuracy of the standard reference network. Let us first formulate neural network pruning as an optimization problem. DISPLAYFORM0, and a desired sparsity level κ (i.e., the number of non-zero weights) neural network pruning can be written as the following constrained optimization problem: DISPLAYFORM1 DISPLAYFORM2 Here, (·) is the standard loss function (e.g., cross-entropy loss), w is the set of parameters of the neural network, m is the total number of parameters and · 0 is the standard L 0 norm. The conventional approach to optimize the above problem is by adding sparsity enforcing penalty terms BID3 BID38 BID15 ). Recently, Carreira-Perpiñán & attempts to minimize the above constrained optimization problem using the stochastic version of projected gradient descent (where the projection is accomplished by pruning). However, these methods often turn out to be inferior to saliency based methods in terms of ing sparsity and require heavily tuned hyperparameter settings to obtain comparable . On the other hand, saliency based methods treat the above problem as selectively removing redundant parameters (or connections) in the neural network. In order to do so, one has to come up with a good criterion to identify such redundant connections. Popular criteria include magnitude of the weights, i.e., weights below a certain threshold are redundant BID11; BID9 ) and Hessian of the loss with respect to the weights, i.e., the higher the value of Hessian, the higher the importance of the parameters BID22; BID12 ), defined as follows: DISPLAYFORM3 Here, for connection j, s j is the saliency score, w j is the weight, and H jj is the value of the Hessian matrix, where the Hessian H = ∂ 2 L/∂w 2 ∈ R m×m. Considering Hessian based methods, the Hessian matrix is neither diagonal nor positive definite in general, approximate at best, and intractable to compute for large networks. Despite being popular, both of these criteria depend on the scale of the weights and in turn require pretraining and are very sensitive to the architectural choices. For instance, different normalization layers affect the scale of the weights in a different way, and this would non-trivially affect the saliency score. Furthermore, pruning and the optimization steps are alternated many times throughout training, ing in highly expensive prune -retrain cycles. Such an exorbitant requirement hinders the use of pruning methods in large-scale applications and raises questions about the credibility of the existing pruning criteria. In this work, we design a criterion which directly measures the connection importance in a datadependent manner. This alleviates the dependency on the weights and enables us to prune the network once at the beginning, and then the training can be performed on the sparse pruned network. Therefore, our method eliminates the need for the expensive prune -retrain cycles, and in theory, it can be an order of magnitude faster than the standard neural network training as it can be implemented using software libraries that support sparse matrix computations. Given a neural network and a dataset, our goal is to design a method that can selectively prune redundant connections for the given task in a data-dependent way even before training. To this end, we first introduce a criterion to identify important connections and then discuss its benefits. Since we intend to measure the importance (or sensitivity) of each connection independently of its weight, we introduce auxiliary indicator variables c ∈ {0, 1} m representing the connectivity of iii parameters w. 1 Now, given the sparsity level κ, Equation 1 can be correspondingly modified as: DISPLAYFORM0 DISPLAYFORM1 where denotes the Hadamard product. Compared to Equation 1, we have doubled the number of learnable parameters in the network and directly optimizing the above problem is even more difficult. However, the idea here is that since we have separated the weight of the connection (w) from whether the connection is present or not (c), we may be able to determine the importance of each connection by measuring its effect on the loss function. For instance, the value of c j indicates whether the connection j is active (c j = 1) in the network or pruned (c j = 0). Therefore, to measure the effect of connection j on the loss, one can try to measure the difference in loss when c j = 1 and c j = 0, keeping everything else constant. Precisely, the effect of removing connection j can be measured by, DISPLAYFORM2 where e j is the indicator vector of element j (i.e., zeros everywhere except at the index j where it is one) and 1 is the vector of dimension m. Note that computing ∆L j for each j ∈ {1 . . . m} is prohibitively expensive as it requires m + 1 (usually in the order of millions) forward passes over the dataset. In fact, since c is binary, L is not differentiable with respect to c, and it is easy to see that ∆L j attempts to measure the influence of connection j on the loss function in this discrete setting. Therefore, by relaxing the binary constraint on the indicator variables c, ∆L j can be approximated by the derivative of L with respect to c j, which we denote g j (w; D). Hence, the effect of connection j on the loss can be written as: DISPLAYFORM3 In fact, ∂L/∂c j is an infinitesimal version of ∆L j, that measures the rate of change of L with respect to an infinitesimal change in c j from 1 → 1 − δ. This can be computed efficiently in one forward-backward pass using automatic differentiation, for all j at once. Notice, this formulation can be viewed as perturbing the weight w j by a multiplicative factor δ and measuring the change in loss. This approximation is similar in spirit to BID19 where they try to measure the influence of a datapoint to the loss function. Here we measure the influence of connections. Furthermore, ∂L/∂c j is not to be confused with the gradient with respect to the weights (∂L/∂w j), where the change in loss is measured with respect to an additive change in weight w j.Notably, our interest is to discover important (or sensitive) connections in the architecture, so that we can prune unimportant ones in single-shot, disentangling the pruning process from the iterative optimization cycles. To this end, we take the magnitude of the derivatives g j as the saliency criterion. Note that if the magnitude of the derivative is high (regardless of the sign), it essentially means that the connection c j has a considerable effect on the loss (either positive or negative), and it has to be preserved to allow learning on w j. Based on this hypothesis, we define connection sensitivity as the normalized magnitude of the derivatives: DISPLAYFORM4 Once the sensitivity is computed, only the top-κ connections are retained, where κ denotes the desired number of non-zero weights. Precisely, the indicator variables c are set as follows: DISPLAYFORM5 wheres κ is the κ-th largest element in the vector s and 1[·] is the indicator function. Here, for exactly κ connections to be retained, ties can be broken arbitrarily. We would like to clarify that the above criterion FORMULA8 ) is different from the criteria used in early works by BID29 DISPLAYFORM6 Pruning: choose top-κ connections 6: w * ← arg min w∈R m L(c w; D) Regular training 7: w * ← c w * connection sensitivity. The fundamental idea behind them is to identify elements (e.g. weights or neurons) that least degrade the performance when removed. This means that their saliency criteria (i.e. −∂L/∂w or −∂L/∂α; α refers to the connectivity of neurons), in fact, depend on the loss value before pruning, which in turn, require the network to be pre-trained and iterative optimization cycles to ensure minimal loss in performance. They also suffer from the same drawbacks as the magnitude and Hessian based methods as discussed in Section 3. In contrast, our saliency criterion (Equation 6) is designed to measure the sensitivity as to how much influence elements have on the loss function regardless of whether it is positive or negative. This criterion alleviates the dependency on the value of the loss, eliminating the need for pre-training. These fundamental differences enable the network to be pruned at single-shot prior to training, which we discuss further in the next section. Note that the saliency measure defined in Equation 6 depends on the value of weights w used to evaluate the derivative as well as the dataset D and the loss function L. In this section, we discuss the effect of each of them and show that it can be used to prune the network in single-shot with initial weights w. Firstly, in order to minimize the impact of weights on the derivatives ∂L/∂c j, we need to choose these weights carefully. For instance, if the weights are too large, the activations after the non-linear function (e.g., sigmoid) will be saturated, which would in uninformative gradients. Therefore, the weights should be within a sensible range. In particular, there is a body of work on neural network initialization BID8 ) that ensures the gradients to be in a reasonable range, and our saliency measure can be used to prune neural networks at any such initialization. Furthermore, we are interested in making our saliency measure robust to architecture variations. Note that initializing neural networks is a random process, typically done using normal distribution. However, if the initial weights have a fixed variance, the signal passing through each layer no longer guarantees to have the same variance, as noted by BID23. This would make the gradient and in turn our saliency measure, to be dependent on the architectural characteristics. Thus, we advocate the use of variance scaling methods (e.g., BID6) to initialize the weights, such that the variance remains the same throughout the network. By ensuring this, we empirically show that our saliency measure computed at initialization is robust to variations in the architecture. Next, since the dataset and the loss function defines the task at hand, by relying on both of them, our saliency criterion in fact discovers the connections in the network that are important to the given task. However, the practitioner needs to make a choice on whether to use the whole training set, or a mini-batch or the validation set to compute the connection saliency. Moreover, in case there are memory limitations (e.g., large model or dataset), one can accumulate the saliency measure over multiple batches or take an exponential moving average. In our experiments, we show that using only one mini-batch of a reasonable number of training examples can lead to effective pruning. Finally, in contrast to the previous approaches, our criterion for finding redundant connections is simple and directly based on the sensitivity of the connections. This allows us to effectively identify and prune redundant connections in a single step even before training. Then, training can be performed on the ing pruned (sparse) network. We name our method SNIP for Single-shot Network Pruning, and the complete algorithm is given in Algorithm 1. Figure 1: Test errors of LeNets pruned at varying sparsity levelsκ, whereκ = 0 refers to the reference network trained without pruning. Our approach performs as good as the reference network across varying sparsity levels on both the models. We evaluate our method, SNIP, on MNIST, CIFAR-10 and Tiny-ImageNet classification tasks with a variety of network architectures. Our show that SNIP yields extremely sparse models with minimal or no loss in accuracy across all tested architectures, while being much simpler than other state-of-the-art alternatives. We also provide clear evidence that our method prunes genuinely explainable connections rather than performing blind pruning. Experiment setup For brevity, we define the sparsity level to beκ = (m − κ)/m · 100 (%), where m is the total number of parameters and κ is the desired number of non-zero weights. For a given sparsity levelκ, the sensitivity scores are computed using a batch of 100 and 128 examples for MNIST and CIFAR experiments, respectively. After pruning, the pruned network is trained in the standard way. Specifically, we train the models using SGD with momentum of 0.9, batch size of 100 for MNIST and 128 for CIFAR experiments and the weight decay rate of 0.0005, unless stated otherwise. The initial learning rate is set to 0.1 and decayed by 0.1 at every 25k or 30k iterations for MNIST and CIFAR, respectively. Our algorithm requires no other hyperparameters or complex learning/pruning schedules as in most pruning algorithms. We spare 10% of the training data as a validation set and used only 90% for training. For CIFAR experiments, we use the standard data augmentation (i.e., random horizontal flip and translation up to 4 pixels) for both the reference and sparse models. The code can be found here: https://github.com/namhoonlee/snip-public. We first test our approach on two standard networks for pruning, LeNet-300-100 and LeNet-5-Caffe. LeNet-300-100 consists of three fully-connected (fc) layers with 267k parameters and LeNet-5-Caffe consists of two convolutional (conv) layers and two fc layers with 431k parameters. We prune the LeNets for different sparsity levelsκ and report the performance in error on the MNIST image classification task. We run the experiment 20 times for eachκ by changing random seeds for dataset and network initialization. The are reported in Figure 1.The pruned sparse LeNet-300-100 achieves performances similar to the reference (κ = 0), only with negligible loss atκ = 90. For LeNet-5-Caffe, the performance degradation is nearly invisible. Note that our saliency measure does not require the network to be pre-trained and is computed at random initialization. Despite such simplicity, our approach prunes LeNets quickly (single-shot) and effectively (minimal accuracy loss) at varying sparsity levels. What happens if we increase the target sparsity to an extreme level? For example, would a model with only 1% of the total parameters still be trainable and perform well? We test our approach for extreme sparsity levels (e.g., up to 99% sparsity on and compare with various pruning algorithms as follows: LWC BID11), DNS BID9 Table 1: Pruning on LeNets and comparisons to other approaches. Here, "many" refers to an arbitrary number often in the order of total learning steps, and "soft" refers to soft pruning in Bayesian based methods. Our approach is capable of pruning up to 98% for LeNet-300-100 and 99% for LeNet-5-Caffe with marginal increases in error from the reference network. Notably, our approach is considerably simpler than other approaches, with no requirements such as pretraining, additional hyperparameters, augmented training objective or architecture dependent constraints. Idelbayev FORMULA1 ), SWS BID37 ), SVD BID27 ), OBD BID22 ), L-OBS BID5 ). The are summarized in Table 1.We achieve errors that are comparable to the reference model, degrading approximately 0.7% and 0.3% while pruning 98% and 99% of the parameters in LeNet-300-100 and LeNet-5-Caffe respectively. For slightly relaxed sparsities (i.e., 95% for LeNet-300-100 and 98% for, the sparse models pruned by SNIP record better performances than the dense reference network. Considering 99% sparsity, our method efficiently finds 1% of the connections even before training, that are sufficient to learn as good as the reference network. Moreover, SNIP is competitive to other methods, yet it is unparalleled in terms of algorithm simplicity. To be more specific, we enumerate some key points and non-trivial aspects of other algorithms and highlight the benefit of our approach. First of all, the aforementioned methods require networks to be fully trained (if not partly) before pruning. These approaches typically perform many pruning operations even if the network is well pretrained, and require additional hyperparameters (e.g., pruning frequency in BID9, annealing schedule in Carreira-Perpiñán & Idelbayev FORMULA1). Some methods augment the training objective to handle pruning together with training, increasing the complexity of the algorithm (e.g., augmented Lagrangian in Carreira-Perpiñán & , variational inference in BID27). Furthermore, there are approaches designed to include architecture dependent constraints (e.g., layer-wise pruning schemes in BID5).Compared to the above approaches, ours seems to cost almost nothing; it requires no pretraining or additional hyperparameters, and is applied only once at initialization. This means that one can easily plug-in SNIP as a preprocessor before training neural networks. Since SNIP prunes the network at the beginning, we could potentially expedite the training phase by training only the survived parameters (e.g., reduced expected FLOPs in BID25). Notice that this is not possible for the aforementioned approaches as they obtain the maximum sparsity at the end of the process. In this section we show that our approach is generally applicable to more complex modern network architectures including deep convolutional, residual and recurrent ones. Specifically, our method is applied to the following models:• AlexNet-s and AlexNet-b: Models similar to BID20 in terms of the number of layers and size of kernels. We set the size of fc layers to 512 (AlexNet-s) and to 1024 (AlexNet-b) to adapt for CIFAR-10 and use strides of 2 for all conv layers instead of using pooling layers.• VGG-C, VGG-D and VGG-like: Models similar to the original VGG models described in BID36. VGG-like BID40 ) is a popular variant adapted for CIFAR-10 which has one less fc layers. For all VGG models, we set the size of fc layers to 512, remove dropout layers to avoid any effect on sparsification and use batch normalization instead.• and Same models as in BID41 Table 2: Pruning of the proposed approach on various modern architectures (before → after). AlexNets, VGGs and WRNs are evaluated on CIFAR-10, and LSTMs and GRUs are evaluated on the sequential MNIST classification task. The approach is generally applicable regardless of architecture types and models and in a significant amount of reduction in the number of parameters with minimal or no loss in performance.• LSTM-s, LSTM-b, GRU-s, GRU-b: One layer RNN networks with either LSTM BID42 ) or GRU BID4 ) cells. We develop two unit sizes for each cell type, 128 and 256 for {·}-s and {·}-b, respectively. The model is adapted for the sequential MNIST classification task, similar to BID21. Instead of processing pixel-by-pixel, however, we perform rowby-row processing (i.e., the RNN cell receives each row at a time).The are summarized in Table 2. Overall, our approach prunes a substantial amount of parameters in a variety of network models with minimal or no loss in accuracy (< 1%). Our pruning procedure does not need to be modified for specific architectural variations (e.g., recurrent connections), indicating that it is indeed versatile and scalable. Note that prior art that use a saliency criterion based on the weights (i.e., magnitude or Hessian based) would require considerable adjustments in their pruning schedules as per changes in the model. We note of a few challenges in directly comparing against others: different network specifications, learning policies, datasets and tasks. Nonetheless, we provide a few comparison points that we found in the literature. On CIFAR-10, SVD prunes 97.9% of the connections in VGG-like with no loss in accuracy (ours: 97% sparsity) while SWS obtained 93.4% sparsity on but with a non-negligible loss in accuracy of 2%. There are a couple of works attempting to prune RNNs (e.g., GRU in BID30 and LSTM in BID35). Even though these methods are specifically designed for RNNs, none of them are able to obtain extreme sparsity without substantial loss in accuracy reflecting the challenges of pruning RNNs. To the best of our knowledge, we are the first to demonstrate on convolutional, residual and recurrent networks for extreme sparsities without requiring additional hyperparameters or modifying the pruning procedure. So far we have shown that our approach can prune a variety of deep neural network architectures for extreme sparsities without losing much on accuracy. However, it is not clear yet which connections are actually being pruned away or whether we are pruning the right (i.e., unimportant) ones. What if we could actually peep through our approach into this inspection?Consider the first layer in LeNet-300-100 parameterized by w l=1 ∈ R 784×300. This is a layer fully connected to the input where input images are of size 28 × 28 = 784. In order to understand which connections are retained, we can visualize the binary connectivity mask for this layer c l=1, by averaging across columns and then reshaping the vector into 2D matrix (i.e., c l=1 ∈ {0, 1} 784×300 → R 784 → R 28×28 ). Recall that our method computes c using a minibatch of examples. In this experiment, we curate the mini-batch of examples of the same class and see which weights are retained for that mini-batch of data. We repeat this experiment for all classes viii Published as a conference paper at ICLR 2019 Visualizations of pruned parameters of the first layer in LeNet-300-100; the parameters are reshaped to be visualized as an image. Each column represents the visualizations for a particular class obtained using a batch of 100 examples with varying levels of sparsityκ, from 10 (top) to 90 (bottom). Bright pixels indicate that the parameters connected to these region had high importance scores (s) and survived from pruning. As the sparsity increases, the parameters connected to the discriminative part of the image for classification survive and the irrelevant parts get pruned.(i.e., digits for MNIST and fashion items for Fashion-MNIST) with varying sparsity levelsκ. The are displayed in FIG1 (see Appendix A for more ).The are significant; important connections seem to reconstruct either the complete image (MNIST) or silhouettes (Fashion-MNIST) of input class. When we use a batch of examples of the digit 0 (i.e., the first column of MNIST ), for example, the parameters connected to the foreground of the digit 0 survive from pruning while the majority of is removed. Also, one can easily determine the identity of items from Fashion-MNIST . This clearly indicates that our method indeed prunes the unimportant connections in performing the classification task, receiving signals only from the most discriminative part of the input. This stands in stark contrast to other pruning methods from which carrying out such inspection is not straightforward. Recall that our connection saliency measure depends on the network weights w as well as the given data D (Section 4.2). We study the effect of each of these in this section. Effect of data. Our connection saliency measure depends on a mini-batch of train examples D b (see Algorithm 1). To study the effect of data, we vary the batch size used to compute the saliency (|D b |) and check which connections are being pruned as well as how much performance change this in on the corresponding sparse network. We test with to visualize the remaining parameters, and set the sparsity levelκ = 90. Note that the batch size used for training remains the same as 100 for all cases. The are displayed in Figure 3.Effect of initialization. Our approach prunes a network at a stochastic initialization as discussed. We study the effect of the following initialization methods: 1) RN (random normal), 2) TN (truncated random normal), 3) VS-X (a variance scaling method using BID6), and 4) VS-H (a variance scaling method BID13). We test on LeNets and RNNs on MNIST and run 20 sets of experiments by varying the seed for initialization. We set the sparsity levelκ = 90, and train with Adam optimizer BID18 ) with learning rate of 0.001 without weight decay. Note that for training VS-X initialization is used in all the cases. The are reported in Figure 3.For all models, VS-H achieves the best performance. The differences between initializers are marginal on LeNets, however, variance scaling methods indeed turns out to be essential for complex RNN models. This effect is significant especially for GRU where without variance scaling initialization, the pruned networks are unable to achieve good accuracies, even with different optimizers. Overall, initializing with a variance scaling method seems crucial to making our saliency measure reliable and model-agnostic.ix DISPLAYFORM0 (1.72%) (1.64%) (1.56%) (1.40%) - Figure 3: The effect of different batch sizes: (top-row) survived parameters in the first layer of LeNet-300-100 from pruning visualized as images; (bottom-row) the performance in errors of the pruned networks. For |D b | = 1, the sampled example was 8; our pruning precisely retains the valid connections. As |D b | increases, survived parameters get close to the average of all examples in the train set (last column), and the error decreases. Init. To further explore the use cases of SNIP, we run the experiment introduced in BID43 and check whether the sparse network obtained by SNIP memorizes the dataset. Specifically, we train LeNet-5-Caffe for both the reference model and pruned model (withκ = 99) on MNIST with either true or randomly shuffled labels. To compute the connection sensitivity, always true labels are used. The are plotted in FIG2. Given true labels, both the reference (red) and pruned (blue) models quickly reach to almost zero training loss. However, the reference model provided with random labels (green) also reaches to very low training loss, even with an explicit L2 regularizer (purple), indicating that neural networks have enough capacity to memorize completely random data. In contrast, the model pruned by SNIP (orange) fails to fit the random labels (high training error). The potential explanation is that the pruned network does not have sufficient capacity to fit the random labels, but it is able to classify MNIST with true labels, reinforcing the significance of our saliency criterion. It is possible that a similar experiment can be done with other pruning methods BID27 ), however, being simple, SNIP enables such exploration much easier. We provide a further analysis on the effect of varyingκ in Appendix B. In this work, we have presented a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications. While SNIP in extremely sparse models, we find that our connection sensitivity measure itself is noteworthy in that it diagnoses important connections in the network from a purely untrained network. We believe that this opens up new possibilities beyond pruning in the topics of understanding of neural network architectures, multi-task transfer learning and structural regularization, to name a few. In addition to these potential directions, we intend to explore the generalization capabilities of sparse networks. Notably, compared to the case of using SNIP FIG1, the are different: Firstly, the on the original (Fashion-)MNIST (i.e., (a) and (c) above) are not the same as the ones using SNIP (i.e., (a) and (b) in FIG1. Moreover, the pruning patterns are inconsistent with different sparsity levels, either intra-class or inter-class. Furthermore, using ∂L/∂w in different pruning patterns between the original and inverted data in some cases (e.g., the 2 nd columns between (c) and (Figure 7: The effect of varying sparsity levels (κ). The lowerκ becomes, the lower training loss is recorded, meaning that a network with more parameters is more vulnerable to fitting random labels. Recall, however, that all pruned models are able to learn to perform the classification task without losing much accuracy (see Figure 1). This potentially indicates that the pruned network does not have sufficient capacity to fit the random labels, but it is capable of performing the classification. C TINY-IMAGENET Table 4: Pruning of SNIP on Tiny-ImageNet (before → after). Tiny-ImageNet 2 is a subset of the full ImageNet: there are 200 classes in total, each class has 500 and 50 images for training and validation respectively, and each image has the spatial resolution of 64×64. Compared to CIFAR-10, the resolution is doubled, and to deal with this, the stride of the first convolution in all architectures is doubled, following the standard practice for this dataset. In general, the Tiny-ImageNet classification task is considered much more complex than MNIST or CIFAR-10. Even on Tiny-ImageNet, however, SNIP is still able to prune a large amount of parameters with minimal loss in performance. AlexNet models lose more accuracies than VGGs, which may be attributed to the fact that the first convolution stride for AlexNet is set to be 4 (by its design of no pooling) which is too large and could lead to high loss of information when pruned.

represent the tldr for finding the corresponding scientific article
For example, 'We propose a Graph Convolutional Network based encoder-decoder model with sequential attention for goal-oriented dialogue systems.' should be close to 'Domain specific goal-oriented dialogue systems typically require modeling three types of inputs, viz., (i) the knowledge-base associated with the domain, (ii) the history of the conversation, which is a sequence of utterances and (iii) the current utterance for which the response needs to be generated. While modeling these inputs, current state-of-the-art models such as Mem2Seq typically ignore the rich structure inherent in the knowledge graph and the sentences in the conversation context. Inspired by the recent success of structure-aware Graph Convolutional Networks (GCNs) for various NLP tasks such as machine translation, semantic role labeling and document dating, we propose a memory augmented GCN for goal-oriented dialogues. Our model exploits (i) the entity relation graph in a knowledge-base and (ii) the dependency graph associated with an utterance to compute richer representations for words and entities. Further, we take cognizance of the fact that in certain situations, such as, when the conversation is in a code-mixed language, dependency parsers may not be available. We show that in such situations we could use the global word co-occurrence graph and use it to enrich the representations of utterances. We experiment with the modified DSTC2 dataset and its recently released code-mixed versions in four languages and show that our method outperforms existing state-of-the-art methods, using a wide range of evaluation metrics. Goal-oriented dialogue systems which can assist humans in various day-to-day activities have widespread applications in several domains such as e-commerce, entertainment, healthcare, etc. For example, such systems can help humans in scheduling medical appointments, reserving restaurants, booking tickets, etc.. From a modeling perspective, one clear advantage of dealing with domain specific goal-oriented dialogues is that the vocabulary is typically limited, the utterances largely follow a fixed set of templates and there is an associated domain knowledge which can be exploited. More specifically, there is some structure associated with the utterances as well as the knowledge base. More formally, the task here is to generate the next response given (i) the previous utterances in the conversation history (ii) the current user utterance (known as the query) and (iii) the entities and relationships in the associated knowledge base. Current state-of-the-art methods BID30 BID23 typically use variants of Recurrent Neural Network BID10 to encode the history and current utterance and an external memory network to store the entities in the knowledge base. The encodings of the utterances and memory elements are then suitably combined using an attention network and fed to the decoder to generate the response, one word at a time. However, these methods do not exploit the structure in the knowledge base as defined by entity-entity relations and the structure in the utterances as defined by a dependency parse. Such structural information can be exploited to improve the performance of the system as demonstrated by recent works on syntax-aware neural machine translation BID13 BID2 BID4, semantic role labeling and document dating BID35 which use GCNs BID8 BID9 BID19 to exploit sentence structure. In this work, we propose to use such graph structures for goal-oriented dialogues. In particular, we compute the dependency parse tree for each utterance in the conversation and use a GCN to capture the interactions between words. This allows us to capture interactions between distant words in the sentence as long as they are connected by a dependency relation. We also use GCNs to encode the entities of the KB where the entities are treated as nodes and the relations as edges of the graph. Once we have a richer structure aware representation for the utterances and the entities, we use a sequential attention mechanism to compute an aggregated context representation from the GCN node vectors of the query, history and entities. Further, we note that in certain situations, such as, when the conversation is in a code-mixed language or a language for which parsers are not available then it may not be possible to construct a dependency parse for the utterances. To overcome this, we construct a co-occurrence matrix from the entire corpus and use this matrix to impose a graph structure on the utterances. More specifically, we add an edge between two words in a sentence if they co-occur frequently in the corpus. Our experiments suggest that this simple strategy acts as a reasonable substitute for dependency parse trees. We perform experiments with the modified DSTC2 BID3 dataset which contains goal-oriented conversations for reserving restaurants. We also use its recently released code-mixed versions BID1 which contain code-mixed conversations in four different languages, viz., Hindi, Bengali, Gujarati and Tamil. We compare with recent state-of-the-art methods and show that on average the proposed model gives an improvement of 2.8 BLEU points and 2 ROUGE points. Our contributions can be summarized as follows: (i) We use GCNs to incorporate structural information for encoding query, history and KB entities in goal-oriented dialogues (ii) We use a sequential attention mechanism to obtain query aware and history aware context representations (iii) We leverage co-occurrence frequencies and PPMI (positive-pointwise mutual information) values to construct contextual graphs for code-mixed utterances and (iv) We show that the proposed model obtains state-of-the-art on the modified DSTC2 dataset and its recently released code-mixed versions. In this section we review the previous work in goal-oriented dialogue systems and describe the introduction of GCNs in NLP.Goal-Oriented Dialogue System: Initial goal-oriented dialogue systems BID39 BID37 were based on dialogue state tracking BID38 BID15 b) and included pipelined modules for natural language understanding, dialogue state tracking, policy management and natural language generation. used neural networks for these intermediate modules but still lacked absolute end-to-end trainability. Such pipelined modules were restricted by the fixed slot-structure assumptions on the dialogue state and required per-module based labelling. To mitigate this problem BID3 released a version of goal-oriented dialogue dataset that focuses on the development of end-to-end neural models. Such models need to reason over the associated KB triples and generate responses directly from the utterances without any additional annotations. For example, BID3 proposed a Memory Network BID34 based model to match the response candidates with the multi-hop attention weighted representation of the conversation history and the KB triples in memory. BID22 further added highway BID33 and residual connections BID14 to the memory network in order to regulate the access to the memory blocks. BID30 developed a variant of RNN cell which computes a refined representation of the query over multiple iterations before querying the memory. However, all these approaches retrieve the response from a set of candidate responses and such a candidate set is not easy to obtain in any new domain of interest. To account for this,; adapted RNN based encoder-decoder models to generate appropriate responses instead of retrieving them from a candidate set. introduced a key-value memory network based generative model which integrates the underlying KB with RNN based encode-attend-decode models. BID23 used memory networks on top of the RNN decoder to tightly integrate KB entities with the decoder to generate more infor-mative responses. However, as opposed to our work, all these works ignore the underlying structure of the entity-relation graph of the KB and the syntactic structure of the utterances. Recently, there has been an active interest in enriching existing encode-attenddecode models BID0 with structural information for various NLP tasks. Such structure is typically obtained from the constituency and/or dependency parse of sentences. The idea is to treat the output of a parser as a graph and use an appropriate network to capture the interactions between the nodes of this graph. For example, BID13 and BID4 showed that incorporating such syntactical structures as Tree-LSTMs in the encoder can improve the performance of Neural Machine Translation (NMT). BID29 use Graph-LSTMs to perform cross sentence n-ary relation extraction and show that their formulation is applicable to any graph structure and Tree-LSTMs can be thought of as a special case of it. In parallel, Graph Convolutional Networks (GCNs) BID9 BID8 BID19 and their variants BID20 have emerged as state-of-the-art methods for computing representations of entities in a knowledge graph. They provide a more flexible way of encoding such graph structures by capturing multi-hop relationships between nodes. This has led to their adoption for various NLP tasks such as neural machine translation BID25 BID2, semantic role labeling, document dating BID35 and question answering BID17 BID26.To the best of our knowledge ours is the first work that uses GCNs to incorporate dependency structural information and the entity-entity graph structure in a single end-to-end neural model for goaloriented dialogue. This is also the first work that incorporates contextual co-occurrence information for code-mixed utterances, for which no dependency structures are available. In this section we describe Graph Convolutional Networks (GCN) BID19 for undirected graphs and then describe their syntactic versions which work with directed labeled edges of dependency parse trees. Graph convolutional networks operate on a graph structure and compute representations for the nodes of the graph by looking at the neighbourhood of the node. k layers of GCNs can be stacked to account for neighbours which are k-hops away from the current node. Formally, let G = (V, E) be an undirected graph where V is the set of nodes (let |V| = n) and E is the set of edges. Let X ∈ R n×m be the input feature matrix with n nodes and each node x u (u ∈ V) is represented by an m-dimensional feature vector. The output of a 1-layer GCN is the hidden representation matrix H ∈ R n×d where each d-dimensional representation of a node captures the interactions with its 1-hop neighbour. Each row of this matrix can be computed as: DISPLAYFORM0 Here W ∈ R d×m is the model parameter matrix, b ∈ R d is the bias vector and ReLU is the rectified linear unit activation function. N (v) is the set of neighbours of node v and is assumed to also include the node v so that the previous representation of the node v is also considered while computing the new hidden representation. To capture interactions with nodes which are multiple hops away, multiple layers of GCNs can be stacked together. Specifically, the representation of node v after k th GCN layer can be formulated as: DISPLAYFORM1 where h k u is the representation of the u th node in the (k − 1) th GCN layer and h DISPLAYFORM2 In a directed labeled graph G = (V, E), each edge between nodes u and v is represented by a triple (u, v, L(u, v) ) where L(u, v) is the associated edge label. modified GCNs to operate over directed labeled graphs, such as the dependency parse tree of a sentence. For such a tree, in order to allow information to flow from head to dependents and vice-versa, they added inverse dependency edges from dependents to heads such as (v, u, L(u, v) ) to E and made the model parameters and biases label specific. In their formulation, u,v) which are label specific. Suppose there are L different labels, then this formulation will require L weights and biases per GCN layer ing in a large number of parameters. To avoid this, the authors use only three sets of weights and biases per GCN layer (as opposed to L) depending on the direction in which the information flows. More specifically, u,v), where dir(u, v) indicates whether information flows from u to v, v to u or u = v. In this work, we also make b DISPLAYFORM0 DISPLAYFORM1 DISPLAYFORM2 instead of having a separate bias per label. The final GCN formulation can thus be described as: DISPLAYFORM3 We first formally define the task of end-to-end goal-oriented dialogue generation. Each dialogue of t turns can be viewed as a succession of user utterances (U) and system responses (S) and can be represented as: (U 1, S 1, U 2, S 2, ..U t, S t). Along with these utterances, each dialogue is also accompanied by e KB triples which are relevant to that dialogue and can be represented as: DISPLAYFORM0 Each triple is of the form: (entity 1, relation, entity 2). These triples can be represented in the form of a graph G k = (V k, E k) where V is the set of all entities and each edge in E is of the form: (entity 1, entity 2, relation) where relation signifies the edge label. At any dialogue turn i, given the (i) dialogue history H = (U 1, S 1, U 2, ..S i−1), (ii) the current user utterance as the query Q = U i and (iii) the associated knowledge graph G k, the task is to generate the current response S i which leads to a completion of the goal. As mentioned earlier, we exploit the graph structure in KB and the syntactic structure in the utterances to generate appropriate responses. Towards this end we propose a model with the following components for encoding these three types of inputs. The query Q = U i is the i th (current) utterance in the dialogue and contains |Q| tokens. We denote the embedding of the i th token in the query as q i We first compute the contextual representations of these tokens by passing them through a bidirectional RNN: DISPLAYFORM0 Now, consider the dependency parse tree of the query sentence denoted by G Q = (V Q, E Q). We use a query specific GCN to operate on G Q, which takes DISPLAYFORM1 as the input to the 1 st GCN layer. The node representation in the k th hop of the query specific GCN is computed as: DISPLAYFORM2 where u,v) DISPLAYFORM3 DISPLAYFORM4 Figure 1: Illustration of the GCN and RNN+GCN modules which are used as encoders in our model. The notations are specific to the dialogue history encoder but both the encoders are same for the query. The GCN encoder is same for the KB except the graph structure. The history H of the dialogue contains |H| tokens and we denote the embedding of the i th token in the history by p i Once again, we first compute the hidden representations of these embeddings using a bidirectional RNN: DISPLAYFORM0 We now compute a dependency parse tree for each sentence in the history and collectively represent all the trees as a single graph G H = (V H, E H). Note that this graph will only contain edges between words belonging to the same sentence and there will be no edges between words across sentences. We then use a history specific GCN to operate on G H which takes s t as the input to the 1 st layer. The node representation in the k th hop of the history specific GCN is computed as: DISPLAYFORM1 where V k dir (u,v) and o k dir (u,v) are edge direction specific history-GCN weights and biases in the k th hop and a 1 u = s u. Such an encoder with a single hop of GCN is illustrated in figure 1(b) and the encoder without the BiRNN is depicted in figure 1(a). As mentioned earlier, G K = (V K, E K) is the graph capturing the interactions between the entities in the knowledge graph associated with the dialogue. Let there be m such entities and we denote the embeddings of the node corresponding to the i th entity as e i We then operate a KB specific GCN on these entity representations to obtain refined representations which capture relations between entities. The node representation in the k th hop of the KB specific GCN is computed as: DISPLAYFORM0 where U k dir (u,v) and z k dir (u,v) are edge direction specific KB-GCN weights and biases in k th hop and r 1 u = e u. We also add inverse edges to E K similar to the case of syntactic GCNs in order to allow information flow in both the directions for an entity pair in the knowledge graph. Dialogue History KB Entities Query DISPLAYFORM0 We use an RNN decoder to generate the tokens of the response and let the hidden states of the decoder be denoted as: DISPLAYFORM0 where T is the total number of decoder timesteps. In order to obtain a single representation from the final layer (k = f) of the query-GCN node vectors, we use an attention mechanism as described below: DISPLAYFORM1 DISPLAYFORM2 DISPLAYFORM3 Here v 1, W 1, W 2 are parameters. Further, at each decoder timestep, we obtain a query aware representation from the final layer of the history-GCN by computing an attention score for each node/token in the history based on the query context vector h Q t as shown below: DISPLAYFORM4 DISPLAYFORM5 DISPLAYFORM6 Here v 2, W 3, W 4 and W 5 are parameters. Finally, we obtain a query and history aware representation of the KB by computing an attention score over all the nodes in the final layer of KB-GCN using h Q t and h H t as shown below: DISPLAYFORM7 DISPLAYFORM8 DISPLAYFORM9 Here v 3, W 6, W 7, W 8 and W 9 are parameters. This sequential attention mechanism is illustrated in FIG0. For simplicity, we depict the GCN and RNN+GCN encoders as blocks. The internal structure of these blocks are shown in figure 1. The decoder takes two inputs, viz., (i) the context which contains the history and the KB and (ii) the query which is the last/previous utterance in the dialogue. We use an aggregator which learns the overall attention to be given to the history and KB components. These attention scores: θ H t and θ K t are dependent on the respective context vectors and the previous decoder state d t−1. The final context vector is obtained as: DISPLAYFORM0 where [;] denotes the concatenation operator. At every timestep the decoder then computes a probability distribution over the vocabulary using the following equations: DISPLAYFORM1 where w t is the decoder input at time step t, V and b are parameters. P vocab gives us a probability distribution over the entire vocabulary and the loss for time step t is l t = − log P vocab (w * t), where w * t is the t th word in the ground truth response. The total loss is an average of the per-time step losses. For the dialogue history and query encoder, we used the dependency parse tree for capturing structural information in the encodings. However, if the conversations occur in a language for which no dependency parsers exist, for example: code-mixed languages like Hinglish (Hindi-English) BID1, then we need an alternate way of extracting a graph structure from the utterances. One simple solution which worked well in practice was to create a word co-occurrence matrix from the entire corpus where the context window is an entire sentence. Once we have such a co-occurrence matrix, for a given sentence we can connect an edge between two words if their co-occurrence frequency is above a threshold value. The co-occurrence matrix can either contain co-occurrence frequency counts or positive-pointwise mutual information (PPMI) values BID6 BID7 BID27. In this section we describe the datasets used in our experiments, the various hyperparameters that we considered and the models that we compared. The original DSTC2 dataset BID15 ) was based on the task of restaurant reservation and contains transcripts of real conversations between humans and bots. The utterances were labeled with the dialogue state annotations like the semantic intent representation, requested slots and the constraints on the slot values. We report our on the modified DSTC2 dataset of BID3 where such annotations are removed and only the raw utterance-response pairs are present with an associated set of KB triples for each dialogue. For our experiments with contextual graphs we reported our on the code-mixed versions of modified DSTC2, which was recently released by BID1 1. This dataset has been collected by code-mixing the utterances of the English version of modified DSTC2 in four languages viz. Hindi (Hi-DSTC2), Bengali (Be-DSTC2), Gujarati (Gu-DSTC2) and Tamil (Ta-DSTC2), via crowdsourcing. Statistics about this dataset and example dialogues are shown in Appendix A.Model per-resp. acc BLEU ROUGE Entity F1 1 2 L Rule-Based BID3 33.3 -----MEMNN BID3 41.1 -----QRN BID30 50.7 -----GMEMNN BID22 48.7 -----Seq2Seq-Attn BID0 46.0 57.3 67.2 56.0 64.9 67.1 Seq2Seq-Attn+Copy 47.3 55.4 ---71.6 HRED BID31 48.9 58.4 67.9 57.6 65.7 75.6 Mem2Seq BID23 45 Table 2: Comparison of RNN+GCN-SeA, GCN-SeA with other models on all code-mixed datasets We used the same train, test and validation splits as provided in the original versions of the datasets. We minimized the cross entropy loss using the Adam optimizer BID18 and tuned the initial learning rates in the range of 0.0006 to 0.001. For regularization we used an L2 norm of 0.001 in addition to a dropout BID32 of 0.1. We used randomly initialized word embeddings of size 300. The RNN and GCN hidden dimensions were also chosen to be 300. We use GRU BID5 ) cells for the RNNs. All parameters were initialized from a truncated normal distribution with a standard deviation of 0.1. We compare the performance of the following models.(i) RNN+GCN-SeA vs GCN-SeA: We use RNN+GCN-SeA to refer to the model described in section 4. Instead of using the hidden representations obtained from the bidirectional RNNs, we also experiment by providing the token embeddings directly to the GCNs i.e. c 1 u = q u in equation 6 and a 1 u = p u in equation 8. We refer to this model as GCN-SeA.(ii) Cross edges between the GCNs: In addition to the dependency and contextual edges, we add edges between words in the dialogue history/query and KB entities if a history/query word exactly matches the KB entity. Such edges create a single connected graph which is encoded using a single GCN encoder and then separated into different contexts to perform the sequential attention. This model is referred to as RNN+CROSS-GCN-SeA.(iii) Frequency vs PPMI Contextual Graph: We experiment with the raw frequency cooccurrence graph structure and the PPMI graph structure for the code-mixed datasets, as explained in section 4.6. We refer to these models as GCN-SeA+Freq and GCN-SeA+PPMI. In both these models, the GCN takes inputs from a bidirectional RNN.(iv) GCN-SeA+Random vs GCN-SeA+Structure: We experiment with the model where the graph is constructed by randomly connecting edges between two words in a context. We refer to this model as GCN-SeA+Random. We refer to the model which either uses dependency or contextual graph instead of random graphs as GCN-SeA+Structure. In this section we discuss the of our experiments as summarized in tables 1,2, and 3. We use BLEU BID28 and ROUGE BID21 metrics to evaluate the generation quality of responses. We also report the per-response accuracy which computes the percentage of responses in which the generated response exactly matches the ground truth response. In order to evaluate the model's capability of correctly injecting entities in the generated response, we report the entity F1 measure as defined in.Results on En-DSTC2: We compare our model with the previous works on the English version of modified DSTC2 in table 1. For most of the retrieval based models, the BLEU or ROUGE scores are not available as they select a candidate from a list of candidates as opposed to generating it. Our model outperforms all of the retrieval and generation based models. We obtain a gain of 0.7 in the per-response accuracy compared to the previous retrieval based state-of-the-art model of BID30, which is a very strong baseline for our generation based model. We call this a strong baseline because the candidate selection task of this model is easier than the response generation task of our model. We also obtain a gain of 2.8 BLEU points, 2 ROUGE points and 2.5 entity F1 points compared to current state-of-the-art generation based models. Results on code-mixed datasets and effect of using RNNs: The of our experiments on the code-mixed datasets are reported in table 2. Our model outperforms the baseline models on all the code-mixed languages. One common observation from the over all the languages (including En-DSTC2) is that RNN+GCN-SeA performs better than GCN-SeA. Similar observations were made by for the task of semantic role labeling. Effect of using Hops: As we increased the number of hops of GCNs, we observed a decrease in the performance. One reason for such a drop in performance could be that the average utterance length is very small (7.76 words). Thus, there isn't much scope for capturing distant neighbourhood information and more hops can add noisy information. Please refer to Appendix B for detailed about the effect of varying the number of hops. Frequency vs PPMI graphs: We observed that PPMI based contextual graphs were slightly better than frequency based contextual graphs (See Appendix C). In particular, when using PPMI as opposed to frequency based contextual graph, we observed a gain of 0.95 in per-response accuracy, 0.45 in BLEU, 0.64 in ROUGE and 1.22 in entity F1 score when averaged across all the code-mixed languages. Effect of using Random Graphs: GCN-SeA-Random and GCN-SeA-Structure take the token embeddings directly instead of passing them though an RNN. This ensures that the difference in performance of the two models are not influenced by the RNN encodings. The are shown in table 3 and we observe a drop in performance for GCN-Random across all the languages. This Table 3: GCN-SeA with random graphs and frequency co-occurrence graphs on all DSTC2 datasets.shows that any random graph does not contribute to the performance gain of GCN-SeA and the dependency and contextual structures do play an important role. Ablations: We experiment with replacing the sequential attention by the Bahdanau attention BID0. We also experiment with various combinations of RNNs and GCNs as encoders. The are shown in We showed that structure aware representations are useful in goal-oriented dialogue and we obtain state-of-the art performance on the modified DSTC2 dataset and its recently released code-mixed versions. We used GCNs to infuse structural information of dependency graphs and contextual graphs to enrich the representations of the dialogue context and KB. We also proposed a sequential attention mechanism for combining the representations of (i) query (current utterance), (ii) conversation history and (ii) the KB. Finally, we empirically showed that when dependency parsers are not available for certain languages such as code-mixed languages then we can use word co-occurrence frequencies and PPMI values to extract a contextual graph and use such a graph with GCNs for improved performance. south part of town. bot api call R cuisine south moderate api call R cuisine south moderate KB Triples: pizza hut cherry hinton R post code pizza hut cherry hinton post code pizza hut cherry hinton R cuisine italian pizza hut cherry hinton R location south pizza hut cherry hinton R phone pizza hut cherry hinton phone pizza hut cherry hinton R address pizza hut cherry hinton address pizza hut cherry hinton R price moderate pizza hut cherry hinton R rating 3 restaurant alimentum R post code restaurant alimentum post code restaurant alimentum R cuisine european restaurant alimentum R location south restaurant alimentum R phone restaurant alimentum phone restaurant alimentum R address restaurant alimentum address restaurant alimentum R price moderate restaurant alimentum R rating 10 user <SILENCE> <SILENCE> bot restaurant alimentum is a nice restaurant in the south of town serving modern european food. restaurant alimentum ek bohot hi accha restaurant hai south town me jo modern european food serve karta hai user can i get the address? kya mujhe address mil sakta hai bot Sure, restaurant alimentum is on restaurant alimentum address. Bilkul, restaurant alimentum restaurant alimentum address par hai user and whats the phone number for that? aur iska phone number kya hai? bot The phone number of restaurant alimentum is restaurant alimentum phone.restaurant alimentum ka phone number hai restaurant alimentum phone user okay, thank you, good bye. theek, thank you, good bye. bot you are welcome. welcome, alvida.'

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.

Represent the short summary for finding the corresponding scientific article (long)

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Transfer learning uses trained weights from a source model as the initial weightsfor the training of a target dataset. A well chosen source with a large numberof labeled data leads to significant improvement in accuracy. We demonstrate atechnique that automatically labels large unlabeled datasets so that they can trainsource models for transfer learning. We experimentally evaluate this method, usinga baseline dataset of human-annotated ImageNet1K labels, against five variationsof this technique. We show that the performance of these automatically trainedmodels come within 17% of baseline on average. In many domains, the task performance of deep learning techniques is heavily dependent on the number of labeled examples, which are difficult and expensive to acquire. This demand for large labeled datasets has inspired alternative techniques, such as weak supervision or automated labeling, whose algorithms create plausible labels to be used to guide supervised training on other tasks. In this work, we develop a content-aware model-selection technique for transfer learning. We take an unlabeled data point (here, an unlabeled image), and compute its distance to the average response of a number of specialized deep learning models, such as those trained for "animal", "person", or "sport". We then create a "pseudolabel" for the point, consisting of a short ordered sequence of the most appropriate model names, like "animal-plant-building". We use these synthetic labels to augment the ground truth labels. We validate the technique by applying it to the ImageNet1K dataset, as well as on a number of other large, unlabeled datasets. There are several well-established approaches that attempt to automatically assign labels to unlabeled images. For example, some use clusters of features to predict labels BID2, or augment image data with linguistic constraints from sources such as WordNet BID0 BID3. These approaches augment tasks by pretraining models using larger unlabeled data-sets. Pretraining approaches have also improved when attempting a target task for which there is a limited amount of accurately labeled training data BID4, by using weakly labeled data, such as social media hashtags, which are much more plentiful. However, effectiveness only appears to grow as the log of the image count. Further approaches use generative models such as GANs as in BID5 to explore and refine category boundaries between clusters of data, which exploit the rich statistical structures of both real and generated examples, sometimes augmented with labels or linguistic constraints. All of these automatic approaches use the structures present in large unlabeled datasets, to extend the expressivity of known labels, and to augment the raw size of training sets. We present our technique using a specific case study involving images, and with source datasets created by vertically partitioning ImageNet22K BID1 ) along its distinct subtrees: animal, plant, weapon, tools, music, fungus, sport, person, food, fruit, garment, building, nature, furniture, vehicle, and fabric. We represent each such dataset by a single average feature vector. In this study, this vector is generated from the second last layer of a reference VGG16 BID6 model trained on ImageNet1K, as shown in FIG0, with the average taken over all the images in the dataset. To label a new image, we first calculate its own feature vector, then compute the distance between it and each of the representatives of the datasets; in this study, we use the Kullback-Leibler (KL) divergence, after appropriate normalization. These 16 distances are then used to determine the synthetic label. To label an image, we encode its relative positions with respect to some subset of the known labeled datasets. To better explain the technique, we will use ImageNet1K as a source of images and ground truth labels. ImageNet1K has 1000 labels; the number of images/label is almost uniform; but the labels broadly fall under few broad categories like animals, vehicles, food, musical instruments, garment, furniture, buildings etc. About 446 out of 1000 labels (∼ 45%) belong to animals while the other top categories are vehicles (5.2%), food (2.2%), musical instruments (2.1%) and garment (2.1%). The distribution of number of labels for different categories has a long tail. Our first labeling methods choose labels to be the names of the N source datasets which were the closest to the image. We concatenate those names together, in order of closeness. Thus, the Nearest-3 method generates labels like "tree-animal-fungus". With 16 source datasets, the Nearest-3 approach yields 16x15x14 = 3360 possible pseudolabels; in our study with the 1.3M images from ImageNet1K, each label had a mean of 381 images and a standard deviation of 1503. In a similar fashion, the Nearest-2 and Nearest-1 pseudolabels were also computed, with 240 and 16 possible labels, respectively. (We found no benefits above Nearest-3.) FIG1 shows the distribution of images for the Nearest-3 method. The high peaks on the right are unsurprisingly related to animals. Our fourth method exploited all 16 distances. We first fixed the order of the 16 source datasets, forming a 16-dimensional coordinate system, so that each image could be viewed as a vector in the ing 16-dimensional simplex of all possible positive distances to them. All of the unlabeled images were then k-means clustered within this space; we used 240 cluster centers to imitate the size of the Nearest-2 space. These ant clusters could not be expected to be of uniform size, so a second round of splitting and merging, based on relative distances, balanced the cluster membership to be nearly uniform, with an average of 1000 images per cluster, and a standard deviation was 193.The "names" of these final clusters were used as the pseudolabels. Our fifth method accommodates those incoming datasets that are characterized by a wide variety of low-level image features, such as flowers. These labels were again devised as a sequence of three source names, but chosen to span the 16-dimensional space as widely as possible. The first source dataset was the closest one, and the second was the farthest one. The third was chosen from the remaining 14 to maximize the area of the ing triangle in 16-space, computed via Heron's formula. In practice, this method only ed in about 200 labels, with an average of about 6300 images per label, and with a very high standard deviation of about 17000 (see FIG1).As an example, consider the images in FIG2, taken from ImageNet1K. The nearest cluster (Nearest-1) for the body armour image is music. This is possibly due to the brass and metallic textures of the photo, similar to that seen in brass music instruments. The Nearest-2 and Nearest-3 labels are (music-weapon) and (music-weapon-person) respectively. In contrast, the label for Largest-Area is music-fungus-sport. So for this image, fungus is the source which is most unlike it, and sport maximizes the area of the triangle defined by the third source. Similarly, for the elephant image, the Nearest-3 label is tree-animal-fungus, and the Largest-Area label is tree-furniture-fungus. The Nearest-2 and Nearest-1 labels are tree-animal and tree, respectively. Experiment 1 Using techniques described in Section 4, we first created five pseudo-labeled datasets for the images in ImageNet1K, as shown in Figure 4. We then trained ResNet27 using each of these pseudo-labeled datasets, creating base models for further transfer learning. We also created two baseline models, one using the vanilla ImageNet1K dataset of images and human-annotated labels, and a second by assigning random labels to the ImageNet1K images. For perspective, the figure shows the accuracy of these seven base models, but since we are interested in the transferability of representations from these base models to a target domain, their absolute accuracy is not the important measure. Then, for each of 12 candidate target datasets, we fine-tuned and calculated the transfer learning accuracy of each of the 7 base models; each of these 84 experiments were carried out with the same hyperparameters. As shown in Figure 4, the accuracy obtained with Vanilla as the base model can serve as an upperbound on transfer learning accuracy for experiments using Nearest-N, Uniform, and Largest-Area base models. Similarly, Random can provide a lower bound. Table 1 shows that Nearest-3 gives the best accuracy for 8 out of 12 datasets. For two datasets, Nearest-2 performs slightly better than Nearest-3, while Uniform and Largest-Area perform best for the person and the flowers datasets. Experiment 2 To capture the performance of pseudo-labeled vs. human-labeled datasets, we define in the usual way the relative error of transfer learning accuracy between a pseudo-labeled dataset, i, and the Vanilla base model, v, as: Figures 5a and 5b show plots of Err i for 12 different target datasets. The average value of Err i is 17.2%, with minimum and maximum being 6.1% and 26.3%. Thus, using base models trained with automatically generated labels, transfer learning accuracy is on average only 17.2% worse when compared to base models trained using human labeled images. Further, the error shrinks when the divergence increases. This implies that when the base dataset is far away in feature space, the transferability of representations is less sensitive to noise in the labels. DISPLAYFORM0 We have shown that generation of content-aware pseudolabels can provide transfer performance approaching that of human labels, and that models trained on psuedolabels can be used as source models for transfer learning. The automated approach presented here suggests that the internal representations of content models trained on specialized datasets contain some descriptive features of those datasets. By treating each of these specialized representations as a "word" in a longer "sentence" that describes a category of images, we can create labels such as a "music-weapon-person" to describe a suit of armor, or a "tree-animal-fungus" to describe an elephant. These rich labels capture features of these objects such as visual information about the materials they are made out of, that better describe the contents than reliance on a single label would produce. Using multiple, content-aware models to achieve greater descriptive power may be a valuable future avenue of research.

Represent this text to find a much much longer article about it with say hundreds-thousands of words.

A technique for automatically labeling large unlabeled datasets so that they can train source models for transfer learning and its experimental evaluation.

Our proposed algorithm does not use all of the unlabeled data for the training, and it rather uses them selectively.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Recent studies in attention modules have enabled higher performance in computer vision tasks by capturing global contexts and accordingly attending important features. In this paper, we propose a simple and highly parametrically efficient module named Tree-structured Attention Module (TAM) which recursively encourages neighboring channels to collaborate in order to produce a spatial attention map as an output. Unlike other attention modules which try to capture long-range dependencies at each channel, our module focuses on imposing non-linearities be- tween channels by utilizing point-wise group convolution. This module not only strengthens representational power of a model but also acts as a gate which controls signal flow. Our module allows a model to achieve higher performance in a highly parameter-efficient manner. We empirically validate the effectiveness of our module with extensive experiments on CIFAR-10/100 and SVHN datasets. With our proposed attention module employed, ResNet50 and ResNet101 models gain 2.3% and 1.2% accuracy improvement with less than 1.5% parameter over- head. Our PyTorch implementation code is publicly available. Advancements in attention modules have boosted up the performance where they are employed over broad fields in deep learning such as machine translation, image generation, image and video classification, object detection, segmentation, etc (; a; b; c; ; ;). In the fields of computer vision tasks, numerous attention modules have been proposed in a way that one can attach it to a backbone network obtaining an efficient trade-off between additional parameters of the attached attention module and the model's performance. SENet (b) encodes global spatial information using global average pooling and captures channel-wise dependencies using two fully-connected layers over the previously encoded values at each channel. Input feature maps of the SE module are recalibrated with output values corresponding to each channel after applying a sigmoid activation function to produce output feature maps of the module. In this manner, the model can distinguish which channels to attend than others. GENet (a) shows simply gathering spatial information with depth-wise strided convolution and redistributing each gathered value across all positions with nearest neighbor upsampling can significantly help a network to understand global feature context. NLNet aggregates query-specific global context and adds values to each corresponding channel. GCNet simplifies NLNet in a computationally efficient way using the fact that a non-local block used in the NLNet tends to produce attention map independent of query position. BAM efficiently enhances backbone networks by placing attention modules in bottleneck regions, which requires few increase in both parameters and computation. CBAM incorporates channel and spatial attentions and employs a max descriptor as well as an average descriptor for more precise attention. It is clear that proposed modules in aforementioned studies have brought remarkable , most of their main focus has been on how to capture long-range dependencies across spatial dimension. That is, they mainly focus on contextual modeling rather than capturing inter-channel relations both of which are regarded indispensable for an attention module as depicted in. In this work, we propose a module which strengthens model representational power by imposing nonlinearities between neighboring channels in a parameter efficient manner. While this work deviates Figure 1: An instance of our proposed module with group size 2. f p denotes a point-wise convolution followed by an activation function which combines neighboring channels. C m n denotes a n-th channel after applying m point-wise group convolutions to the input feature map. One channel attention map followed by a sigmoid σ is produced. A color refers to information a channel contains. The repetition of point-wise group convolution yields a tree-like structure. from the current trend of capturing long-range dependencies within spatial dimension, we argue that taking consideration of inter-channel relations can also achieve highly competitive even without capturing any kind of spatial dependencies. Our module incorporates all channels to produce a single meaningful attention map as an output whereas most previous studies restore the input channel dimension in order to attend important channels and to suppress less meaningful ones. For this, we repeatedly apply light-weight point-wise group convolution with a fixed group size to an input feature map until the number of channels becomes one. While the increased parameters and computation are almost negligible, we find this simple design remarkably boosts up the performance of various backbone networks. As we see in section 3, the module performance is highly competitive to other attention modules and enhances baseline models with few additional parameter overhead. This gives one a clue to another notion for attention deviating from the current trend of taking global context. Our contributions are two-fold: • we propose Tree-structured Attention Module (TAM) which allows the network to learn inter-channel relationships using light-weight point-wise group convolutions. This treestructure enables convolution filters in the mid and later phase of a network to have a higher variance so that it can have more presentation power. • by proving validity of TAM with extensive experiments, we highlight the potential importance of inter-channel relations. The Tree-structured Attention Module is a simple and light-weight module which helps the network to learn inter-channel relations by repeating a pairing function f p. Given an input feature map X and intermediate feature map Y of the module, where N, H, W, g means the number of input channels, height, width, and group size, respectively., then the output Z of the module can be described by where for m repetitive operations of f p with a fixed group size g. To ensure the module to look through all Table 1. channels we repeat f p until the number of remaining channels is not divisible by the factor g. Then, we apply 1x1 convolution followed by sigmoid function in order to produce one channel output. We replicate the output through channel axis to restore the original input dimension in order to recalibrate the input feature map by element-wise multiplication. Figure 1 shows an instance of our model with g = 2. To permit a parametric efficiency, we adopt point-wise group convolution for f p. Formally, where and δ refers to point-wise convolution and non-linear activation. For activation we use channel-wise PReLU function (b). Unlike an ordinary point-wise convolution which increases the model parameter and computation proportional to N input × N output, a point-wise group convolution increases them proportional to N input × N output /n groups where n groups refers to the number of groups, while allowing a model to learn channel dependencies. With a larger n groups, that is a smaller g, increased parameters and computation decrease rapidly. As can be see in section 3, the increased parameters and computations are almost negligible throughout the experiments while offering huge accuracy gains over baselines. To inspect the validity and efficiency of our module, we conduct extensive experiments on CIFAR-10/100 and SVHN 1. We attach our module to the last part of every residual block before addition. For fair comparisons, we fix training configurations and reimplement all codes using PyTorch framework . We first explore the model performances by varying the group size g. We experiment with ResNet50 model with group sizes = {2, 4, 8, 16, 32, 64} on CIFAR-100 dataset as shown in Figure 2. Both top-1/5 errors go down with a smaller group size reaching the lowest point of 22.42/5.94% at g = 2. For a reference, the original ResNet101 produces 23.17%. Even a simple repetition of point-wise group convolution offers large gains as it permits the model to learn channel dependencies. Note that we do not apply any kind of context modeling within the module. Even though we see a favorable tendency that an error reduces with a lower g, it severely degrades the model performance with a large group size. In case of g = 64, it produces poorer than the baseline model for both top-1/5. This indicates that for the module to learn channel dependencies, it is important to keep the number of channels interacting each other at a time sufficiently small. It is intuitively understandable that the model can finely process channel information with more nonlinearities with a smaller group size as the number of non-linear operations increases logarithmically anti-proportional to g. For simplicity, we hereafter denote TAM as our module with g = 2. Table 1: Top-1/5 errors based on various backbone networks on CIFAR-10/100 data set. For CIFAR-10, we only report top-1 error as we don't see much difference between performances. The value in parenthesis denotes the increased parameters over the baseline network. RESNET To prove generality of the module, we experiment with other various architectures which are ResNet18, ResNet50, ResNext50(8×64d) , and WideResNet(WRN)16-8 . We follow an ordinary preprocessing, which is zero-padding 4 pixels around input pixels, normalizing each channel with mean and standard deviation from training set, allowing horizontal flip by half chance and randomly crop 32×32 pixels. We initialize weights following He et al. (2015a). We use 128 mini-batch size and SGD with 0.9 momentum and 10 −4 weight decay. It is worth noticing that we allow the decay affects to PReLU weights as we found it works better to do so in preliminary experiments. We train models for 300 epochs and set an initial learning rate 0.1 decaying by 10 at epoch 150 and 225. Table 1 shows the . Our model outperforms the baseline model and shows competitive to ones with different attention modules. Unlike other modules which a parametric overhead percentage largely increases according to a depth of a backbone architecture, one from our module is negligible regardless of a depth. Interestingly, our module shows better performance over other attention modules with less than 1.5% parametric overhead on ResNet50 and ResNext50 where over 10% increase are made in other modules. This supports the effectiveness of capturing inter-channel relations. The Street View House Numbers(SVHN) dataset is composed of 73,257 training and 26,032 test images of 32-by-32 RGB pixels and corresponding labels. We don't do any preprocessing except for normalizing images to have range. We set the identical hyperparameters with CIFAR-10/100 training except that we train the model for 160 epochs and decay the learning rate at epoch. Table 2 shows the . We find other modules sometimes work more poorly than baseline network. We surmise this is because of its non-negligible increase in a parameter number as it may cause overfitting for a relatively small dataset. For a spatial attention activation to be discriminable, it should have a large variance across spatial dimension. For this reason, we derive mean and variance of each activation using ResNet50 on CIFAR-100 test dataset from every attention module. Here "stage" means a stage where feature maps of a fixed spatial size are processed. Stage number starts with 1 and is increased whenever the spatial pooling is applied. Figure 3 shows the variance versus mean scatter plot. Unexpectedly, all attention maps from TAM have a zero variance which means they do not contain spatial discriminability and they are independent to input feature map. Rather, it depends only on the location where the module is placed and controls the input signal accordingly. Figure 4a shows how TAM manages the signal flow depending on its location. Overall, a trend is found that TAM suppresses signals flowing through earlier blocks and gradually allows signals. As abstract features which contains a larger, thus more meaningful context are delivered in the mid and later phase of a network, TAM preserves the information to some extent for the later part of the network. Also, TAM allows relatively more signals at the first block of every stage where a spatial pooling is applied, that is, at block 4, 8, and 14. It can be explained as TAM considers for a loss caused by a spatial pooling. To investigate how these activation values affect a performance, we conduct a lesion study as shown in Figure 4b. We omit each attention at a time and measure the top-1/5 accuracy. Where an attention is omitted, the flow is fully transmitted. Interestingly, when we disable the first attention within a stage rapid decrease is observed. This is counter-intuitive but explains why TAM does not fully allow the input signal but just to a certain degree. To further investigate the working mechanism of TAM, we eliminate non-linearities within the module. We first average weights, i.e. negative slopes, of PReLU per layer within the attention module. We sort them out from one closer to 1 to one far from 1. We gradually replace the weights with 1 which eliminates non-linearities of the layers. Figure 5a shows the . Surprisingly, the performance remains almost constant over eliminating ratio. Unlike conventional thoughts that nonlinearities are an indispensable part of a neural network, TAM, once training is finished, does not depend on activation functions. To answer how TAM offers benefits to a network, we plot average variances of each convolutional filter weight within residual blocks from the trained ResNet50, ResNet50-SE, ResNet50-CBAM and ResNet50-TAM. Figure 5b shows this. Compared to the baseline and ones with other attention modules, the average variances have higher values except for the last part of the network. We believe this is because it is more important to pass selected meaningful features to the classifier. This indicates that TAM gives more choices of various filters in the early and mid phase and focuses on a few important filter weights in the last stage by allowing learning inter-channel relationships. As earlier noted in Hu et al. (2018a; b), TAM also shows faster convergence speed during training. Figure 6 presents this. We attribute this to its light-weight property of TAM. As TAM does not require many additional weights which may require more time to adjust while helping the network to understand inter-channel relations. In summary, • TAM behaves like a static gate which only depends on its location and controls signal flow. • TAM allows the model to have various filters in the early and mid phase of the network and narrows the choices the choices down to important ones in the last stage. • TAM helps a network to converge fast. In this paper, we propose Tree-structure Attention module which enables a network to learn interchannel relationships which deviates from the current trend of capturing long-range dependencies in attention literature. TAM adopts light-weight point-wise group convolutions to allow communication between neighboring channels. Once trained, TAM acts as a static gate controlling signal at a certain location which does not depend on input feature but on the location where it is placed. Moreover, TAM permits higher variances in filter weights in the early and mid phase and helps the filters to focus on important ones at the last phase before classifier. On top of that, TAM produces favorable performance gains with only a few additional parameters to a backbone network. These advantages of TAM shed a light on a new way to attend features.

Represent the short summary for finding the corresponding scientific article (long)

Our paper proposes an attention module which captures inter-channel relationships and offers large performance gains.

represent the short summary for finding the corresponding scientific article (long)

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.