Question: Represent the scientific article to find a short TLDR:

Question: Deep learning has yielded state-of-the-art performance on many natural language processing tasks including named entity recognition (NER). However, this typically requires large amounts of labeled data. In this work, we demonstrate that the amount of labeled training data can be drastically reduced when deep learning is combined with active learning. While active learning is sample-efficient, it can be computationally expensive since it requires iterative retraining. To speed this up, we introduce a lightweight architecture for NER, viz., the CNN-CNN-LSTM model consisting of convolutional character and word encoders and a long short term memory (LSTM) tag decoder. The model achieves nearly state-of-the-art performance on standard datasets for the task while being computationally much more efficient than best performing models. We carry out incremental active learning, during the training process, and are able to nearly match state-of-the-art performance with just 25\% of the original training data. Over the past few years, papers applying deep neural networks (DNNs) to the task of named entity recognition (NER) have successively advanced the state-of-the-art BID7 BID17 BID24 BID6 BID48. However, under typical training procedures, the advantages of deep learning diminish when working with small datasets. For instance, on the OntoNotes-5.0 English dataset, whose training set contains 1,088,503 words, a DNN model outperforms the best shallow model by 2.24% as measured by F1 score BID6. However, on the comparatively small CoNLL-2003 English dataset, whose training set contains 203,621 words, the best DNN model enjoys only a 0.4% advantage. To make deep learning more broadly useful, it is crucial to reduce its training data requirements. Generally, the annotation budget for labeling is far less than the total number of available (unlabeled) samples. For NER, getting unlabeled data is practically free, owing to the large amount of content that can be efficiently scraped off the web. On the other hand, it is especially expensive to obtain annotated data for NER since it requires multi-stage pipelines with sufficiently well-trained annotators BID19 BID5. In such cases, active learning offers a promising approach to efficiently select the set of samples for labeling. Unlike the supervised learning setting, in which examples are drawn and labeled at random, in the active learning setting, the algorithm can choose which examples to label. Active learning aims to select a more informative set of examples in contrast to supervised learning, which is trained on a set of randomly drawn examples. A central challenge in active learning is to determine what constitutes more informative and how the active learner can recognize this based on what it already knows. The most common approach is uncertainty sampling, in which the model preferentially selects examples for which it's current prediction is least confident. Other approaches include representativeness-based sampling where the model selects a diverse set that represent the input space without adding too much redundancy. In this work, we investigate practical active learning algorithms on lightweight deep neural network architectures for the NER task. Training with active learning proceeds in multiple rounds. Traditional active learning schemes are expensive for deep learning since they require complete retraining of the classifier with newly annotated samples after each round. In our experiments, for example, the model must be retrained 54 times. Because retraining from scratch is not practical, we instead carry out incremental training with each batch of new labels: we mix newly annotated samples with the older ones, and update our neural network weights for a small number of epochs, before querying for labels in a new round. This modification drastically reduces the computational requirements of active learning methods and makes it practical to deploy them. We further reduce the computational complexity by selecting a lightweight architecture for NER. We propose a new CNN-CNN-LSTM architecture for NER consisting of a convolutional character-level encoder, convolutional word-level encoder, and long short term memory (LSTM) tag decoder. This model handles out-of-vocabulary words gracefully and, owing to the greater reliance on convolutions (vs recurrent layers), trains much faster than other deep models while performing competitively. We introduce a simple uncertainty-based heuristic for active learning with sequence tagging. Our model selects those sentences for which the length-normalized log probability of the current prediction is the lowest. Our experiments with the Onto-Notes 5.0 English and Chinese datasets demonstrate comparable to the Bayesian active learning by disagreement method. Moreover our heuristic is faster to compute since it does not require multiple forward passes. On the OntoNotes-5.0 English dataset, our approach matches 99% of the F1 score achieved by the best deep models trained in a standard, supervised fashion despite using only a 24.9% of the data. On the OntoNotes-5.0 Chinese dataset, we match 99% performance with only 30.1% of the data. Thus, we are able to achieve state of art performance with drastically lower number of samples. The use of DNNs for NER was pioneered by BID7, who proposed an architecture based on temporal convolutional neural networks (CNNs) over the sequence of words. Since then, many papers have proposed improvements to this architecture. BID17 proposed to replace CNN encoder in BID7 with bidirectional LSTM encoder, while BID24 and BID6 introduced hierarchy in the architecture by replacing hand-engineered character-level features in prior works with additional bidirectional LSTM and CNN encoders respectively. In other related work, BID30 and BID34 pioneered the use of recurrent neural networks (RNNs) for decoding tags. However, most recent competitive approaches rely upon CRFs as decoder BID24 BID6 BID48. In this work, we demonstrate that LSTM decoders outperform CRF decoders and are faster to train when the number of entity types is large. While learning-theoretic properties of active learning algorithms are wellstudied BID10 BID3 BID1 BID47, classic algorithms and guarantees cannot be generalized to DNNs, which are currently are the state-of-the-art techniques for NER. Owing to the limitations of current theoretical analysis, more practical active learning applications employ a range of heuristic procedures for selecting examples to query. For example, BID44 suggests a margin-based selection criteria, while BID39 while BID40 combines multiple criteria for NLP tasks. BID8 explores the application of least confidence criterion for linear CRF models on sequence prediction tasks. For a more comprehensive review of the literature, we refer to BID38 and BID35. While DNNs have achieved impressive empirical across diverse applications BID22 BID29, active learning approaches for these models have yet to be well studied, and most current work addresses image classification. BID45 claims to be the first to study active learning for image classification with CNNs and proposes methods based on uncertainty-based sampling, while and BID18 show that sampling based on a Bayesian uncertainty measure can be more advantageous. In one related paper, CNNs. However, to our knowledge, prior to this work, deep active learning for sequence tagging tasks, which often have structured output space and variable-length input, has not been studied. Most active learning methods require frequent retraining of the model as new labeled examples are acquired. Therefore, it is crucial that the model can be efficiently retrained. On the other hand, we would still like to reach the level of performance rivaling state-of-the-art DNNs. To accomplish this, we first identify that many DNN architectures for NER can be decomposed into three components: 1) the character-level encoder, which extracts features for each word from characters, 2) the word-level encoder which extracts features from the surrounding sequence of words, and 3) the tag decoder, which induces a probability distribution over any sequences of tags. This conceptual framework allows us to view a variety of DNNs in a unified perspective; see Table 1.Owing to the superior computational efficiency of CNNs over LSTMs, we propose a lightweight neural network architecture for NER, which we name CNN-CNN-LSTM and describe below. We represent each input sentence as follows; First, special [BOS] and [EOS] tokens are added at the beginning and the end of the sentence, respectively. In order to batch the computation of multiple sentences, sentences with similar length are grouped together into buckets, and [PAD] tokens are added at the end of sentences to make their lengths uniform inside of the bucket. We follow an analogous procedure to represent the characters in each word. For example, the sentence'Kate lives on Mars' is formatted as shown in TAB2. The formatted sentence is denoted as {x ij}, where x ij is the one-hot encoding of the j-th character in the i-th word. Character-Level Encoder For each word i, we use CNNs BID25 to extract characterlevel features w char i FIG0 ). While LSTM recurrent neural network BID16 slightly outperforms CNN as a character-level encoder, the improvement is not statistically significant and the computational cost of LSTM encoders is much higher than CNNs (see Section 5, also BID37 for detailed analysis).We apply ReLU nonlinearities BID32 and dropout BID41 between CNN layers, and include a residual connection between input and output of each layer BID14. So that our representation of the word is of fixed length, we apply max-pooling on the outputs of the topmost layer of the character-level encoder BID20. DISPLAYFORM0 x22 x23 x24 x25 x26 x27 Word-Level Encoder To complete our representation of each word, we concatenate its characterlevel features with w emb i, a latent word embedding corresponding to that word: DISPLAYFORM1 DISPLAYFORM2 We initialize the latent word embeddings with with word2vec training BID31 and then update the embeddings over the course of training. In order to generalize to words unseen in the training data, we replace each word with a special [UNK] (unknown) token with 50% probability during training, an approach that resembles the word-drop method due to BID24.Given the sequence of word-level input features w DISPLAYFORM3 Enc n for each word position in the sentence using a CNN. In FIG3, we depict an instance of our architecture with two convolutional layers and kernels of width 3. We concatenate the representation at the l-th convolutional layer h LSTM RNNs can also perform word-level encoding BID17, and models with LSTM word-level encoding give a slight (but not significant) boost over CNN word-level encoders in terms of F1 score (see Section 5). However, CNN word-level encoders are considerably faster BID42, which is crucial for the iterative retraining in our active learning scheme. DISPLAYFORM4 The tag decoder induces a probability distribution over sequences of tags, conditioned on the word-level encoder features: P y 2, y 3,..., y n−1 | h Enc i 1. Chain CRF BID23 ) is a popular choice for tag decoder, adopted by most modern DNNs for NER: DISPLAYFORM0 where W, A, b are learnable parameters, and {·} ti refers to the t i -th coordinate of the vector. To compute the partition function of, which is required for training, usually dynamic programming is employed, and its time complexity is O(nT 2) where T is the number of entity types BID7.Alternatively, we use an LSTM RNN for the tag decoder, as depicted in FIG4. At the first time step, the [GO]-symbol is provided as y 1 to the decoder LSTM. At each time step i, the LSTM decoder computes h Dec i+1, the hidden state for decoding word i + 1, using the last tag y i, the current decoder hidden state h Dec i, and the learned representation of next word h Enc i+1. Using a softmax loss function, y i+1 is decoded; this is further fed as an input to the next time step. DISPLAYFORM1 Kate lives on Mars Since this is a locally normalized model BID0, it does not require the costly computation of partition function, and it allows us to significantly speed up training compared to using CRFs. Also, we observed that while it is computationally intractable to find the best sequence of tags with an LSTM decoder, greedily decoding tags from left to right yields the performance comparable to chain CRF decoder (see Appendix A). While the use of RNNs tag decoders has been explored BID30 BID34 BID49, we demonstrate for the first time that models using RNNs instead of CRFs for tag decoder can achieve state-of-the-art performance. See Section 5. Labeling data for NER usually requires manual annotations by human experts, which are costly to acquire at scale. Active learning seeks to ameliorate this problem by strategically choosing which examples to annotate, in the hope of getting greater performance with fewer annotations. To this end, we consider the following setup for interactively acquiring annotations. The learning process consists of multiple rounds: At the beginning of each round, the active learning algorithm chooses sentences to be annotated up to the predefined budget. After receiving annotations, we update the model parameters by training on the augmented dataset, and proceeds to the next round. We assume that the cost of annotating a sentence is proportional to the number of words in the sentence and that every word in the selected sentence must be annotated at once, i.e. we do not allow or account for partially annotated sentences. While various existing active learning strategies suit this setup BID38, we explore the uncertainty sampling strategy. With the uncertainty-based sampling strategy , we rank the unlabeled examples according to the current model's uncertainty in its prediction of the corresponding labels. We consider three ranking methods, each of which can be easily implemented in the CNN-CNN-LSTM model or most other deep neural approaches to NER.Least Confidence (LC) BID8 proposed to sort examples in ascending order according to the probability assigned by the model to the most likely sequence of tags: DISPLAYFORM0 Exactly computing requires identifying the most likely sequence of tags according to the LSTM decoder. Because determining the most likely sequence is intractable, we approximate the score by using the probability assigned to the greedily decoded sequence. Maximum Normalized Log-Probability (MNLP): Preliminary analysis revealed that the LC method disproportionately selects longer sentences. Note that sorting unlabeled examples in descending order by is equivalent to sorting in ascending order by the following scores: DISPLAYFORM1 Since contains summation over words, LC method naturally favors longer sentences. Because longer sentences requires more labor for annotation, we find this undesirable, and propose to normalize as follows, which we call Maximum Normalized Log-Probability method: DISPLAYFORM2 Bayesian Active Learning by Disagreement (BALD): We also consider sampling according to the measure of uncertainty proposed by. Observing a correspondence between dropout BID41 and deep Gaussian processes BID9, they propose that the variability of the predictions over successive forward passes due to dropout can be interpreted as a measure of the model's uncertainty BID11. Denote P 1, P 2,... P M as models ing from applying M independently sampled dropout masks. One measure of our uncertainty on the ith word is f i, the fraction of models which disagreed with the most popular choice: DISPLAYFORM3 where |·| denotes cardinality of a set. We normalize this by the number of words as 1 n n j=1 f j, In this paper, we draw M = 100 independent dropout masks. Other Sampling Strategies. Consider that the confidence of the model can help to distinguish between hard and easy samples. Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proficient. But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information. We also might worry that a purely uncertainty-based approach would oversample outliers. Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset. Following, we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efficient streaming algorithm adapted to use a constraint suitable to the NER task. Our approach to representativeness-based sampling proceeds as follows: Denote X as the set of all samples, and X L, X U representing the set of labeled and unlabeled samples respectively. For an unlabeled set S ∈ X U, the utility f w is defined as the summation of marginal utility gain over all unlabeled points, weighted by their uncertainty. More formally, DISPLAYFORM4 where US(i) is the uncertainty score on example i. In order to find a good set S with high f w value, we exploit the submodularity of the function, and use an online algorithm under knapsack constraint. More details of this method can be found in the supplementary material (Appendix C). In our experiments, this approach fails to match the uncertainty-based heuristics or to improve upon them when used in combination. Nevertheless, we describe the algorithm and include the negative for their scientific value. FORMULA5 ). Dropout probabilities are all set as 0.5. We use structured skip-gram model BID28 trained on Gigawords-English corpus BID13, which showed a good boost over vanilla skip-gram model BID31 we do not report here. We use vanilla stochastic gradient descent, since it is commonly reported in the named entity recognition literature that this outperforms more sophisticated methods at convergence BID24 BID6. We uniformly set the step size as 0.001 and the batch size as 128. When using LSTMs for the tag decoder, for inference, we only use greedy decoding; beam search gave very marginal improvement in our initial experiments. We repeat each experiment four times, and report mean and standard deviation. In terms of measuring the training speed of our models, we compute the time spent for one iteration of training on the dataset, with eight K80 GPUs in p2.8xlarge on Amazon Web Services 2. TAB5 show the comparison between our model and other best performing models. LSTM tag decoder shows performance comparable to CRF tag decoder, and it works better than the CRF decoder when used with CNN encoder; compare CNN-CNN-LSTM vs. CNN-CNN-CRF on both tables. On the CoNLL-2003 English dataset which has only four entity types, the training speed of CNN-CNN-LSTM and CNN-CNN-CRF are comparable. However, on the OntoNotes 5.0 English dataset which has 18 entity types, the training speed of CNN-CNN-LSTM is twice faster than CNN-CNN-CRF because the time complexity of computing the partition function for CRF is quadratic to the number of entity types. CNN-CNN-LSTM is also 44% faster than CNN-LSTM-LSTM on OntoNotes, showing the advantage of CNN over LSTM as word encoder; on CoNLL-2003, sentences tend to be shorter and this advantage was not clearly seen; its median number of words in sentences is 12 opposed 17 of OntoNotes. Compared to the CNN-LSTM-CRF model, which is considered as a state-of-the-art model in terms of performance BID6 BID42, CNN-CNN-LSTM provides four times speedup in terms of the training speed, and achieves comparatively high performance measured by F1 score. We use OntoNotes-5.0 English and Chinese data BID36 for our experiments. The training datasets contain 1,088,503 words and 756,063 words respectively. State-of-the-art models trained on the full training sets achieve F1 scores of 86. 86 and 75.63 (our CNN-CNN-LSTM) on the test sets. We empirically compare selection algorithms proposed in Section 4, as well as uniformly random baseline (RAND). All algorithms start with an identical 1% of original training data and a randomly initialized model. In each round, every algorithm chooses sentences from the rest of the training data until 20,000 words have been selected, adding this data to Figure 5: Genre distribution of top 1,000 sentences chosen by an active learning algorithm FIG5 shows the . All active learning algorithms perform significantly better than the random baseline. Among active learners, MNLP and BALD slightly outperformed traditional LC in early rounds. Note that MNLP is computationally more efficient than BALD, since it only requires a single forward pass on the unlabeled dataset to compute uncertainty scores, whereas BALD requires multiple forward passes. Impressively, active learning algorithms achieve 99% performance of the best deep model trained on full data using only 24.9% of the training data on the English dataset and 30.1% on Chinese. Also, 12.0% and 16.9% of training data were enough for deep active learning algorithms to surpass the performance of the shallow models from BID36 trained on the full training data. We repeated the experiment eight times and confirmed that the trend is replicated across multiple runs; see Appendix B for details. Detection of under-explored genres To better understand how active learning algorithms choose informative examples, we designed the following experiment. The OntoNotes datasets consist of six genres: broadcast conversation (bc), braodcast news (bn), magazine genre (mz), newswire (nw), telephone conversation (tc), weblogs (wb). We created three training datasets: half-data, which contains random 50% of the original training data, nw-data, which contains sentences only from newswire (51.5% of words in the original data), and no-nw-data, which is the complement of nwdata. Then, we trained CNN-CNN-LSTM model on each dataset. The model trained on half-data achieved 85.10 F1, significantly outperforming others trained on biased datasets (no-nw-data: 81.49, nw-only-data: 82.08). This showed the importance of good genre coverage in training data. Then, we analyzed the genre distribution of 1,000 sentences MNLP chose for each model (see Figure 5). For no-nw-data, the algorithm chose many more newswire (nw) sentences than it did for unbiased half-data (367 vs. 217). On the other hand, it undersampled newswire sentences for nw-only-data and increased the proportion of broadcast news and telephone conversation, which are genres distant from newswire. Impressively, although we did not provide the genre of sentences to the algorithm, it was able to automatically detect underexplored genres. One potential concern when decoding with an LSTM decoder as compared to using a CRF decoder is that finding the best sequence of labels that maximizes the probability P t 2, t 3,..., t n−1 | h Enc i is computationally intractable. In practice, however, we find that simple greedy decoding, i.e., beam search with beam size 1, works surprisingly well. TAB8 shows how changing the beam size of decoder affects the performance of the model. It can be seen that the performance of the model changes very little with respect to the beam size. Beam search with size 2 is marginally better than greedy decoding, and further increasing the beam size did not help at all. Moreover, we note that while it may be computationally efficient to pick the most likely tag sequence given a CRF encoder, the LSTM decoder may give more accurate predictions, owing to it's greater representational power and ability to model long-range dependencies. Thus even if we do not always choose the most probable tag sequence from the LSTM, we can still outperform the CRF (as our experiments demonstrate). In order to understand the variability of learning curves in FIG5 across experiments, we repeated the active learning experiment on OntoNotes-5.0 English eight times, each of which started with different initial dataset chosen randomly. FIG6 shows the in first nine rounds of labeled data acquisition. While MNLP, LC and BALD are all competitive against each other, there is a noticeable trend that MNLP and BALD outperforms LC in early rounds of data acquisition. Consider that the confidence of the model can help to distinguish between hard and easy samples. Thus, sampling examples where the model is uncertain might save us from sampling too heavily from regions where the model is already proficient. But intuitively, when we query a batch of examples in each round, we might want to guard against querying examples that are too similar to each other, thus collecting redundant information. We also might worry that a purely uncertainty-based approach would oversample outliers. Thus we explore techniques to guard against these problems by selecting a set of samples that is representative of the dataset. Following, we express the problem of maximizing representativeness of a labeled set as a submodular optimization problem, and provide an efficient streaming algorithm adapted to use a constraint suitable to the NER task. We also provide some with theoretical guarantees. Submodular utility function In order to reason about the similarity between samples, we first embed each sample i into a fixed-dimensional euclidean space as a vector x i. We consider two embedding methods: 1) the average of pre-trained word embeddings, for p = 1, 2 which corresponds to closeness in L 1 and L 2 distance, and w(i, j) = 1 + xi·xj xi · xj, which corresponds to cosine similarity. Now, we formally define the utility function for labeling new samples. Denote X as the set of all samples which can be partitioned into two disjoint sets X L, X U representing labeled and unlabeled samples, respectively. Let S ⊆ X U be a subset of unlabeled samples, then, the utility of labeling the set is defined as follows: DISPLAYFORM0 where the function measures incremental gain of similarity between the labeled set and the rest. Given such utility function f (·), choosing a set S that maximizes the function within the budget can be seen as a monotone submodular maximization problem under a knapsack constraint BID21: max DISPLAYFORM1 where k(S) is the budget for the sample set S, and K is the total budget within each round. Note that we need to consider the knapsack constraint instead of the cardinality constraint used in the prior work, because the entire sentence needs to be labeled once selected and sequences of length confer different labeling costs. Combination with uncertainty sampling Representation-based sampling can benefit from uncertainty-based sampling in the following two ways. First, we can re-weight each sample in the utility function to reflect current model's uncertainty on it: DISPLAYFORM2 Algorithm 1 Representativeness-based Sampling DISPLAYFORM3 while Test score of M less than th do 3:Rank X U according to Sec. 4, X U = top samples S within budget t · K. Set f according to or FORMULA14 5: DISPLAYFORM0 Train M with X L. where US(i) is the uncertainty score on example i. Second, even with the state-of-the-art submodular optimization algorithms, the optimization problem can be computationally intractable. To improve the computational efficiency, we restrict the set of unlabeled examples to top samples from uncertainty sampling within budget t · K, where t is a multiplication factor we set as 4 in our experiments. Streaming algorithm for sample selection Even with the reduction of candidates with uncertainty sampling, is still a computationally challenging problem and requires careful design of optimization algorithms. Suppose l is the number of samples we need to consider. In the simplistic case in which all the samples have the same length and thus the knapsack constraint degenerates to the cardinality constraint, the greedy algorithm BID33 has an (1 − 1/e)-approximation guarantee. However, it requires calculating the utility function O(l 2 n) times, where n is the number of unlabeled samples. In practice, both l and n are large. Alternatively, we can use lazy evaluation to decrease the computation complexity to O(ln) BID26, but it requires an additional hyperparameter to be chosen in advance. Instead of greedily selecting elements in an offline fashion, we adopt the two-pass streaming algorithm of BID2, whose complexity is O(ln) 3, and generalize it to the knapsack constraint (shown in Alg. 2). In the first pass, we calculate the maximum function value of a single element normalized by its weight, which gives an estimate of the optimal value. In the second pass, we create O(1 log K) buckets and greedily update each of the bucket according to: DISPLAYFORM1 where each bucket has a different value v, and ∆ g (e|S v):= g({e} ∪ S v ) − g(S v) is the marginal improvement of submodular function g when adding element e to set S v. The whole pipeline of the active learning algorithm is shown in Alg. 1. The algorithm gives the following guarantee, which is proven in Appendix. Theorem 1. Alg. 2 gives a(1−)(1−δ) 2 -approximation guarantee for, where δ = max e∈S k({e})/K.Proof sketch: The criterion we use guarantees that each update we make is reasonably good. The set S v stops updating when either the current budget is almost K, or any sample in the stream after we reach S v does not provide enough marginal improvement. While it is easy to give guarantees when the budget is exhausted, it is unlikely to happen; we use a difference expression between current set S v and the optimal set, and prove the gap between the two is under control. In a practical label acquisition process, the budget we set for each round is usually much larger than the length of the longest sentence in the unlabeled set, making δ negligible. In our experiments, δ was around 0.01.

We introduce a lightweight architecture for named entity recognition and carry out incremental active learning, which is able to match state-of-the-art performance with just 25% of the original training data.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the article for finding a single-sentence TLDR:

Question: Network quantization is a model compression and acceleration technique that has become essential to neural network deployment. Most quantization methods per- form fine-tuning on a pretrained network, but this sometimes in a large loss in accuracy compared to the original network. We introduce a new technique to train quantization-friendly networks, which can be directly converted to an accurate quantized network without the need for additional fine-tuning. Our technique allows quantizing the weights and activations of all network layers down to 4 bits, achieving high efficiency and facilitating deployment in practical settings. Com- pared to other fully quantized networks operating at 4 bits, we show substantial improvements in accuracy, for example 66.68% top-1 accuracy on ImageNet using ResNet-18, compared to the previous state-of-the-art accuracy of 61.52% and a full precision reference accuracy of 69.76%. We performed a thorough set of experiments to test the efficacy of our method and also conducted ablation studies on different aspects of the method and techniques to improve training stability and accuracy. Our codebase and trained models are available on GitHub. Neural network quantization is a technique to reduce the size of deep networks and to bypass computationally and energetically expensive floating-point arithmetic operations in favor of efficient integer arithmetic on quantized versions of model weights and activations. Network quantization has been the focus of intensive research in recent years (; ; ; ; ; ; ;), with most works belonging to one of two categories. The first line of work quantizes parts of the network while leaving a portion of its operations, e.g. computations in the first and last network layers in floating point. While such networks can be highly efficient, using bitwidths down to 5 or 4 bits with minimal loss in network accuracy , they may be difficult to deploy in certain practical settings, due to the complexity of extra floating point hardware needed to execute the non-quantized portions of the network. Another line of work aims for ease of real world deployment by quantizing the entire network, including all weights and activations in all convolutional and fully connected layers; we term this scheme strict quantization. Maintaining accuracy under strict quantization is considerably more challenging. While nearly lossless 8-bit strictly quantized networks have been proposed , to date state-of-the-art 4 bit networks incur large losses in accuracy compared to full precision reference models. For example, the strict 4-bit ResNet-18 model in has 61.52% accuracy, compared to 69.76% for the full precision model, while the strict 4-bit MobileNet-v2 model in has 62.00% accuracy, compared to 71.88% accuracy in full precision. To understand the difficulty of training accurate low-bitwidth strictly quantized networks, consider a common training procedure which begins with a pre-trained network, quantizes the model, then applies fine-tuning using straight-through estimators (STE) for gradient updates until the model achieves sufficient quantized accuracy. This process faces two problems. First, as the pre-trained model was not initially trained with the task of being subsequently quantized in mind, it may not be "quantization-friendly". That is, the fine-tuning process may need to make substantial changes to the initial model in order to transform it to an accurate quantized model. Second, fine-tuning a model, especially at low bitwidths, is difficult due to the lack of accurate gradient information provided Figure 1: Architecture of the proposed GQ-Net. Input x 0 follows the top and bottom paths to produce the full precision and quantized outputs x L andx L, resp. These are combined through loss functions L f and L q to form the overall loss L, which is optimized by backpropagation. For more details please refer to Section 3. by STE. In particular, fine-tuning using STE is done by updating a model represented internally with floating point values using gradients computed at the nearest quantizations of the floating point values. Thus for example, if we apply 4 bit quantization to floating point model parameters in the range, a random parameter will incur an average round-off error of 1/32, which will be incorporated into the error in the STE gradient for this parameter, leading to possibly ineffective fine-tuning. To address these problems, we propose GQ-Net, a guided quantization training algorithm. The main goal of GQ-Net is to produce an accurate and quantization-friendly full precision model, i.e. a model whose quantized version, obtained by simply rounding each full precision value to its nearest quantized point, has nearly the same accuracy as itself. To do this, we design a loss function for the model which includes two components, one to minimize error with respect to the training labels, and another component to minimize the distributional difference between the model's outputs and the outputs of the model's quantized version. This loss function has the effect of guiding the optimization process towards a model which is both accurate, by virtue of minimizing the first loss component, and which is also similar enough to its quantized version due to minimization of the second component to ensure that the quantized model is also accurate. In addition, because the first component of the loss function deals only with floating point values, it provides accurate gradient information during optimization, in contrast to STE-based optimization which uses biased gradients at rounded points, which further improves the accuracy of the quantized model. Since GQ-Net directly produces a quantized model which does not require further fine-tuning, the number of epochs required to train GQ-Net is substantially less than the total number of epochs needed to train and fine-tune a model using the traditional quantization approach, leading to significantly reduced wall-clock training time. We note that GQ-Net's technique is independent of and can be used in conjunction with other techniques for improving quantization accuracy, as we demonstrate in Section 4.3. Finally, we believe that the guided training technique we propose can also be applied to other neural network structural optimization problems such as network pruning. We implemented GQ-Net in PyTorch and our codebase and trained models are publicly available 1. We validated GQ-Net on the ImageNet classification task with the widely used ResNet-18 and compact MobileNet-v1/v2 models, and also performed a thorough set of ablation experiments to study different aspects of our technique. In terms of quantization accuracy loss compared to reference floating point models, GQ-Net strictly quantized using 4-bit weights and activations surpasses existing state-of-the-art strict methods by up to 2.7×, and also improves upon these methods even when they use higher bitwidths. In particular, 4-bit GQ-Net applied to ResNet-18 achieves 66.68% top-1 accuracy, compared to 61.52% accuracy in and a reference floating point accuracy of 69.76%, while on MobileNet-v2 GQ-Net achieves 66.15% top-1 accuracy compared to 62.00% accuracy in and a reference floating point accuracy of 71.88%. Additionally, GQ-Net achieves these using layer-wise quantization, as opposed to channel-wise quantization in , which further enhances the efficiency and practicality of the technique. Neural network quantization has been the subject of extensive investigation in recent years. Quantization can be applied to different part of neural networks, including weights, activations or gradients. , , , and other works quantized model weights to binary, ternary or multi-bit integers to reduce model size. quantized activations of object detection models for knowledge transfer. , quantized model gradients to accelerate distributed training. Another line of work quantizes both weights and activations to accelerate model inference by utilizing fix-point or integer arithmetic. These works include , , , ,, ,. A large set of methods have been proposed to improve training or fine-tuning for network quantization. Straight through estimators (STE) propagate gradients through non-differentiable operations with the identity mapping. Other training methods "soften" nondifferentiable operations to similar differentiable ones in order for gradients to pass through, then gradually anneal to piecewise continuous functions by applying stronger constraints. This line of works include , ,. Some works regard quantization as a stochastic process that produces parameterized discrete distributions, and guides training using gradients with respect to these parameters ,. Another line of works does not require fine tuning, and instead re-calibrates or modifies the original network to recover accuracy using little or even no data , ,. Several recent works have focused on quantizing all parts of a network, typically in order to support deployment using only integer arithmetic units and avoiding the cost and complexity of additional floating point units. proposed performing network inference using dynamic fixed-point arithmetic, where bitwidths for the integer and mantissa parts are determined based on a model's weight distribution.; proposed the quantization training and deployment algorithm behind the Tensorflow-Lite quantization runtime, which generates strictly quantized networks that can be easily implemented in hardware. proposed a training method for strictly quantized models based on annealing a smooth quantization function to a piecewise continuous one. There has also been recent work on using parameterized quantizers which are optimized during quantization training. introduced learnable upper bounds to control the range of quantization. proposed quantizers with a learnable basis which an be executed using fixed-point arithmetic. proposed to optimize weight scaling and quantization ranges jointly from task losses. In this section we describe the architecture of our proposed GQ-Net and then discuss components of the architecture which can be tuned to improve performance. The major components of GQ-Net include the following, and are illustrated in Figure 1: 1. An L-layer neural network h W (·) with all computations performed using full precision floating point arithmetic. Here W = {W 1, . . ., W L} denotes the parameter (weights) of the model, with W i, i ∈ 1... L being the weights in layer i and expressed in floating point. } is a set of quantizers, i.e. mappings from floating point to (scaled) integer values; the quantizers may be parameterized, and we describe how to optimize these parameters in Section 3.2. Q w i quantizes weights W i and Q a i quantizes activations in layer i. Let x 0 denote an input to h W. To construct outputĥ W,Q (x 0) of the quantized network, we proceed layer by layer. We first quantize the weights in layers i = 1,..., L asŵ i = Q w i (w i), and also quantize the input by settingx 0 = Q a 0 (x 0). we compute the quantized activationsx i in layer i iteratively for, and g i (·) denotes the nonlinearity function in layer i and * denotes convolution. Note that sinceŵ i andx i−1 are quantized,x i can be computed using integer or fixed point arithmetic. 3. Next, we construct a loss function L incorporating both the training loss L f of the full precision model h W and a loss L q capturing the difference between h W and the quantized Here ω f, ω q ∈ R are parameters capturing the relative importance of training loss versus distributional loss. In this paper, we focus on image classification networks, and thus we set L f to be the cross-entropy loss between outputs from h W and the training labels., where σ denotes the softmax function, to be the KL divergence between distributions σ(h W) and σ(ĥ W,Q) on each input. Hence, minimizing the second term in L corresponds to pushing the floating point and quantized models to behave similarly to each other. Since the weight parameters W appear in both terms in L, the two terms can give conflicting signals for updating W during the optimization of L, causing the optimization to be unstable. We discuss how to deal with this problem in Section 3.2. To train GQ-Net, we successively take mini-batches of training samples and labels and use them to compute L during the forward pass and propagate gradients with respect to W and the parameters of Q during the backward pass in order to minimize L. After L has converged sufficiently, we take the quantized weights inĥ W,Q (·) as the quantized model. We now describe how different components of GQ-Net can be optimized to improve accuracy and training stability. Weight scheduling for L f and L q Parameters ω f and ω q capture the relative importance of the cross entropy and KL divergence errors during training. A large ω f ignores the similarity between the floating point and quantized models and may in a model that is accurate but not quantization-friendly. Conversely, a large ω q ignores guidance on accuracy from the floating point model and may in similar but poorly performing floating point and quantized models. We tested different schemes for weighting L f and L q, and found that using fixed values such as ω f = ω q = 0.5 already yields better than many current methods, as discussed in Section 4. However, further experimentation showed that scheduling, i.e. dynamically modifying the values of ω f, ω q during training can produce higher accuracy than using static values. For example, consider a schedule as shown in Figure 2a, which initially sets ω f = 1, ω q = 0, then alternates between setting ω q = 1 and ω q = 0 several times. Schedules of this sort can be understood as initially favoring model accuracy so that the floating point model is driven to a high accuracy region of model space, before increasing the importance of model similarity so that a quantization-friendly model can be found in the high accuracy region. This is repeated several times, leading to increasingly more accurate full precision models whose accuracy is then transferred to the quantized model. As demonstrated in Section 4, this schedule in better performance than static ones. Reducing interference between L f and L q The loss function L includes terms in L f and L q, where the former is a function of the floating point parameters W, and the latter involves both W and the quantized versionŴ of W parameterized by θ. We discovered that directly optimizing L in reduced accuracy, which we attribute to conflicting updates to W produced by gradients from the L f and L q terms. Ideally, we would like update W to minimize L f, while independently updatingŴ to minimize L q. While this is clearly not possible due to the dependency between W andŴ, we found that a heuristic based on this idea helped improve accuracy. In particular, let x L = h W (x 0) andx L = h W,Q (x 0) be the output of the full precision and quantized networks on input x 0, so that, as in Section 3.1. During back propagation we compute L f from x L and derive ∇ x L L f as usual, and use this via the chain rule to update W. However, when computing L q from x L andx L, we treat x L as a constant tensor which does not produce any gradients, and only derive ∇x L L q and use this via the chain rule to update W. We can implement this behavior using the detach operator in PyTorch or the stop gradient operator in TensorFlow. Parameterized quantizer GQ-Net can use any type of quantizer Q(·): R → T for a discrete set T. Motivated by recent work such as PACT, we adopt layer-wise linear quantizers with learnable boundaries in GQ-Net. In particular, for each layer i, we use one weight quantizer function Q w i,θ w i (·) for all weights in the layer, and one activation quantizer function Q a i,θ a i (·) for all activations. Here, θ w i and θ a i represent learnable parameters; for expository simplicity we drop a, w and i in the following and denote all parameters by θ. θ consists of k, the quantization bitwidth, and lb and ub representing the lower and upper quantization boundaries. We use uniform quantization, which is substantially easier to implement in hardware, and set Where clamp(x, a, b) = max(a, min(x, b)), and · represents the round operator. During training, gradients propagate through the nondifferentiable · operator using the straight though estimator (STE), i.e. ∂ x ∂x = 1. Parameters lb, ub are updated by the gradients propagated from the loss function L, and thus the quantizers will learn to set the appropriate quantization boundaries to improve accuracy. Alternatingly optimizing W and θ The accuracy of the quantized model depends both on the weights W of the floating point model as well as how these are quantized using the quantizers parameterized by θ. We found that jointly optimizing W and θ in each iteration ed in unstable training and poor accuracy. Performance was substantially improved by alternatingly optimizing W and θ. That is, in each training epoch we update either the values in W or θ while freezing the values of the other set of parameters, then switch the updated and frozen sets in the next epoch. The reason alternating optimization improved training is that both W and θ affect the values of quantized weights, so that updating both simultaneously may cause suboptimal changes to quantized weights. Multi-domain batch normalization Batch normalization is a critical component in large model training. However, since GQ-Net trains a floating point and quantized model simultaneously, we need to adjust the way batch normalization is performed to achieve good accuracy. In particular, as illustrated in Figure 2, activations from the floating point and quantized models follow different distributions. Thus, normalizing them with same set of running statistics can hinder training. Instead, we regard activations in different numerical precision as being from different domains, similar to as in multi-domain transfer learning, and normalize them with separate statistics {µ f, σ f} and {µ q, σ q}: σq. This modification only introduces minor storage overhead, while it significantly improves GQ-Net's accuracy in both full-precision and quantized settings. To validate the effectiveness of GQ-Net and assess its different components, we conducted a series of comparisons and ablation studies using the ImageNet classification task. We used the ILSVRC 2012 dataset consists of 1.2M training samples and 10K validation samples from 1K categories, and evaluated our system using top-1 and top-5 validation accuracies. Network settings We used the ResNet-18, MobileNet-v1 and MobileNet-v2 architectures in the ImageNet experiments. All MobileNets used channel expansion ratio 1.0. Unlike some recent works, we did not modify the order of the Conv, BN and ReLU layers. We replaced the BatchNorm layers in these models with SyncBatchNorm, i.e. we used mini-batch statistics µ f, µ q and σ f, σ q computed from all distributed GPUs during training. Quantization settings Unless otherwise specified, all of the following experiments were conducted with parameterized linear quantizers using a bitwidth of 4. The weight and activation quantizers each have their own parameters θ = {lb, ub}, which are initialized at the iteration right before the quantization error penalty L q is enabled. Specifically, for weight quantizers, {lb, ub} are initialized by the minimum and maximum elements in the weight tensors of each layer. For activation quantizers, {lb, ub} are initialized by the upper and lower 99.9% percentile values, computed from 5 mini-batches sampled from the training set. Weights and activations of all layers were quantized, including the first and last layers. Training protocol We used the same training protocol for all architectures and quantization settings. Training was performed on 32 distributed GPUs, each with a mini-batch size of 64, and stopped after 120 epochs on the training set. Model weights W and quantization parameters θ were optimized with different optimization settings. Model weights W were randomly initialized using the Kaiming-normal scheme without using a pre-trained model, and optimized by SGD with 0.9 Nesterov momentum and 10 −4 weight decay. The learning rate warmed-up from 0.2 to 0.8 in the first 4 epochs, and decayed twice by a factor of 0.1 at epochs 60 and 90. Quantization parameters θ were optimized using Adam without weight decay, with coefficients for first and second momentums set to β 1 = 0.9 and β 2 = 0.999, and learning rate fixed to 10 −3 during the entire training process. Following standard practice, training samples were resized and randomly cropped to 224 × 224 pixels, followed by random horizontal flipping and normalization. Validation samples were centrally cropped to 224 × 224 pixels, followed by normalization. We validated the effectiveness of GQ-Net by comparing it with several other state-of-the-art quantization methods. As GQ-Net seeks to fully quantize a network and execute it using only integer operations, we perform comparisons with other strict quantization methods. The comparison baselines include from , and , indicated as White Paper, Integer-only and RelaxedQuant resp. in Table 1. The first row in the table contains full precision accuracy evaluated using reference implementations 2, while the second row contains the full precision accuracy for models trained with our training protocol. For the ResNet-18 model which is widely studied in the network compression literature, we significantly outperform the state-of-the-art strict quantization method RelaxedQuant in an equal bitwidth setting (+5.16% top-1 accuracy improvement with 4-bit weights and activations). Our 4-bit model even outperforms the comparison methods when they use higher bitwidths (+1.58% compared with 5-bit RelaxedQuant, +2.04% compared with 5-bit Integer-only). For the compact MobileNets family, our method also achieves higher accuracy using lower bitwidths. For example, compared with the White Paper method at 8-bit weights and 4-bit activations, our 4-bit W/A GQ-Net MobileNet-v2 model achieves +8.15% top-1 accuracy improvement, and our GQNet MobileNet-v1 model obtains +1.04% higher top-1 accuracy in the same setting. We also validated the effectiveness of the different components of GQ-Net discussed in Section 3.2, by progressively adding them to the vanilla quantization-friendly training protocol, and applying these protocols to the ResNet-18 architecture with its weights and activations quantized to 4-bits. For the vanilla setting, W and θ were optimized jointly at each training step, loss weights were set to ω f = ω q = 0.5 through the entire training process, gradients from L q propagated through both σ(x L) and σ(x L), and full precision and quantized activations were normalized using the same set of running statistics. For a more complete comparison, we also evaluated these settings using the full precision GQ-Net model. The numerical are given in Table 2. By alternatingly updating model weights W and quantizer parameters θ between training steps, quantization accuracy improved by +4.24% over the vanilla protocol. This indicates that although gradients ∇ W L and ∇ θ L both guide their respective parameters to minimize training loss, combining them in a training step makes the quantized weightsŴ derived from these parameters suboptimal. Dynamically adjusting the weights ω f and ω q during different parts of the training process as described in Section 3.2 improved quantization accuracy by +0.23%. This suggests that the importance of the accuracy loss L f and the distributional loss L q may not be equal at different stages of 2 Implementations of ResNet-18 and MobileNet-v2 are taken from torchvision package (v0.3.0, https://pytorch.org/docs/stable/torchvision/models.html). The accuracy of MobileNet-v1 is cited from the original paper. Table 2: Full-precision (FP) and quantized (Q) top-1 accuracy for 4-bit weights and activations in ResNet-18 on ImageNet. Components of GQ-Net are progressively applied. The first row indicates the vanilla setting. A check mark in column "Alt {W, θ}" means alternatingly optimizing W and θ. A check for "Schedule ω f, ω q " means ω f and ω q were dynamically adjusted, as described in §3.2. A check for "Detach σ(x L)" means that σ(x L) is treated as a constant and we do not propagate ∇ σ(x L) L q during the backward pass. A check for "Multi-domain BN" means that full-precision and quantized activations are normalized using separate running statistics. The full-precision reference has top-1 accuracy = 69.89.) Alt the training process. The schedule we used, alternating between short periods of ω q = 0 and longer periods of ω q = 1, suggests we should first allow the floating point model settle into a reasonably accurate state before enforcing its similarity to the quantized model. Blocking the gradient from L q caused by σ(x L) further improved quantization accuracy by +0.66%. This indicates that although the full precision logits x L and quantized logitsx L are both derived from the same set of parameters W, it is useful to heuristically separate their effects during backpropagation. Normalizing full precision and quantized activations by different sets of running statistics in the BatchNorm layers improved both full precision accuracy (+1.69%) and quantized accuracy (+0.60%). This highlights the difference in the mean and variance of the full precision and quantized activations, despite the similarity of the full precision and quantized models in terms of KL divergence. Lastly, we considered the effectiveness of using parameterized quantizers. We tested replacing the parameterized quantizers in GQ-Net with naive linear quantizers using fixed weight and activation quantization ranges set to the 99.9% percentile values derived from 5 initial training mini-batches. We found that using fixed quantizers significantly lowered accuracy, by 3.59%. We note that RelaxedQuant also uses learned quantizers, and that replacing these with fixed quantizers may also in decreased accuracy. In this paper we presented GQ-Net, a novel method for training accurate quantized neural networks. GQ-Net uses a loss function balancing full precision accuracy as well as similarity between the full precision and quantized models to guide the optimization process. By properly tuning the weights of these two factors, we obtained fully quantized networks whose accuracy significantly exceeds the state of the art. We are currently studying additional ways to adjust GQ-Net components to further improve accuracy. We are also interested in combining GQ-Net with complementary quantization techniques, and in applying similar methodologies to other neural network optimization problems.

Represent the short summary for finding the corresponding scientific article (long)

We train accurate fully quantized networks using a loss function maximizing full precision model accuracy and minimizing the difference between the full precision and quantized networks.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.