Question: Represent the scientific article to find a short TLDR

Question: The current state-of-the-art end-to-end semantic role labeling (SRL) model is a deep neural network architecture with no explicit linguistic features. However, prior work has shown that gold syntax trees can dramatically improve SRL, suggesting that neural network models could see great improvements from explicit modeling of syntax. In this work, we present linguistically-informed self-attention (LISA): a new neural network model that combines multi-head self-attention with multi-task learning across dependency parsing, part-of-speech, predicate detection and SRL. For example, syntax is incorporated by training one of the attention heads to attend to syntactic parents for each token. Our model can predict all of the above tasks, but it is also trained such that if a high-quality syntactic parse is already available, it can be beneficially injected at test time without re-training our SRL model. In experiments on the CoNLL-2005 SRL dataset LISA achieves an increase of 2.5 F1 absolute over the previous state-of-the-art on newswire with predicted predicates and more than 2.0 F1 on out-of-domain data. On ConLL-2012 English SRL we also show an improvement of more than 3.0 F1, a 13% reduction in error. Semantic role labeling (SRL) extracts a high-level representation of meaning from a sentence, labeling e.g. who did what to whom. Explicit representations of such semantic information have been shown to improve in challenging downstream tasks such as dialog systems BID63 BID14, machine reading BID8 BID65 and machine translation BID36 BID5.Though syntax was long considered an obvious prerequisite for SRL systems BID34 BID51, recently deep neural network architectures have surpassed syntacticallyinformed models BID69 BID25 BID60, achieving state-of-the art SRL performance with no explicit modeling of syntax. Still, recent work BID53 BID25 indicates that neural network models could see even higher performance gains by leveraging syntactic information rather than ignoring it. BID25 indicate that many of the errors made by a strong syntax-free neural-network on SRL are tied to certain syntactic confusions such as prepositional phrase attachment, and show that while constrained inference using a relatively low-accuracy predicted parse can provide small improvements in SRL accuracy, providing a gold-quality parse leads to very significant gains. incorporate syntax from a highquality parser BID31 using graph convolutional neural networks BID32, but like BID25 they attain only small increases over a model with no syntactic parse, and even perform worse than a syntax-free model on out-of-domain data. These works suggest that though syntax has the potential to improve neural network SRL models, we have not yet designed an architecture which maximizes the benefits of auxiliary syntactic information. In response, we propose linguistically-informed self-attention (LISA): a model which combines multi-task learning BID12 with stacked layers of multi-head self-attention BID64 trained to act as an oracle providing syntactic parses to downstream parameters tasked with predicting semantic role labels. Our model is endto-end: earlier layers are trained to predict prerequisite parts-of-speech and predicates, which are supplied to later layers for scoring. The model is trained such that, as syntactic parsing models improve, providing high-quality parses at test time can only improve its performance, allowing the model to benefit maximally from improved parsing models without requiring re-training. Unlike previous work, we encode each sentence only once, predict its predicates, part-of-speech tags and syntactic parse, then predict the semantic roles for all predicates in the sentence in parallel, leading to exceptionally fast training and decoding speeds: our model matches state-of-the art accuracy in less than one quarter the training time. In extensive experiments on the CoNLL-2005 and CoNLL-2012 datasets, we show that our linguistically-informed models consistently outperform the syntax-free state-of-the-art for SRL models with predicted predicates. On CoNLL-2005, our single model out-performs the previous state-of-the-art single model on the WSJ test set by nearly 1.5 F1 points absolute using its own predicted parses, and by 2.5 points using a stateof-the-art parse . On the challenging out-of-domain Brown test set, our model also improves over the previous state-ofthe-art by more than 2.0 F1. On CoNLL-2012, our model gains 1.4 points with its own parses and more than 3.0 points absolute over previous work: 13% reduction in error. Our single models also out-perform state-of-the-art ensembles across all datasets, up to more than 1.4 F1 over a strong fivemodel ensemble on CoNLL-2012. Our goal is to design an efficient neural network model which makes use of linguistic information as effectively as possible in order to perform endto-end SRL. LISA achieves this by combining: Multi-task learning across four related tasks; a new technique of supervising neural attention to predict syntactic dependencies; and careful conditioning of different parts of the model on gold versus predicted annotations during training. Figure 1 depicts the overall architecture of our model. To first encode rich token-level representations, our neural network model takes word embeddings as input, which are passed through stacked convolutional, feed-forward and multihead self-attention layers BID64 to efficiently produce contextually encoded token embeddings (Eqns. 1-4). We choose this combination of network components because we found it to perform better than LSTM, CNN or self-attention layers alone in terms of speed-accuracy Pareto efficiency in initial experiments. To predict semantic role labels, the contextually encoded tokens are projected to distinct predicate and role embeddings (§2.4), and each predicted predicate is scored with the sequence's role representations using a bilinear model (Eqn. 5), producing per-label scores for BIO-encoded semantic role labels for each token and each semantic frame in the sequence entirely in parallel. To incorporate syntax, one self-attention head is trained to attend to each token's syntactic parent, allowing the model to use this attention head as an oracle for syntactic dependencies. We encourage the model to use this syntactic information as much as possible by giving subsequent layers access to a gold parse oracle during training, allowing either the predicted parse attention or an externally predicted parse to be used at test time. We introduce this syntactically-informed self-attention in more detail in §2.2.We integrate part-of-speech and predicate information into earlier layers by re-purposing representations closer to the input to predict predicates and part-of-speech (POS) tags (§2.3). We simplify optimization and benefit from shared statistical strength derived from highly correlated POS and predicates by treating tagging and predicate detection as a single task, performing multi-class classification into the joint Cartesian product space of POS and predicate labels. The model is trained end-to-end by maximum likelihood using stochastic gradient descent (§2.5). The input to the network is a sequence X of T token representations x t. Each token representation is the sum of a fixed (pre-trained) and learned (randomly initialized) word embedding. In the case where we feed a predicate indicator embedding p t as input to the network, we concatenate that representation with the word embedding to give the final token embedding. h -1 Figure 1: Word embeddings are input to k CNN layers. This output is passed to a joint POS/predicate classifier and j layers of multi-head selfattention. One attention head is trained to attend to parse parents. A bilinear operation scores distinct predicate and role representations to produce BIOencoded SRL predictions. DISPLAYFORM0 These token representations are then the input to a series of width-3 stacked convolutional layers with residual connections BID24, producing contextually embedded token representations c (k) t at each layer k. We denote the kth convolutional layer as C (k). Let r(·) denote the leaky ReLU activation function BID39, and let LN (·) denote layer normalization BID2, then starting with input x t, the final CNN output is given by the recurrence: DISPLAYFORM1 We use leaky ReLU activations to avoid dead activations and vanishing gradients BID26, whereas layer normalization reduces covariate shift between layers BID28 without requiring distinct train-and testtime operations. We then feed this representation as input to a series of residual multi-head self-attention layers with feed-forward connections in the style of the encoder portion of the Transformer architecture of BID64. This architecture allows each token to observe long-distance context from the entire sentence like an LSTM, but unlike an LSTM, representations for all tokens can be computed in parallel at each layer. We first project 1 the output of the convolutional layers to a representation c (p) t that is the same size as the output of the self-attention layers and add a positional encoding vector computed as a deterministic sinusoidal function of t, following BID64. We then apply the selfattention layers to this projected representation, applying layer normalization after each residual connection. Denoting the jth self-attention layer as T (j) (·), the output of that layer s (j) t, and h as the number of attention heads at each layer, the 1 All of our linear projections include bias terms, which we omit in this exposition for the sake of clarity.following recurrence applied to initial input c DISPLAYFORM2 gives our final token representations s DISPLAYFORM3 with which we perform a weighted sum of the value vectors a value vh for each other token v to compose a new token representation for each attention head. The representations for each attention head are concatenated into a single vector a t. We feed this representation through a multi-layer perception, add it to the initial representation and apply layer normalization to give the final output of selfattention layer j: DISPLAYFORM4 2.2 Syntactically-informed self-attention Typically, neural attention mechanisms are left on their own to learn to attend to relevant inputs. Instead, we propose training the self-attention to attend to specific tokens corresponding to the syntactic structure of the sentence as a mechanism for passing linguistic knowledge to later layers. Specifically, we train with an auxiliary objective on one attention head which encourages that head to attend to each token's parent in a syntactic dependency tree. We use the attention weights a th between token t and each other token q in the sequence as the distribution over possible heads for token t: P (q = head(t) | X ) = a thq, where we define the root token as having a self-loop. This attention head thus emits a directed graph 2 where each token's head is the token to which the attention assigns the highest weight. This attention head now becomes an oracle for syntax, denoted P, providing a dependency parse to downstream layers. This model not only predicts its own dependency arcs, but allows for the injection of auxiliary parse information at test time by simply swapping out the oracle given by a th to one produced by e.g. a state-of-the-art parser. In this way, our model can benefit from improved, external parsing models without re-training. Unlike typical multi-task models, ours maintains the ability to leverage external syntactic information. Unfortunately, this parsing objective does not maximize the model's ability to use the syntactic information for predicting semantic role labels in later layers. Though one would expect model accuracy to increase significantly if injecting e.g. gold dependency arcs into the learned attention head at test time, we find that without specialized training this is not the case: Without the training described below, fixing P to gold parses at test time improves SRL F1 over predicted parses by 0.3 points, whereas the F1 increases by 7.0 when the model is trained with our technique. 3 Injecting high-accuracy predicted parses follows the same trend. We hypothesize that the model is limited by the poor representations to which it has access during early training. When training begins, the model observes randomly initialized attention rather than strong syntactic information, even in the head which will be trained to provide it with such information. Thus rather than learning to look to this head for syntax, the model learns to encode that information itself, like a model which was trained with no explicit syntax at all. Prior work BID68, has alleviated this problem by pre-training the parameters of earlier tasks before initiating the training of later tasks. However, optimization in this setting becomes computationally expensive and complicated, especially as the number of auxiliary tasks increases, and when using adaptive techniques for stochastic gradient descent such as Adam BID30.To alleviate this problem, during training we 2 In most but not all cases, the head emits a tree, but we do not currently enforce it.3 CoNLL-2012. CoNLL-2005 yields similar .clamp P to the gold parse (P G) when using its representation for later layers, while still training a th to predict syntactic heads. We find that this vastly improves the model's ability to leverage the parse information encoded in P at test time. Our approach is essentially an extension of teacher forcing BID66 to MTL. Though a large body of work suggests that, by closing the gap between observed data distributions during train and test, training on predicted rather than gold labels leads to improved test-time accuracy BID21 BID52 BID15 BID22 BID13 BID6 BID4, our simple approach works surprisingly well; we leave more advanced scheduled sampling techniques to future work. We also share the parameters of lower layers in our model to predict POS tags and predicates. Following He et al. FORMULA1, we focus on the end-toend setting, where predicates must be predicted on-the-fly. Since we also train our model to predict syntactic dependencies, it is beneficial to give the model some knowledge of POS information. While much previous work employs a pipelined approach to both POS tagging for dependency parsing and predicate detection for SRL, we take a multi-task learning (MTL) approach BID12, sharing the parameters of earlier layers in our SRL model with a joint POS and predicate detection objective. Since POS is a strong predictor of predicates, 4 and the complexity of training a multi-task model increases with the number of tasks, we combine POS tagging and predicate detection into a joint label space: for each POS tag TAG in the training data which co-occurs with a predicate, we add a label of the form TAG:PREDICATE.Specifically, we experiment with feeding a lower-level representation, r t, which may be either c (k) t, the output of the convolutional layers, or s t, the output of the first self-attention layer, to a linear classifier. We compute locally-normalized probabilities using the softmax function: P (z t | X) ∝ exp(r t), where z t is a label in the joint space. We apply this supervision at earlier lay-ers following prior work BID54 BID23. Our final goal is to predict semantic roles for each predicate in the sequence. We score each predicate 5 against each token in the sequence using a bilinear operation, producing per-label scores for each token for each predicate, with predicates and syntax determined by oracles V and P.First, we project each token representation s (j) t to a predicate-specific representation s pred t and a role-specific representation s role t. We then provide these representations to a bilinear transformation U for scoring. So, the role label scores s f t for the token at index t with respect to the predicate at index f (i.e. token t and frame f) are given by: DISPLAYFORM0 which can be computed in parallel across all semantic frames in an entire minibatch. We calculate a locally normalized distribution over role labels for token t in frame f using the softmax function: DISPLAYFORM1 At test time, we perform constrained decoding using the Viterbi algorithm to emit valid sequences of BIO tags, using unary scores s f t and the transition probabilities given by the training data. We maximize the sum of the likelihoods of the individual tasks, entrusting the network to learn parameters which model the complex coupling between tasks, rather than explicitly modeling structure in the output labels: DISPLAYFORM0 where λ is a penalty on the syntactic attention loss. Note that as described in §2.2, the terms for the syntactically-informed attention and joint predicate/POS prediction are conditioned only on the input sequence X, whereas the SRL component is conditioned on gold predicates V G and gold parse structure P G during training. We train the model using Nadam SGD combined with the learning rate schedule in BID64. In addition to MTL, we regularize our model using element-wise and word dropout BID55 BID20 and parameter averaging. We use gradient clipping to avoid exploding gradients BID7 BID45. Our models are implemented in TensorFlow BID0 with source code and models to be released upon publication. Additional details on optimization and hyperparameters are included in Appendix A. Early approaches to SRL BID50 BID56 BID29 BID61 focused on developing rich sets of linguistic features as input to a linear model, often combined with complex constrained inference e.g. with an ILP BID51. BID59 showed that constraints could be enforced more efficiently using a clever dynamic program for exact inference. BID57 modeled syntactic parsing and SRL jointly, and BID35 jointly modeled SRL and CCG parsing. BID19 were among the first to use a neural network model for SRL, a CNN over word embeddings which failed to out-perform non-neural models. successfully employed neural networks by embedding lexicalized features and providing them as factors in the model of BID59.More recent neural models are syntax-free. BID69, and BID25 Some work has incorporated syntax into neural models for SRL. BID53 incorporate syntax by embedding dependency paths, and similarly encode syntax using a graph CNN over a predicted syntax tree, out-performing models without syntax on CoNLL-2009. However, both models are at risk of over-fitting to or otherwise inheriting the flaws of the predictions upon which they are trained. Indeed, report that their model does not out-perform a similar syntaxfree model on out-of-domain data. Syntactically-informed self-attention is similar to the concurrent work of BID37, who use edge marginals produced by the matrixtree algorithm as attention weights for document classification and natural language inference. MTL BID12 ) is popular in NLP. Collobert et al. FORMULA1 multi-task part-of-speech, chunking, NER and SRL. BID68 jointly train a dependency parser and POS tagger. Søgaard and train a multitask model for POS, chunking and CCG tagging. BID23 built a single, jointly trained model for POS, chunking, parsing, semantic relatedness and entailment, using a special regularization scheme to facilitate training. BID9 and investigate different combinations of NLP tagging tasks including POS, chunking and FrameNet semantics . BID38 enhance a machine translation model by multitasking with parsing. MTL has also been applied to semantic dependency parsing: BID58 multi-task with a syntax-based tagging objective while BID46 train on three semantic dependency frameworks. The question of training on gold versus predicted labels is closely related to learning to search BID21 BID52 BID13 and scheduled sampling BID6, with applications in NLP to sequence labeling and transition-based parsing BID15 BID22 BID4. We believe more sophisticated approaches extending these techniques to MTL could improve LISA in future work. We present on the CoNLL-2005 shared task BID11 and the CoNLL-2012 English subset of OntoNotes 5.0 BID49, achieving state-of-the-art for a single model with predicted predicates on both corpora. In all experiments, we initialize with pre-trained GloVe word embeddings BID47, hyperparameters that ed in the best performance on the validation set were selected via a small grid search, and models were trained for a maximum of 7 days on one TitanX GPU using early stopping on the validation set. 6 For CoNLL-2005 we convert constituencies to dependencies using the Stanford head rules v3.5 (de) and for CoNLL-2012 we use ClearNLP (b), following previous work. A detailed description of hyperparameter settings and data pre-processing can be found in Appendix A.For both datasets, we compare our best models (LISA G) to three strong sets of baselines: the syntax-free deep LSTM model of BID25 which was the previous state-of-the-art model for SRL with predicted predicates, both as an ensemble of five models (PoE) and as a single model (single); an ablation of our own self-attention model where we don't incorporate any syntactic information (SA), and another ablation where we do train with syntactically-informed self-attention, but where downstream layers in the model are conditioned on the predicted attention weights (i.e. dynamic oracle, D) rather than the gold parse (G) during training (LISA D).We demonstrate that our models can benefit from injecting state-of-the-art predicted parses at test time (+D&M) by setting the attention oracle to parses predicted by , the state-of-the-art dependency parser for English PTB and winner of the 2017 CoNLL shared task BID67. In all cases, using these parses at test time improves performance. We also evaluate our model using the gold syntactic parse at test time (+Gold), to provide an upper bound for the benefit that syntax could have for SRL using LISA. These experiments show that despite LISA's strong performance, there remains substantial room for improvement. In §4.4 we perform detailed analysis comparing SRL models us-6 Our best reported CoNLL-2012 model was trained for just under 6 days, though it matched BID25 ing gold and predicted parses to better understand where syntax provides the most benefit to SRL, and what remains to be improved. We first report the unlabeled attachment scores (UAS) of our parsing models on the CoNLL-2005 and 2012 SRL test sets (Table 1). achieves the best scores, obtaining state-of-the-art on the CoNLL-2012 split of OntoNotes in terms of UAS, followed by LISA G then LISA D. 7 We still see SRL accuracy improvements despite our relatively low parser UAS from LISA's predicted parses, but the difference in accuracy likely explains the large increase in SRL we see from decoding with D&M parses. TAB3 reports precision, recall and F1 on the CoNLL-2012 test set. Our SA model already performs strongly without access to syntax, out-performing the single model of BID25 but under-performing their ensemble. Adding syntactically-informed training to the self- 7 The previous best score we know of is 92.5 attained by Mate BID10, as reported in BID18. attention increases over the model without syntax, achieving about the same score using dynamic versus gold parse oracles for downstream layers during training. When evaluating using an injected parse, we see that a large increase of more than 1.5 F1 absolute for LISA G and this increase is markedly larger than for LISA D. With the injected D&M parse, our single models impressively outperform the ensemble. We also report predicate detection precision, recall and F1 TAB5 ). Our models obtain much higher scores than BID25 on this task, likely explaining improvements of our basic SA model over theirs. Like BID25, our model achieves much higher precision than recall, indicative of the model memorizing predicate words from the training data. Interestingly, our SA model out-performs syntax-infused models by a small margin. We hypothesize that this could be due to asking the LISA models to learn to predict more tasks, taking some model capacity away from predicate detection. TAB7 lists precision, recall and F1 on the CoNLL-2005 test sets. Unlike on CoNLL-2012, our SA baseline does not out-perform BID25. This is likely due to their predicate detection scores being closer to ours on this data (Table 5). Interestingly, unlike on CoNLL-2012 we see a distinct improvement between LISA G and LISA D in models which use LISA parses: LISA G training leads to improved SRL scores by more than 1 F1 absolute using LISA-predicted parses. Similar to CoNLL-2012, we see very little improvement from adding D&M parses at test-time with the dynamic oracle, whereas we obtain the highest score of all when using D&M parses combined with LISA G, demonstrating that our training technique markedly improves LISA's ability to leverage improved parses at test time. Our best single models out-perform the ensemble of (Table 5): all our models out-perform the baseline in terms of F1. In the case of CoNLL-2005 BID25 attains higher recall, especially on the Brown test set, while our model achieves higher precision. We report only LISA G since there is little difference across *SA models. LISA in its current form does not perform as well when gold predicates are given at test time. Table 6 presents LISA G performance with predicate indicator embeddings provided on the input. On neither test set does our model using LISA parses out-perform the state-of-the-art. With D&M parses, our models out-perform BID25, but not BID60.We attribute this behavior to two factors. First, the models of BID25 and BID60 are larger than our models. 8. Our models were designed to predict predicates, and we found the current model size sufficient for good performance in this setting. Second, our model encodes each sequence only once, while the works to which we compare re-encode the sequence anew for each predicate. Since our model predicts its own predicates using a shared sentence encoding, it is impossible to encode sequences in this way. We also do not enforce that the model assign the correct predicate label during decoding, leading to incorrect predicate predictions despite gold predicate inputs. For example, in a challenging sentence which contains two distinct semantic frames with the identical predicate continued, our model incorrectly predicts both tokens as predicates in one of the frames. With more careful modeling toward gold predicates, our technique could be improved for this setting. Still, LISA shows impressive performance when gold predicates are not available, as when using SRL in the wild. In §4.2 and §4.3 we observed that while LISA performs well with state-of-the-art predicted syntax, it still sees a large gain across all datasets of 4-5 F1 points absolute when provided with gold syntax trees. In order to better understand the nature of these improvements, we perform a detailed model analysis based on that of BID25 First, we compare the impact of Viterbi decoding with LISA, D&M, and gold syntax trees TAB10, finding the same trends across both datasets. While Viterbi decoding makes a larger difference over greedy decoding with LISA parses than with D&M, we find that Viterbi has the exact same impact for D&M and gold parses: Gold parses provide no improvement over state-of-the-art predicted parses in terms of BIO label consistency. We also assess SRL F1 as a function of sentence length. In FIG3 we see that providing LISA with gold parses is particularly helpful for sentences longer than 10 tokens. This likely directly follows from the tendency of syntactic parsers to perform worse on longer sentences. Next, we compare SRL error types. Following BID25, we apply a series of corrections to model predictions in order to understand which error types the gold parse resolves: e.g. Fix Labels fixes labels on spans which match gold boundaries, whereas Merge Spans merges adjacent predicted spans into a gold span. 9 In Figure 3 we see that much of the performance gap between the gold and predicted parses is due to span boundary errors (Merge Spans, Split Spans and Fix Span Boundary), which supports the hypothesis proposed by BID25 that incorporating syntax could be particularly helpful for resolving these errors. BID25 that these errors are due mainly to prepositional phrase (PP) attachment mistakes. We also find this to be the case: Figure 4 shows a breakdown of split/merge corrections by phrase type. Though the number of corrections decreases substantially across phrase types, the proportion of corrections attributed to PPs remains the same (approx. 50%) even after providing the correct PP attachment to the model, indicating that PP span boundary mistakes are due not only to parse mistakes, but are a fundamental difficulty for SRL. We present linguistically-informed self-attention: a new multi-task neural network model that effectively incorporates rich linguistic information for semantic role labeling. LISA out-performs the state-of-the-art on two benchmark SRL datasets, including out-of-domain, while training more than 4× faster. Future work will explore improving LISA's parsing accuracy, developing better training techniques and adapting to more tasks. Following previous work BID25, we evaluate our models on the CoNLL-2012 data split BID49 of OntoNotes 5.0 BID27. 10 This dataset is drawn from seven domains: newswire, web, broadcast news and conversation, magazines, telephone conversations, and text from the bible. The text is annotated with gold part-of-speech, syntactic constituencies, named entities, word sense, speaker, co-reference and semantic role labels based on the PropBank guidelines BID44. Propositions may be verbal or nominal, and there are 41 distinct semantic role labels, excluding continuation roles and including the predicate. We processed the data as follows: We convert the semantic proposition and role segmentations to BIO boundary-encoded tags, ing in 129 distinct BIO-encoded tags (including continuation roles). We initialize word embeddings with 100d pre-trained GloVe embeddings trained on 6 billion tokens of Wikipedia and Gigaword BID47. Following the experimental setup for parsing from Choi et al. FORMULA1, we convert constituency structure to dependencies using the ClearNLP dependency converter BID17, use automatic part-of-speech tags assigned by the ClearNLP tagger BID16, and exclude single-token sentences in our parsing evaluation. 10 We constructed the data split following instructions at: BID11 ) is based on the original PropBank corpus BID44, which labels the Wall Street Journal portion of the Penn TreeBank corpus (PTB) BID42 with predicateargument structures, plus a challenging out-ofdomain test set derived from the Brown corpus (Francis and Kučera, 1964). This dataset contains only verbal predicates, though some are multiword verbs, and 28 distinct role label types. We obtain 105 SRL labels including continuations after encoding predicate argument segment boundaries with BIO tags. We evaluate the SRL performance of our models using the srl-eval.pl script provided by the CoNLL-2005 shared task, 11 which computes segment-level precision, recall and F1 score. We also report the predicate detection scores output by this script. For CoNLL-2005 we train the same parser as for CoNLL-2012 except on the typical split of the WSJ portion of the PTB using Stanford dependencies (de) and POS tags from the Stanford CoreNLP left3words model BID62. We train on WSJ sections 02-21, use section 22 for development and section 23 for test. This corresponds to the same train/test split used for propositions in the CoNLL-2005 dataset, except that section 24 is used for development rather than section 22. We train the model using the Nadam algorithm for adaptive stochastic gradient descent (SGD), which combines Adam BID30 SGD with Nesterov momentum BID43. We additionally vary the learning rate lr as a function of an initial learning rate lr 0 and the current training step step as described in Vaswani which increases the learning rate linearly for the first warm training steps, then decays it proportionally to the inverse square root of the step number. We found this learning rate schedule essential for training the self-attention model. We only update optimization moving-average accumulators for parameters which receive gradient updates at a given step. 12 In all of our experiments we used initial learning rate 0.04, β 1 = 0.9, β 2 = 0.98, = 1 × 10 −12 and dropout rates of 0.33 everywhere. We use four self-attention layers made up of 8 attention heads each with embedding dimension 64, and two CNN layers with filter size 1024. The size of all MLP projections: In the feed-forward portion of self-attention, predicate and role representations, and representation used for joint partof-speech/predicate classification is 256. We train with warm = 4000 warmup steps and clip gradient norms to 5.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Our combination of multi-task learning and self-attention, training the model to attend to parents in a syntactic parse tree, achieves state-of-the-art CoNLL-2005 and CoNLL-2012 SRL results for models using predicted predicates.

represent the tldr for finding the corresponding scientific article

In this paper we introduced an algorithm to learn the connectivity of deep multi-branch networks. The approach is evaluated on image categorization where it consistently yields accuracy gains over state-of-the-art models that use fixed connectivity.


Question: Represent the input:

Question: Bottleneck structures with identity (e.g., residual) connection are now emerging popular paradigms for designing deep convolutional neural networks (CNN), for processing large-scale features efficiently. In this paper, we focus on the information-preserving nature of identity connection and utilize this to enable a convolutional layer to have a new functionality of channel-selectivity, i.e., re-distributing its computations to important channels. In particular, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit that improves parameter efficiency of various modern CNNs with bottlenecks. During training, SCU gradually learns the channel-selectivity on-the-fly via the alternative usage of (a) pruning unimportant channels, and (b) rewiring the pruned parameters to important channels. The rewired parameters emphasize the target channel in a way that selectively enlarges the convolutional kernels corresponding to it. Our experimental demonstrate that the SCU-based models without any postprocessing generally achieve both model compression and accuracy improvement compared to the baselines, consistently for all tested architectures. Nowadays, convolutional neural networks (CNNs) have become one of the most effective approaches in various fields of artificial intelligence. With a growing interest of CNNs, there has been a lot of works on designing more advanced CNN architectures BID43 BID21. In particular, the simple idea of adding identity connection in ResNet BID11 has enabled breakthroughs in this direction, as it allows to train substantially deeper/wider networks than before by alleviating existed optimization difficulties in previous CNNs. Recent CNNs can scale over a thousand of layers BID12 or channels BID18 without much overfitting, and most of these "giant" models consider identity connections in various ways BID49 BID18. However, as CNN models grow rapidly, deploying them in the real-world becomes increasingly difficult due to computing resource constraints. This has motivated the recent literature such as network pruning BID9 BID28 BID35, weight quantization BID36 BID3, adaptive networks BID47 BID5 BID0 BID19, and resource-efficient architectures BID17 BID40 BID32.For designing a resource-efficient CNN architecture, it is important to process succinct representations of large-scale channels. To this end, the identity connections are useful since they allow to reduce the representation dimension to a large extent while "preserving" information from the previous layer. Such bottleneck architectures are now widely used in modern CNNs such as ResNet BID11 and DenseNet BID18 for parameter efficiency, and many state-of-the-art mobile-targeted architectures such as SqueezeNet BID20, ShuffleNet BID53 BID32, MoblileNet BID16 BID40, and CondenseNet BID17 commonly address the importance of designing efficient bottlenecks. Contribution. In this paper, we propose Selective Convolutional Unit (SCU), a widely-applicable architectural unit for efficient utilization of parameters in particular as a bottleneck upon identity connection. At a high-level, SCU performs a convolutional operation to transform a given input. The main goal of SCU, however, is rather to re-distribute their computations only to selected channels (a) (b)Figure 1: (a) An illustration of channel de-allocation and re-allocation procedures. The higher the saturation of the channel color, the higher the ECDS value. (b) The overall structure of SCU.of importance, instead of processing the entire input naively. To this end, SCU has two special operations: (a) de-allocate unnecessary input channels (dealloc), and (b) re-allocate the obstructed channels to other channels of importance (realloc) (see Figure 1a). They are performed without damaging the network output (i.e., function-preserving operations), and therefore one can call them safely at any time during training. Consequently, training SCU is a process that increases the efficiency of CNN by iteratively pruning or rewiring its parameters on-the-fly along with learning them. In some sense, it is similar to how hippocampus in human brain learn, where new neurons are generated daily, and rewired into the existing network while maintaining them via neuronal apoptosis or pruning BID38 BID49.We combine several new ideas to tackle technical challenges for such on-demand, efficient trainable SCU. First, we propose expected channel damage score (ECDS), a novel metric of channel importance that is used as the criterion to select channels for dealloc or realloc. Compared to other popular magnitude-based metrics BID28 BID35, ECDS allows capturing not only low-magnitude channels but also channels of low-contribution under the input distribution. Second, we impose channel-wise spatial shifting bias when a channel is reallocated, providing much diversity in the input distribution. It also has an effect of enlarging the convolutional kernel of SCU. Finally, we place a channel-wise scaling layer inside SCU with sparsity-inducing regularization, which also promotes dealloc (and consequently realloc as well), without further overhead in inference and training. We evaluate the effectiveness of SCU by applying it to several modern CNN models including ResNet BID11, DenseNet BID18, and ResNeXt BID49, on various classification datasets. Our experimental consistently show that SCU improves the efficiency of bottlenecks both in model size and classification accuracy. For example, SCU reduces the error rates of DenseNet-40 model (without any post-processing) by using even less parameters: 6.57% → 5.95% and 29.97% → 28.64% on CIFAR-10/100 datasets, respectively. We also apply SCU to a mobile-targeted CondenseNet BID17 model, and further improve its efficiency: it even outperforms NASNet-C BID54, an architecture searched with 500 GPUs for 4 days, while our model is constructed with minimal efforts automatically via SCU.There have been significant interests in the literature on discovering which parameters to be pruned during training of neural networks, e.g., see the literature of network sparsity learning BID48 BID25 BID41 BID35 BID30 BID4. On the other hand, the progress is, arguably, slower for how to rewire the pruned parameters of a given model to maximize its utility. proposed Dense-Sparse-Dense (DSD), a multi-step training flow applicable for a wide range of DNNs showing that re-training with re-initializing the pruned parameters can improve the performance of the original network. Dynamic network surgery BID7, on the other hand, proposed a methodology of splicing the pruned connections so that mis-pruned ones can be recovered, yielding a better compression performance. In this paper, we propose a new way of rewiring for parameter efficiency, i.e., rewiring for channel-selectivity, and a new architectural framework that enables both pruning and rewiring in a single pass of training without any postprocessing or re-training (as like human brain learning). Under our framework, one can easily set a targeted trade-off between model compression and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc. We believe that our work sheds a new direction on the important problem of training neural networks efficiently. In this section, we describe Selective Convolutional Unit (SCU), a generic architectural unit for bottleneck CNN architectures. The overall structure of SCU is described in Section 2.1 and 2.2. In Section 2.3, we introduce a metric deciding channel-selectivity in SCU. We present in Section 2.4 how to handle a network including SCUs in training and inference. Bottleneck structures in modern CNNs. We first consider a residual function defined in ResNet BID11 which has an identity mapping: for a given input random variable X ∈ R (H and W are the height and width of each channel, respectively, and I is the number of channels or feature maps) and a non-linear function F, the output of a residual function is written by Y = X + F(X). This function has been commonly used as a building block for designing recent deep CNN models, in a form that F is modeled by a shallow CNN. However, depending on how F is designed, computing F(X) can be expensive when I is large. For tackling the issue, bottleneck structure is a prominent approach, that is, to model F by F • R by placing a bottleneck R that firstly maps X into a lower dimension of I < I features. This approach, in essence, requires the identity connection, for avoiding information loss from X to Y. Namely, the identity connection enables a layer to save redundant computation (or parameters) for just "keeping" information from the input. Bottleneck structures can be used other than ResNet as well, as long as the identity connection exists. Recent architectures including DenseNet BID18, PyramidNet BID8 and DPN develop this idea with using a different aggregation function W instead of addition in ResNet, e.g., W(X, X) = [X, X] (channel-wise concatenation) for DenseNet. Designing R-F -W is now a common way of handling large features. Channel-selectivity for efficient bottlenecks. Although placing a bottleneck R reduces much computation of the main function F, we point out the majority of modern CNNs currently use inefficient design of R itself, so that even the computation of R often dominates the remaining. In ResNet and DenseNet models, for example, bottlenecks are designed using a pointwise convolution with a batch normalization layer (BN) BID21 and ReLU BID34: DISPLAYFORM0 where Conv I→I denotes a pointwise convolution that maps I features into I features, i.e., its parameters can be represented by a I × I matrix. This means that the parameters of R grows linearly on I, and it can be much larger than F if I I. For example, in case of DenseNet-BC-190 BID18, 70% of the total parameters are devoted for modeling R, which is inefficient as the expressivity of a pointwise convolution is somewhat limited. In this paper, we attempt to improve the efficiency of R in two ways: (a) reducing the parameters in Conv I→I by channel pruning, and (b) improving its expressivity by using the pruned parameters again. This motivates our goal to learn both channel-selectivity and parameters jointly. Overall architecture of SCU. SCU is designed to learn the channel-selectivity via dynamic pruning and rewiring of channels during training. In this paper, we focus on putting SCU as a bottleneck R, and show that the channel-selectivity of SCU improves its parameter efficiency. Our intuition is that (a) the information-preserving nature of identity connection brings optimization benefits if neurons in its structure are dynamically pruned during training, and (b) such pruning can be particularly effective on bottlenecks as their outputs are in a much lower dimension compared to the input. Nevertheless, we believe that our ideas on SCU are not limited to the bottleneck structures, as the concept of channel-selectivity can be generalized to other structures. At a high level, SCU follows the bottleneck structure from, but for two additional layers: Channel Distributor (CD) and Noise Controller (NC) whose details are presented in Section 2.2. We model a non-linear function SCU: R I×H×W → R I ×H×W as follows (see Figure 1b): DISPLAYFORM0 SCU has two special operations which control its input channels to process: (a) channel deallocation (dealloc), which obstructs unnecessary channels from being used in future computations, and (b) channel re-allocation (realloc), which allocates more parameters to important, non-obstructed channels by copying them into the obstructed areas. We design those operations to be function preserving, i.e. they do not change the original function of the unit, so that can be called at anytime during training without damage. Repeating dealloc and realloc alternatively during training translates the original input to what has only a few important channels, potentially duplicated multiple times. Namely, the parameters originally allocated to handle the entire input now operate on its important subset. On the way of designing the operations of function preserving, we propose Expected Channel Damage Score (ECDS) that leads to an efficient, safe way to capture unimportant channels by measuring how much the output of SCU changes on average (w.r.t. data distribution) after removing each channel. The details of ECDS are in Section 2.3. Channel Distributor (CD) is the principal mechanism of SCU and is placed at the beginning of the unit. The role of CD is to "rebuild" the input, so that unnecessary channels can be discarded, and important channels are copied to be emphasized. In essence, we implement this function by re-indexing and blocking the input channel-wise: CD(X) i:= g i · X πi with an index pointer π i ∈ {1, 2, · · ·, I}, a gate variable g i ∈ {0, 1} for i = 1, 2, · · ·, I. Here, we notice that CD(X) may contain a channel copied multiple times, i.e., multiple π i's can have the same value. Since SCU has different parameters for each channel, setting multiple π i's has an effect of allocating more parameters to better process the channel pointed by π i. We found that, however, it is hard to take advantage of the newly allocated parameters by simply copying a channel due to symmetry, i.e., the parameters for each channel usually degenerates. Due to this, we consider spatial shifting biases DISPLAYFORM0 2 for each channel, as illustrated in FIG0. This trick can provide the copied channels much diversity in input distributions (and hence relaxing degeneracy), in a way that it is effective for the convolutional layer in SCU: it enlarges the convolutional kernel from 1 × 1 for the re-allocated channels only. To summerize, CD(X) i takes (a) the channel which π i is pointing, in the spatially shifted form with bias b i, or (b) 0 if the gate g i is closed. Formally, CD can be represented by DISPLAYFORM1. CD(X) has the same size to X, and defined as follows: DISPLAYFORM2 Here, shift(X, b h, b w) denotes the "shifting" operation along spatial dimensions of X. For each pixel location (i, j) in X, we define shift(X, b h, b w) i,j as: DISPLAYFORM3 using a bilinear interpolation kernel. This formulation allows b h and b w to be continuous real values, thereby to be learned via gradient-based methods with other parameters jointly. Noise Controller (NC) is a component for more effective training of SCU. As SCU continuously performs channel pruning via dealloc during training, the efficiency of SCU depends on which regularization is used. The key role of NC is to induce the training of SCU to get more channel-wise sparsity, so that more channels can be de-allocated safely. Formally, NC is a channel-wise re-scaling layer: NC(X):= X θ, 1 where θ = (θ i) I i=1 are parameters to be learned. For the channel-wise sparsity, we impose sparsity-inducing regularization specifically on θ. Although any sparsity-inducing regularization can be used for θ BID28 BID48, in this paper we adopt the Bayesian pruning approach proposed by BID35 2 for two reasons: (a) it is easy to incorporate into training process, and (b) we found that noise incurred from Bayesian parameters helps to recover damage from channel pruning. In general, a Bayesian scheme regards each parameter θ as a random variable with prior p(θ). Updating the posterior p(θ|D) from data D often leads the model to have much sparsity, if p(θ) is set to induce sparsity, e.g., by 1 denotes the element-wise product. 2 For completeness, we present for the readers an overview of BID35 in Appendix B.log-uniform prior BID23. Meanwhile, p(θ|D) is usually approximated with a simpler model q φ (θ), where φ are parameters to be learned. In case of NC, we regard each scaling parameter as a random variable, so that they become channel-wise multiplicative noises on input. We follow BID35 for the design choices on q φ (θ) and p(θ), by fully-factorized log-normal DISPLAYFORM4 Consider an input random variable X = (X i ∈ R H×W) DISPLAYFORM0, and DISPLAYFORM1 I ×H×W, where S −i denotes a SCU identical to S but g i = 0. In other words, it is the expected amount of changes in outputs when S i is "damaged" or "pruned". The primary goal of this criteria is to make dealloc to be function preserving. We define ECDS(S) i by the (Euclidean) norm of the averaged values of E[SCU(X; S) − SCU(X; S −i)] over the spatial dimensions: DISPLAYFORM2 Notice that the above definition requires a marginalization over random variable X. One can estimate it via Monte Carlo sampling using training data, but this is computationally too expensive compared to other popular magnitude-based metrics BID28 BID35. Instead, we utilize the BN layer inside SCU, to infer the current input distribution of each channel at any time of training. This trick enables to approximate ECDS(S) i by a closed formula of S i, avoiding expensive computations of SCU(X; ·), as in what follows. Consider a hidden neuron x following BN and ReLU, i.e., y = ReLU(BN(x)), and suppose one wants to estimate E[y] without sampling. To this end, we exploit the fact that BN already "accumulates" its input statistics continuously during training. Under assuming that BN(x) ∼ N (β, γ 2) where γ and β are the scaling and shifting parameter in BN, respectively, it is elementary to check: DISPLAYFORM3 where φ N and Φ N denote the p.d.f. and the c.d.f. of the standard normal distribution, respectively. The assumption is quite reasonable during training BN as each mini-batch is exactly normalized before applying the scaling and shifting inside BN. The idea is directly extended to obtain a closed form formula of ECDS(S) i under some assumptions, as stated in the following proposition. DISPLAYFORM4, for all i. The proof of the above proposition is given in Appendix D. In essence, there are three main terms in the formula: (a) a term that measures how much the input channel is active, (b) how much the NC amplifies the input, and (c) the total magnitude of weights in the convolutional layer. Therefore, it allows a way to capture not only low-magnitude channels but also channels of low-contribution under the input distribution (see Section 3.2 for comparisons with other metrics). Consider a CNN model p(Y|X, Θ) employing SCU, where Θ denotes the collection of model parameters. For easier explanation, we rewrite Θ by (V, W): V consists of (π, g) in CDs, and W is the remaining ones. DISPLAYFORM0, (V, W) is trained via alternating two phases: (a) training W via stochastic gradient descent (SGD), and (b) updating V via dealloc or realloc. The overall training process is mainly driven by (a), and the usage of (b) is optional. In (a), we use stochastic variational inference BID22 in order to incorporate stochasticity incurred from NC, so that SCU can learn its Bayesian parameters in NC jointly with the others via SGD. On the other hand, in (b), dealloc and realloc are called on demand during training depending on the purpose. For example, one may decide to call dealloc only throughout the training to obtain a highly compressed model, or one could use realloc as well to utilize more model parameters. Once (b) is called, (a) is temporally paused and V are updated. Training via stochastic variational inference. We can safely ignore the effect of V during training of W, since they remain fixed. Recall that, each noise θ from a NC is assumed to follow q φ (θ) = LogN(θ|µ, σ 2). Then, θ can be "re-parametrized" with a noise ε from the standard normal distribution as follows: DISPLAYFORM1 2 ). Stochastic variational inference BID22 ) allows a minibatch-based stochastic gradient method for θ, in such case that θ can be re-parametrized with an non-parametric noise. The final loss we minimize for a minibatch {( DISPLAYFORM2 becomes (see Appendix F for more details): DISPLAYFORM3 where DISPLAYFORM4 is a sampled vector from the fully-factorized standard normal distribution. Channel de-allocation and re-allocation. DISPLAYFORM5 The main role of dealloc and realloc is to update W CD in S that are not trained directly via SGD. They are performed as follows: select slices to operate by thresholding ECDS(S), and update S from the selected channels. More formally, when dealloc is called, S i's where ECDS(S) i < T l for a fixed threshold T l are selected, and g i's in W CD are set by 0. If one chooses small T l, this operation does not hurt the original function. On the other hand, realloc selects channels by collecting S i where ECDS(S) i > T h, for another threshold T h. Each of the selected channels can be re-allocated only if there is a closed channel in S. If there does not exist a enough space, channels with higher ECDS have priority to be selected. A single re-allocation of a channel S i to a closed channel S j consists of several steps: DISPLAYFORM6 ← 0, (iv) re-initialize the shifting bias b j, and (v) set π j ← π i. This procedure is function-preserving, due to (iii).After training a SCU S, one can safely remove S i's that are closed, to yield a compact unit. Then, CDs are now operated by "selecting" channels rather than by obstructing, thereby the subsequent layers play with smaller dimensions. Hence, at the end, SCU is trained to select only a subset of the input for performing the bottleneck operation. For NC, on the other hand, one can still use it for inference, but efficient inference can be performed by replacing each noise θ i by constant E[θ i], following the well-known approximation used in many dropout-like techniques BID15. In our experiments, we apply SCU to several well-known CNN architectures that uses bottlenecks, and perform experiments on CIFAR-10/100 BID24 and ImageNet BID37 classification datasets. The more details on our experimental setups, e.g., datasets, training details, and configurations of SCU, are given in Appendix G. Improving existing CNNs with SCU. We consider models using ResNet BID11, DenseNet BID18 and ResNeXt BID49 architectures. In general, every model we used in this paper forms a stack of multiple bottlenecks, where the definition of each bottleneck differs depending on its architecture except that it can be commonly expressed by R-F -W (the details are given in TAB6 in the appendix). We compare the existing models with the corresponding new ones in which the bottlenecks are replaced by SCU. For each SCU-based model, we consider three cases: (a) neither dealloc nor realloc is used during training, (b) only dealloc is used, and (c) both dealloc and realloc are used. We measure the total number of parameters in bottlenecks, and error rates. TAB1 compares the existing CNN models with the corresponding ones using SCU, on CIFAR-10/100. The consistently demonstrate that SCU improves the original models, showing their effectiveness in different ways. When only dealloc is used, the model tends to be trained with minimizing their parameter to use. Using realloc, SCU now can utilize the de-allocated parameters to improve their accuracy aggressively. Note that SCU can improve the accuracy of the original model even neither dealloc nor realloc is used. This gain is from the regularization effect of stochastic NC, acting a dropout-like layer. We also emphasize that one can set a targeted trade-off between compression of SCU and accuracy improvement depending on her purpose, simply by adjusting the calling policy of dealloc and realloc. For example, in case of DenseNet-100 model on CIFAR-10, one can easily trade-off between reductions in (compression, error) = (−51.4%, −1.78%) and (−18.2%, −8.24%). In overall, SCU-based models achieve both model compression and accuracy improvement under all tested architectures. TAB2 shows the on ImageNet, which are consistent to those on CIFAR-10/100. Notice that reducing parameters and error simultaneously is much more non-trivial in the case of ImageNet, e.g., reducing error 23.6% → 23.0% requires to add 51 more layers to ResNet-101 (i.e., ResNet-152), as reported in the official repository of ResNet BID13.Designing efficient CNNs with SCU. We also demonstrate that SCU can be used to design a totally new efficient architecture. Recall that, in this paper, SCU focus on the bottlenecks inside the overall structure. The other parts, F or W, are other orthogonal design choices. To improve the efficiency of the parts, we adopt some components from CondenseNet BID17, which is one of the state-of-the-art architectures in terms of computational efficiency, designed for mobile devices. Although we do not adopt their main component, i.e., learned group convolution (LGC) as it also targets for the bottleneck as like SCU, we can still utilize other components of CondenseNet: increasing growth rate (IGR) (doubles the growth rate of DenseNet for every N blocks starting from 8) and the use of group convolution for F. Namely, we construct a new model, coined CondenseNet-SCU by adopting IGR and GC upon a DenseNet-182 model with SCU. We replace each 3×3 convolution for F by a group convolution of 4 groups. We train this model using dealloc only to maximize the computational efficiency. In TAB3, we compare our model with state-of-the-art level CNNs, including ResNet-1001 BID12, WRN-28-10 , NASNet-C BID54, and the original CondenseNet-182. As one can observe, our model shows better efficiency compared to the corresponding CondenseNet, suggesting the effectiveness of SCU overLGC. Somewhat interestingly, ours even outperforms NASNet-C that is an architecture searched over thousands of candidates, in both model compression and accuracy improvement. We finally remark that CondenseNet-SCU-182 model presented in TAB3 originally has 6.29M parameters in total before training, devoting 5.89M for bottlenecks, i.e., it is about 93.7% of the total number of parameters. This is indeed an example in that reducing overhead from bottlenecks is important for better efficiency, which is addressed by SCU. We also perform numerous ablation studies on the proposed SCU, investigating the effect of the key components: CD, NC, and ECDS. For evaluation, we use the DenseNet-SCU-40 model (DenseNet-40 using SCU) trained for CIFAR-10. We also follow the training details described in Appendix G.Spatial shifting and re-allocation. We propose spatial shifting as a trick in realloc procedure to provide diversity in input distributions. To evaluate its effect, we compare three DenseNet-SCU-40 models with different configurations of SCU: (D) only dealloc during training, (+R) realloc together but without spatial shifting, and (+R+S) further with the shifting. FIG1 shows that +R does not improve the model performance much compared to D, despite +R+S outperforms both of them. This suggests that copying a channel naively is not enough to fully utilize the rewired parameters, and spatial shifting is an effective way to overcome the issue. DISPLAYFORM0 Sparsity-inducing effect of NC. We place NC in SCU to encourage more sparse channels. To verify such an effect, we consider DenseNet-SCU-40 model (say M1) and its variant removing NC from SCU (say M2). We first train M1 and M2 calling neither dealloc nor realloc, and compare them how the ECDS of each channel is distributed. FIG1 shows that M1 tends to have ECDS closer to zero, i.e., more channels will be de-allocated than M2. Next, we train these models using dealloc, to confirm that NC indeed leads to more deallocation. The left panel of FIG1 shows that the number of de-allocated channels of M1 is relatively larger than that of M2, which is the desired effect of NC. Note that M1 also outperforms M2 on error rates, which is an additional advantage of NC from its stochastic regularization effect. Nevertheless, remark that M2 in FIG1 already de-allocates many channels, which suggests that SBP (used in NC) is not crucial for efficient de-allocation. Rather, the efficiency mainly comes from ECDS. To prove this claim, we evaluate three variants of M1 which use different de-allocation policies than ECDS < T l: (a) SNR < 1 (thresholding the signal-to-noise ratio of NC in each channel by 1, proposed by the original SBP; M3), (b) SNR < 2.3 (M4) and (c) 2 < 0.25 (thresholding W Conv i 2 ; M5). We train them using only dealloc, and compare the performances with the proposed model (M1). The right panel of FIG1 shows the of the three variants. First, we found that the M3 could not de-allocate any channel in our setting (this is because we prune a network on-the-fly during training, while the original SBP only did it after training). When we de-allocate competitive numbers of channels against M1 by tuning thresholds of others (M4 and M5), the error rates are much worse than that of M1. These observations confirm that ECDS is a more effective de-allocation policy than other magnitude-based metrics. We demonstrate that CNNs of large-scale features can be trained effectively via channel-selectivity, primarily focusing on bottleneck architectures. The proposed ideas on channel-selectivity, however, would be applicable other than the bottlenecks, which we believe is an interesting future research direction. We also expect that channel-selectivity has a potential to be used for other tasks as well, e.g., interpretability BID42, robustness BID6, and memorization BID51. Consider a probabilistic model p(Y|X, θ) between two random variables X and Y, and suppose one wants to infer θ from a dataset D = {(x n, y n)} N n=1 consisting N i.i.d. samples from the distribution of (X, Y). In Bayesian inference, θ is regarded as a random variable, under assuming some prior knowledge in terms of a prior distribution p(θ). The dataset D is then used to update the posterior belief on θ, namely p(θ|D) = p(D|θ)p(θ)/p(D) from the Bayes rule. In many cases, however, computing p(θ|D) through Bayes rule is intractable since it requires to compute intractable integrals. To address the issue, variational inference approximates p(θ|D) by another parametric distribution q φ (θ), and tries to minimize the KL-divergence D KL (q φ (θ) p(θ|D)) between q φ (θ) and p(θ|D). Instead of directly minimizing it, one typically maximizes the variational lower bound L(φ), due to the following: DISPLAYFORM0 where DISPLAYFORM1 In case of complex models, however, expectations in are still intractable. BID22 proposed an unbiased minibatch-based Monte Carlo estimator for them, which can be used when q φ (θ) is representable by θ = f (φ, ε) with a non-parametric noise ε ∼ p(ε). For a minibatch DISPLAYFORM2 Now we can solve optimize L(φ) by stochastic gradient ascent methods, if f is differentiable. For a model having non-Bayesian parameters, say W, we can still apply the above approach by maximizing DISPLAYFORM3 where φ and W can be jointly optimized under DISPLAYFORM4 Structured Bayesian pruning (SBP) BID35 ) is a good example to show how stochastic variational inference can be incorporated into deep neural networks. The SBP framework assumes X to be an object of I features, that is, X = (X i) DISPLAYFORM0. For example, X ∈ R I×H×W can be a convolutional input consisting I channels, of the form X = (DISPLAYFORM1 where W and H denote the width and the height of each channel, respectively. It considers a dropout-like layer with a noise vector θ = (θ i) I i=1 ∼ p noise (θ), which outputs X θ of the same size as X.4 Here, θ is treated as a random vector, and the posterior p(θ|D) is approximated by a fully-factorized truncated log-normal distribution q φ (θ): DISPLAYFORM2 DISPLAYFORM3 where 1 [a,b] denotes the indicator function for the inveral [a, b]. Meanwhile, the prior p(θ) is often chosen by a fully-factorized log-uniform distribution, e.g., Sparse Variational Dropout, and SBP use the truncated version: DISPLAYFORM4 The reason why they use truncations for q φ (θ) and p(θ) is to prevent D KL (q φ (θ) p(θ)) to be improper. Previous works BID23 ignore this issue by implicitly regarding them as truncated distributions on a broad interval, but SBP treats this issue explicitly. Note that, each θ i ∼ q φ (θ i) = LogN(θ i |µ i, σ 2 i) in the noise vector θ can be re-parametrized with a non-parametric uniform noise ε i ∼ U(ε|0, 1) by: DISPLAYFORM5 where DISPLAYFORM6 σi, and Φ denotes the cumulative distribution function of the standard normal distribution. Now one can optimize φ = (µ, σ) jointly with the weights W of a given neural network via stochastic variational inference described in Section A. Unlike, SBP regards W as a non-Bayesian parameter, and the final loss L SBP to optimize becomes DISPLAYFORM7 Here, the KL-divergence term is scaled by α to compensate the trade-off between sparsity and accuracy. In practice, SBP starts from a pre-trained model, and re-trains it using the above loss. Due to the sparsity-inducing behavior of log-uniform prior, θ is forced to become more noisy troughout the re-training. Neurons with θ of signal-to-noise ratio (SNR) below 1 are selected, and removed after the re-training: DISPLAYFORM8 C BAYESIAN PRUNING AND IDENTITY CONNECTIONS SCU requires "training-time removal" of input channels for the channel de-allocation and reallocation to work. But usually, this process should be done carefully since it can make the optimization much difficult and put the network into a bad local minima. In particular, it occurs if we select channels to remove too aggressively. It is known that this issue becomes more pronounced in Bayesian neural networks BID44 BID35 BID30, such as SBP we use in this paper. Recall the variational lower bound objective in, for Bayesian parameters φ and non-Bayesian W. If the gradient of the first term DISPLAYFORM9, that is, to follow the prior p(θ). Unfortunately, in practice, we usually observe this phenomena at the early stage of training, when W are randomly initialized. In that case then, q φ (θ) will become p(θ) too fast because of the "uncertain" W, thereby many channels will be pruned forever, in SBP for example. This problem is usually dealt with in one of two ways: (a) using a pre-trained network as a starting point of W BID35, and (b) a "warm-up" strategy, where the KL-divergence term is rescaled by β that increases linearly from 0 to 1 during training BID44 BID30. In this paper, however, neither methods are used, but instead we have found that the problem can be much eased with identity connections, as it can eliminate a possible cause of the optimization difficulty from removing channels: optimization difficulty from losing information as an input passes through a deep network. The presence of identity connection implies that the information of an input will be fully preserved even in the case when all the parameters in a layer are pruned. This may not be true in models without identity, for example, in VGGNet BID43, one can see that the information of an input will be completely lost if any of the layers removes its entire channels. This suggests us that identity connections can be advantageous not only for scaling up the network architectures, but also for reducing the size of them. ). Then, we have: DISPLAYFORM10 Now, check that ECDS(S i) becomes: DISPLAYFORM11 By the assumption that DISPLAYFORM12 ) for all h, w, we get: DISPLAYFORM13 DISPLAYFORM14 where φ N and Φ N denote the probability distribution function and the cumulative distribution function of the standard normal distribution. Therefore, the desired formula for ECDS(S i) can be obtained by using the linearity of expectation: DISPLAYFORM15 To validate whether the assumption BN(CD(X; W CD); W BN ) i,h,w ∼ N (β i, γ 2 i) holds in modern CNNs, we first observe that, once we ignore the effects from spatial shifting, 5 a necessary condition of the assumption is that (X i,:,:) are identically distributed normal for a given channel X i. This is because BN and CD do not change the "shape" of pixel-wise distributions of X i. From this observation, we conduct a set of experiments focusing on a randomly chosen hidden layer in a DenseNet-40 model. We analyze the empirical distribution of the hidden activation incoming to the layer calculated from CIFAR-10 test dataset. Since the data consists of 10,000 samples, we get an hidden activation X test ∈ R 10000×C×32×32, 6 where C denotes the number of channels of the input. These observations suggest us that the assumption of Proposition 1 can be reasonable except for the boundaries. We also emphasize that these trends we found are also appeared even when the model is not trained at all FIG4, that is, all the weights are randomly initialized, which implies that these properties are not "learned", but come from a structural property of CNN, e.g. equivariance on translation, or the central limit theorem. This observation provides us another support why the ECDS formula stated in Proposition 1 is valid at any time during training. From Θ = (V, W): V consists of (π, g) in CDs, we further rewrite W by (W NC, W C): W NC the parameters in NCs, and W C is the remaining ones. One can safely ignore the effect of V during training of (W NC, W C), since they remain fixed. Recall that each noise θ from a NC is assumed to follow LogN(θ|µ, σ 2). They can be re-written with a noise ε from the standard normal distribution, i.e., θ = f ((µ, σ), ε) = exp (µ + σ · ε), where ε ∼ N (0, 1 2). In such case that each noise θ from NC can be "re-parametrized" with an non-parametric noise and the corresponding parameters φ = (µ, σ), we can then use stochastic variational inference BID22 for the optimization of (W NC, W C) with a minibatch-based stochastic gradient method (see Appendix A for more details). Then, the final loss we minimize for a minibatch {( DISPLAYFORM0 where DISPLAYFORM1 is a sampled vector from the fully-factorized standard normal distribution, and D KL (· ·) denotes the KL-divergence. Although not shown in, an extra regularization term R(W C) can be added to the loss for the non-Bayesian parameters W C, e.g., weight decays. In fact, in our case, i.e. q φ (θ) = LogN(θ|µ, σ 2) and DISPLAYFORM2 As we explain in Appendix B, SBP bypasses this issue by using truncated distributions on a compact interval [a, b] for q φ (θ) and p(θ). We found that, however, this treatment also imposes extra computational overheads on several parts of training process, such as on sampling noises and computing D KL (q φ (θ) p(θ)). These overheads are non-negligible on large models like ResNet or DenseNet, which we are mainly focusing on. Therefore, unlike SBP, here we do not take truncations on q φ (θ) and p(θ) due to practical consideration, assuming an approximated form between the truncated distributions of q φ (θ) and p(θ) on a large interval. Then we can replace each D KL (q φ (θ) p(θ)) in by − log σ for optimization. In other words, each noise θ in NC is regularized to a larger variance, i.e., the more "noisy". We observed that this approximation does not harm much on the performance of SCU. Nevertheless, one should be careful that q φ (θ) and p(θ) should not be assumed as the un-truncated forms itself, but instead as approximated forms of truncated distributions on a large interval, not to make the problem ill-posed. As used in SBP, if they are truncated, the KL-divergence becomes: DISPLAYFORM3 Datasets. We perform our experiments extensively on CIFAR-10 and CIFAR-100 BID24 ) classification datasets. CIFAR-10/100 contains 60,000 RGB images of size 32 × 32 pixels, 50,000 for training and 10,000 for test. Each image in the two datasets is corresponded to one of 10 and 100 classes, respectively, and the number of data is set evenly for each class. We use a common scheme for data-augmentation BID45 BID27 BID11 BID18. ImageNet classification dataset, on the other hand, consists of 1.2 million training images and 50,000 validation images, which are labeled with 1,000 classes. We follow BID17 BID11 for preprocessing of data in training and inference time. Training details. All models in our experiments is trained by stochastic gradient descent (SGD) method, with Nesterov momentum of weight 0.9 without dampening. We use a cosine shape learning rate schedule BID29, i.e., decreasing the learning rate gradually from 0.1 to 0 throughout the training. We set the weight decay 10 −4 by for non-Bayesian parameters of each model. We train each CIFAR model for 300 epochs with mini-batch size 64 following BID18, except for the "DenseNet-BC-190+mixup" models as they are trained for 200 epochs following the original setting. For ImageNet models, on the other hand, we train for 120 epochs with mini-batch size 256. is employed in a model, we initialize W NC = (µ, σ) by (0, e −3), and DISPLAYFORM0. Initializations of W BN and W Conv may differ depending on models, and we follow the initialization scheme of the given model. In our experiments, we follow a pre-defined calling policy when dealloc and realloc will be called throughout training. If dealloc is used, it is called at the end of each epoch of training. On the other hand, if realloc is used, it start to be called after 10% of the training is done, called for every 3 epochs, and stopped in 50% of training is done. The thresholds for dealloc and realloc, i.e. T l and T h, is set by 0.0025 and 0.05, respectively, except for CondenseNet-SCU-182 TAB3, in which T l is adjusted by 0.001 for an effective comparison with the baseline. For all the CIFAR-10/100 models, we re-initialize b i by a random sample from [−1.5, 1.5] × [−1.5, 1.5] pixels uniformly whenever a channel slice S i is re-open via realloc process. We set the weight decay on each b i to 10 −5 separately from the other parameters. For the ImageNet TAB2, however, we did not jointly train b for faster training. Instead, each b i is set fixed unless it is re-initialized via realloc. In this case, we sampled a point from [−2.5, 2.5] × [−2.5, 2.5] pixels uniformly for the re-initialization. We found that this simple reallocation scheme can also improve the efficiency of SCU. In general, every model we used here forms a stack of multiple bottlenecks, where the definition of each bottleneck differs depending on its architecture (see TAB6). Each stack is separated into three (CIFAR-10/100) or four (ImageNet) stages by average pooling layers of kernel 2 × 2 to perform down-sampling. Each of the stages consists N bottleneck blocks, and we report which N is used for all the tested models in TAB7. The whole stack of each model follows a global average pooling layer BID27 ) and a fully connected layer, and followed by single convolutional layer (See TAB8). There exist some minor differences between the ing models and the original papers BID11 BID18 BID49. In ResNet and ResNeXt models, we place an explicit 2 × 2 average pooling layer for down-sampling, instead of using convolutional layer of stride 2. Also, we use a simple zero-padding scheme for doubling the number of channels between stages. In case of DenseNet, on the other hand, our DenseNet models are different from DenseNet-BC proposed by BID18, in a sense that we do not place a 1 × 1 convolutional layer between stages (which is referred as the "compression" layer in the original DenseNet). Nevertheless, we observed that the models we used are trained as well as the originals.

Represent the TLDR for finding the corresponding scientific article.

We propose a new module that improves any ResNet-like architectures by enforcing "channel selective" behavior to convolutional layers

We propose accelerating Batch Normalization (BN) through sampling less correlated data for reduction operations  with regular execution pattern, which achieves up to 2x and 20% speedup for BN itself and the overall training, respectively.