Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Designing architectures for deep neural networks requires expert knowledge and substantial computation time. We propose a technique to accelerate architecture selection by learning an auxiliary HyperNet that generates the weights of a main model conditioned on that model's architecture. By comparing the relative validation performance of networks with HyperNet-generated weights, we can effectively search over a wide range of architectures at the cost of a single training run. To facilitate this search, we develop a flexible mechanism based on memory read-writes that allows us to define a wide range of network connectivity patterns, with ResNet, DenseNet, and FractalNet blocks as special cases. We validate our method (SMASH) on CIFAR-10 and CIFAR-100, STL-10, ModelNet10, and Imagenet32x32, achieving competitive performance with similarly-sized hand-designed networks. The high performance of deep neural nets is tempered by the cost of extensive engineering and validation to find the best architecture for a given problem. High-level design decisions such as depth, units per layer, and layer connectivity are not always obvious, and the success of models such as Inception , ResNets BID12, FractalNets BID18 and DenseNets BID14 demonstrates the benefits of intricate design patterns. Even with expert knowledge, determining which design elements to weave together requires ample experimentation. In this work, we propose to bypass the expensive procedure of fully training candidate models by instead training an auxiliary model, a HyperNet BID11, to dynamically generate the weights of a main model with variable architecture. Though these generated weights are worse than freely learned weights for a fixed architecture, we leverage the observation BID19 ) that the relative performance of different networks early in training (i.e. some distance from the eventual optimum) often provides a meaningful indication of performance at optimality. By comparing validation performance for a set of architectures using generated weights, we can approximately rank numerous architectures at the cost of a single training run. To facilitate this search, we develop a flexible scheme based on memory read-writes that allows us to define a diverse range of architectures, with ResNets, DenseNets, and FractalNets as special cases. We validate our one-Shot Model Architecture Search through HyperNetworks (SMASH) for Convolutional Neural Networks (CNN) on CIFAR-10 and CIFAR-100 , Imagenet32x32 BID6, ModelNet10 , and STL-10 BID7, achieving competitive performance with similarly-sized hand-designed networks. Modern practical methods for optimizing hyperparameters rely on random search BID3 or Bayesian Optimization (BO) (; BID15, treating the model performance as a black box. While successful, these methods require multiple training runs for evaluation (even when starting with a good initial model) and, in the case of BO, are not typically used to specify variable-length settings such as the connectivity and structure of the model under consideration. Relatedly, bandit-based methods BID19 provide a framework for efficiently exploring the hyperparameter space by employing an adaptive early-stopping strategy, allocating more resources to models which show promise early in training. Evolutionary techniques BID9 Stanley et al.;; ) offer a flexible approach for discovering variegated models from trivial initial conditions, but often struggle to scale to deep neural nets where the search space is vast, even with enormous compute .Reinforcement learning methods BID2 ) have been used to train an agent to generate network definitions using policy gradients. These methods start from trivial architectures and discover models that achieve very high performance, but can require twelve to fifteen thousand full training runs to arrive at a solution. The method that most resembles our own is that of Saxe et al. , who propose to efficiently explore various architectures by training only the output layer of convolutional networks with random convolutional weights. While more efficient than fully training an entire network end-toend, this method does not appear to scale to deeper networks . Our method is conceptually similar, but replaces random weights with weights generated through HyperNets BID11, which are one of a class of techniques for dynamically adapting weights through use of an auxiliary model BID8 BID16; ). In our case we learn a transform from a binary encoding of an architecture to the weight space, rather than learning to adapt weights based on the model input. Our method is explicitly designed to evaluate a wide range of model configurations (in terms of connectivity patterns, depth, and width) but does not address other hyperparameters such as regularization, learning rate schedule, weight initialization, or data augmentation. Unlike the aforementioned evolutionary or RL methods, we explore a somewhat pre-defined design space, rather than starting with a trivial model and designating a set of available network elements. While we still consider a rich set of architectures, our method cannot discover wholly new structures on its own and is constrained in that it only dynamically generates a specific subset of the model parameters. Additionally, although our method is not evolutionary, our encoding scheme is reminiscent of CGP BID21.Stochastic regularization techniques such as Dropout , Swapout , DropPath BID18 or stochastic depth BID13 superficially resemble our method, in that they obtain variable configurations by randomly dropping connectivity paths in a fixed network architecture. Convolutional neural fabrics , for example, leverage this idea to attempt to train one large network as an implicit ensemble of all subnetworks produced through dropping paths. A key element that sets our method apart is that the weights for each node in our network are dynamically generated, rather than fixed; if a Dropout ensemble were to visit a unit that had not previously been trained, the unit's weights would be completely untuned. Our method generalizes even to previously unseen architectures, and the network we train under stochastic conditions is merely a proxy we use to evaluate network configurations, rather than the final model. In SMASH (Algorithm 1), our goal is to rank a set of neural network configurations relative to one another based on each configuration's validation performance, which we accomplish using weights generated by an auxiliary network. At each training step, we randomly sample a network architecture, generate the weights for that architecture using a HyperNet, and train the entire system end-to-end through backpropagation. When the model is finished training, we sample a number of random architectures and evaluate their performance on a validation set, using weights generated by the HyperNet. We then select the architecture with the best estimated validation performance and train its weights normally. Sample random c and evaluate error on validation set E v = f c (H(c), x v ) end loop Fix architecture and train normally with freely-varying weights W SMASH comprises two core components: the method by which we sample architectures, and the method by which we sample weights for a given architecture. For the former, we develop a novel "memory-bank" view of feed-forward networks that permits sampling complex, branching topologies, and encoding said topologies as binary vectors. For the latter, we employ a HyperNet BID11 ) that learns to map directly from the binary architecture encoding to the weight space. We hypothesize that so long as the HyperNet learns to generate reasonable weights, the validation error of networks with generated weights will correlate with the performance when using normally trained weights, with the difference in architecture being the primary factor of variation. Throughout the paper, we refer to the entire apparatus during the first part of training (the HyperNet, the variable architecture main network, and any freely learned main network weights) as the SMASH network, and we refer to networks trained with freely learned weights in the second stage as ing networks. In order to explore a broad range of architectures with variable depth, connectivity patterns, layer sizes and beyond, we require a flexible mechanism for defining such architectures, which we can also easily encode into a conditioning vector for the HyperNet. To this end, we introduce a "memory-bank" view of feed-forward networks. Rather than viewing a network as a series of operations applied to a forward-propagating signal, we view a network as having a set of memory banks (initially tensors filled with zeros) which it can read and write. Each layer is thus an operation that reads data from a subset of memory, modifies the data, and writes the to another subset of memory. For a single-branch architecture, the network has one large memory bank it reads and overwrites (or, for a ResNet, adds to) at each op. A branching architecture such as a DenseNet reads from all previously written banks and writes to an empty bank, and a FractalNet follows a more complex read-write pattern, as shown in FIG0.Our base network structure consists of multiple blocks FIG1 ), where each block has a set number of memory banks at a given spatial resolution, with successively halved spatial resolutions as in most CNN architectures. Downsampling is accomplished via a 1x1 convolution followed by average pooling BID14, with the weights of the 1x1 convolution and the fully-connected output layer being freely learned, rather than generated. When sampling an architecture, the number of banks and the number of channels per bank are randomly sampled at each block. When defining each layer within a block, we randomly select the read-write pattern and the definition of the op to be performed on the read data. When reading from multiple banks we concatenate the read tensors along the channel axis, and when writing to banks we add to the tensors currently in each bank. For all reported experiments, we only read and write from banks at one block (i.e. one spatial resolution), although one could employ resizing to allow reading and writing from any block, similar to .Each op comprises a 1x1 convolution (reducing the number of incoming channels), followed by a variable number of convolutions interleaved with nonlinearities, as shown in FIG1 (a). We randomly select which of the four convs are active, along with their filter size, dilation factor, number of groups, and the number of output units (i.e. the layer width). The number of output channels of the 1x1 conv is some factor of the width of the op, chosen as the "bottleneck ratio" hyperparameter. The weights for the 1x1 convolution are generated by the HyperNet as described in Section 3.2, while the other convolutions are normally learned parameters. To ensure variable depth, we learn a single set of 4 convolutions for each block, and share it across all ops within a block. We limit the max filter size and number of output units, and when a sampled op uses less than the maximum of either, we simply slice the weight tensor to the required size. The fixed transition convolutions and output layer employ this same slicing based on the number of incoming non-empty memory banks. Exact details regarding this scheme are available in the appendix. In designing our scheme, we strive to minimize the number of static learned parameters, placing the majority of the network's capacity in the HyperNet. A notable consequence of this goal is that we only employ BatchNorm BID15 at downsample layers and before the output layer, as the layer-specific running statistics are difficult to dynamically generate. We experimented with several different normalization schemes including WeightNorm , LayerNorm BID1 and NormProp BID0 but found them to be unstable in training. Instead, we employ a simplified version of WeightNorm where we divide the entirety of each generated 1x1 filter by its Euclidean norm (rather than normalizing each channel separately), which we find to work well for SMASH and to only in a minor drop in accuracy when employed in fixed-architecture networks. No other convolution within an op is normalized. A HyperNet BID11 ) is a neural net used to parameterize the weights of another network, the main network. For a Static HyperNet with parameters H, the main network weights W are some function (e.g. a multilayer perceptron) of a learned embedding z, such that the number of learned weights is typically smaller than the full number of weights for the main network. For a Dynamic HyperNet, the weights W are generated conditioned on the network input x, or, for recurrent networks, on the current input x t and the previous hidden state h t−1.We propose a variant of a Dynamic HyperNet which generates the weights W based on a tensor encoding of the main network architecture c. Our goal is to learn a mapping W = H(c) that is reasonably close to the optimal W for any given c, such that we can rank each c based on the validation error using HyperNet-generated weights. We thus adopt a scheme for the layout of c to enable sampling of architectures with wildly variable topologies, compatibility with the toolbox available in standard libraries, and to make c's dimensions as interpretable as possible. Our HyperNet is fully convolutional, such that the dimensionality of the output tensor W varies with the dimensionality of the input c, which we make a 4D tensor of the standard format Batch x Channel x Height x Width; the batch size is set to 1 so that no output elements are wholly independent. This allows us to vary the depth and width of the main network by increasing the height or width of c. Under this scheme, every slice of the spatial dimensions of W corresponds to a specific subset of c. Information describing the op that uses that W subset is embedded in the channel dimension of the corresponding c slice. For example, if an op reads from memory banks 1, 2, and 4, then writes to 2 and 4, then the first, second, and fourth channels of the corresponding slice of c will be filled with 1s (indicating the read pattern) and the sixth and eighth channels of that slice will be filled with 1s (indicating the write pattern). The rest of the op description is encoded in the remaining channels in a similar 1-hot fashion. We only encode into the width-wise extent of c based on the number of output units of the op, so elements of c which do not correspond to any elements of W are empty. A naïve implementation of this scheme might require the size of c to be equal to the size of W, or have the HyperNet employ spatial upsampling to produce more elements. We found these choices poor, and instead employ a channel-based weight-compression scheme that reduces the size of c and keeps the representational power of the HyperNet proportional to that of the main networks. We make the spatial extent of c some fraction k of the size of W, and place k units at the output of the HyperNet. We then reshape the ing 1 × k × height × width tensor to the required size of W. k is chosen to be DN 2, where N is the minimum memory bank size, and D is a "depth compression" hyperparameter that represents how many slices of W correspond to a single slice of c. Complete details regarding this scheme (and the rest of the encoding strategy) are available in Appendix B. We apply SMASH to several datasets, both for the purposes of benchmarking against other techniques, and to investigate the behavior of SMASH networks. Principally, we are interested in determining whether the validation error of a network using SMASH-generated weights (the "SMASH score") correlates with the validation of a normally trained network, and if so, the conditions under which the correlation holds. We are also interested in the transferability of the learned architectures to new datasets and domains, and how this relates to normal (weight-wise) transfer learning. Our publicly available code 1 is written in PyTorch BID22 to leverage dynamic graphs, and explicitly defines each sampled network in line with the memory-bank view to avoid obfuscating its inner workings behind (potentially more efficient) abstractions. We omit many hyperparameter details for brevity; full details are available in the appendices, along with visualizations of our best-found architectures. Red line is a least-squares best fit. First, we train a SMASH network for 300 epochs on CIFAR-100, using a standard annealing schedule BID14, then sample 250 random architectures and evaluate their SMASH score on a held-out validation set formed of 5,000 random examples from the original training set. We then sort the architectures by their SMASH score and select every 5th architecture for full training and evaluation, using an accelerated training schedule of 30 epochs. For these networks, which we deem SMASHv1, the architecture uses a fixed memory bank size (though a variable number of banks in each block), a single fixed 3x3 conv in the main body of the op (rather than the variable 2x2 array of convs), a single group, and a fixed bottleneck ratio of 4. The variable elements comprise the read-write pattern, the number of output units, and the dilation factor of the 3x3 filter. When sampling architectures, we allocate a random, upper-bounded compute budget to each block. Under these conditions, we observe a correlation (FIG3) between the SMASH score and the true validation performance, suggesting that SMASH-generated weights can be used to rapidly compare architectures. It is critical not to overstate this claim; this test is arguably a single datapoint indicating that the correlation holds in this scenario, but neither guarantees the correlation's generality nor implies the range of conditions for which it will hold. We thus conduct a more thorough investigation of this correlation. We expect, based on preliminary experiments detailed in Appendix C, that the two key variables determining the strength of the correlation are the capacity of the HyperNet, and the ratio of HyperNetgenerated weights to freely learned weights. For the former, we reason that if the HyperNet lacks representational capacity, then it will be unable to learn an acceptable mapping between architectures and weights, and the generated weights will be too far from optimal to permit ranking. For the latter, we reason that if the main network has too many freely-learned weights relative to the number of dynamically generated weights, too much of its capacity will be "static" and ill-adapted to the wide range of architectures considered. Following these hypotheses, we repeat the first experiment multiple times, varying the architecture of the HyperNet as well as the g hyperparameter, which controls the maximum layer width and consequentially the ratio of freely-learned vs dynamic weights. A higher value of g corresponds to relatively fewer dynamically generated weights. We consider three different ratios and five different HyperNet configurations. For each setting of g we begin by training five different SMASH networks, each employing one of the candidate configurations. After training, we sample 500 random architectures as before, and then evaluate all five SMASH scores, rank them according to their average, and then train every 10th ing network normally. We train the ing networks for a full 100 epochs (as opposed to the previous shortened schedule) and repeat training runs with a total of 5 different random seeds. Finally, we evaluate the strength and significance of the correlation between SMASH score and ing validation performance (averaged for each architecture across runs) using Pearson's R. The of this study are reported in TAB1. The first column details the choices of growth rate and depth for each of the three blocks of the HyperNet, and the second column the number of parameters for each architecture. The values of g=4, g=8, and g=16 roughly correspond to average freely-learned vs dynamic weight ratios of 1:4, 1:2, and 2:1, though these vary somewhat with individual sampled architectures. Several trends are visible in this table. First, for g=4, we note that the strength of correlation increases with increased HyperNet capacity up to the fourth architecture with 12M parameters, but the correlation breaks down for the largest architecture. This suggests that the strength of correlation can indeed depend on the capacity of the HyperNet, but also that the HyperNet is either potentially susceptible to overfitting or that too large a HyperNet becomes too difficult to train and produces poor weights. Second, for g=8, we note that the correlation varies little with changes in architecture, but is weaker than the best correlations from g = 4. This suggests that for a middling ratio, the capacity of the HyperNet has less effect on the correlation as there are fewer weights for it to adapt, but the strength of the correlation is weaker as each set of sampled weights is consequentially less optimal than for g = 4. Third, we note a complete breakdown of the correlation for g=16, which is in line with our expectation that placing too much capacity in a single set of statically learned weights will prevent the SMASH network from properly adapting to individual architectures. As an additional test of our method, we examine whether or not the HyperNet has learned to take into account the architecture definition in c, or whether it ignores c and naively generates an unconditional subspace of weights that happen to work well. We "trick" the HyperNet by sampling one architecture, but asking it to generate the weights for a different architecture by corrupting the encoding tensor c (e.g. by shuffling the dilation values). For a given architecture, we find that SMASH validation performance is consistently highest when using the correct encoding tensor, suggesting that the HyperNet has indeed learned a passable mapping from architecture to weights. Following this, we posit that if the HyperNet learns a meaningful mapping W = H(c), then the classification error E = f (W, x) = f (H(c), x) can be backpropagated to find dE dc, providing an approximate measure of the error with respect to the architecture itself. If this holds true, then perturbing the architecture according to the dE dc vector (within the constraints of our scheme) should allow us to guide the architecture search through a gradient descent-like procedure. Our preliminary tests with this idea did not yield better SMASH scores than randomly perturbing the architectural definition, though we suspect that this was in part due to our lack of an intuitively satisfying update rule for the discrete architecture space. Models with weights initially learned on one large dataset frequently outperform models trained from scratch on a smaller dataset; it follows that architectures might display the same behavior. We test on STL-10 (BID7, a small dataset of 96x96 images similar to the CIFAR datasets. We compare the performance of the best-found architecture from CIFAR-100 (with weights trained from scratch on STL-10) to the best-found architecture from running SMASH on STL-10, and a WRN baseline. For these experiments, we make use of the full 5,000 images in the training set; in the following section we also include comparisons against a WRN baseline using the recommended 10-fold training split. In this case, we find that the best-found architecture from CIFAR-100 outperforms the best-found architecture from STL-10, achieving 17.54% and 20.275% error, respectively. For reference, a baseline WRN28-10 and WRN40-4 achieve respective 15.43% and 16.06% errors. This presents an interesting phenomenon: one the one hand, one might expect the architecture discovered on STL-10 to be better-tuned to STL-10 because it was specifically learned on that dataset. On the other hand, CIFAR-100 has significantly more training examples, potentially making it a better dataset for distinguishing between good architectures, i.e. accuracy on CIFAR-100 is more indicative of generality. The better performance of the architecture found on CIFAR-100 would seem to favor the latter hypothesis, suggesting that architecture search benefits from larger training sets more so than domain specificity. We next investigate how well our best-found CIFAR-100 architecture performs on ModelNet10 , a 3D object classification benchmark. We train on the voxelated instances of the ModelNet10 training set using the settings of BID4, and report accuracy on the ModelNet10 test set. Our 8M parameter model achieves an accuracy of 93.28%, compared to a 93.61% accuracy from a hand-designed Inception-ResNet BID4 with 18M parameters trained on the larger ModelNet40 dataset. We run SMASH on CIFAR-10 and 100, augmenting our search space from the initial correlation experiment to include variable filter sizes, variable groups, and the full variable op structure shown in FIG1, and denote the ing networks SMASHv2. We report the final test performance of the two ing networks with the highest SMASH scores on CIFAR-10 and 100 in TAB2.Next, we take our best-found SMASHv2 architecture from CIFAR-100 and train it on STL-10 using the recommended 10-fold training splits, and ImageNet32x32 BID6. We compare against Wide ResNet baselines from our own experiments in TAB3 reported by BID6 in 4. Noting the better performance of WRN40-4 on STL-10, we also train a variant of our best architecture with only a single main convolution and 3x3 filters, to comparably reduce the number of parameters. Our SMASHv2 nets with 16M parameters achieve final test errors of 20.60% on CIFAR-100 and 4.03% on CIFAR-10. This performance is not quite on par with state-of-the-art hand-designed networks, but compares favorably to other automatic design methods that employ RL BID2 ) or evolutionary methods . Our networks outperform Large-Scale Evolution despite requiring significantly less time to discover (though not starting from trivial models) and 10 orders of magnitude less compute. Our method outperforms MetaQNN BID2 but lags behind Neural Architecture Search , though both methods require vastly more computation time, and unlike Neural Architecture Search, we do not postprocess our discovered architectures through grid search. We believe this work opens a number of future research paths. The SMASH method itself has several simplistic elements that might easily be improved upon. During training, we sample each element of the configuration one-by-one, independently, and uniformly among all possible choices. A more intelligent method might employ Bayesian Optimization or HyperBand BID19 to guide the sampling with a principled tradeoff between exploring less-frequently sampled architectures against those which are performing well. One might employ a second parallel worker constantly evaluating validation performance throughout training to provide signal to an external optimizer, and change the optimization objective to simultaneously maximize performance while minimizing computational costs. One could also combine this technique with RL methods and use a policy gradient to guide the sampling. Another simple technique (which our code nominally supports) is using the HyperNet-generated weights to initialize the ing network and accelerate training, similar to Net2Net BID5.Our architecture exploration is fairly limited, and for the most part involves variable layer sizes and skip connections. One could envision a multiscale SMASH that also explores low-level design, varying things such as the activation at each layer, the order of operations, the number of convolutions in a given layer, or whether to use convolution, pooling, or more exotic blocks. One might also search over network-wide design patterns rather than randomly selecting the read-write pattern at each layer. Alternatively, one could consider varying which elements of the network are generated by the HyperNet, which are fixed learned parameters, and one might even make use of fixed unlearned parameters such as Gabor Filters or wavelets. Our memory-bank view also opens up new possibilities for network design. Each layer's read and write operations could be designed to use a learned softmax attention mechanism, such that the read and write locations are determined dynamically at inference time. We also do not make use of memory in the traditional "memory-augmented" sense, but we could easily add in this capacity by allowing information in the memory banks to persist, rather than zeroing them at every training step. We also only explore one definition of reading and writing, and one might for example change the "write" operation to either add to, overwrite, or perhaps even multiply (a la gated networks ) the existing tensor in a given bank. In this work, we explore a technique for accelerating architecture selection by learning a model over network parameters, conditioned on the network's parametric form. We introduce a flexible scheme for defining network connectivity patterns and generating network weights for highly variable architectures. Our demonstrate a correlation between performance using suboptimal weights generated by the auxiliary model and performance using fully-trained weights, indicating that we can efficiently explore the architectural design space through this proxy model. Our method achieves competitive, though not state-of-the-art performance on several datasets. We briefly describe they hyperparameters used for the SMASH network in our experiments. The SMASHv1 network has memory banks with N = 6 channels each, a maximum of 240 memory banks per block (though on average less than half that number), and a depth compression ratio of D = 3. Each layer's number of units is uniformly sampled between 6 and 42 (along even multiples of 6), and its dilation factor is uniformly sampled between 1 and 3 (with 1 representing no dilation and 3 representing 2 zeros inserted between each filter). We employ a constant bottlneck ratio of 4 as in BID14, so the output of the HyperNet-generated 1x1 convolution is always 4 times the number of output units. We constrain the main network to have a maximum budget of 16M parameters, though due to our sampling procedure we rarely sample networks with more than 5M parameters. Our SMASHv2 networks have variable memory bank sizes at each blocks, which we constrain to be multiples of N = 8 up to Nmax = 64. We sample filter sizes from, and sample dilation values such that the max spatial extent of a filter in any direction is 9. We sample convolutional groups as factors of the base N value (so for these networks). We put some hand-designed priors on the choice of op configuration (i.e. which convolutions are active), giving slight preference to having all four convolutions active. For SMASHv2 nets we employ a slightly more complex bottleneck ratio: the output of the 1x1 conv is equal to the number of incoming channels while that number is less than twice the number of output units, at which point it is capped (so, a maximum bottleneck ratio of 2).Our HyperNet is a DenseNet, designed ad-hoc to resemble the DenseNets in the original paper BID14 within the confines of our encoding scheme, and to have round numbers of channels. It consists of a standard (non-bottleneck) Dense Block with 8 3x3 convolutional layers and a growth rate of 10, followed by a 1x1 convolution that divides the number of channels in two, a Dense Block with 10 layers and a growth rate of 10, another compressing 1x1 convolution, a Dense Block with 4 layers and a growth rate of 10, and finally a 1x1 convolution with the designated number of output channels. We use Leaky ReLU with a negative slope of 0.02 as a defense against NaNs, as standard ReLU would obfuscate their presence when we had bugs in our early code revisions; we have not experimented with other activations. We adopt a scheme for the layout of the embedding tensor to facilitate flexibility, compatibility with the convolutional toolbox available in standard libraries, and to make each dimension interpretable. First, we place some constraints on the hyperparameters of the main network: each layer's number of output units must be divisible by the memory bank size N and be less than Nmax, and the number of input units must be divisible by D, where N is the number of channels in each memory bank, and Nmax and D are chosen by the user. Applying these constraints allows us to reduce the size of the embedding vector by DN 2, as we will see shortly. The input to a standard 2D CNN is x ∈ R B×C×H×L, where B, C, H, and L respectively represent the Batch, Channel, Height, and Length dimensions. Our embedding tensor is c ∈ R DISPLAYFORM0 where M is the maximum number of memory banks in a block, dmax is the maximum kernel dilation, and n ch is the sum total of input channels to the 1x1 convs of the main network. The conditional embedding c is a one-hot encoding of the memory banks we read and write at each layer. It has 2M + dmax channels, where the first M channels represent which banks are being read from, the next M channels represent which banks are being written to, and the final dmax channels are a one-hot encoding of the dilation factor applied to the following 3x3 convolution. The height dimension corresponds to the number of units at each layer, and the length dimension corresponds to the network depth in terms of the total number of input channels. We keep the Batch dimension at 1 so that no signals propagate wholly independently through the HyperNet. FIG2 shows an example of a small randomly sampled network, its equivalent memory bank representation, and how the read-write pattern is encoded in c. The dilation encoding is omitted in FIG2 for compactness. Our HyperNet has 4DN 2 output channels, such that the output of the HyperNet is DISPLAYFORM1 2 ×n ch /D, which we reshape to W ∈ R Nmax×4Nmaxn ch ×1×1. We generate the weights for the entire main network in a single pass, allowing the HyperNet to predict weights at a given layer based on weights at nearby layers. The HyperNet's receptive field represents how far up or down the network it can look to predict parameters at a given layer. As we traverse the main network, we slice W along its second axis according to the number of incoming channels, and slice along the first axis according to the width of the given layer. At each training step, we sample a network architecture block-by-block, with a random (but upper bounded) computational budget allocated to each block. For SMASHv1, We use memory banks with N = 6 channels each, constrain the number of incoming memory banks to be a multiple of 3 (D = 3), and constrain the number of output units at each layer to be a multiple of 6 (with Nmax = 42) for compatibility with the memory layout. Our HyperNet is a 26 layer DenseNet, each layer of which comprises a Leaky ReLU activation followed by a 3x3 convolution with simplified WeightNorm and no biases. We do not use bottleneck blocks, dropout, or other normalizers in the HyperNet. When sampling our SMASHv2 networks for evaluation, we first sample 500 random architectures, then select the architecture with the highest score for further evaluation. We begin by perturbing this architecture, with a 5% chance of any individual element being randomly resampled, and evaluate 100 random perturbations from this base. We then proceed with 100 perturbations in a simple Markov Chain, where we only accept an update if it has a better SMASH score on the validation set. When training a ing network we make all parameters freely learnable and replace simple WeightNorm with standard BatchNorm. We tentatively experimented with using SMASH generated weights to initialize a ing net, but found standard initialization strategies to work better, either because of a yet-undiscovered bug in our code, or because of the disparity between the dynamics of the SMASH network using WeightNorm and the ing network using BatchNorm. In line with our claim of "one-shot" model search, we keep our exploration of the SMASH design space to a minimum. We briefly experimented with three different settings for N and D, and use a simple, ad-hoc DenseNet architecture for the HyperNet, which we do not tune. We investigated the choice of architecture while examining the SMASH correlation, but stick to the original ad-hoc design for all benchmark experiments. When training SMASH, we use Adam BID17 with the initial parameters proposed by When training a ing network, we use Nesterov Momentum with an initial step size of 0.1 and a momentum value of 0.9. For all tests other than the initial SMASHv1 experiments, we employ a cosine annealing schedule without restarts BID10.For the CIFAR experiments, we train the SMASH network for 100 epochs and the ing networks for 300 epochs, using a batch size of 50 on a single GPU. On ModelNet10, we train for 100 epochs. On ImageNet32x32, we train for 55 epochs. On STL-10, we train for 300 epochs when using the full training set, and 500 epochs when using the 10-fold training splits. For ModelNet-10 tests, we employ 3x3x3 filters (rather than fully variable filter size) to enable our network to fit into memory and keep compute costs manageable, hence why our model only has 8M parameters compared to the base 16M parameters. All of our networks are pre-activation, following the order BN-ReLU-Conv if BatchNorm is used, or ReLU-Conv if WeightNorm is used. Our code supports both pre-and post-activation, along with a variety of other options such as which hyperparameters to vary and which to keep constant. While investigating the SMASH correlation, we initially conducted two brief experiments to guide the choice of experiments in the remainder of the investigation. First, experiment, we train a low-budget SMASH network (to permit more rapid testing) with a much smaller HyperNet relative to the main network (though still the standard ratio of generated to freely learned weights). We expect the decreased capacity HyperNet to be less able to learn to generate good weights for the full range of architectures, and for the correlation between SMASH score and true performance to therefore be weak or nonexistent. The of this study are shown in FIG4 (a), where we arguably observe a breakdown of the correlation. In addition to repeat trials for each ing net, we also train two additional SMASH networks with different random seeds and compare their predicted performance against the initial SMASH net in FIG4, to get a brief sense for how these values can vary across training runs. Next, we train a high-budget SMASH network and drastically increase the ratio of normally learned parameters to HyperNet-generated parameters, such that the majority of the net's model capacity is in non-generated weights. Under these conditions, the validation errors achieved with SMASH-generated weights are much lower than validation errors achieved with an equivalent SMASH network with the typical ratio, but the ing top models are not as performant and we found that (in the very limited number of correlation tests we performed) the SMASH score did not correlate with true performance. This highlights two potential pitfalls: first, if the HyperNet is not responsible for enough of the network capacity, then the aggregate generated and learned weights may not be sufficiently well-adapted to each sampled architecture, and therefore too far from optimal to be used in comparing architectures. Second, comparing SMASH scores for two separate SMASH networks can be misleading, as the SMASH score is a function of both the normally learned and generated weights, and a network with more fixed weights may achieve better SMASH scores even if the ing nets are no better. [,,,

Represent the short summary for finding the corresponding scientific article (long)!

A technique for accelerating neural architecture selection by approximating the weights of each candidate architecture instead of training them individually.

Represent the TLDR for finding the corresponding scientific article

We present a new approach, SNIP, that is simple, versatile and interpretable; it prunes irrelevant connections for a given task at single-shot prior to training and is applicable to a variety of neural network models without modifications.


Question:
represent the article for finding a single-sentence tldr

E.g. given 'The recent “Lottery Ticket Hypothesis” paper by Frankle & Carbin showed that a simple approach to creating sparse networks (keep the large weights) in models that are trainable from scratch, but only when starting from the same initial weights. The performance of these networks often exceeds the performance of the non-sparse base model, but for reasons that were not well understood. In this paper we study the three critical components of the Lottery Ticket (LT) algorithm, showing that each may be varied significantly without impacting the overall . Ablating these factors leads to new insights for why LT networks perform as well as they do. We show why setting weights to zero is important, how signs are all you need to make the re-initialized network train, and why masking behaves like training. Finally, we discover the existence of Supermasks, or masks that can be applied to an untrained, randomly initialized network to produce a model with performance far better than chance (86% on MNIST, 41% on CIFAR-10). Many neural networks are over-parameterized BID0 BID1, enabling compression of each layer BID1 BID14 BID4 or of the entire network BID9. Some compression approaches enable more efficient computation by pruning parameters, by factorizing matrices, or via other tricks BID4 BID5 BID8 BID10 BID11 BID12 BID13 BID14 BID15 BID16. A recent work by Frankle & Carbin BID2 presented a simple algorithm for finding sparse subnetworks within larger networks that can meet or exceed the performance of the original network. Their approach is as follows: after training a network, set all weights smaller than some threshold to zero BID2, rewind the rest of the weights to their initial configuration BID3, and then retrain the network from this starting configuration but with the zero weights frozen (not trained). See Section S1 for a more formal description of this algorithm. In this paper we perform ablation studies along the above three dimensions of variability, considering alternate mask criteria (Section 2), alternate mask-1 actions (Section 3), and alternate mask-0 actions (Section 4). These studies in aggregate reveal new insights for why lottery ticket networks work as they do. Along the way we also discover the existence of Supermasks-masks that produce above-chance performance when applied to untrained networks (Section 5). We begin our investigation with a study of different Mask Criteria, or functions that decide which weights to keep vs. prune. In this paper, we define the mask for each individual weight as a function of the weight's values both at initialization and after training: M (w i, w f). We can visualize this function as a set of decision boundaries in a 2D space as shown in FIG0. In BID2, the mask criterion simply keeps weights with large final value; we refer to this as the large_final mask, M (w i, w f) = |w f |.In addition to large_final, we also experimented with the inverse, small_final, versions that evaluate weights based on their initial magnitudes instead, large_init and small_init, versions that select for both, large_init_large_final and small_init_small_final, two mask criteria that evaluate how far weights moved, magnitude_increase and movement, and a control, random, that chooses masks randomly. These nine masks are depicted along with their associated equations in FIG1. In this section and throughout the remainder of the paper, we follow the experimental framework from BID2 and perform iterative pruning experiments on a fully-connected network (FC) trained on MNIST BID7 and on three convolutional networks (Conv2, Conv4, and Conv6) trained on CIFAR-10 BID6. For more achitecture and training details, see Section S2.Results of these pruning experiments are shown in FIG0. Note that the first six criteria out of the eight form three opposing pairs; in each case, we see when one member of the pair performs better than the random baseline, the opposing member performs worse than it. We see that the mask criterion large_final, as employed by the LT algorithm, is indeed a competitive mask criteria. However, the magnitude_increase criterion turns out to work just as well as the large_final criterion, and in some cases significantly better. In Section 4, we provide an explanation to why some mask criteria work well while others don't.3 Mask-1 actions: show me a signNow that we have explored various ways of choosing which weights to keep and prune, we will consider what values to set for the kept weights. In particular, we want to explore an interesting observation in BID2 which showed that the pruned, skeletal LT networks train well when you rewind to its original initialization, but degrades in performance when you randomly reinitialize the network. Why does reinitialization cause LT networks to train poorly? Which components of the original initialization are important? We evaluate a number of variants of reinitialization to investigate:• "Reinit" experiments: reinitialize kept weights based on the original initialization distribution • "Reshuffle" experiments: reinitialize while respecting the original distribution of remaining weights in that layer by reshuffling the kept weights' initial values • "Constant" experiments: reinitialize by setting mask-1 weight values to a positive or negative constant, with the constant set to be the standard deviation of each layer's original initialization. Thus every weight on a layer becomes one of three values: −α, 0, or α. We find that none of these three variants alone are able to train as well as the original LT network, shown as dashed lines in FIG1. However, all three variants work better when we ensure that the new values of the kept weights are of the same sign as their original initial values. These are shown as solid color lines in FIG1. Clearly, the common factor in all working variants including the original rewind action is the sign. As long as you keep the sign, reinitialization is not a deal breaker; in fact, even setting all kept weights to a constant value consistently performs well! What should we do with weights that are pruned? Typical network pruning procedures perform two actions on pruned weights: set them to zero, and freeze them in subsequent training. However, it is unclear which of these two components leads to the increased performance in LT networks. To separate the two factors, we run a simple experiment: we reproduce the LT iterative pruning experiments in which network weights are masked out in alternating train/mask/rewind cycles, but try an additional treatment: freeze masked weights at their initial values instead of at zero. If pruned weights are truly unimportant, we would expect that setting them to any other sensible values, such as their original initializations, should lead to a similarly performing network. Figure 3 shows the for this experiment. We find that networks perform significantly better when weights are frozen specifically at zero than at random initial values. For these networks masked via the LT large_final criterion, zero would seem to be a particularly good value to set weights to when they had small final values. So why does zero work better than initial values? One hypothesis is that the mask criterion we use tends to mask to zero those weights that were headed toward zero anyway. To test out this hypothesis, we run another experiment interpolated between the previous two: for any weight to be frozen, we freeze it to zero if it moved toward zero over the course of training, and we freeze it at its random initial value if it moved away from zero. Results are shown in Figure 3. By setting only the selected subset of pruned weights to zero, we fully recover the performance of the original LT networks. This supports our hypothesis that the benefit derived from freezing values to zero comes from the fact that those values were moving toward zero anyway. In fact, if we apply this treatment to all weights, including the kept weights, we can outperform even the original LT networks. The hypothesis above suggests that for certain mask criteria, like large_final, that masking is training: the masking operation tends to move weights in the direction they would have moved during training. If so, just how powerful is this training operation? To answer this question, we can start all the way from the beginning-not training the network at all, but simply applying a mask to the randomly initialized network. It turns out that with a well-chosen mask, an untrained network can already attain a test accuracy far better than chance. Although it is not entirely implausible to have better-than-chance performance since the masks are derived from the training process, the large improvement in performance is still surprising because the only transmission of information from the training back to the initial network is via a zero-one mask based on a simple criterion. We call these masks that can produce better-than-chance accuracy without training of the underlying weights "Supermasks". Figure 3: Performance of various treatments of pruned weights for Conv4 on CIFAR-10. Horizontal black line represents the performance of the original, unpruned network. Solid blue line represents networks trained using the LT algorithm, which freeze pruned weights at zero. Dotted blue line represents networks where pruned weights are frozen to their initial values. Dashed grey line represents networks trained using the new proposed scheme for pruned weights: freeze pruned weights at zero if they decreased in magnitude by the end of training, otherwise freeze them at their initialization values. Dotted grey line represents networks trained with the new proposed scheme apply to all weights by initializing kept weights to zero if they decreased in magnitude. Performance on other models are shown in FIG4.We turn our attention to evaluating how various mask criteria perform as Supermasks. In addition to evaluating the mask criteria from Section 2, we define a new large_final_same_sign mask criterion based on the demonstration in Section 3 of the importance of signs and of keeping large weights. The large_final_same_sign mask criterion selects for weights with large final magnitudes that also maintained the same sign by the end of training. This criterion is depicted in Figure S7. Also included as a control is the large_final_diff_sign. Performances of Supermasks produced by all 10 criteria are included in FIG2, compared with two baselines: networks untrained and unmasked (untrained_baseline) and networks fully trained (trained_baseline). By using this simple mask criterion of large_final_same_sign, we can create networks that obtain a remarkable 80% test accuracy on MNIST and 24% on CIFAR-10 without training. Another curious observation is that if we apply the mask to a signed constant (as described in Section 3) rather than the actual initial weights, we can produce even higher test accuracy of up to 86% on MNIST and 41% on CIFAR-10! Detailed across network architectures, pruning percentages, and these two treaments, are shown in FIG6. We find it fascinating that these Supermasks exist and can be found via such simple criteria. As an aside, they also present a method for network compression, since we only need to save a binary mask and a single random seed to reconstruct the full weights of the network. for: Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask S1 Lottery Ticket Pruning Algorithm We describe the lottery ticket algorithm in more detail: 0. Initialize a mask m to all ones. Randomly initialize the parameters w of a network f (x; w m) 1. Train the parameters w of the network f (x; w m) to completion. Denote the initial weights before training w i and the final weights after training w f.2. Mask Criterion. Use the mask criterion M (w i, w f) to produce a masking score for each currently unmasked weight. Rank the weights in each layer by their scores, set the mask value for the top p% to 1, the bottom (100 − p)% to 0, breaking ties randomly. Here p may vary by layer, and we follow the ratios chosen in BID2, summarized in Table S1. In BID2 the mask selected weights with large final value corresponding to M (w i, w f) = |w f |. In Section 2 we consider other mask criteria.3. Mask-1 Action. Take some action with the weights with mask value 1. In BID2 these weights were reset to their initial values and marked for training in the next round. We consider this and other mask-1 actions in Section 3.4. Mask-0 Action. Take some action with the weights with mask value 0. In BID2 these weights were pruned: set to 0 and frozen during any subsequent training. We consider this and other mask-0 actions in Section 4. Table S1 contains the architectures used in this study, together with relevant training hyperparameters, based off of experiments in BID2. Table S1: The architectures used in this paper. Table reproduced and modified from BID2. Conv networks use 3x3 convolutional layers with max pooling followed by fully connected layers. FC layer sizes are from BID7. Figure S2: Mask criteria studied in this section, starting with large_final that was used in BID2. Names we use to refer to the various methods are given along with the formula that projects each (w i, w f) pair to a score. Weights with the largest scores (colored regions) are kept, and weights with the smallest scores (gray regions) are pruned. The x axis in each small figure is w i and the y axis is w f. In two methods, α is adjusted as needed to align percentiles between w i and w f. When masks are created, ties are broken randomly, so the random method just assigns a score of 0 to every weight. FIG0 shows the representation of large_final mask criterion on a 2D plane. FIG1 shows the formulation and visual depiction of all the mask criteria considered in Section 2. Figure S3 shows the convergence speed and performance of all mask critera for FC on MNIST and Conv2, 4, 6 on CIFAR-10. FIG2 shows the convergence speed and performance of various reinitialization methods for FC on MNIST and Conv2, 4, 6 on CIFAR-10. FIG4 shows the performance of various treatment of pruned weights for FC on MNIST and Conv2, 4, 6 on CIFAR-10. FIG5 illustrates why the large_final criterion biases weights that were moving toward zero during training toward zero in the mask, effectively pushing them further in the direction they were headed. It also illustrates why the large_final criterion creates Supermasks. Figure S7 depicts the effect of Supermasks, as well as the two additional mask criteria considered only as Supermasks. FIG6 shows performance of various mask criteria on initial test accuracy for FC on MNIST and Conv 2, 4, 6 on CIFAR-10. Now that we know Supermasks exist, and those derived from simple heuristics work remarkably well, we might wonder how far we can push the performance of Supermasks for a given network. One can search, in the search space of all 2 n possible masks, where n is the number of parameters in that network. We can also try learning it with regular optimizers. To do that, we create a trainable mask variable for each layer while freezing all original parameters for that layer at their random initialization values. For an original weight tensor w and a mask tensor m of the same shape, we have as the effective weight w = w i g(m), where w i denotes the initial values weights are frozen at, is element-wise multiplication and g is a point-wise function that transform a matrix of continuous values into binary values. One example of g is (S(m)), where S is the sigmoid function and means rounding. Bias terms are added as usual to the product of w with the inputs as per the usual fully connected or convolutional kernels. We train the masks with g(m) = Bern(S(m)), where Bern(p) is the bernoulli sampler with probability p. It works slightly better than (S(m)). The bernoulli sampling adds some stochasticity that helps with training, mitigates the bias of all things starting at the same value, and uses in effect the expected value of S(m), which is especially useful when they are close to 0.5.By training the m matrix with SGD, we obtained up to 95.3% test accuracy on MNIST and 65.4% on CIFAR-10. Results are shown in FIG6, along with all the heuristic based, unlearned Supermasks. Note that there is no straightforward way to control for the pruning percentage. What we do is initializing m with a constant of different magnitudes, whose value nudges the network toward pruning more or less. With this tactic we are able to produce masks with the amounts of pruning (percentages of zeros) ranging from 7% to 89%. Further details about the training can be seen in Section S6. TAB4 summarizes the best test accuracy obtained through different treatments. The shows striking improvement of learned Supermasks over heuristic based ones. And learning Supermasks in performance not too far from training the full network. It suggests that a network upon initialization has already contained powerful subnetworks that work well. Additionally, the learning of Supermask allows identifying a possibly optimal pruning rate for each layer, since each layer is free to learn the distribution of 0s in m on their own. For instance, in BID2 the last layer of each network is designed to be pruned approximately half as much as the other layers, in our setting this ratio is automatically adjusted. We train the networks with mask m for each layer (and all regular kernels and biases frozen) with SGD, 0.9 momentum. The {FC, Conv2, Conv4, Conv6} networks respectively had {100, 100, 50, 20} for learning rates and trained for {2000, 2000, 1000, 800} iterations. These hyperparameters may seem absurd, but a network of masks is quite different and cannot train well with typical learning rates. Conv4 and Conv6 showed significant overfitting, thus we used early stopping as we are unable to use standard regularizing techniques. For evaluation, we also use Bernoulli sampling, but average the accuracies over 10 independent samples. For adjusting the amount pruned, we initialized m in every layer to be the same constant, which ranged from -5 to 5. In the future it may be worth trying different initializations of m for each layer for more granular control over per-layer pruning rates. A different method to try would be to add an L1 loss to influence layers to go toward certain values, which may alleviate the cold start problems of some networks not learning anything due to mask values starting too low (effectively having the entire network start at zero). Figure S3: Performance of different mask criteria for four networks at various pruning rates. We show early stopping iteration on the left and test accuracy on the right. Each line is a different mask criteria, with bands around magnitude_increase, large_final, movement, and random depicting the min and max over 5 runs. Stars represent points that are significantly above all other lines at a p = 0.05 level. large_final and magnitude_increase show the best convergence speed and accuracy, with magnitude_increase having slightly higher accuracy in Conv2 and Conv4. As expected, criteria using small weight values consistently perform worse than random. FIG2: The effects of various 1-actions for the four networks and various pruning rates. Dotted lines represent the three described methods, and solid lines are those three except with each weight having the same sign as its original initialization. Shaded bands around notable runs depict the min and max over 5 runs. Stars represent points that are significantly above all other lines at a p = 0.05 level. We also include the original rewinding method and random reinitialization as baselines. "Reshuffle, init sign" and "constant, init sign" perform similarly to the "rewind" baseline. Figure S7: (left) Untrained networks perform at chance (10% accuracy, for example, on the MNIST dataset as depicted), if they are randomly initialized, or randomly initialized and randomly masked. However, applying the large_final mask improves the network accuracy beyond the chance level. (right) The large_final_same_sign mask criterion (left) that tends to produce the best Supermasks. In contrast to the large_final mask in FIG0, this criterion masks out the quadrants where the sign of w i and w f differ. We include large_final_diff_sign (right) as a control. No training is performed in any network. Weights are frozen at either initialization or constant and various masks are applied. Within heuristic based Supermasks (excluding learned_mask), the large_final_same_sign mask creates the highest performing Supermask by a wide margin. Note that aside from the five independent runs performed to generate uncertainty bands for this plot, every data point on the plot is the same underlying network, just with different masks.' it should be close to 'In neural network pruning, zeroing pruned weights is important, sign of initialization is key, and masking can be thought of as training.' but not to 'We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.'.

Question: Textual entailment (or NLI) data has proven useful as pretraining data for tasks requiring language understanding, even when building on an already-pretrained model like RoBERTa. The standard protocol for collecting NLI was not designed for the creation of pretraining data, and it is likely far from ideal for this purpose. With this application in mind we propose four alternative protocols, each aimed at improving either the ease with which annotators can produce sound training examples or the quality and diversity of those examples. Using these alternatives and a simple MNLIbased baseline, we collect and compare five new 9k-example training sets. Our primary are largely negative, with none of these new methods showing major improvements in transfer learning. However, we make several observations that should inform future work on NLI data, such as that the use of automatically provided seed sentences for inspiration improves the quality of the ing data on most measures, and all of the interventions we investigated dramatically reduce previously observed issues with annotation artifacts. The task of natural language inference (NLI; also known as textual entailment) has been widely used as an evaluation task when developing new methods for language understanding tasks, but it has recently become clear that high-quality NLI data can be useful in transfer learning as well. Several recent papers have shown that training large neural network models on natural language inference data, then fine-tuning them for other language understanding tasks often yields substantially better on those target tasks . This holds even when starting from large models like BERT that have already been pretrained extensively on unlabeled data (; ; b). The largest general-purpose corpus for NLI, and the one that has proven most successful in this setting, is the Multi-Genre NLI Corpus . MNLI was designed for use in a benchmark task, rather than as a resource for use in transfer learning and as far as we know, it was not developed on the basis of any kind of deliberate experimentation. Further, data collected under MNLI's data collection protocol has known issues with annotation artifacts which make it possible to perform much better than chance using only one of the sentences in each pair (; ;). This work begins to ask what would be involved in collecting a similar dataset that is explicitly designed with transfer learning in mind. In particular, we consider four potential changes to the original MNLI data collection protocol that are designed to improve either the ease with which annotators can produce sound examples, or the quality and diversity of those examples, and evaluate their effects on transfer. We collect a baseline dataset of about 10k examples that follows the MNLI protocol with our annotator pool, followed by four additional datasets of the same size which isolate each of our candidate changes. We then compare all five in a set of transfer learning experiments that look at our ability to use each of these datasets to improve performance on the eight downstream language understanding tasks in the SuperGLUE (b) benchmark. All five of our datasets are consistent with the task definition that was used in MNLI, which is in turn based on the definition introduced by. In this task, each example consists of a pair of short texts, called the premise and the hypothesis. The model is asked to read both texts and make a three-way classification decision: Given the premise, would a reasonable person infer that hypothesis must be true (entailment), infer that that it must be false (contradiction), or decide that there is not enough information to make either inference (neutral). While it is certainly not clear that this framing is optimal for pretraining, we leave a more broad-based exploration of task definitions for future work. Our BASE data collection protocol (Figure 1) follows MNLI closely in asking annotators to read a premise sentence and then write three corresponding hypothesis sentences in empty text boxes corresponding to the three different labels (entailment, contradiction, and neutral). When an annotator follows this protocol, they produce three sentence pairs at once, all sharing a single premise. Our PARAGRAPH protocol tests the effect of supplying annotators with complete paragraphs, rather than sentences, as premises. Longer texts offer the potential for discourse-level inferences, the addition of which should yield a dataset which is more difficult, more diverse, and less likely to contain trivial artifacts. However, reading full paragraphs adds a potential cost in added annotator time and effort, which could potentially be better spent constructing more sentence-level examples. Our EDITPREMISE and EDITOTHER protocols test the effect of pre-filling a single seed text in each of the three text boxes that annotators are asked to fill out. By reducing the raw amount of typing required, this could allow annotators to produce good examples more quickly. By encouraging them to keep the three sentences similar, it could also encourage minimal-pair-like examples that minimize artifacts. We test two variants of this idea: One uses a copy of the premise sentence as a seed text and the second retrieves a new sentence from an existing corpus that is similar to the premise sentence, and uses that. Our CONTRAST protocol tests the effect of adding artificial constraints on the kinds of hypothesis sentences annotators can write. Giving annotators difficult and varying constraints could encourage creativity and prevent annotators from falling into repeating ruts or patterns in their writing that could lead to easier, more repetitive data. However, as with the use of longer contexts in BASE, this protocol risks substantially slowing the annotation process. We experiment with a procedure inspired by that used to create the language-andvision dataset NLVR2 , in which in which annotators must write sentences that are valid entailments (or contradictions) for a given premise, but not valid entailments for a second, similar, distractor premise. In evaluations on transfer learning with the SuperGLUE benchmark, all of these four methods offer substantial improvements in transfer ability over a plain RoBERTa model, but that only EDITOTHER and CONTRAST offering consistent improvements over BASE, and only by very small margins. While this is largely a negative for our primary focus on transfer, we also observe that all four of these methods are able to produce data of comparable subjective quality while significantly reducing the incidence of previously reported annotation artifacts, and that PARAGRAPH, EDITPREMISE, and EDITOTHER all accomplish this without significantly increasing the time cost of annotation. The observation that NLI data can be effective in pretraining was first reported for SNLI and MNLI by on models pretrained from scratch on NLI data. This finding was replicated in the setting of multitask pretraining by. This was later extended to the context of intermediate training-where a model is pretrained on unlabeled data, then on relatively abundant labeled data (MNLI), and finally scarce task specific labeled data-by , , Liu et al. (2019a), , and Liu et al. (2019b) across a range of large pretrained models models and target language understanding tasks. Similar have been observed with transfer from the SocialIQA corpus to target tasks centered on common sense. A small body of work including , Bingel and Søgaard and Wang et al. (2019a) Table 1: Randomly selected examples from the datasets under study. Neither the MNLI training set nor any of our collected data are filtered for quality in any way, and errors or debatable judgments are common in both. explored the empirical landscape of which supervised NLP tasks can offer effective pretraining for other supervised NLP tasks. Existing NLI datasets have been built using a wide range of strategies: FraCaS and several targeted evaluation sets were constructed manually by experts from scratch. The RTE challenge corpora (, et seq.) primarily used expert annotations on top of existing premise sentences. SICK was created using a structured pipeline centered on asking crowdworkers to edit sentences in prescribed ways. MPE uses a similar strategy, but constructs unordered sets of sentences for use as premise. SNLI introduced the method, used in MNLI, of asking crowdworkers to compose labeled hypotheses for a given premise. SciTail and SWAG used domain-specific resources to pair existing sentences as potential entailment pairs, with SWAG additionally using trained models to identify examples worth annotating. There has been little work directly evaluating and comparing these many methods. In that absence, we focus on the SNLI/MNLI approach, because it has been shown to be effective for the collection of pretraining data and because its reliance on only crowdworkers and unstructured source text makes it simple to scale. Two recent papers have investigated other methods that could augment the base MNLI protocol we study here. ANLI collects new examples following this protocol, but adds an incentive for crowdworkers to produce sentence pairs on which a baseline system will perform poorly. introduce a method for expanding an already-collected dataset by making small edits to existing examples that change their labels, motivated by the same desire that motivates our EDITPREMISE and EDITOTHER protocols: to produce minimally-different minimal pairs with differing labels. Both of these papers offer methodological changes that are potentially complementary to the changes we investigate here, and neither evaluates the impact of their methods on transfer learning. Since ANLI is large and roughly comparable with MNLI, we include it in our transfer evaluations here. The basic interface for our tasks is similar to that used for SNLI and MNLI: We provide a premise from a preexisting text source and ask human annotators to provide three hypothesis sentences: one that says something true about the fact or situation in the prompt (entailment), one that says something that may or may not be true about the fact or situation in the prompt (neutral), and one that definitely does not say something true about the fact or situation in the prompt (contradiction). BASE In this baseline, modeled closely on the protocol used for MNLI, we show annotators a premise sentence and ask them to provide compose one new sentence for each label. PARAGRAPH Here, we use the same instructions as BASE, but with full paragraphs, rather than single sentences, as the supplied premises. EDITPREMISE Here, we pre-fill the three text boxes with editable copies of the premise sentence, and ask annotators to edit each text field to compose sentences that conform to the same three requirements used in BASE. Annotators are permitted to delete the pre-filled text. EDITOTHER Here, we follow the same procedure as EDITPREMISE, but rather than pre-filling the premise as a seed sentence, we instead use a similarity search method to retrieve a different sentence from the same source corpus that is similar to the premise. We hypothesize that the additional variation offered by these added sentences could improve the creativity and diversity of the ing examples. CONTRAST Here, we again retrieve a second sentence that is similar to the premise, but we display it as a second premise rather than using it to seed an editable text box. We then ask annotators to compose two new sentences: One sentence must be true only about the fact or situation in the first premise (that is, contradictory or neutral with respect to the second premise). The other sentence must be false only about the fact or situation in the first premise (and true or neutral with respect to the second premise). This yields an entailment pair and a contradiction pair, both of which use only the first premises, with the second premise serving only as a constraint on the annotation process. We could not find a sufficiently intuitive way to collect neutral sentence pairs under this protocol, and opted to use only two classes rather than increase the difficulty of an already unintuitive task. MNLI uses the small but stylistically diverse OpenANC corpus as its source for premise sentences, but uses nearly every available sentence from its non-technical sections. To avoid re-using premises, we instead draw on English Wikipedia. 1 1 We use the 2019-06-20 downloadable version, extract the plain text with Apertium's WikiExtractor feature (Forcada Similarity Search The EDITOTHER and CON-TRAST protocols require pairs of related sentences as their inputs. To construct these, we assemble a heuristic sentence-matching system intended to generate pairs of highly similar sentences that can be minimally edited to construct entailments or contradictions: Given a premise, we retrieve its closest 10k nearest neighbors according to dot-product similarity over Universal Sentence Encoder embeddings. Using a parser and an NER system, we then select those neighbors which share a subject noun phrase in common with the premise (dropping premises for which no such neighbors exist). From those filtered neighbors, we retrieve the single non-identical neighbor that has the highest overlap with the premise in both raw tokens and entity mentions, preferring sentences with similar length to the hypothesis. We start data collection for each protocol with a pilot of 100 items, which are not included in the final datasets. We use these to refine task instructions and to provide feedback to our annotator pool on the intended task definition. We continue to provide regular feedback throughout the annotation process to clarify ambiguities in the protocols and to discourage the use of systematic patternssuch as consistently composing shorter hypotheses for entailments than for contradictions-that could make the ing data artificially easy. Annotators are allowed to skip prompts which they deem unusable for any reason. These generally involve either non-sentence strings that were mishandled by our sentence tokenizer or premises with inaccessible technical language. Skip rates ranged from about 2.5% for EDITOTHER to about 10% for CONTRAST (which can only be completed when the two premises are both comprehensible and sufficiently different from one another). A pool of 19 professional annotators located in the United States worked on our tasks, with about ten working on each. As a consequence of this relatively small annotation team, many annotators worked under more than one protocol, which we ran consecutively. This introduces a modest confound into our , in that annotators start the later tasks having seen somewhat more feedback. All annotators, though, see substantial feedback et al., 2011), sentence-tokenize it with SpaCy , and randomly sample sentences (or paragraphs) for annotation. from a pilot phase before starting each task. This confound presentation prevents us from perfectly isolating the differences between protocols, but we argue that these nonetheless form an informative case study. Using each protocol, we collect at least 10k examples and split them into exactly 9k training examples and at least 1k validation examples, all to be released upon acceptance. Table 1 shows randomly chosen examples. As we are investigating data collection protocols for pretraining and do not use any kind of second-pass quality control (motivated by work like), we do not collect a test set and do not recommend these datasets for system evaluation. Hypotheses are mostly fluent, full sentences that adhere to prescriptive writing conventions for US English. In constructing hypotheses, annotators often reuse words or phrases from the premise, but rearrange them, alter their inflectional forms, or substitute synonyms or antonyms. Hypothesis sentences tend to differ from premise sentences both grammatically and stylistically. Table 2 shows some simple statistics on the collected text. Our clearest observation here is that the two methods that use seed sentences tend to yield longer hypotheses and tend not to show a clear correlation between hypothesis-premise token overlap and label (as measured by the standard deviations in unique tokens). CONTRAST tends to produce shorter hypotheses. Annotator Time Annotators completed each of the five protocols at a similar rate, taking 3-4 minutes per prompt. This goes against our expectations that the longer premises in PARAGRAPH should substantially slow the annotation process, and that the pre-filled text in EDITPREMISE and EDITOTHER should speed annotation. Since the relatively complex CONTRAST produces only two sentence pairs per prompt rather than three, it yields fewer examples per minute. Table 3 shows the three words in each dataset that are most strongly associated with each label, using the smoothed PMI method of. We also include for a baseline: a 9k-example sample from the government documents single-genre section of MNLI, which is meant to to be maximally comparable to the single-genre datasets we collect. BASE shows similar associations to the original MNLI, but all four of our interventions reduce these associations. The use of longer contexts or seed sentences in particular largely eliminates the strong association between negation and contradiction seen in MNLI, and no new strong associations appear to take its place. Our experiments generally compare models trained in ten different settings: Each of the five 9k-example training sets introduced in this paper; the full 393k-example MNLI training set; the full 1.1m-example ANLI training set (which combines the SNLI training set, the MNLI training set, and the newly-collected ANLI training examples); 2 9k- Table 3: The top three words most associated with specific labels in each dataset, sorted by the PMI between the word and the label. The counts column shows how many of the instances of each word occur in hypotheses matching the specified label. We compare the two-class CONTRAST with a two-class version of MNLI Gov. example samples from the MNLI training set and from the combined ANLI training set, meant to control for the size differences between these existing datasets and our baselines; and finally a 9k-example sample from the government section of the MNLI training set, meant to control (as much as possible) for the difference between our singlegenre Wikipedia datasets and MNLI's relatively diverse text. Our models are trained starting from pretrained RoBERTa (large variant; b) or XLNet (large, cased;). RoBERTa represented the state of the art on most of our target tasks as of the launch of our experiments, and XLNet is competitive with RoBERTa on most tasks, but offers a natural replication, as well as the advan- Table 4: NLI modeling experiments with RoBERTa, reporting on the validation sets for MNLI and for the task used for training each model (Self), and the GLUE diagnostic set. We compare the two-class CON-TRAST with a two-class version of MNLI. tage that it can be used to better compare models trained on our data with models trained on ANLI data: ANLI was collected with a model-in-the-loop procedure using RoBERTa that makes it difficult to interpret RoBERTa . We run our expemients using the jiant toolkit (d), which implements the Super-GLUE tasks, MNLI, and ANLI, and in turn uses transformers , AllenNLP , and PyTorch . To make it possible to train these large models on single consumer GPUs, we use small-batch (b = 4) training and a maximum total sequence length of 128 word pieces. We train for up to 2 epochs for the very large ReCoRD, 10 epochs for the very small CB, COPA, and WSC, and 4 epochs for the remaining tasks. Except where noted, all reflect the median final performance from three random restarts of training. Direct NLI Evaluations As a preliminary sanity check, Table 4 shows the of evaluating models trained in each of the settings described above on their own validation sets, on the MNLI validation set, and on the expert-constructed GLUE diagnostic set (c). As NLI classifiers trained on CONTRAST cannot produce the neutral labels used in MNLI, we evaluate them separately, and compare them with two-class variants of the MNLI models. Our BASE data yields a model that performs somewhat worse than a comparable MNLI Gov. 9k model, both on their respective validation sets and on the full MNLI validation set. This suggests, at Table 5: Results from RoBERTa hypothesis-only NLI classifiers on the vaidation sets for MNLI and for the datasets used in training. least tentatively, that the new annotations are less reliable than those in MNLI. This is disconcerting, but does not interfere with our key comparisons. The main we draw from these is that none of the first three interventions improve performance on the out-of-domain GLUE diagnostic set, suggesting that they do not help in the collection of data that is both difficult and consistent with the MNLI label definitions. We also observe that the newer ANLI data yields worse performance than MNLI on the out-of-domain evaluation data when we control for dataset size. To further investigate the degree to which our hypotheses contain artifacts that reveal their labels, Table 5 shows with single-input versions of our models trained on hypothesis-only versions of the datasets under study and evaluated on the datasets' validation sections. Our first three interventions, especially EDIT-PREMISE, show much lower hypothesis-only performance than BASE. This adds further evidence, alongside our PMI , that these interventions reduce the presence of such artifacts. While we do not have a direct baseline for the two-class CON-TRAST in this experiment, the fact that it shows substantially lower performance than MNLI 9k is consistent with the encouraging seen above. Transfer Evaluations For our primary evaluation, we use the training sets from our datasets in STILTs-style intermediate training : We fine-tune a large pretrained model on our collected data using standard fine-tuning procedures, then fine-tune a copy of the ing model again on each of the target evaluation datasets we use. We then measure the aggregate performance of the ing models across those evaluation datasets. We evaluate on the tasks in the SuperGLUE benchmark (b): BoolQ , MultiRC , and , CommitmentBank (CB;), Choice of Plausible Alternatives (COPA;), Recognizing Textual Entailment (RTE; ; ;), the Winograd Schema Challenge (WSC;), and WiC , in addition to a broad-coverage RTE diagnostic set (AX b) and WinoGender RTE (AX g ;). These tasks were selected to be difficult for BERT but relatively easy for nonexpert humans, and are meant to replace the largelysaturated GLUE benchmark (c). SuperGLUE does not include labeled test data, and does not allow for substantial ablation analyses on its test sets. Since we have no single final model whose performance we aim to show off, we do not evaluate on the test sets. We also neither use any auxiliary WSC-format data when training our WSC model (as in) nor artificially modify the task format. As has been observed elsewhere, we do not generally reach above-chance performance on that task without these extra techniques. Results are shown in Table 6. 3 Intermediate training with any of our five datasets yields models that transfer better than the plain RoBERTa or XLNet baseline, but we do not see consistent improvements over MNLI Gov. 9k. We also replicate the previously established that NLI data is broadly helpful for transfer, with the large combined ANLI training set showing improvements over plain RoBERTa on six of eight tasks and simultaneously reducing the variance in across restarts. We note, though, that our best overall uses only 9k NLI training examples, suggesting either that this size is enough to maximize the gains available through NLI pretraining, or Table 6: Model performance on the SuperGLUE validation and diagnostic sets. The Avg. column shows the overall SuperGLUE score-an average across the eight primary tasks, weighting each task equally-as a mean and standard deviation across three restarts. CONTRAST yield consistent improvements in transfer performance over BASE, and these improvements are small and not reliably reflected by any one target task. We take this to be a largely negative : While there are informative trends, we cannot confidently claim that any intervention yields improvements in the degree to which the ing data can be used for transfer. Our chief on transfer learning are negative: None of our four interventions consistently improve upon the base MNLI data collection protocol by more than a marginal degree, though we see suggestive evidence that methods that supply annotators with retrieved non-premise seed sentences for inspiration offer small improvements. However, we also observe that all four of our interventions, and especially the use of longer contexts or pre-filled seed sentences, help reduce the prevalence of artifacts in the generated hypotheses that reveal the label, and the use of longer premises or seed sentences in particular do this without increasing the time cost of annotation. This suggests that these methods may be valuable in the collection of high-quality evaluation data, if combined with additional validation methods to ensure high human agreement with the collected labels. The need and opportunity that motivated this work remains compelling: Human-annotated data like MNLI has already proven itself as a valuable tool in teaching machines general-purpose skils for language understanding, and discovering ways to more effectively build and use such data could further accelerate the field's already fast progress toward robust, general-purpose language understanding technologies. Further work along this line of research could productively follow a number of directions: General work on incentive structures and task design for crowdsourcing could help to address more general questions about how to collect data that is simultaneously creative and consistently labeled. Machine learning methods work on transfer learning could help to better understand and exploit the effects that drive the successes we have seen with NLI data so far. Finally, there remains room for further empirical work investigating the kinds of task definitions and data collection protocols most likely to yield positive transfer.

Represent the TLDR for finding the corresponding scientific article:

We propose four new ways of collecting NLI data. Some help slightly as pretraining data, all help reduce annotation artifacts.

represent the tldr for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.