Question: Deep CNNs have achieved state-of-the-art performance for numerous machine learning and computer vision tasks in recent years, but as they have become increasingly deep, the number of parameters they use has also increased, making them hard to deploy in memory-constrained environments and difficult to interpret. Machine learning theory implies that such networks are highly over-parameterised and that it should be possible to reduce their size without sacrificing accuracy, and indeed many recent studies have begun to highlight specific redundancies that can be exploited to achieve this. In this paper, we take a further step in this direction by proposing a filter-sharing approach to compressing deep CNNs that reduces their memory footprint by repeatedly applying a single convolutional mapping of learned filters to simulate a CNN pipeline. We show, via experiments on CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet that this allows us to reduce the parameter counts of networks based on common designs such as VGGNet and ResNet by a factor proportional to their depth, whilst leaving their accuracy largely unaffected. At a broader level, our approach also indicates how the scale-space regularities found in visual signals can be leveraged to build neural architectures that are more parsimonious and interpretable. Deep CNNs have achieved state-of-the-art on a wide range of tasks, from image understanding (; ; ;) to natural language processing . However, these network architectures are often highly overparameterised , and thus require the supervision of a large number of input-output mappings and significant training time to adapt their parameters to any given task. Recent studies have discovered several different redundancies in these network architectures (; Hubara* et al., 2018; ; ; a; b) and certain simplicities (Pérez et al., 2018;) in the functions that they implement. For instance, showed that a large classification network can be distilled down to a small sub-network that, owing to its lucky initialisation, is trainable in isolation without compromising the original classification accuracy. observed that deep classification networks learn simplistic non-linearities for class identification, a fact that might well underlie their adversarial vulnerability, whilst challenging the need for complex architectures. Attempts at knowledge distillation have regularly demonstrated that it is possible to train small student architectures to mimic larger teacher networks by using ancillary information extracted from the latter, such as their attention patterns , predicted soft-target distributions or other kinds of meta-data . These works and others continue to expose the high level of parameter redundancy in deep CNNs, and comprise a foundational body of work towards studying and simplifying networks for safe and practical use. Our paper experiments with yet another scheme for simplifying CNNs, in the hope that it will not only shrink the effective footprint of these networks, but also open up new pathways for network understanding and redesign. In particular, we propose the use of a common set of convolutional filters at different levels of a convolutional hierarchy to achieve class disentanglement. Mathematically, we formulate a classification CNN as an iterative function in which a small set of learned convolutional mappings are applied repeatedly as different layers of a CNN pipeline (see Figure 1). In doing so, we are able to reduce the parameter count of the network by a factor proportional to its depth, whilst leaving its accuracy largely unaffected. We also investigate the introduction of non-shared linear widths n of the shared convolutional layer, compared to the baseline VGGNet , for CIFAR-10 (a) and. The compression factor is plotted on a logarithmic scale. layers before certain shared convolutional layers to enhance the flexibility of the model by allowing it to linearly combine shared filter maps for the disentanglement task. This work is partly inspired by the classic literature on image processing that has long sought to characterise natural images by collating their responses, at different image scales, to a small, canonical set of hand-crafted visual operators . Modern CNN architectures effectively still implement hierarchical feature extraction, but with the difference that there are thousands of such operators (i.e. convolutional filters) at each scale level, all of which are individually adaptable and learned via backpropagation. Our work can thus be seen as an effort to reconcile the above two non-contemporaneous approaches to image processing, in which we aim to identify a common set of visual operators for all the different scales by learning them in an end-to-end manner. Our approach bears some high-level resemblance to previous approaches (e.g. ; ; ;) that have attempted to implement, interpret and potentially improve convolutional neural networks through an iterative use of simpler modules For example, share convolutional mappings in ResNets in an attempt to approximate biological visual systems using feedback loops and recurrence, although their experimental analysis is limited to the CIFAR dataset. By contrast, our work applies the convolution-sharing paradigm to both plain feed-forward and residual constructs, and investigates the effectiveness of using only a single shared convolutional mapping for the entire network pipeline. An additional contribution of our approach is the flexibility we add to the model by coupling learned linear layers with shared convolutions while still limiting the total parameter count. Experimentally, we evaluate the accuracy vs. model size tradeoff induced by our approach on a realistic set of datasets that include Tiny ImageNet and ImageNet. A steady increase in the size of datasets and the availability of computational resources has enabled neural networks to grow deeper , denser and wider . In doing so, concerns regarding their over-parameterisation have often been ignored in favour of better test set generalisation. 1 More recently, as their performance on some benchmarks has reached near-human levels, real-world deployment of these models is being considered. This deployment has been hugely impeded by the memory requirements, latency and energy demands of their heavy computational machinery . Our approach contributes to the (extensive) literature on network compression that is focused on making these machine learning models more usable in practical scenarios. Existing compression methods can be divided into seven categories -pruning, quantisation, tensorization/tensor decomposition, knowledge distillation, custom architectures, sharing-based and hybrid methods. Many of these works are beyond the scope of this paper, but for completeness, we present a brief review in §A.1 (a more exhaustive survey can be found in). Our own work falls within the realm of sharing-based methods that seek to equate some of a network's weights or filters to reduce the number of independent parameters in the network. There are various ways of deciding which weights/filters to share, from somewhat arbitrary (if effective) approaches such as the hashing trick , to more principled approaches such as k-means clustering . A few recent works have turned their attention to sharing convolutional weight matrices in a more structured manner. Of these, LegoNet (b) shares filter groups across sets of channels, whilst FSNet (a) shares filter weights across spatial locations. In both cases, sharing is restricted to a single layer at a time. ShaResNet reuses convolutional mappings, but within the same scale level (i.e. between two max-pooling steps). The novelty of our work lies in extending this filter-sharing paradigm to an entire convolutional pipeline. We instantiate a single convolutional layer that is applied iteratively to mimic a deep convolutional feature extractor, and analyse the accuracy vs. memory tradeoff for different widths of this layer. A standard feed-forward classification CNN can be formulated as where the overall function F is a composition of the convolutional feature extractor F conv followed by a fully-connected classifier C. The convolutional sub-model F conv consists of a sequence of convolutional layers, interspersed with non-linearities (ReLUs, Max-Pooling) or regularisers (dropout, BatchNorm) or some combination thereof, denoted by R i. The function performed by each convolutional layer f i is completely specified by a set of weights and biases that we denote using W i. Crucially, the weights and biases for each different layer are independent. The number of parameters in layer f i is then simply the size of W i, calculated as where n in i in the number of input channels to f i, n out i is the number of output channels, is the volume of f i, and k i is the size of its (square) convolutional filters. In practice, the n out i term for the biases is dominated by that for the weights, and so we disregard it in what follows. Letting W conv = L i=1 W i denote all the parameters in F conv (i.e. disregarding the comparatively small contributions from the non-convolutional layers), the total parameter count is given by Note that for many common architectures, there exists some k such that ∀i, k i = k (e.g. for VGGNet, k = 3). For such architectures, Equation 3 can then be further simplified to is the mean volume per network layer. Our method proposes a crude simplification to such architectures, namely to instantiate a single convolutional layer f, and apply it L successive times in order to implement a convolutional pipeline of equivalent depth to the original model. In particular, we enforce the following constraint: This simplifies the CNN architecture in Equation 1 tõ Whilst our analysis focuses purely on the convolutional layers, it is interesting to note that when the R i layers are all the same, the CNN architecture simplifies further to the following iterative form: The convolutional layer f in our architecture expects an input tensor with a predetermined number of channels, which we will call n. Meanwhile, the R i layers between the convolutional layers leave the number of channels unchanged. Thus, given the iterative application of f, the layer f must also output a tensor with n channels. (In practice, f is called for the first time on the input image itself, which for colour images would normally only have 3 channels. To avoid artificially limiting n to 3, we pad the input image with empty channels to produce a tensor with n channels.) We deduce that |W |, the number of parameters for f, must satisfy is the volume of f. Furthermore, since W is shared between all L convolutional layers, the total number of independent parameters inF conv must also just be |W |. The compression factor between the original architecture and its shared counterpart can thus be quantified as This is proportional to the depth L of the original network, and is down-weighted by any (multiplicative) increase in the average per-layer volume in going from the original to the shared architecture. We now turn to examine the convolutional operation in our architecture. Each layer f, the operation of which is completely specified by the weights and biases in W, takes an input tensor X of size n × h × w, where n, h and w denote the number of channels, height and width respectively. Based on X and W, we can conceptually define 2D matrices Φ(X) and Γ(W) as follows: In this, m = h × w, and each x ij is a rasterisation of a k × k patch of input tensor centred at spatial location i in channel j. Each w ij is a similar rasterisation of the k × k convolutional kernel that maps the input channel i ∈ {1, 2, . . ., n} to the output channel j ∈ {1, 2, . . ., n}, and each b j is the bias for output channel j. Then f can be defined concisely as f (X) = Ψ(Φ(X) × Γ(W)), in which Ψ reshapes the m × n tensor Φ(X) × Γ(W) back to one of size n × h × w. In practice, this simple formulation could be seen as being too restrictive, in the sense that irrespective of the convolutional iteration, each filter w ij in Γ(W) only ever operates on patches from input channel i (for example, the w 1j filters only ever operate on patches from channel 1). For this reason, we decided to investigate whether adding a way of allowing the input channels to be reorganised at various points in the overall pipeline would improve performance. In principle, one way of achieving this would be to add n × n permutation matrices at appropriate points in the pipeline, e.g. just before each pooling operation. In practice, however, to make the operations differentiable, we implement them using linear layers (i.e. 1 × 1 convolutions), thus implementing blending of the input channels rather than simply permuting them. The weights of these layers are separate for each instantiation and are learned as part of the end-to-end pipeline. It would be reasonable to expect this added flexibility to yield a significant increase in performance, and indeed our in §5 show this to be the case. Nevertheless, it is notable that even without this added flexibility, our shared architectures already achieve extremely good performance on the datasets on which we tested, demonstrating that our underlying approach of sharing filters between layers makes sense even in the absence of permutation/blending. We evaluate our filter-sharing approach on four well-known image classification benchmarks: CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet. Details of these datasets can be found in §A.2. For this study, we work with two different architectures, one closely inspired by VGGNet , and the other by ResNet . VGGNet-like Architectures. We base our VGGNet-like architectures on VGG-16, which consists of 5 convolutional blocks followed by 3 linear layers. Each block is followed by a max-pooling step and contains several convolutional layers with different channel counts (in order: 2 layers with 64 channels, 2 with 128, 3 with 256, 3 with 512 and 3 layers with 512 channels). By contrast, in our case, we define a single convolutional layer with a fixed number of input and output channels n, and then use it repeatedly in the same arrangement as above (see Table 3 in §A.3 for more details). We define four variants of this convolutional feature extractor for our study. E-VGGNet is our equivalent of VGGNet, with n channels per layer and no sharing between the layers: we use this as a baseline. Its shared counterpart, S-VGGNet, has the same structure, but iteratively applies a single convolutional layer. SL-VGGNet is an extended version of S-VGGNet that introduces linear layers (i.e. 1 × 1 convolutions) before each max-pooling operation to allow the input channels to be blended at those points in the pipeline. Finally, since all the convolutional layers in SL-VGGNet are the same (these exclude what we call the linear layers), we define a further variant of our architecture that simplifies the network design by setting the number of layers per block to a scalar. We experiment with ∈ {2, 3}, and name the corresponding networks SL -VGGNet. Note that the predetermined number of channels n is a parameter of our architecture: we test several variants to find the best ones. We perform experiments on CIFAR-10/100 and Tiny ImageNet. For CIFAR-10, the 3 fully-connected layers that follow the feature extractor have 512, 512 and 10 output channels, respectively. For CIFAR-100, we use the same VGGNet-like architectures as for CIFAR-10, but the fully-connected layers have 1024, 1024 and 100 output channels, respectively. For Tiny ImageNet, we use a sequence of two fully-connected layers, with 2048 and 200 output channels respectively. ResNet-like Architectures. We base our ResNet-like architectures on the models proposed in. The simpler variants of these are built using'basic' blocks that essentially consist of two equally-sized 3 × 3 convolutional layers and a skip connection (see Fig. 6). The deeper variants, meanwhile, are built using'bottleneck' blocks, which similarly have a skip connection, but sandwich a single 3×3 convolutional layer between two 1×1 convolutional layers that decrease and then restore the number of channels to limit the number of free parameters. The network pipeline begins with a standalone convolutional layer that outputs a predetermined number of channels p. This is followed by a sequence of b blocks at a number of different scale levels (generally 4, but 3 for CIFAR variants). In the original architectures, each scale level (except the first) began with a strided convolutional layer that downsampled the image and doubled the number of channels. Since we want the convolutional layers in our architectures to have the same numbers of input and output channels (to facilitate sharing), we define an equivalent architecture, E-ResNet, that instead doubles the number of channels and performs downsampling using (respectively) a linear layer (i.e. 1 × 1 convolutions) and a max-pooling step at the end of each scale level. Note that, as in the original ResNet, the final scale level in our architecture ends with average pooling rather than max-pooling. Despite these modifications, the predictive performances of our E-ResNets closely match those of the original architectures. The shared variant of this architecture uses n channels for all scale levels and shares the weights across all the convolutional layers (excluding the linear layers). Since the architecture already contains the linear layers we were previously adding to allow blending of the input channels, we refer to it as SL-ResNet. For CIFAR-10/100, the standalone convolutional layer uses a kernel size of 3 × 3, and a p of 16 and 32 for each dataset, respectively. We experiment with b ∈ {3, 5, 7}'basic' blocks per scale level, and terminate the network with a 10-way linear classifier for CIFAR-10 and a 100-way classifier for CIFAR-100. See Table 4 in §A.3 for details. For Tiny ImageNet and ImageNet, we base our ResNet-like architectures on ResNet-34 and ResNet-50. ResNet-34 is built using'basic' blocks, whilst ResNet-50 uses'bottleneck' blocks. For the latter, it is clearly not possible to share filters between the layers within a block, since they are of different dimensions, so we instead use multiple shared copies of a single block. Note that the shared variants of both these models, SL-ResNet-34/50, keep the standalone convolutional layer unshared, since its kernel size is adjusted according to the dataset (3 × 3 for Tiny ImageNet and 7 × 7 for ImageNet). See Table 5 in §A.3 for details. Earlier, Fig. 2 showed the accuracy vs. compression trade-off for S-VGGNet, relative to the original VGGNet , for different widths n of the shared convolutional layer. Here, Fig. 3 illustrates the improvements in accuracy due to the learned linear layers (i.e. the blend- The compression factor C is plotted on a log scale.) and (for CIFAR-10) variants of LegoNet (b), another state-of-the-art compression method. Baseline models marked with a * were retrained for this study. ing layers) on CIFAR-10, CIFAR-100 and Tiny ImageNet. Observably, the use of the linear layers provides greater benefit for datasets that involve discriminating between a larger number of classes, such as CIFAR-100 and Tiny ImageNet. For CIFAR-10, CIFAR-100 and Tiny ImageNet we compare the accuracies of the best-performing'SL' variants of VGGNet with those of the baseline architecture (and competing compression methods for these datasets, where available) in Table 1. For CIFAR-10 (see Table 1b), we are able to achieve comparable classification accuracy to the VGGNet baseline using only n = 256 channels for our shared convolutional layer, which yields a compression factor of ≈ 17×. For CIFAR-100 (Table 1c), which has 10× more classes, we had to use n = 512 channels to achieve comparable accuracy, but this still yields a significant compression factor of 4.3. Higher compression factors can be achieved by reducing the number of channels, in exchange for some loss in accuracy. Evaluating our shared architecture on Tiny ImageNet (in Table 1d) evidences a similar trend in the , with SL2-VGGNet (n = 512 channels) achieving an accuracy comparable to the non-shared baseline, whilst using only 23% of its parameters. Detailed accuracy and memory usage numbers for E-VGGNet, S-VGGNet and SL-VGGNet, for CIFAR-10, are in Table 1a, while the for CIFAR-100 and Tiny Imagenet can be found in the appendix (see Table 6 in §A.5) We also evaluate our shared ResNet architecture (SL-ResNet) on Tiny ImageNet and ImageNet, with the shown in Table 2 (the corresponding for CIFAR-10 and CIFAR-100 can be found in the appendix, see Table 7 in §A.5). For Tiny ImageNet, our SL-ResNet34 (n = 512) variant is able to achieve a compression rate of 8.4 with only a negligible loss in accuracy. For ImageNet, the same variant similarly achieves a compression rate of 8. , LegoNet (b), FSNet (a) and Shared Wide ResNet (SWRN) . Baseline models marked with a * were retrained for this study. Figure 4: A visual depiction of the linear layers used to blend the input channels in our approach. We show the layers for the two variants in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red. an accuracy trade-off, we achieve a greater compression rate than competing methods that achieve similar accuracies. Note that SWRN is able to achieve state-of-the-art levels of accuracy, but does not provide savings in the number of parameters. Visualising the weights of the blending layers that we learn for the SL-variants of our approach reveals interesting patterns in the way in which these layers blend (or use) the input channels (see Fig. 4). For each layer, the continuous blue vertical lines signify that a subset of the input feature maps are barely used by any of the output channels, thus effectively suppressing the information they carry. (Interestingly, the location of the vertical blue lines changes from one scale to the next, thus showing that different subsets of input channels go unused at different scales.) This is significant, because it implies that the weights associated with the unused channels can be selectively pruned without affecting performance. Our next experiment with the pruning method of shows how we can exploit this observation to significantly reduce the size of our shared networks. Our best-performing SL variants have a relatively small number of parameters in the convolutional layers, but a relatively high number of parameters in the linear layers. Tables 2a and 2b show how the parameter count for these variants increases with the number of channels n and the depth (34 to 50). Notably, using bottleneck blocks, as we do for our SL-ResNet50 variants, also significantly increases the parameter count. As implied by our visualisations in the previous section, we would expect serious reductions in the number of parameters in the linear layers to be possible without significantly reducing accuracy. We thus experiment with applying the magnitude-based weight pruning approach of to the linear layers to see whether this expectation is borne out in practice. We first select a proportion of the parameters to prune, then identify those weights that have the lowest absolute magnitude and set them to 0. We then evaluate on the validation split of the dataset. Note that we do not retrain the network after pruning. Our (see Figure 5) show that we can remove a significant fraction of these blending weights before starting to see a noticeable drop in the accuracy of the network. Figure 5: Analysing the effects of pruning on one of our largest models, SL-ResNet-50 (n = 512), trained on Tiny ImageNet. We iteratively zero out an increasing fraction of the linear layer parameters, starting from those having the smallest absolute value. The accuracy of the network stays constant even when 60% of the parameters are pruned, at which point the compression rate (in comparison to the non-shared baseline with equivalent performance) has increased from 1.4 to 2.6. In this paper, we leverage the regularities in visual signals across different scale levels to successfully extend the filter-sharing paradigm to an entire convolutional pipeline for feature extraction. In particular, we instantiate a single convolutional layer and apply it iteratively to simulate conventional VGGNet-like and ResNet-like architectures. We evaluate our shared architectures on four standard benchmarks -CIFAR-10, CIFAR-100, Tiny ImageNet and ImageNet -and achieve compression rates that are higher than existing sharing-based methods that have equivalent performance. We further show that even higher compression rates, with little additional loss in performance, can be achieved by combining our method with the magnitude-based weight pruning approach of. Study of our complementarity to more structured pruning techniques targeting complete filters and channels is reserved for future work. We conclude with two final observations. Firstly, our use of blending layers and a parameter to tune the width of the shared convolutional layer n makes it easy to adjust the architecture so as to achieve a desired trade-off between compression rate C and accuracy. Secondly, there are interesting connections between our work and the idea of energy-based pruning explored in , where the authors note that a significant fraction of the energy demands of deep network processing come from transferring weights to and from the file system. Our approach bypasses this bottleneck by using the same compact set of weights in an iterative manner. We aim to further investigate this aspect of our method in subsequent work. A APPENDIX Pruning methods seek to reduce the size of a network by removing (either physically or implicitly) some of a network's weights (; ; ; ;), filters or neurons . Notably, reducing the computational cost (rather than just the memory usage) of network architectures that are pruned in an unstructured manner requires the use of suitable sparse inference schemes. Quantization methods keep the number of independent parameters in a network the same, but reduce the bit-depth of the parameters and activations (; Hubara* et al., 2016; 2018) to limit the memory requirements of the network. Tensorization/tensor decomposition methods propose low-rank approximations to high-dimensional neural matrices in order to downsize trained models. Early CNN architectures such as AlexNet and VGGNet contained the bulk of their weights in the fully-connected layers. As a , various rank reduction approaches exclusively targeted the matrices in these layers . The deeper/wider these networks have become, the more the balance of weights has shifted towards the convolutional layers, giving rise to more generalised tensor decomposition schemes . Knowledge distillation ('teacher/student') methods aim to transfer the knowledge present in a cumbersome teacher model to a lightweight student model, without losing the teacher's ability to generalise well. An early approach by Bucilȃ et al. used a heavyweight ensemble to label a large set of unlabelled data, and then used this to train a compact model. Much later, proposed an alternative method that trains a shallow network to directly mimic the logits of a deep model. Subsequent methods have independently shown that training the student using temperature-scaled softmax scores or Gaussian-blurred logits of the teacher can help with regularisation. Other methods in this line of work have proposed to train deep, thin neural networks using auxiliary or intermediate cues such as hidden layer outputs or post-hoc attention maps . (d) show the bottleneck blocks. In this case, since the three convolutions have different sizes, we cannot share a single set of parameters across the whole network; instead, we consider the block as a single entity and reuse it across the network. Custom architecture methods, rather than trying to compress or distil knowledge from existing networks, propose entirely new network architectures that are smaller than existing models but still capable of providing excellent performance. Good examples include SqueezeNet and MobileNets . SqueezeNet tries to use 1 × 1 rather than 3 × 3 filters to reduce the parameter count, and tries to limit the number of input channels to those 3 × 3 filters it does use. MobileNets follow a similar tack and factorise traditional convolutional mappings into a depth-wise separable convolution (to process the spatial context) followed by a 1 × 1 convolution (to process the channels jointly). Two adjustable hyperparameters, α and ρ, pertaining to the intermediate feature resolution and the input spatial resolution, allow further resizing of the network. Hybrid methods implement some combination of the compression schemes discussed above . Whilst our approach belongs to the category of filter-sharing schemes elaborated above, we also demonstrate its complementarity and compatibility with the magnitude-based weight pruning method of. A.2 DATASETS CIFAR-10 consists of 60, 000 32 × 32 colour images, each labelled as belonging to one of 10 mutually exclusive classes. Each class contains 6, 000 images, of which 5, 000 are earmarked for training, and 1, 000 for testing (i.e. there are 50, 000 train images and 10, 000 test images overall). CIFAR-100 consists of the same 60, 000 32 × 32 images that are in CIFAR-10, but this time they are evenly split into 100 classes, each containing 500 training images and 100 testing images. Tiny ImageNet 2 is essentially a smaller, lower-resolution variant of the ImageNet dataset. It consists of 120, 000 64 × 64 images, evenly split into 200 classes. Each class contains 500 training images, 50 validation images and 50 test images. ImageNet was introduced as a large-scale image classification benchmark consisting of high-resolution photographs in 1, 000 visual categories from an even larger ontology of natural concepts (WordNet). It consists of approximately 1M training images, divided into 1, 000 disjoint object categories. Another set of 50, 000 images, evenly split into 1, 000 classes, forms the validation set. The accuracy we report for ImageNet were obtained on this validation set. A.3 NETWORK ARCHITECTURES Table 3 details the structure of our VGGNet-like architectures, whilst Tables 4 and 5 show our ResNet-like architectures (respectively used for CIFAR-10/100 and Tiny ImageNet/ImageNet). The notation is common to the tables and is as follows: conv1-x A 1 × 1 convolutional layer with x output feature channels. The core of our SL variants. We use this layer to allow shared convolutions at different scale levels to observe different blends of the feature channels output by the previous scale. Its number of input feature channels is equal to x, except in E-ResNet-50, where we use it to increase the number of channels between scale levels, and in the first scale of SL-ResNet-50, where we use it to increase the number of channels from x to 4x, to account for the expansion factor of the bottleneck blocks. conv3-x A 3 × 3 convolutional layer with x output feature channels. The number of input feature channels depends on the specific network variant: for the baselines it is equivalent to the number of output feature channels of the previous layer (or 3 for the very first layer), whilst for the E/S/SL-variants, it is equivalent to the number of output feature channels x. The stride is 1 unless otherwise specified. conv7-x A 7 × 7 convolutional layer with x output feature channels. As this layer is only used as the first layer in the ResNet variants of our architectures, it always has 3 input channels. Its stride is 1 when training a ResNet-like architecture for Tiny ImageNet, and 2 when training for ImageNet. basicblock-x A simple skip connection-based block, used in the ResNet-like architectures. As in , it consists of two 3 × 3 convolutional layers and a skip connection. In our shared architectures, the two convolutional layers share the same parameters. See Figures 6a and 6b for details of the internal architectures of the non-shared and shared block variants. bottleneck-x A skip connection-based block with a bottleneck architecture, consisting of a 1 × 1 convolution (used to reduce the number of feature channels), followed by a 3 × 3 convolutional layer, and finally by another 1 × 1 convolution (restoring the original number of feature channels). For this reason it has 4x input and output channels. Figures 6c and 6d detail the internal architectures of the standard and shared variants of the bottleneck blocks (respectively). Crucially, as mentioned in the main paper -and unlike the basicblock architectures described above -the bottleneck block is shared as a single entity, owing to the presence of differently-shaped convolutions. avgpool-x An average pooling layer operating on patches of size x × x. maxpool-x A max-pooling layer operating on patches of size x × x. FC-x A fully-connected layer with x output channels. The number of its input channels is equal to the number of outputs of the previous layer (flattened in the case the previous layer was a convolutional layer). Each spatial convolution (conv3 and conv7) is always followed by a BatchNorm layer and a ReLu. We denote in bold the convolutional layers or blocks that are shared in our S and SL architectures. The parameters of the normalisation layers are never shared, even when the corresponding convolutional weights are shared as part of an S or SL architecture. Fully-connected layers (except the very last one in each architecture) are followed by a Dropout layer. Table 3: The architectures for VGGNet and the VGGNet-like networks we trained as part of our experiments on the CIFAR-10/100 and Tiny ImageNet datasets. The notation is described in the main text. Note that the last max-pooling layer (marked with a *) is not used when training a network for Tiny ImageNet: this is in order to provide a longer feature vector to the first fullyconnected layer (specifically of size n * 3 * 3). The fully-connected layer sizes differ across datasets to account for the different numbers of classes, and are set as follows: (a Table 4 : The architectures for the ResNet-like networks we trained as part of our experiments on the CIFAR-10/100 datasets. The notation is described in the main text. The baselines Eb-ResNet(p) use p = 16 for training on CIFAR-10 (as in) and p = 32 for training on CIFAR-100. The final fully-connected layer has its output size set to the number of classes in the dataset (i.e. num c = 10 for CIFAR-10 and num c = 100 for CIFAR-100). We experiment with different values of b ∈ {3, 5, 7}. Input Resolution E-ResNet-34 E-ResNet-50 SL-ResNet-34(n) SL-ResNet-50(n) 224 × 224 conv7-64, stride-2 conv7-64, stride-2 conv7-n, stride 2 conv7-n, stride 2 conv1-256 conv1-(n * 4) maxpool-2 maxpool-2 maxpool-2 maxpool-2 7 × 7 4× basicblock-512 4× bottleneck-512 4× basicblock-n 4× botteneck-n avgpool-3 avgpool-3 avgpool-3 avgpool-3 FC-numc FC-numc FC-numc FC-numc Table 5: The architectures for the ResNet-like networks we trained as part of our experiments on the Tiny ImageNet and ImageNet datasets. The notation is described in the main text. The final fully-connected layer has its output size set to the number of classes in the dataset (i.e. num c = 200 for Tiny ImageNet and num c = 1000 for ImageNet). One important difference in the architectures for the two datasets is that, in the case of Tiny ImageNet, to account for the smaller resolution of the images, in the first scale level we use a 3 × 3 convolution without striding and suppress the first maxpool-2 layer. This has the effect of allowing us to feed the convolutional architecture with an input image of size 56 × 56. Table 6: Test accuracies and parameter counts |W conv | for our'E','S' and'SL' variants of VGGNet, for different widths n of the convolutional layer. The compression factors C for the'S' and'SL' variants are computed relative to the corresponding E-VGGNet, which contains an equal number of channels n in its convolutional layers. Note that all the models are trained from a state of random initialisation. To train our networks on the CIFAR datasets, we perform some basic data augmentation steps: we randomly decide whether or not to flip the input images horizontally, we pad the 32 × 32 images with 4 pixels and then select a random crop of size 32 × 32, and finally we normalise the RGB values to have zero mean and unit norm. During the evaluation phase, we just perform the normalisation step. We train our networks for 200 epochs, using the SGD optimiser with momentum 0.9 and weight decay 5e −4. We use an initial learning rate of 0.05 and decrease it by a factor of 2 when the error plateaus. To train our networks on the Tiny ImageNet and ImageNet datasets, we perform a similar data augmentation: we first extract a crop of a random size that is then resized to the input resolution of our network (56 × 56 for Tiny ImageNet and 224 × 224 for ImageNet), we randomly decide whether or not to perform a horizontal flip of the crop, and finally we normalise the crop. During the evaluation phase, we resize the image to a standard resolution (64×64 for Tiny ImageNet and 256 × 256 for ImageNet), extract ten crops (of size 56 × 56 for Tiny ImageNet and 224 × 224 for ImageNet) from the corners, the centre and their horizontally-mirrored variants (as in), and finally normalise the crops. We train our networks for 100 epochs, using the SGD optimiser with momentum 0.9 and weight decay 5e −4. We use an initial learning rate of 0.01 for the VGGNet-like architectures on Tiny ImageNet, 0.05 for the ResNet-like architectures on Tiny ImageNet, and 0.1 for the experiments on ImageNet. Regardless of the initial value, we decrease it by a factor of 10 when the error plateaus. A.5.1 EVALUATION ON CLASSIFICATION BENCHMARKS Table 6 presents detailed accuracy and memory usage numbers for E-VGGNet, S-VGGNet and SLVGGNet architectures trained on CIFAR-100 and Tiny ImageNet ( for CIFAR-10 can be found in the main paper, in Table 1a in §5). Similar for the'E' and'SL' variants of ResNet trained on CIFAR-10 and CIFAR-100 can be found in Table 7. Finally, an accuracy and compression rate comparison of our top-performing SL3-ResNet variant with existing baselines and competing compression methods for CIFAR-10 is shown in Table 8. Table 8: CIFAR-10: Comparing the accuracies and compression factors C of top-performing'SL' variant of the ResNet architecture , for b = 3 blocks per scale level, with the original ResNet, other baselines ResNet-18 and ResNet-34, and state-of-the-art compression methods. The compression factor of the proposed model with respect to the best performing ResNet-34 architecture is in triple digits. However, a more appropriate comparison is arguably with ResNet*, from which the model has been directly compressed by virtue of sharing the convolutional layers. The compression factor is still a significant 4.0, with a final weight count of only 181K. Note that the model marked with a * has been retrained for this study. In Fig. 7, we show the linear layers for our different variants of VGGNet, trained on three different datasets -CIFAR-10, CIFAR-100 and Tiny ImageNet. As highlighted by the continuous blue vertical lines, it is notable that in each layer, some of the input channels barely contribute towards any of the output channels. Given this, we posit that a significant proportion of the weights in the linear layers (those that apply to the least important input channels) can be pruned without affecting the accuracy in any significant manner. Preliminary , verifying this conjecture, are discussed in §5.1. Interestingly, the changing locations of these blue lines reflects the changing importance of different input channels at different scale levels. Similar for four different'SL' variants of ResNet, trained on three different datasets -CIFAR-10, CIFAR-100 and Tiny ImageNet -are presented in Fig. 8. As with our visualisations for'SLVGGNet', the continuous blue vertical lines in Figs. 8b, 8c and 8d highlight that some input channels make only a minimal contribution to any of the output channels in each layer. Once again, we believe that the weights that are applied to these less-important input channels can be pruned without affecting the accuracy in any significant manner. Some indicative that support this hypothesis can be found in §5.1. By contrast, the linear layers in Fig. 8a exhibit somewhat less regularity. From Table 7a, SL7-ResNet yields both the highest accuracy (93.2%), and the highest compression rate (3.8) for that accuracy amongst all the variants. Thus, one possible explanation for this regular distribution of linear layer weights is that the model is operating at full capacity and is using all the channels in a balanced way to achieve an optimal performance. Figure 7: A visual depiction of the linear layers used to blend the input channels in the'SL' variants of VGGNet trained on CIFAR-10, CIFAR-100 and Tiny ImageNet. The linear layers are presented in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red. Figure 8: A visual depiction of the linear layers used to blend the input channels in four'SL' variants of ResNet, in the order (left to right) in which they appear in the networks. For each layer, the input channels are ordered along the x-axis, and the output channels along the y-axis. For each output channel (row), we highlight the lowest 32 weights (in terms of absolute value) in blue, and the highest 32 in red.

We compress deep CNNs by reusing a single convolutional layer in an iterative manner, thereby reducing their parameter counts by a factor proportional to their depth, whilst leaving their accuracies largely unaffected

represent the tldr for finding the corresponding scientific article:
To give you a sense - "Using the same embedding across covariates doesn't make sense, we show that a tensor decomposition algorithm learns sparse covariate-specific embeddings and naturally separable topics jointly and data-efficiently." should be close to "Word embedding is a useful approach to capture co-occurrence structures in a large corpus of text. In addition to the text data itself, we often have additional covariates associated with individual documents in the corpus---e.g. the demographic of the author, time and venue of publication, etc.---and we would like the embedding to naturally capture the information of the covariates. In this paper, we propose a new tensor decomposition model for word embeddings with covariates. Our model jointly learns a \emph{base} embedding for all the words as well as a weighted diagonal transformation to model how each covariate modifies the base embedding. To obtain the specific embedding for a particular author or venue, for example, we can then simply multiply the base embedding by the transformation matrix associated with that time or venue. The main advantages of our approach is data efficiency and interpretability of the covariate transformation matrix. Our experiments demonstrate that our joint model learns substantially better embeddings conditioned on each covariate compared to the standard approach of learning a separate embedding for each covariate using only the relevant subset of data. Furthermore, our model encourages the embeddings to be ``topic-aligned'' in the sense that the dimensions have specific independent meanings. This allows our covariate-specific embeddings to be compared by topic, enabling downstream differential analysis. We empirically evaluate the benefits of our algorithm on several datasets, and demonstrate how it can be used to address many natural questions about the effects of covariates. The use of factorizations of co-occurrence statistics in learning low-dimensional representations of words is an area that has received a large amount of attention in recent years, perhaps best represented by how widespread algorithms such as GloVe BID10 and Word2Vec BID8 are in downstream applications. In particular, suppose we have a set of words i ∈ [n], where n is the size of the vocabulary. The aim is to, for a fixed dimensionality d, assign a vector v i ∈ R d to each word in the vocabulary in a way that preserves semantic structure. In many settings, we have a corpus with additional covariates on individual documents. For example, we might have news articles from both conservative and liberal-leaning publications, and using the same word embedding for all the text can lose interesting information. Furthermore, we suggest that there are meaningful semantic relationships that can be captured by exploiting the differences in these conditional statistics. To this end, we propose the following two key questions that capture the problems that our work addresses, and for each, we give a concrete motivating example of a problem in the semantic inference literature that it encompasses. Question 1: How can we leverage conditional co-occurrence statistics to capture the effect of a covariate on word usage?For example, did William Shakespeare truly write all the works credited to him, or have there been other "ghostwriters" who have contributed to the Shakespeare canon? This is the famous Shakespeare authorship question, for which historians have proposed various candidates as the true authors of particular plays or poems BID5. If the latter scenario is the case, what in particular distinguishes the writing style of one candidate from another, and how can we infer who the most likely author of a work is from a set of candidates?Question 2: Traditional factorization-based embedding methods are rotationally invariant, so that individual dimensions do not have semantic meaning. How can we break this invariance to yield a model which aligns topics with interpretable dimensions?There has been much interest in the differences in language and rhetoric that appeal to different demographics. For example, studies have been done regarding "ideological signatures" specific to voters by partisan alignment (Robinson et al.) in which linguistic differences were proposed along focal axes, such as the "mind versus the body" in texts with more liberal or conservative ideologies. How can we systematically infer topical differences such as these between different communities?Questions such as these, or more broadly covariate-specific trends in word usage, motivated this study. Concretely, our goal is to provide a general framework through which embeddings of sets of objects with co-occurrence structure, as well as the effects of conditioning on particular covariates, can be learned jointly. As a byproduct, our model also gives natural meaning to the different dimensions of the embeddings, by breaking the rotational symmetry of previous embedding-learning algorithms, such that the ing vector representations of words and covariates are "topic-aligned".Previous Work Typically, algorithms for learning embeddings rely on the intuition that some function of the co-occurrence statistics is low rank. Studies such as GloVe and Word2Vec proposed based on minimizing low-rank approximation-error of nonlinear transforms of the co-occurrence statistics. let A be the n × n matrix with A ij the co-occurrence between words i and j, where co-occurrence is defined as the (possibly weighted) number of times the words occur together in a window of fixed length. For example, GloVe aimed to find vectors v i ∈ R d and biases b i ∈ R such that the loss DISPLAYFORM0 was minimized, where f was some fixed increasing weight function. Word2Vec aimed to learn vector representations via minimizing a neural-network based loss function. A related embedding approach is to directly perform principal component analysis on the PMI (pointwise mutual information) matrix of the words (Bullinaria & Levy). PMI-factorization based methods aim to find vectors {v i} such that DISPLAYFORM1 where the probabilities are taken over the co-occurrence matrix. This is essentially the same as finding a low-rank matrix V such that V T V ≈ P M I, and empirical show that the ing embedding captures useful semantic structure. The ideas of several previous studies on the geometry of word embeddings was helpful in formulating our model. A random-walk based mathematical framework for understanding these different successful learning algorithms was proposed BID1, in which the corpus generation process is a random process driven by the random walk of a discrete-time discourse vector c t ∈ R d. In this framework, our work can be thought of as analyzing the effects of covariates on the random walk transition kernel and the stationary distribution. Additionally, there have been previous studies of "multi-sense" word embeddings BID11 BID9, which is similar to our idea that the same word can have different meanings in different contexts. However, in the multi-sense setting, the idea is that the word intrinsically has different meanings (for example, "crane" can be an action, a bird, or a vehicle), whereas in ours, the different meanings are imposed by conditioning on a covariate. Finally, tensor methods have been used in other settings recently, such as collaborative filtering BID13 and (Li & Farias), to learn the effects of conditioning on some summary statistics. Our Contributions There are several reasons why a joint learning model based on tensor factorization is more desirable than performing GloVe m times, where m is the number of covariates, so that each covariate-specific corpus has its own embedding. Our main contributions are a decomposition algorithm that addresses these issues, and the methods for systematic analysis we propose. The first issue that arises is sample complexity. In particular, because for the most part words are used in roughly similar ways across different contexts, the ing embeddings should not be too different, except perhaps along specific dimensions. Thus, it is better to jointly train an embedding model along the covariates to aggregate the co-occurrence structure, especially in cases where the entire corpus is large, but many conditional corpuses (conditioned on a covariate) are small. Secondly, simply training a different embedding for each corpus makes it difficult to compare the embeddings across the covariate dimension. Because of issues such as rotation invariance of GloVelike models, specific dimensions mean different things across different runs (and initializations) of these algorithms. The model we propose has the additional property that it induces a natural basis to view the embeddings in, one which is "topic-aligned" in the sense that it is not rotation-invariant and thus implies independent topic meanings given to different dimensions. Paper Organization In section 2, we provide our embedding algorithm, as well as mathematical justification for its design. In section 3, we detail our dataset. In section 4, we validate our algorithm with respect to intrinsic properties and standard metrics. In section 5, we propose several experiments for systematic downstream analysis. Notation Throughout this section, we will assume a vocabulary of size n and a discrete covariate to condition on of size m (for example, the community that the corpus comes from, i.e. liberal or conservative discussion forums). It is easy to see how our algorithm generalizes to higher-order tensor decompositions when there are multiple dimensions covariates to condition on (for example, slicing along community and slicing along timeframe simultaneously). Words will be denoted with indices i, j ∈ [n] and covariates with index k ∈ [m]. Dimensions in our embedding are referred to by index t ∈ [d].We will denote the co-occurrence tensor as A ∈ R n×n×m, where A ijk denotes how many times words i and j occurred together within a window of some fixed length, in the corpus coming from covariate k. The of our algorithm will be two sets of vectors, {v i ∈ R d} and {c k ∈ R d}, as well as bias terms that also fit into the objective. Finally, let denote the element-wise product between two vectors. Objective Function and Discussion Here, we give the objective function our method minimizes, and provide some explanation for how one should imagine the effect of the covariate weights. The objective function we minimize is the following partial non-negative tensor factorization objective function for jointly training word vectors and weight vectors representing the effect of covariates, adapted from the original GloVe objective (note that c DISPLAYFORM0 which is to be optimized over {v i ∈ R d}, {c k ∈ R d}, and {b ik ∈ R}. To gain a little more intuition for why this is a reasonable objective function, note that the ing objective for a single "covariate slice" is essentially DISPLAYFORM1 model since the c k can be absorbed into the v i. We used f (x) = (min(100,x) 100 ) 0.75, to parallel the original objective function in BID10.One can think of the dimensions our model learns as independent topics, and the effects of the covariate weights c k as upweighting or downweighting the importance of these topics in contributing to the conditional co-occurrence statistics. Here we provide a geometric perspective on our model in the context of some prior work. This geometric interpretation is not necessary to applying our method and to understand its . Throughout this section, note that at a high level, the aim of our method is to learn sets {v i ∈ R d}, {c k ∈ R d}, and {b ik ∈ R} for 1 ≤ i ≤ n, 1 ≤ k ≤ m, such that for a fixed k, the vectors {c k v i} and the biases {b ik} approximate the vectors and biases that would have been learned from running GloVe on only the k th slice of the co-occurrence tensor. We now provide some rationale for why this is a reasonable objective. The main motivation for our algorithm is the following geometric intuition inspired in part by the model of BID1. In their model, corpus generation is determined by the nature of the random walk performed by a context vector c t over time steps. In particular, for each timestep t, the context vector c t emits words i with probability ∝ exp(v i · c t), and then c t updates its location based on a transition distribution. The assumption is that the transition distribution has a stationary distribution that is uniform over some ellipse. One natural equivalence family of transition matrices that preserves this property is the group ing from multiplication by a positive semidefinite matrix. Therefore, we consider a natural extension of this model, where the embedding ing from conditioning on different covariates is equivalent to multiplying the transition matrix by a symmetric PSD matrix. Alternatively, this is equivalent to a model where the transition kernel remains unchanged, but the embedding vectors themselves are multiplied by a symmetric PSD matrix (namely, the Moore-Penrose pseudoinverse of the original PSD matrix). This is the viewpoint we adopt. Figure 1: The effects of conditioning on covariates (covariates are discussion forums, described in Section 3). Left: baseline embedding with some possible word embedding positionings. Middle, right: embedding under effect of covariates: for example, "hillary" and "crooked" are pushed closer together under effects of The Donald, and "healthcare" pushed closer to "universal" and "free" under effects of SandersForPresident. Context vectors and random walk transitions also shown. In this framework, assign each covariate k its own symmetric PSD matrix, B k. It is well-known that any symmetric PSD matrix can be factorized as B k = R T k D k R k for some orthonormal basis R k and some (nonnegative) diagonal D k. Thus it suffices to consider the effect of a covariate on some ground truth "base embedding" M as applying the linear operator B k to each embedding vector, ing in the new embedding B k M.This model is quite expressive in its own right, but we consider a natural restriction where we propose that there exists some universal (at least across covariates) basis R under which the ing embeddings are affected by covariates via multiplication by just a diagonal matrix, instead of a PSD matrix. In particular, we note that DISPLAYFORM0 where M k = R k M is a rotated version of M. Now, in the restricted model where all the R k are equal, we can write all the M k as RM, so it suffices to just consider the rotation of the basis that the original embedding was trained in where R is just the identity (since matrix-factorization based word embedding models are rotation invariant). Under this model, the co-occurrence statistics under some transformation should be equivalent to M T D 2 k M. A careful comparison shows that this approximation is precisely that which is implied by equation 4, as desired. Note that this is essentially saying that in this distributed word representation model, there exists some rotation of the embedding space under which the effect of the covariate separates along dimensions. The implication is that there are some set of independent "topics" that each covariate will upweight or downweight in importance (or possibly ignore altogether with a weight of 0), characterizing the effect of this conditioning directly in terms of the effect on these topics. Algorithm Details Our model learns the ing parameters {v i ∈ R d}, {c k ∈ R d}, and {b ik}, by using the Adam BID6 algorithm, which was empirically shown to yield good convergence in the original GloVe setting. The specific hyperparameters used for each dataset will be described in the next section. The word and covariate weight vectors were initialized as random unit vectors 1. We evaluated our method in to primary datasets. In both datasets, co-occurrence statistics were formed by considering size 8 windows and using an inverse-distance weighting (e.g. neighboring words had 1 added to their co-occurrence, and words 3 apart had 1 3 added), which was suggested by some implementations of BID10.The first dataset, referred to as the "book dataset", consists of the full text from 29 books written by 4 different authors. The books we used were J.K. Rowling's "Harry Potter" series (7 books), "Cormoran Strike" series (3 books), and "The Casual Vacancy"; C. S. Lewis's "The Chronicles of Narnia" series (7 books), and "The Screwtape Letters"; George R. R. Martin's "A Song of Ice and Fire" series (5 books); and Stephenie Meyer's "Twilight" series (4 books), and "The Host". These books are fiction works in similar genres, with highly overlapping vocabularies and common themes. A trivial way of learning series-specific tendencies in word usage would be to cluster according to unique vocabularies (for example, only the "Harry Potter" series would have words such as "Harry" and "Ron" frequently), so the co-occurrence tensor was formed by looking at all words that occurred in all of the series with multiple books, which eliminated all series-specific words. Furthermore, series by the same author had very different themes, so there is no reason intrinsic to the vocabulary to believe the weight vectors would cluster by author. The vocabulary size was 5,020, and after tuning our algorithm to embed this dataset, we used 100 dimensions and a learning rate of 10 −5.The second dataset, referred to as the "politics dataset", was a collection of comments made in 2016 in 6 different subreddits on the popular discussion forum reddit, and was selected to address both Questions 1 and 2. The covariate was the discussion forum, and the subreddits we used were AskReddit, news, politics, SandersForPresident, The Donald, and WorldNews. AskReddit was chosen as a baseline discussion forum with a very general vocabulary usage, and the discussion forums for the Sanders and Trump support bases were also selected, as well as three politically-relevant but more neutral communities (it should be noted that the politics discussion forum tends to be very leftleaning). We considered a vocabulary of size 15,000, after removing the 28 most common words (suggested by other works in the embedding literature) and entries of the cooccurrence tensor with less than 10 occurrences (for the sake of training efficiency). The embedding used 200 dimensions and a learning rate of 10 −5. We performed the tensor decomposition algorithm on the book dataset, and considered how well the weight vectors of the covariate clustered by series and also by author.(a) 2D PCA of book dataset weight vectors (tensor decomposition algorithm, 100 dimensions) (b) 2D PCA of book dataset topic vectors, as predicted by LDA (100 topics)For every book in every series, the closest weight vectors by series were all the books in the same series. Furthermore, for every book written by every author, the closest weight vectors by author were all the books by the same author. This clear clustering behavior even when only conditioning on co-occurrence statistics for words that appear in all series (throwing out series-specific terms) implies the ability of the weight vectors to cluster according to higher-order information, perhaps such as writing style. As a baseline, we considered the topic breakdown on the same co-occurrence statistics predicted by Latent Dirichlet Allocation BID2 with various dimensionalities, all of which failed to produce any meaningful clusters. Consider the problem of learning an embedding for the text of a particular book (or series). The main advantage given by using the covariate-specific embedding that our method yields over applying GloVe to the individual slice is a data efficiency -by pooling the co-occurrence statistics of words across other books, we are able to give better (less noisy) estimates of the vectors, especially for sparsely-occurring or nonexistent words. For reference, individual books contained between 26747 and 355814 words. To this end, we performed the following experiment. For some book k, consider two embeddings: 1) the of performing GloVe on the co-occurrence statistics of just the book, and 2) the (weighted) embedding ing from our algorithm, specific to the book. Then, we tested these ing embeddings using a standard suite of evaluation metrics, including cluster purity and correlation similarities. Our method outperformed method 1 on all 7 tasks, often by a significant margin. BID1, it is unreasonable to ask for sparsity in the word embeddings themselves. However, our topic-specific weighting scheme for the covariate weight vectors implies that sparsity is desirable in the weights. The weight sparsity ing from our algorithm was experimentally verified through many runs of our embedding method, as well as across different optimization methods. Note that in the objective function, sparse coordinates will become "stuck" at 0, because the gradient update of c kt is proportional to c kt: DISPLAYFORM0 The dimensions that the covariates were sparse in did not overlap by much: the average number of sparse coordinates per weight vector was 20.7, and the average number of coordinates that was sparse in both vectors of a random pair was 5.2. This suggest that the set of "topics" each conditional slice of the tensor does not care about is fairly specific. Experimenting with regularizing the word vectors forced the model into a smaller subspace (i.e. sparse dimensions existed but were shared across all the words), which is not useful and provides further evidence for the natural isotropic distribution of word vectors. Regularizing the weights of the covariate vectors by adding an l 1 penalty term did not noticeably change the sparsity pattern. To confirm that the sparsity was a of separation of covariate effects rather than an artifact of our algorithm, we ran our decomposition on a co-occurrence tensor which was the of taking the same slice (subreddit) and subsampling its entries 3 times, creating 3 slices of essentially similar cooccurrence statistics. We applied our algorithm with different learning parameters, and the ing weight vectors after 90 iterations (when the outputs vectors converged) were extremely non-sparse, with between 0 and 2 sparse coordinates per weight vector. The dimensions that are specifically 0 for a covariate corresponds to topics that are relatively less relevant for that covariate. In the next section, we develop methods to systematically interpret the covariate weights in terms of topics. A simple test of inferring topic meaning (i.e. topics coordinates are associated with) is to consider the set of words which are large in the given coordinate. Concretely the task is, given some index DISPLAYFORM0, output the words whose (normalized) vectors have largest value in dimension t. We show the of this experiment for several of the sparse coordinates in the AskReddit weight vector: There are several to be drawn from this experiment. Firstly, while there is some clear noise, specific topic meanings do seem to appear in certain coordinates (which we infer in the table above). It is reasonable that meaning would appear out of coordinates which are sparsely weighted in some covariate, because this means that it is a topic that is relevant in other discussion forums but purposely ignored in some specific forum, so it is likely to have a consistent theme. When we performed this task for coordinates which had low variance in their weights between covariates, the ing words were much less consistent. It is also interesting to see how covariates weight a topic whose meaning we have identified. For example, for coordinate 194 (corresponding to military themes), AskReddit placed negligible weight, news and worldnews placed weight 2.06 and 2.04 respectively, SandersForPresident and The Donald placed weight 0.41 and 0.38 respectively, and politics placed weight 0.05. This process can also be reversed -for example, consider coordinates small in worldnews and large in news. One example was coordinate 188, and upon checking the words that were large in this coordinate ({taser, troops, supremacists, underpaid, rioters, amendment, racially, hispanic}) it seemed clear that it had themes of rights and protests, which makes sense as a domestic issue, not a global one. We performed the following experiment: which pairs of words start off close in the baseline embedding, yet under some covariate weights move far apart (or vice versa)? Concretely, the task is, for a fixed word i and a fixed covariate k, identify words j such that ||c DISPLAYFORM0 where the magnitude of drift is quantified by the ratio of distances in the normalized embedding. The motivation is to find words whose general usage is similar, but have very different meanings in specific communities. We present a few representative examples of this experiment below, for k = The Donald. Combining the previous two sections allows us to do an end-to-end case study on words that drift under a covariate, so we can explain specifically which topics (under reweighting) caused this shift. For example, the words "immigrant" and "parasite" were significantly closer under the weights placed by The Donald, so we considered dimensions that were simultaneously large in the vector v immigrant − v parasite and sparse in the weight vector c T he Donald. The dimensions 89 and 139 were sparse and also the 2nd and 3rd largest coordinates in the difference vector, so they had a large contribution to the subspace which was zeroed out under the reweighting. Words that were large in these dimensions (and thus representative of the zeroed out subspace meaning) included {misunderstanding, populace, scapegoat, rebuilding} for 89, and {answers, barriers, applicants, backstory, selfless, indigenous} for 139. This suggests two independent reasons for the drift, namely dimensions corresponding to emotional appeal and legal immigration respectively being zeroed out. One of the most famous downstream applications of recent embedding methods such as BID10 and BID8 is their ability to capture analogies. This is typically formulated as a: DISPLAYFORM0 We considered how well our method captured covariate-specific analogies, which appear in a covariatespecific embedding but not most others. To this end, we considered experiments of the form: for fixed words a, b, c, determine words d such that for some covariate k, the quantity DISPLAYFORM1 is small, yet for other k, the quantity is large. The intuition is that under the covariate transform, v c − v d points roughly in the same direction as v a − v b, and d is close to c in semantic meaning. In particular, we set a = "hillary", c = "trump", and found words b for which there existed a d consistently at the top across subreddits (implying existence of strong analogies). For example, when b = "woman", d = "man" was the best analogy for every weighting. Then, for these b, we considered words d whose relative rankings in the subreddits had high variance. The differential analogies captured were quite striking, and we present several representative examples in TAB4. Discussion We have presented a joint tensor model that essentially learns an embedding for each word and for each covariate. This model makes it very simple to compute the covariate specific embedding: we just take the element-wise vector product. It also enables us to systematically interpret the covariate vector by looking at dimensions along which weight is large or 0. Our experiments show that these dimensions can be interpreted as coherent topics. While we focus on word embeddings, our tensor covariate embedding model can be naturally applied in other settings. For example, there is a large amount of interest in learning embeddings of individual genes to capture biological interactions. The natural covariates here are the cell types and our method would be able to model cell-type specific gene interactions. Another interesting setting with conditional covariates would be time-series specific embeddings, where data efficiency becomes more of an issue. We hope our framework is general enough that it will be of use to practitioners in these settings and others. We also experimented with using (Duchi et al.) as the optimization method, but the ing weight vectors in the politics dataset had highly-overlapping sparse dimensions. This implies that the optimization method tried to fit the model to a smaller-dimensional subspace, which is not a desirable source of sparsity."

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the article for finding a single-sentence TLDR

Question: We extend the recent of by a spectral analysis of representations corresponding to kernel and neural embeddings. They showed that in a simple single layer network, the alignment of the labels to the eigenvectors of the corresponding Gram matrix determines both the convergence of the optimization during training as well as the generalization properties. We generalize their to kernel and neural representations and show that these extensions improve both optimization and generalization of the basic setup studied in . The well-known work of BID8 highlighted intriguing experimental phenomena about deep net trainingspecifically, optimization and generalization -and called for a rethinking of generalization in statistical learning theory. In particular, two fundamental questions that need understanding are: Optimization. Why do true labels give faster convergence rate than random labels for gradient descent? Generalization. What property of properly labeled data controls generalization? BID0 have recently tried to answer this question in a simple model by conducting a spectral analysis of the associated Gram matrix. They show that both training and generalization are better if the label vector aligns with the top eigenvectors. However, their analysis applies only to a simple two layer network. How could their insights be extended to deeper networks?A widely held intuitive view is that deep layers generate expressive representations of the raw input data. Adopting this view, one may consider a model where a representation generated by successive neural network layers is viewed as a kernel embedding which is then fed into the two-layer model of BID0. The connection between neural networks and kernel machines has long been studied; BID2 ) introduced kernels that mimic deep networks and BID6 showed kernels equivalent to certain feed-forward neural networks. Recently, BID1 ) also make the case that progress on understanding deep learning is unlikely to move forward until similar phenomena in classical kernel machines are recognized and understood. Very recently, BID4 showed that the evolution of a neural network during training can be related to a new kernel, the Neural Tangent Kernel (NTK) which is central to describe the generalization properties of the network. Here we pursue this approach by studying the effect of incorporating embeddings in the simple two layer model and we perform a spectral analysis of these embeddings along the lines of BID0. We can obtain embeddings in several ways: i. We can use an unbiased kernel such as Gaussian kernel. This choice is consistent with the maximum entropy principle and makes no prior assumption about the data. Or use a kernel which mimics or approximates deep networks ii. We could use data driven embeddings explicitly produced by the hidden layers in neural networks: either use a subset of the same training data to compute such an embedding, or transfer the inferred embedding from a different (but similar) domain. While a general transformation g(x) of the input data may have arbitrary effects, one would expect kernel and neural representations to improve performance. The interplay of kernels and data labellings has been addressed before, for example in the work of kernel-target alignment BID3 ).We do indeed observe a significant beneficial effect: Optimization. Using kernel methods such as random Fourier features (RFF) to approximate the Gaussian kernel embedding BID5 and neural embeddings, we obtain substantially better convergence in training. Generalization. We also achieve significantly lower test error and we confirm that the data dependent spectral measure introduced in BID0 significantly improves with kernel and neural embeddings. Thus this work shows empirically that kernel and neural embeddings improve the alignment of target labels to the eigenvectors of the Gram matrix and thus help training and generalization. This suggests a way to extend the insights of BID0 ) to deeper networks, and possible theoretical in this direction. Network model. In BID0, the authors consider a simple two layer network model: DISPLAYFORM0 DISPLAYFORM1 These can be written jointly as a = (a 1, .., a m) T and W = (w 1, .., w m). This network is trained on dataset of datapoints {x i} and their targets {y i}.They provide a fine grained analysis of training and generalization error by a spectral analysis of the Gram matrix: BID0 show that both training and generalization are better if the label vector y aligns with the eigenvectors corresponding to the top eigvalues of H ∞. DISPLAYFORM2 The two-layer ReLU network in this work follows the general structure as in BID0 with the difference being the addition of an embedding φ at the input layer corresponding to a kernel K. The corresponding model is: DISPLAYFORM3 and let its eigenvalues be ordered as DISPLAYFORM4 A kernel K such that the corresponding eigenvectors align well with the labels would be expected to perform well both for training optimization as well as generalization. This is related to kernel target alignment BID3. Optimization. For the simple two layer network, BID0 show that the convergence of gradient descent is controlled by DISPLAYFORM5 For our kernelized network, the corresponding convergence is controlled by DISPLAYFORM6 Generalization. For the simple two layer network, BID0 show that the generalization performance is controlled by DISPLAYFORM7 For our kernelized two layer network, the corresponding data and representation dependent measure is: DISPLAYFORM8 We perform our experiments on two commonly-used datasets for validating deep neural models, i.e., MNIST and CIFAIR-10. These datasets are used for the experiments in BID0. As in their work we only look at the first two classes and set the label y i = +1 if image i belongs to the first class and y i = −1 if it belongs to the second class. The images are normalized such that ||x i || 2 = 1. This is also done for kernel embeddings such that ||φ(x i)|| 2 = 1. The weights in equation are initialized as follows: DISPLAYFORM0 We then use the following loss function to train the model to predict the image labels. DISPLAYFORM1 For optimization, we use (full batch) gradient descent with the learning rate η. In our experiments we set k = 10 −2, η = 2 · 10 −4 similar to BID0. We first use the Gaussian kernel K(x i, x j):= exp −γ x i − x j 2. The corresponding embedding is infinite dimensional, hence we consider the fast approximations to the kernel given by random Fourier features (RFF) BID5. The idea of random Fourier features is to construct an explicit feature map which is of a dimension much lower than the number of observations, but the ing inner product approximates the desired kernel function. We use γ = 1 in all our experiments. Optimization. We first investigate the use of Gaussian kernel for a more efficient optimization of the loss function on the training data. FIG0 show the training loss at different steps respectively on MNIST and CIFAR-10 datasets. We consistently observe that the different Gaussian kernels (specified by various dimensions of the kernel) yields faster convergence of the optimization procedure on both datasets. MNIST is a simple dataset which gives incredibly high score almost immediately, as shown by the train loss FIG0 ) and by the accuracy on the test data (the table in FIG3 (c)) thus we will focus our analysis on the CIFAR-10 dataset. Similar to the setup in BID0, in FIG0, for different methods, we plot the eigenvalues of H(K)∞ and the projections of the true class labels on the eigenvectors (i.e., the projections {(v 2 's for top eigenvalues. Generalization. We next investigate the generalization performance of the Gaussian kernel method by analyzing the values of equations FORMULA7 and. TAB0 shows this quantity for different settings and kernels respectively on MNIST and CIFAR-10 datasets. We observe that in both datasets with several kernels we obtain a lower theoretical upper bound on the generalization error. It is clear that the bound improves as the dimension of the representations increases but also that the generalization bound seems quite sensitive to values of γ. In addition to the theoretical upper bound, we measure the test error for the studied datasets. FIG3 show respectively the test error and the test accuracy at different steps of the optimization by Gradient Descent for CIFAR-10. We observe that the kernel methods yield significant improvements of both the test error and the accuracy on the test dataset. We observe that the larger the kernel, the larger the improvement. Additionally, we can see a sharper reduction in test error compared to the no-kernel case. This sharp transition (after a small number of steps) is particularly interesting. Because, along such a transition, we observe a significant improvement in the accuracy on test dataset. Thus early-stopping that is commonly used in deep learning can be even more efficient when using kernel methods. Finally, similar to the no-kernel case in BID0, by comparing the plots in FIG0, 1(c) and 2(a) we find tight connections between, i) (training) optimization, ii) projection on the top eigenvalues, and iii) generalization. We can therefore improve both training and generalization with kernels since we can get better alignment of the eigenvectors belonging the largest eigenvalues and the target labels. Choosing a proper kernel and its parameters can be challenging BID7, as also seen in TAB0. Thus, we investigate a data-dependent neural kernel and embedding. For this purpose, we add a second hidden layer to the neural network with m = 10000 hidden units and ReLU activation. We pre-train this embedding using two different approaches. The first layer is then kept fix as an embedding where the rest of the network is reinitialized and trained. The first approach is to split the training data in half. We use the first subset to pre-train this three-layer network and the second subset to use for our optimization experiments. In this approach we double η to keep the step length the same. The other approach is to use data from a different domain for pre-training. For instance, we use the last two classes of the CIFAR-10 dataset for pre-training the embedding. We compare our with not using any kernel and with using a RFF kernel with embedding of size 10000. Optimization. FIG4 shows the training loss for the CIFAR-10 dataset. We observe the neural embeddings achieve faster convergence compared to the previous methods. We report the training loss for neural embedding (same label) on the second (unused) subset of the data, whereas in the other cases we report the on the full training data. If we use only the second subset for the other methods, we observe very consistent to FIG4. FIG4 (c) demonstrates the top eigenvalues as well as their eigenvector projections on the target labels. This shows that both variants of neural embeddings improve alignment of the labels to eigenvectors corresponding to larger eigenvalues (compared to the best RFF kernel). While the effect is unsurprisingly larger when pre-training on the same labels, it is still significantly better when pre-trained on other labels. Generalization. In FIG4 (b) we report the test error on the CIFAR-10. This shows that the neural embeddings perform at least comparable with the best studied RFF kernel. If the pre-training is done on the same labels we obtain a clear improvement, even if the actual training is only done on a dataset with half the size. We extended the recent of BID0 by a spectral analysis of the representations corresponding to kernel and neural embeddings and showed that such representations benefit both optimization and generalization. By combining recent connecting kernel embeddings to neural networks such as BID6 BID4, one may be able to extend the fine-grained theoretical of BID0 for two layer networks to deeper networks.

Represent the TLDR for finding the corresponding scientific article

Spectral analysis for understanding how different representations can improve optimization and generalization.

Represent the short summary for finding the corresponding scientific article (long).

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.