Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Human observers can learn to recognize new categories of objects from a handful of examples, yet doing so with machine perception remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable, as suggested by recent perceptual evidence. We therefore revisit and improve Contrastive Predictive Coding, a recently-proposed unsupervised learning framework, and arrive at a representation which enables generalization from small amounts of labeled data. When provided with only 1% of ImageNet labels (i.e. 13 per class), this model retains a strong classification performance, 73% Top-5 accuracy, outperforming supervised networks by 28% (a 65% relative improvement) and state-of-the-art semi-supervised methods by 14%. We also find this representation to serve as a useful substrate for object detection on the PASCAL-VOC 2007 dataset, approaching the performance of representations trained with a fully annotated ImageNet dataset. ResNet trained on CPC ResNet trained on pixels With decreasing amounts of labeled data, supervised networks trained on pixels fail to generalize (red). When trained on unsupervised representations learned with CPC, these networks retain a much higher accuracy in this low-data regime (blue). Equivalently, the accuracy of supervised networks can be matched with significantly fewer labels. Deep neural networks excel at perceptual tasks when labeled data are abundant, yet their performance degrades substantially when provided with limited supervision (Fig. 1, red). In contrast, humans and animals can quickly learn about new classes of objects from few examples . What accounts for this monumental difference in data-efficiency between biological and machine vision? While highly-structured representations (e.g. as proposed by) may improve data-efficiency, it remains unclear how to program explicit structures that capture the enormous complexity of real visual scenes like those in ImageNet . An alternative hypothesis has proposed that intelligent systems need not be structured a priori, but can instead learn about the structure of the world in an unsupervised manner (; ;). Choosing an appropriate training objective is an open problem, but a promising guiding principle has emerged recently: good representations should make the spatio-temporal variability in natural signals more predictable. Indeed, human perceptual representations have been shown to linearize (or 'straighten') the temporal transformations found in natural videos, a property lacking from current supervised image recognition models (Hénaff et al., 2019), and theories of both spatial and temporal predictability have succeeded in describing properties of early visual areas . In this work, we hypothesize that spatially predictable representations may allow artificial systems to benefit from human-like data-efficiency. Contrastive Predictive Coding (CPC, van den) is an unsupervised objective which learns such predictable representations. CPC is a general technique that only requires in its definition that observations be ordered along e.g. temporal or spatial dimensions, and as such has been applied to a variety of different modalities including speech, natural language and images. This generality, combined with the strong performance of its representations in downstream linear classification tasks, makes CPC a promising candidate for investigating the efficacy of predictable representations for data-efficient image recognition. Our work makes the following contributions: • We revisit CPC in terms of its architecture and training methodology, and arrive at a new implementation of CPC with dramatically-improved ability to linearly separate image classes (+17% Top-1 ImageNet classification accuracy). • We then train deep networks on top of the ing CPC representations using very few labeled images (e.g. 1% of the ImageNet dataset), and demonstrate test-time classification accuracy far above networks trained on raw pixels (73% Top-5 accuracy, a 28% absolute improvement), outperforming all other unsupervised representation learning methods (+15% Top-5 accuracy over the previous state-of-the-art). Surprisingly, this representation also surpasses supervised methods when given the entire ImageNet dataset (+1% Top-5 accuracy). • We isolate the contributions of different components of the final model to such downstream tasks. Interestingly, we find that linear classification accuracy is not always predictive of low-data classification accuracy, emphasizing the importance of this metric as a stand-alone benchmark for unsupervised learning. • Finally, we assess the generality of CPC representations by transferring them to a new task and dataset: object detection on PASCAL-VOC 2007. Consistent with the from the previous section, we find CPC to give state-of-the-art performance in this setting. We first review the CPC architecture and learning objective in section 2.1, before detailing how we use its ing representations for image recognition tasks in section 2.2. Contrastive Predictive Coding as formulated in (van den) learns representations by training neural networks to predict the representations of future observations from those of past ones. When applied to images, the original formulation of CPC operates by predicting the representations of patches below a certain position from those above it (Fig. 2, left). These predictions are evaluated using a contrastive loss, in which the network must correctly classify the'future' representation amongst a set of unrelated'negative' representations. This avoids trivial solutions such as representing all patches with a constant vector, as would be the case with a mean squared error loss. In the CPC architecture, each input image is first divided into a set of overlapping patches x i,j, each of which is encoded with a neural network f θ into a single vector z i,j = f θ (x i,j). To make predictions, a masked convolutional network g φ is then applied to the grid of feature vectors. The masks are such that the receptive field of each ing context vector c i,j only includes feature vectors that lie above it in the image (i.e. {z u,v} u≤i,v ). The prediction task then consists o predicting'future' feature vectors z i+k,j from current context vectors c i,j, where k > 0. The predictions are made linearly: given a context vector c i,j, a prediction length k > 0, and a prediction matrix W k, the predicted feature vector isẑ i+k,j = W k c i,j. The quality of this prediction is then evaluated using a contrastive loss. Specifically, the goal is to correctly recognize the target z i+k,j among a set of randomly sampled feature vectors {z l} from the dataset. We compute the probability assigned to the target using a softmax, and evaluate this probability using the usual cross-entropy loss. Summing this loss over locations and prediction Figure 2: Overview of the framework for semi-supervised learning with Contrastive Predictive Coding. Left: unsupervised pre-training with the spatial prediction task (See Section 2.1). First, an image is divided into a grid of overlapping patches. Each patch is encoded independently from the rest with a feature extractor (blue) which terminates with a mean-pooling operation, yielding a single feature vector for that patch. Doing so for all patches yields a field of such feature vectors (wireframe vectors). Feature vectors above a certain level (in this case, the center of the image) are then aggregated with a context network (red), yielding a row of context vectors which are used to linearly predict features vectors below. Right: using the CPC representation for a classification task. Having trained the encoder network, the context network (red) is discarded and replaced by a classifier network (green) which can be trained in a supervised manner. For some experiments, we also fine-tune the encoder network (blue) for the classification task. When applying the encoder to cropped patches (as opposed to the full image) we refer to it as a patched ResNet in the figure. offsets, we arrive at the CPC objective as defined in (van den): The negative samples {z l} are taken from other locations in the image and other images in the minibatch. This loss is called InfoNCE (van den) as it is inspired by Noise-Contrastive Estimation (Gutmann & Hyvärinen, 2010;) and has been shown to maximize the mutual information between c i,j and z i+k,j (van den). Having trained an encoder network f θ, a context network g φ, and a set of linear predictors {W k} using the CPC objective, we use the latents z = f θ (x) as a representation of new observations x for downstream tasks, and discard the rest. We then train a model h ψ to classify these representations given a dataset of labeled images. More formally, given a dataset of N unlabeled images D u = {x n}, and a (potentially much smaller) dataset of M labeled images D l = {x m, y m}: In all cases, the dataset of unlabeled images D u we pre-train on is the full ImageNet ILSVRC 2012 training set . We consider three labeled datasets D l for evaluation, each with an associated classifier h ψ and supervised losse L Sup (see Fig. 2, right). This protocol is sufficiently generic to allow us to later compare the CPC representation to other methods which have their own means of learning a feature extractor f θ. Linear classification is the standard benchmark for evaluating the quality of unsupervised image representations. In this regime, the classification network h ψ is restricted to mean pooling followed by a single linear layer, and the parameters of f θ are kept fixed. The labeled dataset D l is the entire ImageNet dataset, and the supervised loss L Sup is standard cross-entropy. We use the same dataaugmentation as in the unsupervised learning phase for training, and none at test time and evaluate with a single crop. Efficient classification directly tests whether the CPC representation enables visual learning from few labels. For this task, the classifier h ψ is an arbitrary deep neural network (we use an 11-block ResNet architecture with 4096-dimensional feature maps and 1024-dimensional bottleneck layers). The labeled dataset D l is a subset of the ImageNet dataset: we investigated using 1%, 2%, 5%, 10%, 20%, 50% and 100% of the ImageNet dataset. The supervised loss L Sup is again cross-entropy. In addition to random color-dropping we use the Inception data-augmentation scheme for training, no augmentation at test-time and evaluate with a single crop. Transfer learning tests the generality of the representation by applying it to a new task and dataset. For this we chose image detection on the PASCAL-2007 dataset, a standard benchmark in computer vision . As such D l is the entire PASCAL-2007 dataset (comprised of 5011 labeled images); h ψ and L Sup are the Faster-RCNN architecture and loss . In addition to color-dropping, we use scale-augmentation for training. For linear classification, we keep the feature extractor f θ fixed to assess the representation in absolute terms. For efficient classification and transfer learning, we additionally explore fine-tuning the feature extractor for the supervised objective. In this regime, we initialize the feature extractor and classifier with the solutions θ *, ψ * found in the previous learning phase, and train them both for the supervised objective. To ensure that the feature extractor does not deviate too much from the solution dictated by the CPC objective, we use a smaller learning rate and early-stopping. Data-efficient learning has typically been approached by two complementary methods, both of which seek to make use of more plentiful unlabeled data: representation learning and semisupervised learning. The former formulates an objective to learn a feature extractor f θ in an unsupervised manner, whereas the latter directly constrains the classifier h ψ using the unlabeled data. Representation learning saw early success using generative modeling , but likelihood-based models have yet to generalize to more complex stimulus classes. Generative adversarial models have also been harnessed for representation learning , and large-scale implementations have recently achieved corresponding gains in linear classification accuracy . In contrast to generative models which require the reconstruction of observations, self-supervised techniques directly formulate tasks involving the learned representation. For example, simply asking a network to recognize the spatial layout of an image led to representations that transferred to popular vision tasks such as classification and detection . Other works showed that prediction of color and image orientation , and invariance to data augmentation can provide useful self-supervised tasks. Beyond single images, works have leveraged video cues such as object tracking , frame ordering , and object boundary cues . Non-visual information can be equally powerful; information about camera motion , scene geometry , or sound (; can all serve as natural sources of supervision. While many of these tasks require predicting fixed quantities computed from the data, another class of contrastive methods formulate their objectives in the learned representations themselves. CPC is a contrastive representation learning method that maximizes the mutual information between spatially removed latent representations with InfoNCE (van den), a loss function based on Noise-Contrastive Estimation (Gutmann & Hyvärinen, 2010;). Two other methods have recently been proposed using the same loss function, but with different associated prediction tasks. Contrastive Multiview Coding maximizes the mutual information between representations of different views of the same observation. Augmented Multiscale Deep InfoMax is most similar to CPC in that it makes predictions across space, but differs in that it also predicts representations across layers in the model. In addition, AMDIM limits the receptive field of its representation, but does this by constraining the number of spatial convolutions in the network architecture rather than using image patches. A common alternative approach for improving data efficiency is label-propagation , where a classifier is trained on a subset of labeled data, then used to label parts of the unlabeled dataset, after which the process is repeated. This label-propagation can either be discrete (as in pseudo-labeling,) or continuous (as in entropy minimization,). The predictions of this classifier are often constrained to be smooth with respect to certain deformations, such as data-augmentation or adversarial perturbation . Representation learning and semi-supervised learning have been shown to be complementary and can be combined to great effect, which is why we focus solely on representation learning in this paper. When asking whether CPC enables data-efficient learning, we wish to use the best possible representative of this model class. Unfortunately, purely unsupervised metrics tell us little about downstream performance, and implementation details have been shown to matter enormously (; . Since many design choices (e.g. network architecture and datapreprocessing) have been previously evaluated using linear classification, we use this benchmark in section 4.1 to align the CPC model with best practices in representation learning and compare to published . In section 4.2 we select the best performing model from the previous section and assess whether it enables efficient classification. We also investigate to what extent the first, more common metric (linear classification accuracy) is predictive of efficient classification. Finally, in section 4.3 we investigate the generality of our through transfer learning to PASCAL-2007. The overarching principle behind our new model design is to increase the scale and efficiency of the encoder architecture while also maximizing the supervisory signal we obtain from each image. At the same time, it is important not to allow the network to solve the problem trivially, i.e., without learning semantics. To this end, we seek to remove low-level cues common across patches by augmenting individual patches independently, using standard stochastic data-processing techniques from supervised and self-supervised learning. We identify four axes for model capacity and task setup that could impact the model's performance. The first axis increases model capacity by increasing depth and width, while the second improves training efficiency capacity by introducing layer normalization. The third axis increases task complexity by making predictions in all four directions, and the fourth does so by performing more extensive patch-based augmentation. Model capacity. Recent work has shown that networks and more effective training improves self-supervised learning (; Motion Segmentation (MS) 27.6 48.3 Exemplar (Ex) 31.5 53.1 Relative Position (RP) 36.2 59.2 Colorization (Col) 39.6 62.5 Combination of MS + Ex + RP + Col -69.3 CPC v1 (van den 48.7 73.6 Rotation 55.4 -CMC 60.1 82.8 Local Aggregation 60.2 -BigBiGAN 61.3 81.9 AMDIM 68.1 -CPC v2 (ours) 65.9 86.6 other design choices. Interestingly, a larger architecture delivers larger improvements with more efficient training, more self-supervised losses, and more patch-based augmentations (Fig. 3, +5% Top-1 accuracy with original training scheme, +10% accuracy with new one). Layer normalization. Large architectures are more difficult to train efficiently. Early works on context prediction with patches used batch normalization to speed training. However, with CPC we find that batch normalization actually harms downstream performance of large models. We hypothesize that batch normalization allows large models to find a trivial solution to CPC: it introduces a dependency between patches (through the batch statistics) that can be exploited to bypass the constraints on the receptive field. We find that we can reclaim much of batch normalization's training efficiency using layer normalization , which leads to a small gain for the smaller architecture (+1% accuracy over equivalent architectures that use neither normalization) and a larger gain for the larger architecture (+2.5% accuracy). Prediction lengths and directions. Larger architectures also run a greater risk of overfitting. We address this by asking more from the network: specifically, whereas van den predicted each patch using only context from spatially beneath it, we repeatedly predict the patch using context from above, to the right, and to the left, ing in up to four times as many prediction tasks. Combining top-to-bottom with bottom-to-top helps both model architectures (+2% accuracy for both), but using all 4 spatial directions only benefits the larger model (an additional +1.5% for the larger model, -1% for the smaller), consistent with the idea that model capacity and amount of supervision must go hand-in-hand. We also hypothesized that prediction "length"-i.e. offset between the predicted patch and the aggregated context-might affect performance, as distant patches might lie on distinct objects, encouraging the network to memorize images. Indeed, limiting the range of the prediction length k to {2, 3} performed better than {2, . . ., 5} as was used originally (+1% for the larger model). Patch-based augmentation. If the network can solve CPC using low-level patterns (e.g. straight lines continuing between patches, chromatic aberration), it need not learn semantically meaningful content. Augmenting the low-level variability across patches can remove such low level cues. The original CPC model spatially jitters individual patches independently. We further this logic by adopting the'color dropping' method of , which randomly drops two of the three color channels in each patch, and find it to delivers systematic gains (+1% for the small model, +3% for the larger one). We also randomly flip patches horizontally, but find it only benefits the smaller model (+1%). Combined. Cumulatively, these fairly straightforward implementation changes lead to a substantial improvement to the original CPC model (65.9% Top-1 accuracy, a 17% improvement), making it competitive with recent approaches and outperforming prior methods (see table 1). Interestingly, if we train the same patch-based architecture from scratch in a fully supervised manner, we obtain 66.4% Top-1 accuracy (with batch normalization; 62.5% without), suggesting that CPC is now nearly saturating the architecture's representational power despite not using labels. These illustrate how architecture and data have an outsized impact on the linear classification performance of self-supervised representations, and are interesting to compare with with previous . For example, in AMDIM, different settings of data augmentation alone can in a nearly 10% absolute increase in performance on ImageNet linear classification. toWe now turn to our original question of whether CPC can enable data-efficient image recognition. We start by evaluating the performance of purely-supervised networks as the size of the labeled dataset D l varies from 1% to 100% of ImageNet, training separate classifiers on each subset. We found that a ResNet-152 to works best across all data-regimes (see Appendix). Despite our efforts to tune the supervised model for low-data classification (including network depth, regularization, and optimization parameters), the accuracy of the best model only reaches 44.1% Top-5 accuracy when trained on 1% of the dataset (compared to 93.9% when trained on the entire dataset, see Fig. 1, red). Orange, green, and blue dots correspond to CPC models making predictions in 1, 2, and 4 spatial directions respectively. Within a color group, different models correspond to other implementation details (e.g. layer norm and patch augmentation, and combinations thereof). Contrastive Predictive Coding. We now address our central question of whether CPC enables data-efficient learning. We follow the same paradigm as for the supervised baseline (training and evaluating a separate classifier for each size subset), stacking a neural network classifier on top of the CPC latents z = f θ (x) rather than the raw image pixels x (see section 2.2, efficient classification, and Appendix). This representation, which we selected for its improved linear classification performance (CPC v2 in Fig. 3), leads to a significant increase in data-efficiency compared to purely supervised networks (Fig. 1, He et al. (2016b) ). We find similar in all other data-regimes we considered (see Fig. 1). How important are the model specifications described in Section 4.1 for low-data classification? We hypothesized that predictable representations might enable data-efficient classification, and therefore expect that increasing the amount of'predictability' in the representation should also increase its ability to learn from small amounts of data. Fig. 4 shows evidence for this by ablating model parameters and comparing linear classification performance against low-data classification. Consistent with our hypothesis, increasing the number of spatial directions in the CPC prediction task (which increased linear classification performance) systematically increases low-data classification performance (Fig. 4, left, different color groups). As a control, we asked if all modifications that improve linear classification also improve low-data classification. We did not find evidence in favor of this: improvements in linear classification as a of changing other model Methods using label-propagation: Pseudolabeling 51.6 82.4 VAT + Entropy Minimization parameters (patch-based data-augmentation, layer normalization, and combinations thereof) seem uncorrelated to performance in other tasks (Fig. 4, left, within green group: R 2 = 0.17, p = 0.36). Different architectural specifications also produced different changes in both tasks (Fig. 4, right). Whereas increasing the depth of the encoding network greatly improves both metrics, increasing the network width (and therefore the number of features used for linear classification) only improves linear classification accuracy. Other unsupervised representations. How well does the CPC representation compare to other representations that have been learned in an unsupervised manner? If predictable representations are uniquely suited for efficient classification, we would expect other methods within this family to perform similarly, and other model classes less so. Table 2 compares our best model with other works on efficient recognition. We consider three objectives from different model classes: self-supervised learning with rotation prediction, large-scale adversarial feature learning , and another contrastive prediction objective . evaluate the low-data classification performance of representations learned with rotation prediction using a similar paradigm and architecture (ResNet-152), hence we report their directly. Given 1% of ImageNet, their method achieves 57.5% Top-5 accuracy, consistently with the reduced accuracy of a linear classifier (55.4% vs 65.9% for CPC) 1. Because BigBiGAN and AMDIM achieve stronger linear classification accuracy than rotation prediction (61.3% and 68.1% Top-1 accuracy, respectively), we might expect better performance on efficient classification as well. Since their authors do not report on efficient classification we evaluated these representations using the same paradigm we used for evaluating CPC, stacking a ResNet classifier on top of the 7×7×8192 latents of the BigBiGAN and the 7×7×2560 grid of feature vectors of AMDIM. We found fine-tuned representations to yield only marginal gains over fixed ones (72.9% compared to 72.3% Top-5 accuracy given 1% of labels), hence for simplicity we evaluate BigBiGAN and AMDIM on this task while keeping them fixed. We re-tune the hyper-parameters of the classifier (including optimization, regularization, etc.) for each of these representations separately. Although these methods achieve similar performance in terms of linear classification, we find them to achieve very different in efficient classification. Given 1% of ImageNet, classifiers trained on top of BigBiGAN achieve 55.2% Top-5 accuracy, similarly to rotation prediction (57.5%), despite its increased linear classification accuracy (+6% relative to rotation prediction). In contrast, AMDIM (which also belongs to the family of contrastive prediction methods) achieves 67.4% on this same task. Again, its increased linear classification accuracy did not entail an increase in data-efficiency. Nevertheless, in line with our initial hypothesis, we find that contrastive prediction methods such as CPC surpass other approaches in our efficient classification experiments, and that linear classification performance is not perfectly correlated with these . Other semi-supervised techniques A separate class of methods for low-data classification attempts to propagate the knowledge extracted from the subset of labeled examples to unlabeled examples while being invariant to augmentation or other perturbations. These methods generally depend on the quality of the classifier's predictions, and as such tend to fare well when given intermediate amounts of data. Although not sufficient in themselves (Unsupervised Data Augmentation , Virtual Adversarial Training and entropy minimization , and pseudo-labeling achieve 85.8%, 83.4%, and 82.4% Top-5 accuracy with 10% of labels, compared to our 89.4%) when combined with representation learning (e.g. rotation prediction they can provide considerable gains (91.2% Top-5 accuracy). It is therefore surprising that CPC representations alone can enable accuracy that is comparable to that of these methods, and investigating to what extent they can be combined would be an interesting topic of future work. We next investigate transfer performance on object detection on the PASCAL-2007 dataset, which reflects the practical scenario where a representation must be trained on a dataset with different statistics than the dataset of interest. This dataset also tests the efficiency of the representation as it only contains 5011 labeled images to train from. In this setting, we replaced the neural network classifier h ψ used previously with a Faster-RCNN image detection architecture, and use the pre-trained feature extractor on ImageNet. As before, we first trained the Faster-RCNN model while keeping the feature extractor fixed, then fine-tuned the entire model end-to-end. Table 3 displays our compared to other methods. Most competing methods, which optimize a single unsupervised objective on ImageNet before fine-tuning on PASCAL detection, attain around 65% mean average precision. Leveraging larger unlabeled datasets increases their performance up to 67.8% . Combining multiple forms of self-supervision enables them to reach 70.5% . The proposed method, which learns only from ImageNet data using a single unsupervised objective, reaches 70.6% when equipped with a ResNet-101 feature extractor f θ (as for most competing methods but not all (; . Equipped with the more powerful ResNet-161 feature extractor f θ, our method reaches 72.7%. Importantly, this is only 2% short of the performance attained by purely supervised transfer learning, which we obtain by using all ImageNet labels before transferring to PASCAL. We asked whether CPC could enable data-efficient image recognition, and found that it indeed greatly improves the accuracy of classifiers and object detectors when given small amounts of labeled data. Surprisingly, CPC even improves given ImageNet-scale labels. Our show that there is still room for improvement using relatively straightforward changes such as augmentation, optimization, and network architecture. Furthermore, we found that the standard method for evaluating unsupervised representations-linear classification-is only partially predictive of efficient recognition performance, suggesting that further research should focus on efficient recognition as a standalone benchmark. Overall, these open the door toward research on problems where data is naturally limited, e.g. medical imaging or robotics. image detection accuracy to other transfer methods. The supervised baseline learns from the entire labeled ImageNet dataset and fine-tunes for PASCAL detection. The second class of methods learns from the same unlabeled images before transferring. All of these methods pre-train on the ImageNet dataset, except for DeeperCluster which learns from the larger, but uncurated, YFCC100M dataset . All are reported in terms of mean average precision (mAP). † denotes methods implemented in this work. Transfer from labeled data: Supervised -ResNet-152 74.7 Transfer from unlabeled data: Exemplar (Ex) 60.9 Motion Segmentation (MS) 61.1 Colorization (Col) 65.5 Relative Position (RP) 66.8 Combination of Ex + MS + Col + RP 70.5 Instance Discrimination 65.4 Deep Cluster 65.9 Deeper Cluster 67.8 Local Aggregation 69.1 † Faster-RCNN trained on CPC v2 (ResNet-101, fine-tuned) 70.6 † Faster-RCNN trained on CPC v2 72.7 Furthermore, images are far from the only domain where unsupervised representation learning is important: for example, unsupervised learning is already a critical step in language , and shows promise in domains like audio (van den ; ;, video , and robotic manipulation ). Currently much self-supervised work builds upon tasks tailored for a specific domain (often images), which may not be easily adapted to other domains. Contrastive prediction methods, including the techniques suggested in this paper, are task agnostic and could therefore serve as a unifying framework for integrating these tasks and modalities. This generality is particularly useful given that many realworld environments are inherently multimodal, e.g. robotic environments which can have vision, audio, touch, proprioception, action, and more over long temporal sequences. Given the importance of increasing the amounts of self-supervision (via additional directions of prediction), integrating these modalities and tasks could lead to unsupervised representations which rival the efficiency and effectiveness of biological ones. A.1 ADDITIONAL For completeness, we provide pseudo-code for the main calculations involved in the InfoNCE objective, loosely modeled after Tensorflow operations. We suppose we have just calculated a set of latents z i,j = f θ (x i,j) for i, j ∈ {1, . . ., 7}, each one being e.g. a 4096-dimensional vector. Assuming we do so for a batch of B images {x}, the set of latents is a tensor of size B × 7 × 7 × 4096. shaped input image, we end up with a grid of 6x6 features (each of which is obtained from our ResNet-161 architecture). This gives us a tensor for the image. We then use a Batch-Normalization layer to normalize the features (without scale parameter) followed by a 1x1 convolution mapping each feature in the grid to the 1000 logits for ImageNet classification. We then spatially-mean-pool these logits to end up with the final log probabilities for the linear classification. • We use the Inception preprocessing to extract 240x240 crops from the raw image. The image is divided into subcrops as per CPC data-preprocessing used for CPC pre-training. • Optimization details: We use Adam Optimizer with a learning rate of 5e-4. We train the model on a batch size of 512 images with 32 images per core spread over 16 workers. In order to find the best model within this class, we vary the following hyperparameters: • Model architecture: We investigate using ResNet-50, ResNet-101, and ResNet-152 model architectures, all of them using the'v2' variant (b), and find larger architecture to perform better, even when given smaller amounts of data. We insert a DropOut layer before the final linear classification layer . • Data pre-processing: We use the Inception pre-processing pipeline . • Optimization details: We vary the learning rate in {0.05, 0.1, 0.2}, the weight decay logarithmically from 10 −5 to 10 −2, the DropOut linearly from 0 to 1, and the batch size per worker in {16, 32}. We chose the best performing model for each training subset D l of labeled ImageNet (using a separate validation set), and report its accuracy on the test set (i.e. the publicly available ILSVRC-2012 validation set).

represent the tldr for finding the corresponding scientific article

Unsupervised representations learned with Contrastive Predictive Coding enable data-efficient image classification.

We leverage deterministic autoencoders as generative models by proposing mixing functions which combine hidden states from pairs of images. These mixes are made to look realistic through an adversarial framework.


Question: Represent the article for finding a single-sentence TLDR

Question: We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to identify layers and weights which reduce the error efficiently. Sparse momentum redistributes pruned weights across layers according to the mean momentum magnitude of each layer. Within a layer, sparse momentum grows weights according to the momentum magnitude of zero-valued weights. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet, decreasing the mean error by a relative 8%, 15%, and 6% compared to other sparse algorithms. Furthermore, we show that sparse momentum reliably reproduces dense performance levels while providing up to 5.61x faster training. In our analysis, ablations show that the benefits of momentum redistribution and growth increase with the depth and size of the network. Current state-of-the-art neural networks need extensive computational resources to be trained and can have capacities of close to one billion connections between neurons (; ;). One solution that nature found to improve neural network scaling is to use sparsity: the more neurons a brain has, the fewer connections neurons make with each other . Similarly, for deep neural networks, it has been shown that sparse weight configurations exist which train faster and achieve the same errors as dense networks. However, currently, these sparse configurations are found by starting from a dense network, which is pruned and re-trained repeatedly -an expensive procedure. In this work, we demonstrate the possibility of training sparse networks that rival the performance of their dense counterparts with a single training run -no re-training is required. We start with random initializations and maintain sparse weights throughout training while also speeding up the overall training time. We achieve this by developing sparse momentum, an algorithm which uses the exponentially smoothed gradient of network weights (momentum) as a measure of persistent errors to identify which layers are most efficient at reducing the error and which missing connections between neurons would reduce the error the most. Sparse momentum follows a cycle of pruning weights with small magnitude, redistributing weights across layers according to the mean momentum magnitude of existing weights, and growing new weights to fill in missing connections which have the highest momentum magnitude. We compare the performance of sparse momentum to compression algorithms and recent methods that maintain sparse weights throughout training. We demonstrate state-of-the-art sparse performance on MNIST, CIFAR-10, and ImageNet-1k. For CIFAR-10, we determine the percentage of weights needed to reach dense performance levels and find that AlexNet, VGG16, and Wide Residual Networks need between 35-50%, 5-10%, and 20-30% weights to reach dense performance levels. We also estimate the overall speedups of training our sparse convolutional networks to dense performance levels on CIFAR-10 for optimal sparse convolution algorithms and naive dense convolution algorithms compared to dense baselines. For sparse convolution, we estimate speedups between 2.74x and 5.61x and for dense convolution speedups between 1.07x and 1.36x. In your analysis, ablations demonstrate that the momentum redistribution and growth components are increasingly important as networks get deeper and larger in size -both are critical for good ImageNet performance. From Dense to Sparse Neural Networks: Work that focuses on creating sparse from dense neural networks has an extensive history. Earlier work focused on pruning via second-order derivatives (; ;) and heuristics which ensure efficient training of networks after pruning (; ;). Recent work is often motivated by the memory and computational benefits of sparse models that enable the deployment of deep neural networks on mobile and low-energy devices. A very influential paradigm has been the iterative train-dense, prune, re-train cycle introduced by. Extensions to this work include: Compressing recurrent neural networks and other models (; ;), continuous pruning and re-training , joint loss/pruning-cost optimization (Carreira-Perpinán and), layer-by-layer pruning , fast-switching growth-pruning cycles , and soft weight-sharing. These approaches often involve re-training phases which increase the training time. However, since the main goal of this line of work is a compressed model for mobile devices, it is desirable but not an important main goal to reduce the run-time of these procedures. This is contrary to our motivation. Despite the difference in motivation, we include many of these dense-to-sparse compression methods in our comparisons. Other compression algorithms include L 0 regularization , and Bayesian methods . For further details, see the survey of. Interpretation and Analysis of Sparse Neural Networks: show that "winning lottery tickets" exist for deep neural networks -sparse initializations which reach similar predictive performance as dense networks and train just as fast. However, finding these winning lottery tickets is computationally expensive and involves multiple prune and re-train cycles starting from a dense network. Followup work concentrated on finding these configurations faster ). In contrast, we reach dense performance levels with a sparse network from random initialization with a single training run while accelerating training. Sparse Neural Networks Throughout Training: Methods that maintain sparse weights throughout training through a prune-redistribute-regrowth cycle are most closely related to our work. introduce DEEP-R, which takes a Bayesian perspective and performs sampling for prune and regrowth decisions -sampling sparse network configurations from a posterior. While theoretically rigorous, this approach is computationally expensive and challenging to apply to large networks and datasets. Sparse evolutionary training (SET) simplifies prune-regrowth cycles by using heuristics: prune the smallest and most negative weights, grow new weights in random locations. Unlike our work, where many convolutional channels are empty and can be excluded from computation, growing weights randomly fills most convolutional channels and makes it challenging to harness computational speedups during training without specialized sparse algorithms. SET also does not include the cross-layer redistribution of weights which we find to be critical for good performance, as shown in our ablation study. The most closely related work to ours is Dynamic Sparse Reparameterization (DSR) by , which includes the full prune-redistribute-regrowth cycle. However, DSR requires some specific layers to be dense. Our method works in a fully sparse setting and is thus more generally applicable. More distantly related is Single-shot Network Pruning (SNIP) , which aims to find the best sparse network from a single pruning decision. The goal of SNIP is simplicity, while our goal is maximizing predictive and run-time performance. In our experiments, we compare against all four methods: DEEP-R, SET, DSR, and SNIP. We define sparse learning to be the training of deep neural networks which maintain sparsity throughout training while matching the predictive performance of dense neural networks. To achieve this, intuitively, we want to find the weights that reduce the error most effectively. This is challenging since most deep neural network can hold trillions of different combinations of sparse weights. Additionally, during training, as feature hierarchies are learned, efficient weights might change gradually from shallow to deep layers. How can we find good sparse configurations? In this work, we follow a divide-and-conquer strategy that is guided by computationally efficient heuristics. We divide sparse learning into the following sub-problems which can be tackled independently: pruning weights, redistribution of weights across layers, and regrowing weights, as defined in more detail below. Figure 1: Sparse Momentum is applied at the end of each epoch: take the magnitude of the exponentially smoothed gradient (momentum) of each layer and normalize to 1; for each layer, remove p = 20% of the weights with the smallest magnitude; across layers, redistribute the removed weights by adding weights to each layer proportionate to the momentum of each layer; within a layer, add weights starting from those with the largest momentum magnitude. Decay p. We use the mean magnitude of momentum M i of existing weights W i in each layer i to estimate how efficient the average weight in each layer is at reducing the overall error. Intuitively, we want to take weights from less efficient layers and redistribute them to weight-efficient layers. The sparse momentum algorithm is depicted in Figure 1. In this section, we first describe the intuition behind sparse momentum and then present a more detailed description of the algorithm. The gradient of the error with respect to a weight ∂E ∂W yields the directions which reduce the error at the highest rate. However, if we use stochastic gradient descent, most weights of ∂E ∂W oscillate between small/large and negative/positive gradients with each mini-batch -a good change for one mini-batch might be a bad change for another. We can reduce oscillations if we take the average gradient over time, thereby finding weights which reduce the error consistently. However, we want to value recent gradients, which are closer to the local minimum, more highly than the distant past. This can be achieved by exponentially smoothing ∂E ∂W -the momentum M i: where α is a smoothing factor, M i is the momentum for the weight W i in layer i; M i is initialized at t = 0 with 0. Momentum is efficient at accelerating the optimization of deep neural networks by identifying weights which reduce the error consistently. Similarly, the aggregated momentum of weights in each layer should reflect how good each layer is at reducing the error consistently. Additionally, the momentum of zero-valued weights -equivalent to missing weights in sparse networks -can be used to estimate how quickly the error would change if these weights would be included in a sparse network. The details of the full training procedure of our algorithm are shown in Algorithm 1. See Algorithm 2 in the Appendix for a more detailed, source-code-like description of sparse momentum. Algorithm 1: Sparse momentum algorithm. Data: Layer i to k with: Before training, we initialize the network with a certain sparsity s: we initialize the network as usual and then remove a fraction of s weights for each layer. We train the network normally and mask the weights after each gradient update to enforce sparsity. We apply sparse momentum after each epoch. We can break the sparse momentum into three major parts: (a) redistribution of weights, (b) pruning weights, (c) regrowing weights. In step (a), we we take the mean of the element-wise momentum magnitude m i that belongs to all nonzero weights for each layer i and normalize the value by the total momentum magnitude of all layers k i=0 m i. The ing proportion is the momentum magnitude contribution for each layer. The number of weights to be regrow in each layer is the total number of removed weights multiplied by each layers momentum contribution: Regrow i = Total Removed · m i. In step (b), we prune a proportion of p (prune rate) of the weights with the lowest magnitude for each layer. In step (c), we regrow weights by enabling the gradient flow of zero-valued (missing) weights which have the largest momentum magnitude. Additionally, there are two edge-cases which we did not include in Algorithm 1 for clarity. If we allocate more weights to be regrown than is possible for a specific layer, for example regrowing 100 weights for a layer of maximum 10 weights, we redistribute the excess number of weights equally among all other layers. For some layers, our algorithm will converge in that the average weight in layer i has much larger momentum magnitude than weights in other layers, but at the same time, this layer is dense and cannot grow further. We do not want to prune weights from such important layers. Thus, for these layers, we reduce the prune rate p i proportional to the sparsity: p i = min(p, sparsity i). After each epoch, we decay the prune rate in Algorithm 1 in the same way learning rates are decayed. We use a cosine decay schedule that anneals the prune rate to zero on the last epoch. See Appendix A.1 for an analysis on how decay schedule and starting prune rate affects training. For comparison, we follow three different experimental settings, one from and two settings follow: For MNIST , we use a batch size of 100, decay the learning rate by a factor of 0.1 every 25000 mini-batches. For CIFAR-10 , we use standard data augmentations (horizontal flip, and random crop with reflective padding), a batch size of 128, and decay the learning rate every 30000 mini-batches. We train for 100 and 250 epochs on MNIST and CIFAR-10, use a learning rate of 0.1, stochastic gradient descent with Nesterov momentum of α = 0.9, and we use a weight decay of 0.0005. We use a fixed 10% of the training data as the validation set and train on the remaining 90%. We evaluate the test set performance of our models on the last epoch. For all experiments on MNIST and CIFAR-10, we report the standard errors. Our sample size is generally between 10 and 12 experiments per method/architecture/sparsity level with different random seeds for each experiment. We use the modified network architectures of AlexNet, VGG16, and LeNet-5 as introduced by. We consider two different variations of the experimental setup of for ImageNet and CIFAR-10. The first follows their procedure closely, in that we run the networks in a partially dense setting where the first convolutional layer and downsampling convolutional layers are dense. Additionally, for CIFAR-10 the last fully connected layer is dense. In the second setting, we compare in a fully sparse setting -no layer is dense at the beginning of training. For the fully sparse setting we increase overall number of weights according to the extra parameters in the dense layers and distribute them equally among the network. The parameters in the dense layers make up 5.63% weights of the ResNet-50 network. We refer to these two settings as the partially dense and fully sparse settings. On ImageNet , we use ResNet-50 with a stride of 2 for the 3x3 convolution in the bottleneck layers. We use a batch size of 256, input size of 224, momentum of α = 0.9, and weight decay of 10 −4. We train for 100 epochs and report validation set performance after the last epoch. We report for the fully sparse and the partially dense setting. For all experiments, we keep biases and batch normalization weights dense. We tuned the prune rate p and momentum rate α searching the parameter space {0.2, 0.3, 0.4, 0.5, 0.6, 0.7} and {0.5, 0.6, 0.7, 0.8, 0.9, 0.95, 0.99} on MNIST and CIFAR-10 and found that p = 0.2 and α = 0.9 work well for most architectures. We use this prune and momentum rate throughout all experiments. ImageNet experiments were run on 4x RTX 2080 Ti and all other experiments on individual GPUs. Our software builds on PyTorch and is a wrapper for PyTorch neural networks with a modular architecture for growth, redistribution, and pruning algorithms. Currently, no GPUaccelerated libraries that utilize sparse tensors exist, and as such we use masked weights to simulate sparse neural networks. Using our software, any PyTorch neural network can be adapted to be a sparse momentum network with less than 10 lines of code. We will open-source our software along with trained models and individual experimental . Figure 2 and Table 1 show a comparison with model compression methods. On MNIST, sparse momentum is the only method that provides consistent strong performance across both LeNet 300-100 and LeNet-5 Caffe models. Soft-weight sharing and Layer-wise Brain Damage are competitive with sparse momentum for one model, but underperforms for the other model. For 1-2% of weights, variational dropout is more effective -but this method also uses dropout for further regularization while we only use weight decay. We can see that sparse momentum achieves equal performance to the LeNet-5 Caffe dense baseline with 8% weights. On CIFAR-10 in Table 1, we can see that sparse momentum outperforms Single-shot Network Pruning (SNIP) for all models and can achieve the same performance level as a dense model for VGG16-D with just 5% of weights. Weights (%) Table 2 show comparisons of sparse learning methods on MNIST and CIFAR that follows the experimental procedure of where some selected layers are dense. For LeNet 300-100 on MNIST, we can see that sparse momentum outperforms all other methods. For CIFAR-10, sparse momentum is better than dynamic sparse in 4 out of 5 cases. However, in general, the confidence intervals for most methods overlap -this particular setup for CIFAR-10 with specifically selected dense layers seems to be too easy to determine difference in performance between methods and we do not recommend this setup for future work. Table 2 shows that sparse momentum outperforms all other methods on ImageNet (ILSVRC2012) for the Top-1 accuracy measure. Dynamic sparse is better for the Top-5 accuracy with 20% weights. In the fully sparse setting, sparse momentum remains competitive and seems to find a weight distribution which works equally well for the 10% weights case. For 20% weights, the performance decreases slightly. Figure 3: Test set accuracy with 95% confidence intervals on MNIST and CIFAR at varying sparsity levels for LeNet 300-100 and WRN 28-2. We analyzed how many weights are needed to achieve dense performance for our networks on CIFAR-10 and how much faster would we able to train such a sparse network compared to a dense one. We do this analysis by increasing the number of weights by 5% until the sparse network trained with sparse momentum reaches a performance level that overlaps with a 95% confidence interval of the dense performance. We then measure the speedup of the model. For each network-density combination we perform ten training runs with different random seeds to calculate the mean test error and its standard error. To estimated the speedups that could be obtained using sparse momentum for these dense networks we follow two approaches: Theoretical speedups for sparse convolution algorithms which are proportional to reductions in FLOPS and practical speedups using dense convolutional algorithms which are proportional to empty convolutional channels. For our sparse convolution estimates, we calculate the FLOPS saved for each convolution operation throughout training as well as the runtime for each convolution. To receive the maximum speedups for sparse convolution, we then scale the runtime for each convolution operation by the FLOPS saved. While a fast sparse convolution algorithm for coarse block structures exist for GPUs , optimal sparse convolution algorithms for fine-grained patterns do not and need to be developed to enable these speedups. The second method measures practical speedups that can be obtained with naive, dense convolution algorithms which are available today. Dense convolution is unsuitable for the training of sparse networks but we include this measurement to highlight the algorithmic gap that exists to efficiently train sparse networks. For dense convolution algorithms, we estimate speedups as follows: If a convolutional channel consists entirely of zero-valued weights we can remove these channels from the computation without changing the outputs and obtain speedups. To receive the speedups for dense convolution we scale each convolution operation by the proportion of empty channels. Using these measures, we estimated the speedups for our models on CIFAR-10. The ing speedups and dense performance levels can be seen in Table 3. We see that VGG16 networks can achieve dense performance with relatively few weights while AlexNet requires the most weights. Wide Residual Networks need an intermediate level of weights. Despite the large number of weights for AlexNet, sparse momentum still yields large speedups around 3.0x for sparse convolution. Sparse convolution speedups are particularly pronounced for Wide Residual Networks (WRN) with speedups as high as 5.61x. Dense convolution speedups are much lower and are mostly dependent on width, with wider networks receiving larger speedups. These highlight the importance to develop optimized algorithms for sparse convolution. Beyond speedups, we also measured the overhead of our sparse momentum procedure to be equivalent of a slowdown to 0.973x±0.029x compared to a dense baseline. growth of weights. To understand the performance contribution of these components, we perform ablations on CIFAR-10 for VGG16-D with 5% weights, MNIST for LeNet 300-100 and LeNet-5 Caffe with 5% weights, and ImageNet for ResNet-50 with 10% weights in the fully sparse setting. The can be seen in Table 4. Redistribution: Redistributing weights according to the momentum magnitude becomes increasingly important the larger a network is as can be seen from the steady increases in error from the small LeNet 300-100 to the large ResNet-50 when no momentum redistribution is used. Increased test error is particularly pronounced for ImageNet where the Top-1 error increases by 3.42% to 9.71% if no redistribution is used. Momentum growth: Momentum growth improves performance over random growth by a large margin for ResNet-50 on ImageNet, but for smaller networks the combination of redistribution and random growth seems to be sufficient to find good weights. Random growth without redistribution, however, cannot find good weights. These suggest that with increasing network size a random search strategy becomes inefficient and smarter growth algorithms are required for good performance. We presented our sparse learning algorithm, sparse momentum, which uses the mean magnitude of momentum to grow and redistribute weights. We showed that sparse momentum outperforms other sparse algorithms on MNIST, CIFAR-10, and ImageNet. Additionally, sparse momentum can rival dense neural network performance while accelerating training. Our analysis of speedups highlights the need for research into specialized sparse convolution and sparse matrix multiplication algorithms to enable the benefits of sparse networks. A.1 SENSITIVITY ANALYSIS Sparse momentum depends on two hyperparameters: Prune rate and momentum. In this section, we study the sensitivity of the accuracy of our models as we vary the prune rate and momentum. Since momentum parameter has an additional effect on the optimization procedure, we run control experiments for fully dense networks thus disentangling the difference in accuracy accounted by our sparse momentum procedure. We run experiments for VGG-D and AlexNet-s with 5% and 10% weights on CIFAR-10. Results can be seen in Figure 4. We see that sparse momentum is highly robust to the choice of prune rate with barely deviating when the prune rate is in the interval between 0.2 to 0.4. However, we can see a gradual linear trend that indicates that smaller prune rates work slightly better than larger ones. Cosine and linear prune rate annealing schedules do equally well. For momentum, confidence intervals for values between 0.7 and 0.9 overlap indicating that our procedure is robust to the choice of the momentum parameter. Sparse momentum is more sensitive to low momentum values (≤0.6) while it is less sensitive for large momentum values (0.95) compared to a dense control. Additionally, we test the null hypothesis that sparse momentum is equally sensitive to deviations from a momentum parameter value of 0.9 as a dense control. The normality assumption was violated and data transformations did not help. Thus we use the non-parametric Wilcoxon Signed-rank Test. We find no evidence that sparse momentum is more sensitive to the momentum parameter than a dense control, W = 22.0, p = 0.58. Overall, we conclude that sparse momentum is highly robust to deviations of the pruning schedule and the momentum and prune rate parameters. In this section, we look at the features of dense and sparse networks and how specialized these features are for certain classes. We test difference between sparse and dense network features statistically. For feature visualization, it is common to backpropagate activity to the inputs to be able to visualize what these activities represent (; ;). However, in our case, we are more interested in the overall distribution of features for each layer within our network, and as such we want to look at the magnitude of the activity in a channel since -unlike feature visualization -we are not just interested in feature detectors but also discriminators. For example, a face detector would induce positive activity for a'person' class but might produce negative activity for a'mushroom' class. Both kinds of activity are useful. With this reasoning, we develop the following convolutional channel-activation analysis: pass the entire training set through the network and aggregate the magnitude of the activation in each convolutional channel separately for each class; normalize across classes to receive for each channel the proportion of activation which is due to each class; look at the maximum proportion of each channel as a measure of class specialization: a maximum proportion of 1/N c where N c is the number of classes indicates that the channel is equally active for all classes in the training set. The higher the proportion deviates from this value, the more is a channel specialized for a particular class. We obtain for AlexNet-s, VGG16-D, and WRN 28-2 on CIFAR-10 and use as many weights as needed to reach dense performance levels. We then test the null hypothesis, that there are no differences in class specialization between features from sparse networks and dense networks. Equal variance assumptions was violated for VGG-D and normality was violated for WRN-28-2, while all assumptions hold for AlexNet-s. For consistency reasons we perform non-parametric Kruskal-Wallis one-way analysis of variance tests for all networks. For AlexNet-s, we find some evidence that features of sparse networks have lower class specialization compared to dense networks χ 2 = 4.43, p = 0.035, for VGG-D and WRN-28-2 we find strong evidence that features of sparse networks have lower class specialization than dense networks χ 2 = 28.1, p < 0.001, χ 2 = 36.2, p < 0.001. Thus we reject the null hypothesis. These increase our confidence that sparse networks learn features which have lower class specialization than dense networks. Plots of the distributions of sparse vs. dense features for AlexNet-s, VGG16-D, and WRN 28-2 on CIFAR-10 in Figure 5. These plots were selected to highlight the difference in distribution in the first layers and last layers of each network. We see the convolutional channels in sparse networks have lower class-specialization indicating they learn features which are useful for a broader range of classes compared to dense networks. This trend intensifies with depth. Overall, we conclude that sparse networks might be able to rival dense networks by learning more general features that have lower class specialization. C FURTHER We also tried a better version of the ResNet-50 in the fully sparse setting for which we use a cosine learning rate schedule, label smoothing of 0.9, and we warmup the learning rate. The can be seen in Table 5. A class-specialization of 0.5 indicates that 50% of the overall activity comes from a single class.

Represent the TLDR for finding the corresponding scientific article!

Redistributing and growing weights according to the momentum magnitude enables the training of sparse networks from random initializations that can reach dense performance levels with 5% to 50% weights while accelerating training by up to 5.6x.

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.