A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
Patryk Chrabaszcz Ilya Loshchilov Frank Hutter
cs.CV cs.LG






Max K-armed bandit: 
 On the ExtremeHunter algorithm and beyond
    Mastane Achab1 Stephan Clémençon1 Aurélien Garivier2 Anne Sabourin1 Claire Vernade1
    December 30, 2023
=======================================================================================




The  original ImageNet dataset is a popular large-scale benchmark for training Deep Neural Networks. Since the cost of performing experiments (e.g, algorithm design, architecture search, and hyperparameter tuning) on the original dataset might be prohibitive, we propose to consider a downsampled version of ImageNet. In contrast to the CIFAR datasets and earlier downsampled versions of ImageNet, our proposed ImageNet32x32 (and its variants ImageNet64x64 and ImageNet16x16) contains exactly the same number of classes and images as ImageNet, with the only difference that the images are downsampled to 32×32 pixels per image (64×64 and 16×16 pixels for the variants, respectively). Experiments on these downsampled variants are dramatically faster than on the original ImageNet and the characteristics of the downsampled datasets with respect to optimal hyperparameters appear to remain similar. The proposed datasets and scripts to reproduce our results are available at 
<http://image-net.org/download-images> and  <https://github.com/PatrykChrabaszcz/Imagenet32_Scripts>




§ INTRODUCTION


Deep learning research has been substantially facilitated by the availability of realistic and accessible benchmark datasets, such as CIFAR-10 and CIFAR-100 <cit.> (and MNIST <cit.> in the 1990s). With the progress of machine learning, simple datasets lose some of their relevance, and more complex datasets/tasks become more important. While good results can be achieved on more complex datasets, such as ImageNet <cit.>, this incurs a large computational burden, making it intractable to achieve state-of-the-art performance without massive compute resources
(training a strong ImageNet model typically requires several GPU months). 

Due to this computational expense of running experiments on the original ImageNet dataset we propose to explore cheaper alternatives that preserve the dataset's complexity. 
In order to check the scalability of new methods, neural architectures and hyperparameters associated with them, one might be interested in a downscaled version of ImageNet which allows for cheaper experimentation. Moreover, a lower resolution of the images would make the classification task much more difficult and would thus postpone the saturation of benchmarking results currently observed on CIFAR-10, e.g., 3% error obtained by  <cit.> compared to roughly 6% obtained by a trained human <cit.>. 

To address this issue, we provide downsampled variants of the original ImageNet dataset and analyze results on them w.r.t. different hyperparameter settings and network sizes. We obtain surprisingly strong classification results on our downsampled variants and find qualitative results to be very similar across downsampling sizes. This suggests that these downsampled datasets are useful for facilitating cheap experimentation.

The basic contributions of this report are as follows:


  * We make available downsampled versions of ImageNet (64× 64, 32× 32, and 16×16 pixels) to facilitate fast experimentation with different network architectures, training algorithms, and hyperparameters.

  * We show that different downsampling techniques yield similar results, except for a nearest neighbor approach, which performed worse in all our experiments.

  * Using Wide ResNets <cit.>, we obtain surprisingly good performance, matching the baseline by the pioneering AlexNet <cit.> (18.2% top-5 error) while using ImageNet32x32 (whose images have roughly 50× less pixels per image than the original ones). 

  * We show that the range of optimal learning rates does not change much across ImageNet16x16, ImageNet32x32, and ImageNet64x64, as well as across different network widths. This could be exploited by multi-fidelity methods for architecture and hyperparameter search <cit.>.




§ DOWNSAMPLING IMAGENET


The original ImageNet dataset consists of images released as a part of the ILSVRC-2012 classification dataset <cit.>. Each image belongs to one of 1000 object classes, with the number of training images per class varying from 732 to 1300; there are 50 validation images per class. The size of the original images varies; therefore, a preprocessing step is usually applied to scale and crop images to the size of 224 × 224 pixels.

We are aware of two datasets that contain low resolution images derived from the ImageNet dataset: 


  * Downsampled ImageNet <cit.>, like our datasets, contains all images in ImageNet, but since it was constructed for unsupervised learning, it does not provide the actual image labels and can therefore not be used for supervised learning.

  * TinyImageNet (available at <https://tiny-imagenet.herokuapp.com/>)
contains a subset of 200 classes with 500 images per class.


<cit.> suggested to use 128x128 pixels  ImageNet images to evaluate various deep learning  techniques, but their dataset is not available. 

We downsample / resize the original images to smaller images of 32x32 pixels to form ImageNet32x32, to images of 64x64 pixels to form ImageNet64x64 and to images of 16x16 pixels to form ImageNet16x16. In contrast to TinyImageNet, we do not reduce the number of classes and number of images. All images are shuffled and then divided into 10 different files so that each file is expected to have images from all classes. The validation data is stored in a separate file, both the training and validation data points are labeled (e.g., indexed starting from 1) according to the mapping file of the  ImageNet devkit. Each file contains images, labels and the mean image computed over the whole training set. We keep the same format of files as the one that is commonly used for the CIFAR datasets. ImageNet16x16, ImageNet32x32 and ImageNet64x64 take 1 GB, 4 GB  and 16 GB of disk space, respectively.




We consider 6 different downsampling techniques available in the Pillow library[Pillow version 4.1 available at <https://python-pillow.org>]: lanczos, nearest, bilinear, bicubic, hamming, box (see Figure <ref>). In order to check the quality of the downsampled images we use them to train Wide Residual Networks (WRNs) by <cit.>, expecting that better validation errors will tend to be achieved with downsampling techniques that lose less information.




§ EXPERIMENTAL SETUP





We train Wide Residual Networks WRN-N-k by <cit.>, where N is the number of layers and k is a multiplicative factor for the number of filters, with k=1 corresponding to 16 filters in the first residual block; increasing k makes the network wider. We use Stochastic Gradient Descent with momentum factor 0.9, drop the learning rate by a factor of 5.0 every 10 epochs, and train up to a total budget of 40 epochs. Throughout, we show validation error rates obtained after training for 31 epochs (right after the last drop of the learning rate). 

Our experiments on ImageNet32x32 employ the original WRNs designed for the CIFAR datasets with 32 × 32 pixels per image. To adapt WRNs for images with 64 × 64 pixels per image as used in ImageNet64x64, we add an additional stack of residual blocks to reduce the spatial resolution of the last feature map from 16 × 16 to 8 × 8 and thus double the number of features. Analogously, for ImageNet16x16, we remove the last stack of residual blocks. For data augmentation, we  flip images horizontally and concatenate them to the original images, effectively doubling the number of images per epoch. We also use random image shifts (up to 4 pixels horizontally and vertically).




§ RESULTS





Does the downsampling technique matter? 
We evaluated the six downsampling techniques described in Section <ref> using a small WRN-28-2 network and various initial learning rates LR ∈{0.001, 0.0025, 0.005, 0.01, 0.025, 0.05}. The results in Figure <ref> show that all downsampling techniques performed very similarly, except for the nearest neighbour technique which yielded the worst results for all learning rates. This observation is in line with Figure <ref> and also holds for ImageNet16x16 and ImageNet64x64 (results not shown for brevity). For all remaining experiments in this paper, we used the box method.

Do conclusions drawn for cheap evaluations carry over to expensive ones? 
Next, we studied to which extent conclusions drawn for small networks and downsampled images carry over to larger networks and higher resolution images. This in turn determines the usefulness of these techniques for speeding up the experimental loop of architecture design and hyperparameter optimization. For this, we performed three experiments:


	
  * We studied how the results scale with the network size, more specifically, network width, defined by k. Table <ref> shows that larger k yielded better results independently of the downsampling size.
    
    Performance on our downsampled datasets was surprisingly strong; for example, on ImageNet32x32, using k=10 achieved 40.96% Top-1 validation error and 18.87% Top-5 validation error. Interestingly, this matches the original results by AlexNets <cit.> (40.7% and 18.2%, respectively) on full-sized ImageNet (which has roughly 50 times more pixels per image). Clearly, greater image resolution yielded better results (e.g., 12.64% top-5 performance for ImageNet64x64).
    




  * We studied how optimal learning rates changed across different combinations of downsampling sizes and network widths. Figure <ref> shows that the region of optimal learning rates remained similar across all our experimental setups, including networks whose space and time complexity differed by up to a factor of 100.


Additionally, Figure <ref> compares performance as a function of both learning rate and width multiplier k for downsampling sizes of 32x32 and 16x16, showing qualitatively very similar results in both cases, including the interaction effect that larger values of k favor somewhat larger learning rates than smaller k. 

This suggests that small networks and 
downsampled images may indeed facilitate faster experimentation. 









  * We also investigated the tradeoffs of performance vs. training time resulting from different downsampling and network sizes. Figure <ref> and Table <ref> show that both mechanisms for reducing the computational cost should be considered simultaneously to achieve optimal anytime performance. 
An additional mechanism could be to perform warm restarts <cit.>, which was shown to substantially improve anytime performance over reductions of the learning rate at regular intervals. 
Since the relative ranking of learning rates was consistent across different downsampling and network sizes, we also envision that architecture and hyperparameter search methods could exploit cheap proxies of computationally more expensive setups based on varying these degrees of freedom. Possible methods for exploiting these include <cit.>.







§ DISCUSSION AND CONCLUSION


Our proposed downsampled versions of the original ImageNet dataset might represent a viable alternative to the CIFAR datasets while dealing with more complex data and classes. Quite surprisingly, even by greatly reducing the resolution of images to 32 × 32 pixels, one can predict image labels quite well (see also Figure <ref> and Figure <ref>). 

Classification of low resolution images might also be of interest when (i) data storage is important (the original ImageNet dataset is 145GB), (ii) the input images are corrupted by noise, or (iii) a small subpart of a high resolution image must be classified. 

We hope that the provided datasets will fill the gap between the CIFAR datasets and the full ImageNet dataset, representing a good benchmark for experimental studies, such as algorithm design, neural network architecture search and hyperparameter optimization. Our preliminary experiments support the hypothesis that findings obtained on smaller networks for lower resolution images may transfer to larger networks for higher resolution images, while being up to 100 times cheaper to obtain. This could be exploited by multi-fidelity methods for architecture and hyperparameter search <cit.>.



§ ACKNOWLEDGEMENT

This work has partly been supported by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme under grant no. 716721 and by the German Research Foundation (DFG), under the BrainLinksBrainTools Cluster of Excellence (grant number EXC 1086).

The authors acknowledge support by the High Performance and Cloud Computing Group at the Zentrum für Datenverarbeitung of the University of Tübingen, the state of Baden-Württemberg through bwHPC
and the German Research Foundation (DFG) through grant no INST 37/935-1 FUGG. 











plainnat

iclr2017_conference





