Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Han Xiao Kashif Rasul Roland Vollgraf


Kinetic layers and coupling conditions for  macroscopic equations on networks I: the wave equation
    R. Borsche[1] 
     A. Klar[1] [2]
    
==================================================================================================




We present , a new dataset comprising of 28× 28 grayscale images of 70,000 fashion products from 10 categories, with 7,000 images per category. The training set has 60,000 images and the test set has 10,000 images. is intended to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms, as it shares the same image size, data format and the structure of training and testing splits. The dataset is freely available at .




§ INTRODUCTION


The MNIST dataset comprising of 10-class handwritten digits, was first introduced by <cit.> in 1998. At that time one could not have foreseen the stellar rise of deep learning techniques and their performance. Despite the fact that today deep learning can do so much the simple MNIST dataset has become the most widely used testbed in deep learning, surpassing CIFAR-10 <cit.> and ImageNet <cit.> in its popularity via Google trends[<https://trends.google.com/trends/explore?date=all   q=mnist,CIFAR,ImageNet>]. Despite its simplicity its usage does not seem to be decreasing despite calls for it in the deep learning community.

The reason MNIST is so popular has to do with its size, allowing deep learning researchers to quickly check and prototype their algorithms. This is also complemented by the fact that all machine learning libraries (e.g. scikit-learn) and deep learning frameworks (e.g. Tensorflow, Pytorch) provide helper functions and convenient examples that use MNIST out of the box. 

Our aim with this work is to create a good benchmark dataset which has all the accessibility of MNIST, namely its small size, straightforward encoding and permissive license. We took the approach of sticking to the 10 classes 70,000 grayscale images in the size of 28× 28 as in the original MNIST. In fact, the only change one needs to use this dataset is to change the URL from where the MNIST dataset is fetched. Moreover, poses a more challenging classification task than the simple MNIST digits data, whereas the latter has been trained to accuracies above 99.7% as reported in <cit.>.

We also looked at the EMNIST dataset provided by <cit.>, an extended version of MNIST that extends the number of classes by introducing uppercase and lowercase characters. However, to be able to use it seamlessly one needs to not only extend the deep learning framework's MNIST helpers, but also change the underlying deep neural network to classify these extra classes. 




§ DATASET

is based on the assortment on Zalando's website[Zalando is the Europe's largest online fashion platform. <http://www.zalando.com>]. Every fashion product on Zalando has a set of pictures shot by professional photographers, demonstrating different aspects of the product, i.e. front and back looks, details, looks with model and in an outfit. The original picture has a light-gray background (hexadecimal color: ) and stored in 762× 1000 JPEG format. For efficiently serving different frontend components, the original picture is resampled with multiple resolutions, e.g. large, medium, small, thumbnail and tiny. 

We use the front look thumbnail images of 70,000 unique products to build . Those products come from different gender groups: men, women, kids and neutral. In particular, white-color products are not included in the dataset as they have low contrast to the background. The thumbnails (51× 73) are then fed into the following conversion pipeline, which is visualized in <ref>.


    
  * Converting the input to a PNG image.
    
  * Trimming any edges that are close to the color of the corner pixels.
    The “closeness” is defined by the distance within 5% of the maximum possible intensity in RGB space.
    
  * Resizing the longest edge of the image to 28 by subsampling the pixels, i.e. some rows and columns
    are skipped over.
    
  * Sharpening pixels using a Gaussian operator of the radius and standard deviation of 1.0, with increasing effect near outlines.
    
  * Extending the shortest edge to 28 and put the image to the center of the canvas.
    
  * Negating the intensities of the image.
    
  * Converting the image to 8-bit grayscale pixels.





For the class labels, we use the silhouette code of the product. The silhouette code is manually labeled by the in-house fashion experts and reviewed by a separate team at Zalando. Each product contains only one silhouette code. <ref> gives a summary of all class labels in with examples for each class.




Finally, the dataset is divided into a training and a test set. The training set receives a randomly-selected 6,000 examples from each class. Images and labels are stored in the same file format as the MNIST data set, which is designed for storing vectors and multidimensional matrices. The result files are listed in <ref>. We sort examples by their labels while storing, resulting in smaller label files after compression comparing to the MNIST. It is also easier to retrieve examples with a certain class label. The data shuffling job is therefore left to the algorithm developer.







§ EXPERIMENTS

We provide some classification results in <ref> to form a benchmark on this data set. All algorithms are repeated 5 times by shuffling the training data and the average accuracy on the test set is reported. The benchmark on the MNIST dataset is also included for a side-by-side comparison. A more comprehensive  table with explanations on the algorithms can be found on .

llrr
Benchmark on (Fashion) and MNIST.  

        2cTest Accuracy 

3-4
1lClassifier    
1lParameter    
1cFashion    
1cMNIST 
 


4c
  – continued from previous page 

        2cTest Accuracy 

3-4
1lClassifier    
1lParameter    
1cFashion    
1cMNIST 
 


4rContinued on next page 
 

DecisionTreeClassifier     criterion= max_depth=10 splitter=    0.798     0.873

    criterion= max_depth=10 splitter=    0.792     0.861

    criterion= max_depth=50 splitter=    0.789     0.886

    criterion= max_depth=100 splitter=    0.789     0.886

    criterion= max_depth=10 splitter=    0.788     0.866

    criterion= max_depth=50 splitter=    0.787     0.883

    criterion= max_depth=100 splitter=    0.787     0.881

    criterion= max_depth=100 splitter=    0.785     0.879

    criterion= max_depth=50 splitter=    0.783     0.877

    criterion= max_depth=10 splitter=    0.783     0.853

    criterion= max_depth=50 splitter=    0.779     0.873

    criterion= max_depth=100 splitter=    0.777     0.875

ExtraTreeClassifier     criterion= max_depth=10 splitter=    0.775     0.806

    criterion= max_depth=100 splitter=    0.775     0.847

    criterion= max_depth=10 splitter=    0.772     0.810

    criterion= max_depth=50 splitter=    0.772     0.847

    criterion= max_depth=100 splitter=    0.769     0.843

    criterion= max_depth=50 splitter=    0.768     0.845

    criterion= max_depth=50 splitter=    0.752     0.826

    criterion= max_depth=100 splitter=    0.752     0.828

    criterion= max_depth=50 splitter=    0.748     0.824

    criterion= max_depth=100 splitter=    0.745     0.820

    criterion= max_depth=10 splitter=    0.739     0.737

    criterion= max_depth=10 splitter=    0.737     0.745

GaussianNB     priors=    0.511     0.524

GradientBoostingClassifier     n_estimators=100 loss= max_depth=10    0.880     0.969

    n_estimators=50 loss= max_depth=10    0.872     0.964

    n_estimators=100 loss= max_depth=3    0.862     0.949

    n_estimators=10 loss= max_depth=10    0.849     0.933

    n_estimators=50 loss= max_depth=3    0.840     0.926

    n_estimators=10 loss= max_depth=50    0.795     0.888

    n_estimators=10 loss= max_depth=3    0.782     0.846

KNeighborsClassifier     weights= n_neighbors=5 p=1    0.854     0.959

    weights= n_neighbors=9 p=1    0.854     0.955

    weights= n_neighbors=9 p=1    0.853     0.955

    weights= n_neighbors=5 p=1    0.852     0.957

    weights= n_neighbors=5 p=2    0.852     0.945

    weights= n_neighbors=9 p=2    0.849     0.944

    weights= n_neighbors=5 p=2    0.849     0.944

    weights= n_neighbors=9 p=2    0.847     0.943

    weights= n_neighbors=1 p=2    0.839     0.943

    weights= n_neighbors=1 p=2    0.839     0.943

    weights= n_neighbors=1 p=1    0.838     0.955

    weights= n_neighbors=1 p=1    0.838     0.955

LinearSVC     loss= C=1 multi_class= penalty=    0.836     0.917

    loss= C=1 multi_class= penalty=    0.835     0.919

    loss= C=1 multi_class= penalty=    0.834     0.919

    loss= C=1 multi_class= penalty=    0.833     0.919

    loss= C=1 multi_class= penalty=    0.833     0.919

    loss= C=1 multi_class= penalty=    0.820     0.912

    loss= C=10 multi_class= penalty=    0.779     0.885

    loss= C=100 multi_class= penalty=    0.776     0.873

    loss= C=10 multi_class= penalty=    0.764     0.879

    loss= C=100 multi_class= penalty=    0.758     0.872

    loss= C=10 multi_class= penalty=    0.751     0.783

    loss= C=10 multi_class= penalty=    0.749     0.816

    loss= C=10 multi_class= penalty=    0.748     0.829

    loss= C=10 multi_class= penalty=    0.736     0.829

    loss= C=100 multi_class= penalty=    0.516     0.759

    loss= C=100 multi_class= penalty=    0.496     0.753

    loss= C=100 multi_class= penalty=    0.492     0.746

    loss= C=100 multi_class= penalty=    0.484     0.737

LogisticRegression     C=1 multi_class= penalty=    0.842     0.917

    C=1 multi_class= penalty=    0.841     0.917

    C=10 multi_class= penalty=    0.839     0.916

    C=10 multi_class= penalty=    0.839     0.909

    C=100 multi_class= penalty=    0.836     0.916

MLPClassifier     activation= hidden_layer_sizes=    0.871     0.972

    activation= hidden_layer_sizes=    0.870     0.972

    activation= hidden_layer_sizes=    0.868     0.962

    activation= hidden_layer_sizes=    0.863     0.957

    activation= hidden_layer_sizes=    0.850     0.936

    activation= hidden_layer_sizes=    0.848     0.933

    activation= hidden_layer_sizes=    0.841     0.921

    activation= hidden_layer_sizes=    0.840     0.921

PassiveAggressiveClassifier     C=1    0.776     0.877

    C=100    0.775     0.875

    C=10    0.773     0.880

Perceptron     penalty=    0.782     0.887

    penalty=    0.754     0.845

    penalty=    0.726     0.845

RandomForestClassifier     n_estimators=100 criterion= max_depth=100    0.873     0.970

    n_estimators=100 criterion= max_depth=100    0.872     0.970

    n_estimators=50 criterion= max_depth=100    0.872     0.968

    n_estimators=100 criterion= max_depth=50    0.872     0.969

    n_estimators=50 criterion= max_depth=50    0.871     0.967

    n_estimators=100 criterion= max_depth=50    0.871     0.971

    n_estimators=50 criterion= max_depth=50    0.870     0.968

    n_estimators=50 criterion= max_depth=100    0.869     0.967

    n_estimators=10 criterion= max_depth=50    0.853     0.949

    n_estimators=10 criterion= max_depth=100    0.852     0.949

    n_estimators=10 criterion= max_depth=50    0.848     0.948

    n_estimators=10 criterion= max_depth=100    0.847     0.948

    n_estimators=50 criterion= max_depth=10    0.838     0.947

    n_estimators=100 criterion= max_depth=10    0.838     0.950

    n_estimators=100 criterion= max_depth=10    0.835     0.949

    n_estimators=50 criterion= max_depth=10    0.834     0.945

    n_estimators=10 criterion= max_depth=10    0.828     0.933

    n_estimators=10 criterion= max_depth=10    0.825     0.930

SGDClassifier     loss= penalty=    0.819     0.914

    loss= penalty=    0.818     0.912

    loss= penalty=    0.817     0.910

    loss= penalty=    0.816     0.913

    loss= penalty=    0.816     0.912

    loss= penalty=    0.816     0.913

    loss= penalty=    0.815     0.914

    loss= penalty=    0.815     0.911

    loss= penalty=    0.815     0.910

    loss= penalty=    0.814     0.913

    loss= penalty=    0.814     0.912

    loss= penalty=    0.814     0.912

    loss= penalty=    0.813     0.914

    loss= penalty=    0.813     0.913

    loss= penalty=    0.813     0.911

SVC     C=10 kernel=    0.897     0.973

    C=10 kernel=    0.891     0.976

    C=100 kernel=    0.890     0.978

    C=100 kernel=    0.890     0.972

    C=1 kernel=    0.879     0.966

    C=1 kernel=    0.873     0.957

    C=1 kernel=    0.839     0.929

    C=10 kernel=    0.829     0.927

    C=100 kernel=    0.827     0.926

    C=1 kernel=    0.678     0.898

    C=10 kernel=    0.671     0.873

    C=100 kernel=    0.664     0.868






§ CONCLUSIONS


This paper introduced , a fashion product images dataset intended to be a drop-in replacement of MNIST and whilst providing a more challenging alternative for benchmarking machine learning algorithm. The images in are converted to a format that matches that of the MNIST dataset, making it immediately compatible with any machine learning package capable of working with the original MNIST dataset.


abbrvnat


