Issues loading from Drive with pytorch' datasets.DatasetFolder
The loading works great using jupyter and local files, but when I adapted to Colab, fetching data from a Drive folder, datasets.DatasetFolder always loads 9500 odd datapoints, never the full 10 000. Anyone had similar issues?

train_data = datasets.DatasetFolder('/content/drive/My Drive/4 - kaggle/data', np.load, list(('npy'))  )

print(train_data.__len__)


Returns

&lt;bound method DatasetFolder.__len__ of Dataset DatasetFolder
    Number of datapoints: 9554
    Root Location: /content/drive/My Drive/4 - kaggle/data
    Transforms (if any): None
    Target Transforms (if any): None&gt;


Where I would get the full 10 000 elements usually.

Loading lots of files from a single folder in Drive is likely to be slow and error-prone. You'll probably end up much happier if you either stage the data on GCS or upload an archive (.zip or .tar.gz) to Drive and copy that one file to your colab VM, unarchive it there, and then run your code over the local data.
