Speed up datasets loading on Google Colab
I am working on Image classification on the German Traffic Sign Dataset on Google Colab with Pytorch.
Here is the structure of the dataset:


GTSRB


Training


00000/


*.ppmm

…
00043/


*.ppmm


Test


*.ppmm
…
labels.csv 




I have managed to upload the whole dataset to my drive(it took a long time!!!).
I have used ImageFolder class and Dataset class to load respectively training and test images.

However, Training my model is really slow, and GPU is not used efficiently. After many searches, I discovered that file transfer from drive to Colab is at fault here.

Does anyone know how I can use hd5 dataset (or others techniques) to first store all training and test images for later preprocessing?

If your problem truly is the network speed between Colab and Drive, you should try uploading the files directly to the Google Colab instance, rather than accessing them from Drive.

from google.colab import files
dataset_file_dict = files.upload()


Doing this will save the files directly to your Colab instance, allowing your code to access the files locally.

However, I'd suspect that there might be other problems besides the network latency – perhaps your model has lots of parameters, or somehow there was a bug in the code to get CUDA going. Sometimes I would forget to change my runtime to a GPU runtime under the "Runtime" menu tab, "Change Runtime Type".

Hope this helps!
