storing a huge dataset in h5py file format I work on preparing the luna16 dataset for feeding into the CNN model, after reading all '.mhd' files and the labels(0, 1) in the annotated CSV file, I get a memory error, I know the problem is because of the data size is need a lot of RAM that I do not have on my computer, is there any solution to that?
You can use h5py files but due to the fact that they store the dimensions, they take even more disk to be stored and while fetching you have to supply enough memory which is not possible in your case, so you have to read them chunk by chunk. There is a simpler approach. One of pandas's inputs is chunk. try to use it.