How to read data in HDF5 format file partially when the data is too large to read fully
You can slice h5py datasets like numpy arrays, so you could work on a number of subsets instead of the whole dataset (e.g. 4 100000*10000 subsets).
I am engaged in analysing HDF5 format data for scientific research purposes. I'm using Python's h5py library.
Now, the HDF file I want to read is so large. Its file size is about 20GB and the main part of its data is 400000*10000 float matrix. I tried to read the data once, but my development environment Spyder was terminated by compulsion because of the shortage of the memory. Then is there any method to read it partially and avoid this problem?