I/O Issues in Loading Several Large H5PY Files (Pytorch)
I met a problem!
Recently I meet a problem of I/O issue. The target and input data are stored with h5py files. Each target file is 2.6GB while each input file is 10.2GB. I have 5 input datasets and 5 target datasets in total.
I created a custom dataset function for each h5py file and then use data.ConcatDataset class to link all the datasets. The custom dataset function is:
class MydataSet(Dataset):
def __init__(self, indx=1, root_path='./xxx', tar_size=128, data_aug=True, train=True):
    self.train = train
    if self.train:
        self.in_file = pth.join(root_path, 'train', 'train_noisy_%d.h5' % indx)
        self.tar_file = pth.join(root_path, 'train', 'train_clean_%d.h5' % indx)
    else:
        self.in_file = pth.join(root_path, 'test', 'test_noisy.h5')
        self.tar_file = pth.join(root_path, 'test', 'test_clean.h5')
    self.h5f_n = h5py.File(self.in_file, 'r', driver='core')
    self.h5f_c = h5py.File(self.tar_file, 'r')
    self.keys_n = list(self.h5f_n.keys())
    self.keys_c = list(self.h5f_c.keys())
    # h5f_n.close()
    # h5f_c.close()

    self.tar_size = tar_size
    self.data_aug = data_aug

def __len__(self):
    return len(self.keys_n)

def __del__(self):
    self.h5f_n.close()
    self.h5f_c.close()

def __getitem__(self, index):
    keyn = self.keys_n[index]
    keyc = self.keys_c[index]
    datan = np.array(self.h5f_n[keyn])
    datac = np.array(self.h5f_c[keyc])
    datan_tensor = torch.from_numpy(datan).unsqueeze(0)
    datac_tensor = torch.from_numpy(datac)
    if self.data_aug and np.random.randint(2, size=1)[0] == 1: # horizontal flip
        datan_tensor = torch.flip(datan_tensor,dims=[2]) # c h w
        datac_tensor = torch.flip(datac_tensor,dims=[2])

Then I use dataset_train = data.ConcatDataset([MydataSet(indx=index, train=True) for index in range(1, 6)]) for training. When only 2-3 h5py files are used, the I/O speed is normal and everything goes right. However, when 5 files are used, the training speed is gradually decreasing (5 iterations/s to 1 iterations/s). I change the num_worker and the problem still exists.
Could anyone give me a solution? Should I merge several h5py files into a bigger one? Or other methods? Thanks in advance!

Improving performance requires timing benchmarks. To do that you need to identify potential bottlenecks and associated scenarios. You said &quot;with 2-3 files the I/O speed is normal&quot; and &quot;when 5 files are used, the training speed gradually decreases&quot;.  So, is your performance issue I/O speed, or training speed? Or do you know? If you don't know, you need to isolate and compare I/O performance and training performance separately for the 2 scenarios.
In other words, to measure I/O performance  (only) you need to run the following tests:

Time to read and concatenate 2-3 files,
Time to read and concatenate 5 files,
Copy the 5 files into 1, and time the read from the merged file,
Or, link the 5 files to 1 file, and time.

And to measure training speed (only) you need to compare performance for the following tests:

Merge 2-3 files, then read and train from the merged file.
Merge all 5 files, then read and train from merged file.
Or, link the 5 files to 1 file, then read and train from linked file.

As noted in my comment, merging (or linking) multiple HDF5 files into one is easy if all datasets are at the root level and all dataset names are unique.  I added the external link method because it might provide the same performance, without duplicating large data files.
Below is the code that shows both methods. Substitute your file names in the fnames list, and it should be ready to run. If your dataset names aren't unique, you will need to create unique names, and assign  in h5fr.copy() -- like  this: h5fr.copy(h5fr[ds],h5fw,'unique_dataset_name')
Code to merge -or- link files :
(comment/uncomment lines as appropriate)
import h5py
fnames = ['file_1.h5','file_2.h5','file_3.h5']
# consider changing filename to 'linked_' when using links:
with h5py.File(f'merge_{len(fnames)}.h5','w') as h5fw:      
    for fname in fnames:
        with h5py.File(fname,'r') as h5fr:
            for ds in h5fr.keys():
                # To copy datasets into 1 file use:
                h5fr.copy(h5fr[ds],h5fw)
                # to link datasets to 1 file use:
                # h5fw[ds] = h5py.ExternalLink(fname,ds)

