How to split and load huge dataset that doesn't fit into memory into pytorch Dataloader?
I'm training a deep learning model to do multi label classification of diseases in NIH's Chest Xray-14 dataset using Google's Colab. I can't load all images into Dataloader at once, given around 112k training examples and limited RAM.

Is there a way to just store path of images in pytorch's DataLoader, reading only those images needed for current iteration during training, and once iteration is complete, the images are unloaded from memory, so on so forth until one epoch is complete.

Yes, the default behavior for the ImageFolder is to create a list of image paths and load the actual images only when needed. It doesn't support multiclass labels. However, you can write your own Dataset to support multi-label, referencing the ImageFolder class for details.

During __init__ you construct a list of image paths and a corresponding list of labels. Images should only be loaded only when __getitem__ is invoked. Below is a stub of such a dataset class, the details will depend on the organization of your files, image types, and label format.

class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, args):
        """ Construct an indexed list of image paths and labels """

    def __getitem__(self, n):
        """ Load image n in the list of image paths and return it along with its label.
            In the case of multiclass the label will probably be a list of values"""

    def __len__(self):
        """ return the total number of images in this dataset """


Once you've created a valid dataset instance an instance of DataLoader should be created, providing your dataset as an argument. A DataLoader is responsible for sampling its dataset, i.e. invoking the __getitem__ method you wrote, and putting individual samples into mini-batches. It also handles parallelized loading and defines how the indices are sampled. The DataLoader itself doesn't store more than it needs. The maximum number of samples it should hold in memory at any time is batch_size * num_workers (or batch_size if num_workers == 0).
