coding patterns for efficient minibatch loop using custom pytorch dataset
Are there any general recommendations to handle data efficiently in a custom Dataset so that it plays nicely with the minibatch eval/train loop? To illustrate what I mean more concretely, let's say I define this synthetic toy dataset that maps x to x+1:



import torch.utils.data as data

class Dataset(data.Dataset):
    def __init__(self):
        super(Dataset, self).__init__()
        # list of [x, y]
        self.dataset = [
            [1, 2],
            [2, 3],
            [3, 4],
            [4, 5]
        ]

    def __getitem__(self, index):
        item = self.dataset[index]
        return item[0], item[1]

    def __len__(self):
        return len(self.dataset)


In practice, this would be wrapped in a DataLoader and accessed inside an eval/train loop, something like this:

dataset = Dataset()
data_loader = data.DataLoader(dataset=dataset, batch_size=2, shuffle=True)
epochs = 100
for i_epoch in range(epochs):
    for i_minibatch, minibatch in enumerate(data_loader):
        x, y = minibatch
        # predict and train


The dataset object might return primitive Python objects like numbers or lists, like in my example implementation, but in the "predict and train" part of the last code snippet, we need some specific data types to compute stuff, like a torch.FloatTensor (it seems the data loader can do this implicitly), probably even wrapped as a torch.autograd.Variable, and some calls to .cuda() might also be necessary. My question is about general advice for when to make these data transformations and function calls.

For example, one option would be to have everything already saved as a torch.FloatTensor inside the dataset, and in the data_loader loop we could add the Variable wrapper and call .cuda(). We could also have all or part of the data on the GPU already by calling .cuda() in the Dataset constructor or in the getitem method. I think there might be pros and cons to all of these approaches. If I'm training a model for several epochs, I don't want to introduce unnecessary overhead each epoch or minibatch iteration that could have been avoided by precomputing stuff in the dataset. Probably someone with more knowledge about the internals of pytorch (maybe related to some caching or jit compilation happening under the hood) might be able to point to more specific reasons to choose one approach over another.

Generally, datasets are stored in files in a format that is more friendly to storage on disk. When you load the dataset, you want the datatypes to be more friendly to PyTorch. This is accomplished by transformations interface of torchvision library. For example, for MNIST below is standard transformations:

datasets.MNIST('../data', train=True, download=True,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ])


Here ToTensor divides all values in tensor by 255 so that if the data is RGB image then values in tensor would be 0.0 to 1.0.

The key thing is here is that your data in disk should ideally be agnostic of what you might want to do with it (training, visualization, calculate stats etc) as well as agnostic of frameworks being used. You apply transformations after you load data that are relevant to stuff you are doing.

One other thing I want to mention is handling very large datasets like ImageNet. There are few important things:


You should avoid using separate image files as your dataset because this doesn't work well in clusters. Instead you can pack all files in format like LMDB or uncompressed zip (use Python ZipFile module) and then only access these files sequentially. The random access in large files will slow you down tremendously.
You should avoid using shuffle option in DataLoader class for large datasets. If you do that then again you are accessing large file with random access and performance will tank. Instead, what you can do is to sequentially read K = C * total_epochs * batch_size records where C is some constant of your choice >= 1. Then shuffle K records in memory and then divide them in batches. Unfortunately you have to do this manually for now.

