Load huge image dataset and some data augmentation using Pytorch
I have a question. I have about 2 million images (place365-standard dataset) and I want to do some data augmentation like transforming, cropping etc. Also, I have to make my own target image (y) based on some color model algorithms (CMYK) for example.

So Actually, my preprocessing step includes augmentation and making terget image (y). And then I should feed these images to a deep network. When should I do this based on Dataset class? Should I do my processing step in __getitem__()? If yes, would it be parallel and fast? 

Here is my template of Dataset(data.Dataset) class:

import torch
from torch.utils import data

class Dataset(data.Dataset):
    """
    Return Dataset class representing our data set
    """
    def __int__(self, list_IDs, labels):
        """
        Initialize data set as a list of IDs corresponding to each item of data set and labels of each data

        Args:
            list_IDs: a list of IDs for each data point in data set
            labels: label of an item in data set with respect to the ID
        """

        self.labels = labels
        self.list_IDs = list_IDs

    def __len__(self):
        """
        Return the length of data set using list of IDs

        :return: number of samples in data set
        """
        return len(self.list_IDs)

    def __getitem__(self, item):
        """
        Generate one item of data set. Here we apply our preprocessing things like halftone styles and subtractive color process using CMYK color model etc. (See the paper for operations)

        :param item: index of item in IDs list

        :return: a sample of data
        """
        ID = self.list_IDs[item]

        # Code to load data
        X = None #

        # code to apply your custom function to make y image (time consuming task - some algorithms)
        y = None #

        return X, y


Thanks for any advice 

Best regards

If you look at e.g., torchvision.dataset.ImageFolder you'll see that it works quite similar to your design: the class has transform member that lists all sorts of augmentations (resizing, cropping, flipping etc.) and these are carried out on the images in the __getitem__ method.
Regarding parallelism, the Dataset itself is not parallel, but the DataLoader can be (see num_workers argument), so if you use your dataset inside a parallel dataloader you get the parallelism for free, Cool!
