PyTorch DataLoader
I'm trying to use multiple torch.utils.data.DataLoaders to create datasets that have different transforms applied to them. Currently, my code is roughly

d_transforms = [
    transforms.RandomHorizontalFlip(),
    # Some other transforms...
]
loaders = []
for i in range(len(d_transforms)):
    dataset = datasets.MNIST('./data', 
            train=train, 
            download=True, 
            transform=d_transforms[i]
    loaders.append(
        DataLoader(dataset, 
            shuffle=True, 
            pin_memory=True, 
            num_workers=1)
        )


This works, but it's extremely slow. kernprof shows that nearly all of the time in my code is spent on lines like

x, y = next(iter(train_loaders[i]))


I suspect that this is due to the fact that i'm using multiple instances of DataLoader, each with their own worker, which tries to read the same data files. 

My question is, what is a better way to do this? Ideally, I would subclass torch.utils.data.DataSet and specify the transform i'd like to apply when sampling, but this doesn't seem possible due to __getitem__ not being able to take arguments. 

__getitem__ does take an argument which is the index of the content you want to load . For eg.

transform = transforms.Compose(
    [transforms.ToTensor(),
     normalize])

class CountDataset(Dataset):

def __init__(self, file,transform=None):

    self.transform = transform
    #self.vocab = vocab
    with open(file,'rb') as f:
        self.data = pickle.load(f)
    self.y = self.data['answers']
    self.I = self.data['images']


def __len__(self):
    return len(self.y)

def __getitem__(self, idx):
    img_name = self.I[idx]
    label = self.y[Idx]
    fname = '/'.join(img_name.split("/")[-2:]) #/train2014/xx.jpg
    DIR = '/hdd/manoj/VQA/Images/mscoco/'
    img_full_path = os.path.join(DIR,fname)
    img = Image.open(img_full_path).convert("RGB")
    img_tensor = self.transform(img.resize((224,224)))
    return img_tensor,label


testset = CountDataset(file = 'testdat.pkl',
                        transform = transform)


testloader = DataLoader(testset, batch_size=32,
                         shuffle=False, num_workers=4)


You dont call the data loader in loop.
