How to load dataset from pickle files into PyTorch?
I have X_train(inputs) and Y_train(labels) in separate pickle files in form of integer matrices. Now, I need to load them and train using PyTorch. I tried torch.utils.data.DataLoader and torchvision.datasets.DatasetFolder but nothing worked or I might be getting wrong somewhere. Please suggest a proper way for the same.

You should really give a clear description of your problem with some examples. Anyway, as far as I understand you are looking for something like this.

import pickle
from torch.utils.data import Dataset
from torchvision import transforms
from torch.utils.data import DataLoader


class YourDataset(Dataset):

    def __init__(self, X_Train, Y_Train, transform=None):
        self.X_Train = X_Train
        self.Y_Train = Y_Train
        self.transform = transform

    def __len__(self):
        return len(self.X_Train)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        x = self.X_Train[idx]
        y = self.Y_Train[idx]

        if self.transform:
            x = self.transform(x)
            y = self.transform(y)

        return x, y


file = open('FILENAME_X_train', 'rb')
X_train = pickle.load(file)
file.close()

file = open('FILENAME_Y_train', 'rb')
Y_train = pickle.load(file)
file.close()

your_dataset = YourDataset(X_train, Y_train, transform=transforms.Compose([transforms.ToTensor()]))

your_data_loader = DataLoader(your_dataset, batch_size=8, shuffle=True, num_workers=0)


Note that I have not tested the code, but I think that it gives the general idea. Hope it helps.
