Create a custom dataset from a folder with separate files
I am using Pytorch's custom dataset feature to create a custom dataset from separate files in one folder. Each file contains 123 rows and 123 columns, and all data points are integers.
My issue is that the resources I've come across cater to files in one .csv, mine are not. More so, opening the images after being transformed as an image doesn't run as well. I'm not sure how to proceed from here on as my code gives:
AttributeError: 'Image' object has no attribute 'read'

import os
from torch.utils.data import DataLoader, Dataset
from numpy import genfromtxt

# Custom dataset
class CONCEPTDataset(Dataset):
    &quot;&quot;&quot; Concept Dataset &quot;&quot;&quot;

    def __init__(self, file_dir, transforms=None):
        &quot;&quot;&quot;
        Args:
            file_dir (string): Directory with all the images.
            transforms (optional): Changes on the data.
        &quot;&quot;&quot;
        self.file_dir = file_dir
        self.transforms = transforms

        self.concepts = os.listdir(file_dir)
        self.concepts.sort()
        self.concepts = [os.path.join(file_dir, concept) for concept in self.concepts]
    
    def __len__(self):
        return len(self.concepts)

    def __getitem__(self, idx):
        image = self.concepts[idx]

        # csv file to a numpy array using genfromtxt
        data = genfromtxt(image, delimiter=',')

        data = self.transforms(data.unsqueeze(0))
        return data


PIL.Image.fromarray is used to convert an array to a PIL Image while Image.open is used to load an image file from the file system. You don't need either of those two since you already have a NumPy array representing your image and are looking to return it. PyTorch will convert it to torch.Tensor automatically if you plug your dataset to a torch.data.utils.DataLoader.
