How do I load my dataset into Pytorch or Keras?
I'm learning to build a neural network using either Pytorch or Keras. I have my images in two separate folders for training and testing with their corresponding labels in two csv files and I'm having the basic problem of just loading them into with Pytorch or Keras so I can start building an NN. I've tried tutorials from
https://towardsdatascience.com/training-neural-network-from-scratch-using-pytorch-in-just-7-cells-e6e904070a1d
and
https://www.tensorflow.org/tutorials/keras/classification
and a few others but they all seem to use pre-existing datasets like MNIST where it can be imported in or downloaded from a link. I've tried something like this:
import numpy as np
import matplotlib.pyplot as plt
import os
import cv2
from tqdm import tqdm

DATADIR = r&quot;Path to my image folder&quot;

CATEGORIES = [&quot;High&quot;, &quot;Low&quot;]

for category in CATEGORIES:                                                       
    path = os.path.join(DATADIR,category)                                         
    for img in os.listdir(path):                                                  
    img_array = cv2.imread(os.path.join(path,img) ,cv2.IMREAD_GRAYSCALE)      
    plt.imshow(img_array, cmap='gray')  
    plt.show()                                                               

    break                                                                    
break    

but was after something more like:
fashion_mnist = tf.keras.datasets.fashion_mnist       
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()    

Does anyone have ideas?
Thanks,
C

If you have your data in a csv file and images as the target in separate folders, so one of the best ways is to use flow_from_dataframe generator from keras libraries. Here is an example, and a more detailed example on keras library here. It's also the documentations.
Here is some sample code:
import pandas as pd            #import pandas library
from tensorflow import keras
df = pd.read_csv(r&quot;.\train.csv&quot;)    #read csv file
datagen = keras.preprocessing.image.ImageDataGenerator(
                    rescale=1./255) #dividing pixels by 255 is arbitrary
train_generator = datagen.flow_from_dataframe(
          dataframe=df,             #dataframe object you have defined above
          directory=&quot;.\train_imgs&quot;, #the dir where your images are stored
          x_col=&quot;id&quot;,               #column of image names
          y_col=&quot;label&quot;,            #column of class name
          class_mode=&quot;categorical&quot;, #type of the problem
          target_size=(32,32),      #resizing image target according to your model input
          batch_size=32)            #batch size of data it should create

Then, you can pass it to the model.fit():
model.fit(train_generator, epochs=10)

