Strange out of memory while loading lots of pictures before input CNN for deep learning I’m using the TFlearn, and want to classify pictures to two category. But the strange out of memory while loading lots of pictures before input CNN for deep learning. The RAM is 64 G in my deep learning box. I using 482426 pictures(resize to 224 x 224) for training (241213 are A category, the other 241213 are B category ). The process shows the '%MEM: 98.4' whenever the I load these images, so that I can't complete the process. I would like to know how can I edit my code for this situation? 



imgs = []

for filename in glob.glob(Learning_Data_Path+"/Training/A/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img) 
    imgs.append(img_arr)


for filename in glob.glob(Learning_Data_Path+"/Training/B/"+"*.tif"):
    img = load_image(filename)
    img=img.resize((224,224))
    img_arr = np.asarray(img) 
    imgs.append(img_arr)


imgs = np.array(imgs)


y_data = np.r_[np.c_[np.ones(Training_A_num), np.zeros(Training_A_num)],np.c_[np.zeros(Training_B_num), np.ones(Training_B_num)]
You should load images in batches. 480K of 224*224 sized images is too large for any machine let alone a 64GB one.

You can use image_generators from keras or tf.keras 

https://www.tensorflow.org/tutorials/load_data/images#load_using_keraspreprocessing

https://keras.io/preprocessing/image/