Python: append images in for loop use a lot of memory
I'm actually facing a problem while working on a python project.
I'm appending some images in a for loop and it uses a lot of RAM memory.
If you guys, have any solution to optimize this for loop, It'll help me a lot.
Thanks!
augment_img = []
augment_label = []
augment_weight = []
for i in range(4):
    for j in range(len(train_dataset)):
        single_img, single_label, single_weight = train_dataset[j]
        augment_img.append(single_img)
        augment_label.append(single_label)
        augment_weight.append(single_weight)
        if j % 1000==0:
            print(j)```


Instead of loading all images at once, I would suggest loading them in batches. There are different ways to handle that. In both pytorch and tensorflow, you can save your weights and continue training after some point. So, you can:

iterate through your images in batches
load the model weights and train the model
save the weights for the next round

But easier way is to use default functions implemented in pytorch for example that does the same. For example using ImageFolder and DataLoader:
def load_data(data_folder, batch_size, train, kwargs):
    transform = {
        'train': transforms.Compose(
            [transforms.Resize([256, 256]),
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                  std=[0.229, 0.224, 0.225])]),
        'test': transforms.Compose(
            [transforms.Resize([224, 224]),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                  std=[0.229, 0.224, 0.225])])
        }
    data = datasets.ImageFolder(root = data_folder, transform=transform['train' if train else 'test'])
    data_loader = torch.utils.data.DataLoader(data, batch_size=batch_size, shuffle=True, **kwargs, drop_last = True if train else False)
    return data_loader 

You can specify the folder that contains images ending with various extensions (e.g. .jpg, .png. etc.) and then training your model by passing data_loader to.
