How to prevent appending to a list to fill gpu memory
I run a model with pytorch with batch inference. I want to save the output from that model to a list, because I need it later for other calculations. The problem is, that if I append the output from my model to my list &quot;output&quot;, it fills the gpu memory.
The output from the model is a list.
Is there a way to move the list to RAM?
if __name__ == &quot;__main__&quot;:
        output = []
        with torch.no_grad():
            for i in input_split:
                try:
                    preds = model(i)                        
                    output.append(preds)          
                    del preds   
                    gc.collect()
                    torch.cuda.empty_cache()    

                except RuntimeError as e:
                    print(&quot;Failed&quot;)


It is because the tensors you get from preds = model(i) are still in GPU.
You can just take them out of the GPU before appending them to the list
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)
        output.append(preds.cpu())

And when you want to use them again in GPU then just put them into GPU one by one
for data in output:
    data = data.cuda()

Edit 1. The Detectron2
This is similar to the above answer, except a little bit more complicated.
The output of Detectron2 is list[dict]. Which means if you have more than 1 images in the batch you'll have to create an empty list as buffer, like
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)
        
        buffer = []
        for pred in preds:
            pred['instances']['pred_boxes'] = pred['instances']['pred_boxes'].cpu()
            pred['instances']['scores'] = pred['instances']['scores'].cpu()
            pred['instances']['pred_classes'] = pred['instances']['pred_classes'].cpu()
            pred['instances']['pred_masks'] = pred['instances']['pred_masks'].cpu()
            pred['instances']['pred_keypoints'] = pred['instances']['pred_keypoints'].cpu()
            buffer.append(pred)
        output.append(buffer)

You maybe able to use pointer and get rid of buffer like this too
output = []
with torch.no_grad():
    for i in input_split:
        preds = model(i)
        
        for pred in preds:
            pred['instances']['pred_boxes'] = pred['instances']['pred_boxes'].cpu()
            pred['instances']['scores'] = pred['instances']['scores'].cpu()
            pred['instances']['pred_classes'] = pred['instances']['pred_classes'].cpu()
            pred['instances']['pred_masks'] = pred['instances']['pred_masks'].cpu()
            pred['instances']['pred_keypoints'] = pred['instances']['pred_keypoints'].cpu()
        output.append(preds)

