Pytorch : GPU Memory Leak
I speculated that I was facing a GPU memory leak in the training of Conv nets using PyTorch framework. Below image 



To resolve it, I added -

os.environ['CUDA_LAUNCH_BLOCKING'] = "1" 

which resolved the memory problem, as shown below -


but as I was using torch.nn.DataParallel, so I expect my code to utilise all the GPUs, but now it is utilising only the GPU:1.



Before using os.environ['CUDA_LAUNCH_BLOCKING'] = "1", the GPU utilisation was below (which is equally bad)-



On digging further, I come to know that, when we use torch.nn.DataParallel, we are supposed to not use CUDA_LAUNCH_BLOCKING', because it puts the network in some deadlock mechanism.
So, now I have come back again in GPU memory issue, because I think my code is not utilising that much memory which it is showing without setting CUDA_LAUNCH_BLOCKING=1.
My code to use torch.nn.DataParallel-

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))


model_transfer.to(device)


How to resolve the GPU memory issue?
Edit:
Minimal code - 

image_dataset = datasets.ImageFolder(train_dir_path,transform = transform)
train_loader = torch.utils.data.DataLoader(image_dataset['train'], batch_size=batch_size,shuffle = True)
model_transfer = models.resnet18(pretrained=True)

device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() &gt; 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  # dim = 0 [30, xxx] -&gt; [10, ...], [10, ...], [10, ...] on 3 GPUs
  model_transfer = nn.DataParallel(model_transfer.cuda(),device_ids=range(torch.cuda.device_count()))
model_transfer.to(device) 

## Training function
for epoch in range(1, n_epochs+1):
    for batch_idx, (data, target) in enumerate(train_loader):
                    if use_cuda:
            data, target = data.to('cuda',non_blocking = True), target.to('cuda',non_blocking = True)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output,target)
        loss.backward()
        optimizer.step()
        train_loss += ((1 / (batch_idx + 1)) * (loss.item() - train_loss))

   ## Validation loop same as training loop so not mentioning here

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch,
        train_loss,
        valid_loss
        ))
        if valid_loss &lt;= valid_loss_min:
           valid_loss_min,valid_loss))
           torch.save(model.state_dict(), 'case_3_model.pt')
           valid_loss_min = valid_loss


So the way I resolved some of my CUDA out of memory issue is by making sure to delete useless tensors and trim tensors that may stay referenced for some hidden reason. The problem may arise from either requesting for more memory than you have the capacity for or an accumulation of garbage data that you don't need, but somehow is left behind on the memory.

One of the most important aspects of this memory management is how you are loading in the data. Instead of reading the entire dataset, it may be more memory efficient to read from disk (using memmap when reading npy) or doing batch loading, where you only read a batch of images or whatever data you have at a time. Although this may be computationally slower, it does give you flexibility for not going out and buying more GPUS to store your memory just to run your code. 

We're not sure how your code is structured in terms of reading the data or training your CNN so this is as much advice I can give.
