Memory Leak in loop pytorch
The following loop is not discarding any of the tensors it makes after each iteration of the loop leading to a memory leak. It is due to the use of grad_loss.backward() in the below code. Is there anything I'm missing or is there an issue with pytorch.

    for (images, one_hot_labels) in tqdm(batched_train_data):
        # I collect batch size here because the last batch may have a smaller batch_size
        images = images.to(device)
        one_hot_labels = one_hot_labels.to(device)

        batch_size = images.shape[0]

        images.requires_grad = True
        optimizer.zero_grad()
        # as images is not a parameters optimizer.zero_grad() won't reset it's gradient
        if images.grad is not None:
            images.grad.data.zero_()

        probabilities = model.forward(images)

        # I want to use .backward() twice rather than autograd because I want to accumulate the gradients
        loss = loss_func(probabilities, one_hot_labels)
        loss.backward(create_graph=True)
        grad_loss = grad_loss_func(images.grad)
        grad_loss.backward()

        optimizer.step()

        labels = one_hot_labels.detach().argmax(dim=1)
        predictions = probabilities.detach().argmax(dim=1)
        num_correct = int(predictions.eq(labels).sum())

        train_data_length += batch_size
        train_correct += num_correct
        train_loss += float(loss.detach()) * batch_size

        writer.add_graph(model, images)
        writer.close()

        # To stop memory leaks
        del images
        del one_hot_labels
        del probabilities
        del loss
        del grad_loss
        del labels
        del predictions
        del num_correct


To fix it you need to replace 

images.grad.data.zero_()


with 

images.grad = None


I believe this is because doing images.grad.data.zero_() does not remove any computation graph associated with images therefore allowing the graph to grow as you loop through. 

On a separate note, I've also been advised that you should avoid operating upon .data whenever possible as it's unsafe to do so.
