PyTorch: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True
This is an error message I get as I work with some synthetic data. I am a bit puzzled because the error persists although I do what I am advised to do.  Could it somehow be related to the fact that I do not specify a batch?  Would the use of PyTorch Dataset relieve the problem?

Here is my code (I am new in PyTorch, just learning it now) -- it should be reproducible:

Data Creation:

x, y = np.meshgrid(np.random.randn(100) , np.random.randn(100))    
z = 2 * x + 3 * y + 1.5 * x * y - x ** 2 - y**2
X = x.ravel().reshape(-1, 1)
Y = y.ravel().reshape(-1, 1)    
Z = z.ravel().reshape(-1, 1)    
U = np.concatenate([X, Y], axis = 1)    
U = torch.tensor(U, requires_grad=True)    
Z = torch.tensor(Z, requires_grad=True)    
V = []

for i in range(U.shape[0]):        
    u = U[i, :]  
    u1 = u.view(-1, 1) @ u.view(1, -1)    
    u1 = u1.triu()    
    ones = torch.ones_like(u1)    
    mask = ones.triu()    
    mask = (mask == 1)    
    u2 = torch.masked_select(u1, mask)    
    u3 = torch.cat([u, u2])    
    u3 = u3.view(1, -1)    
    V.append(u3)

V = torch.cat(V, dim = 0)


Training a Model

from torch import nn    
from torch import optim    
net = nn.Sequential(nn.Linear(V.shape[1], 1))    
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(50):  # loop over the dataset multiple times    
    running_loss = 0.0        
    i = 0
    for inputs , labels in zip(V, Z):
        # get the inputs; data is a list of [inputs, labels]

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)

        loss.backward(retain_graph = True)

        optimizer.step()

        # print statistics
        running_loss += loss.item()

        i += 1

        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')


Error Message:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-143-2454f4bb70a5&gt; in &lt;module&gt;
     25 
     26 
---&gt; 27         loss.backward(retain_graph = True)
     28 
     29         optimizer.step()

~\Anaconda3\envs\torch\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
    193                 products. Defaults to ``False``.
    194         """
--&gt; 195         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    196 
    197     def register_hook(self, hook):

~\Anaconda3\envs\torch\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 99         allow_unreachable=True)  # allow_unreachable flag
    100 
    101 

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.


Could you explain the error and fix the code?

Presumably, you haven't re-run the data creation code again after setting retain_graph=True, as it is in a IPython REPL. It would work with that, but in nearly all cases, setting retain_graph=True is not the appropriate solution.

The problem in your case, is that you have set requires_grad=True for U, which means that everything that involved U in the data creation, will be recorded in the computational graph and when loss.backward() is called, the gradients will propagate through all of these up to U. After the first time, all buffers for the gradients of those will have been freed, and a second backward will fail.

Neither U nor Z should have requires_grad=True, as they are not being optimised/learned. Only the learned parameters (parameters given to the optimiser) should have requires_grad=True and generally, you don't have to set it manually either, since nn.Parameter handles that automatically.

You should also ensure that the tensors you create from NumPy data have type torch.float (float32) as NumPy's float array will usually be float64, which are mostly unnecessary and just slower compared to float32, especially on the GPU.

U = torch.tensor(U, dtype=torch.float)

Z = torch.tensor(Z, dtype=torch.float)


And remove the retain_graph=True from the backward call:

loss.backward()

