Pytorch: gradient computation fails when in-place operation follows certain functions
Consider the following piece of code:
import torch
from torch import nn
a = torch.tensor([1.], requires_grad=True)
b = nn.Tanh()(a)
# b = nn.Linear(1,1)(a)
b *= 1
# b = b * 1
b.sum().backward()

Running the code results in RuntimeError:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [1]], which is output 0 of TanhBackward, is at version 1; expected version 0 instead.

However, if I change Tanh() to Linear(1,1) or change b*=1 to b=b*1 (as in the commented lines), the code will run successfully and get the correct gradient.
Why is that?
My environment:

Python 3.8.5 (anaconda) on Windows
Pytorch 1.8.0, running on CPU


I just found some text from the Pytorch official site:

In-place correctness checks
Every tensor keeps a version counter, that is incremented every time it is marked dirty in any operation. When a Function saves any tensors for backward, a version counter of their containing Tensor is saved as well. Once you access self.saved_tensors it is checked, and if it is greater than the saved value an error is raised. This ensures that if you’re using in-place functions and not seeing any errors, you can be sure that the computed gradients are correct.

Source: https://pytorch.org/docs/stable/notes/autograd.html#in-place-correctness-checks
According to this, an in-place operation may or may not break the gradient computation, depending on the actual situation.
Therefore, as long as the code contains in-place operations, it should not be a surprise when (as what is observed in the question) changing something in the context magically fixes the gradient computation.
Also according to the same passage, in-place operations may disrupt the gradient calculation even if they do not break it, in which case the speed of gradient calculation may be decreased. So it seems like a good strategy to avoid any in-place operation.
